In the following two years, OpenAI focused on more general AI research and development.
In 2018, the company published a paper titled “Improving Language Understanding by Generative Pre-Training”, which introduced the concept of Generative Pre-trained Transformer GPT.
In short, GPT is a neural network — a machine learning model spain mobile database nspired by the structure and function of the human brain — trained on a large dataset of human-generated text. It can perform many functions, such as generating and answering questions.
It can also write haiku about itself, like this:
ChatGPT has a broad mind.
The answer is smooth,
The AI finally spoke.
Anyway…the OpenAI team invested a lot of money and developed GPT-1, their first language model “trained” on BookCorpus, which contains more than 7,000 unpublished books.
The model eventually evolved into GPT-2, a more powerful version that was trained on 8 million web pages and contains 1.5 billion parameters training values that make text predictions possible.
GPT-2 playground with model and decoder setup on the left
At that point, the company strayed from its lofty goal of “open” AI pun not intended and initially decided not to release GPT-2 to the public.