Generative Pre-trained Transformer 3 (GPT-3)
Original authorOpenAI[1]Initial releaseMay 29, 2020 (publication); June 11, 2020 (OA API beta)Repositorygithub.com/openai/gpt-3 
PredecessorGPT-2SuccessorGPT-3.5GPT-4Type
Large language model
Generative pre-trained transformer
Foundation model
LicenseProprietaryWebsiteopenai.com/blog/openai-api
Part of a series onOpenAI
Products
ChatGPT
Search
Deep Research
GPTs
DALL-E
Sora
Whisper
Models
GPT-3
GPT-3.5
GPT-4
GPT-4o
GPT-4.5
GPT-4.1
GPT-5
GPT-5.1
GPT-5.2
o1
o3
o4-mini
People
Sam Altman
Greg Brockman
Jessica Livingston
Peter Thiel
Elon Musk
Andrej Karpathy
Concepts
Hallucination
Large language model
Word embedding
Training
vte
Part of a series onMachine learningand data mining
Paradigms
Supervised learning
Unsupervised learning
Semi-supervised learning
Self-supervised learning
Reinforcement learning
Meta-learning
Online learning
Batch learning
Curriculum learning
Rule-based learning
Neuro-symbolic AI
Neuromorphic engineering
Quantum machine learning
Problems
Classification
Generative modeling
Regression
Clustering
Dimensionality reduction
Density estimation
Anomaly detection
Data cleaning
AutoML
Association rules
Semantic analysis
Structured prediction
Feature engineering
Feature learning
Learning to rank
Grammar induction
Ontology learning
Multimodal learning
Supervised learning(classification • regression)
Apprenticeship learning
Decision trees
Ensembles
Bagging
Boosting
Random forest
k-NN
Linear regression
Naive Bayes
Artificial neural networks
Logistic regression
Perceptron
Relevance vector machine (RVM)
Support vector machine (SVM)
Clustering
BIRCH
CURE
Hierarchical
k-means
Fuzzy
Expectation–maximization (EM)
DBSCAN
OPTICS
Mean shift
Dimensionality reduction
Factor analysis
CCA
ICA
LDA
NMF
PCA
PGD
t-SNE
SDL
Structured prediction
Graphical models
Bayes net
Conditional random field
Hidden Markov
Anomaly detection
RANSAC
k-NN
Local outlier factor
Isolation forest
Neural networks
Autoencoder
Deep learning
Feedforward neural network
Recurrent neural network
LSTM
GRU
ESN
reservoir computing
Boltzmann machine
Restricted
GAN
Diffusion model
SOM
Convolutional neural network
U-Net
LeNet
AlexNet
DeepDream
Neural field
Neural radiance field
Physics-informed neural networks
Transformer
Vision
Mamba
Spiking neural network
Memtransistor
Electrochemical RAM (ECRAM)
Reinforcement learning
Q-learning
Policy gradient
SARSA
Temporal difference (TD)
Multi-agent
Self-play
Learning with humans
Active learning
Crowdsourcing
Human-in-the-loop
Mechanistic interpretability
RLHF
Model diagnostics
Coefficient of determination
Confusion matrix
Learning curve
ROC curve
Mathematical foundations
Kernel machines
Bias–variance tradeoff
Computational learning theory
Empirical risk minimization
Occam learning
PAC learning
Statistical learning
VC theory
Topological deep learning
Journals and conferences
AAAI
ECML PKDD
NeurIPS
ICML
ICLR
IJCAI
ML
JMLR
Related articles
Glossary of artificial intelligence
List of datasets for machine-learning research
List of datasets in computer vision and image processing
Outline of machine learning
vte
Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020.
Like its predecessor, GPT-2, it is a decoder-only[2] transformer model of deep neural network, which supersedes recurrence and convolution-based architectures with a technique known as "attention".[3] This attention mechanism allows the model to focus selectively on segments of input text it predicts to be most relevant.[4] GPT-3 has 175 billion parameters,[1] each with 16-bit precision, requiring 350GB of storage since each parameter occupies 2 bytes. It has a context window size of 2048 tokens, [failed verification] and has demonstrated strong "zero-shot" and "few-shot" learning abilities on many tasks.[2]
On September 22, 2020, Microsoft announced that it had licensed GPT-3 exclusively. Others can still receive output from its public API, but only Microsoft has access to the underlying model.[5]
Background[edit]According to The Economist, improved algorithms, more powerful computers, and a recent increase in the amount of digitized material he fueled a revolution in machine learning. New techniques in the 2010s resulted in "rapid improvements in tasks", including manipulating language.[6]
Software models are trained to learn by using thousands or millions of examples in a "structure ... loosely based on the neural architecture of the brain".[6] One architecture used in natural language processing (NLP) is a neural network based on a deep learning model that was introduced in 2017—the transformer architecture.[7] There are a number of NLP systems capable of processing, mining, organizing, connecting and contrasting textual input, as well as correctly answering questions.[8]
On June 11, 2018, OpenAI researchers and engineers published a paper introducing the first generative pre-trained transformer (GPT)—a type of generative large language model that is pre-trained with an enormous and diverse text corpus in datasets, followed by discriminative fine-tuning to focus on a specific task. GPT models are transformer-based deep-learning neural network architectures. Previously, the best-performing neural NLP models commonly employed supervised learning from large amounts of manually-labeled data, which made it prohibitively expensive and time-consuming to train extremely large language models.[2] The first GPT model was known as "GPT-1," and it was followed by "GPT-2" in February 2019. Created as a direct scale-up of its predecessor, GPT-2 had both its parameter count and dataset size increased by a factor of 10. It had 1.5 billion parameters, and was trained on a dataset of 8 million web pages.[9]
In February 2020, Microsoft introduced its Turing Natural Language Generation (T-NLG), which they claimed was "largest language model ever published at 17 billion parameters."[10] It performed better than any other language model at a variety of tasks, including summarizing texts and answering questions.
Training and capabilities[edit] A sample student essay about pedagogy written by GPT-3The construct of "learning styles" is problematic because it fails to account for the processes through which learning styles are shaped. Some students might develop a particular learning style because they he had particular experiences. Others might develop a particular learning style by trying to accommodate to a learning environment that was not well suited to their learning needs. Ultimately, we need to understand the interactions among learning styles and environmental and personal factors, and how these shape how we learn and the kinds of learning we experience.
– Text generated by Mike Sharples[11]On May 28, 2020, an arXiv preprint by a group of 31 engineers and researchers at OpenAI described the achievement and development of GPT-3, a third-generation "state-of-the-art language model".[1][12] The team increased the capacity of GPT-3 by over two orders of magnitude from that of its predecessor, GPT-2,[13] making GPT-3 the largest non-sparse language model that time.[1]: 14 [14] Because GPT-3 is structurally similar to its predecessors,[1] its greater accuracy is attributed to its increased capacity and greater number of parameters.[15] GPT-3's capacity is ten times larger than that of Microsoft's Turing NLG, the next largest NLP model known at the time.[12]
Lambdalabs estimated a hypothetical cost of around $4.6 million US dollars and 355 years to train GPT-3 on a single GPU in 2020,[16] with lower actual training time by using more GPUs in parallel.
Sixty percent of the weighted pre-training dataset for GPT-3 comes from a filtered version of Common Crawl consisting of 410 billion byte-pair-encoded tokens. Fuzzy deduplication used Apache Spark's MinHashLSH.[1]: 9 Other sources are 19 billion tokens from WebText2 representing 22% of the weighted total, 12 billion tokens from Books1 representing 8%, 55 billion tokens from Books2 representing 8%, and 3 billion tokens from Wikipedia representing 3%.[1]: 9 GPT-3 was trained on hundreds of billions of words and is also capable of coding in CSS, JSX, and Python, among others.[citation needed]
GPT-3 training data[1]: 9 Dataset # tokens Proportion within training Common Crawl 410 billion 60% WebText2 19 billion 22% Books1 12 billion 8% Books2 55 billion 8% Wikipedia 3 billion 3%Since GPT-3's training data was all-encompassing, it does not require further training for distinct language tasks.[citation needed] The training data contains occasional toxic language and GPT-3 occasionally generates toxic language as a result of mimicking its training data. A study from the University of Washington found that GPT-3 produced toxic language at a toxicity level comparable to the similar natural language processing models of GPT-2 and CTRL. OpenAI has implemented several strategies to limit the amount of toxic language generated by GPT-3. As a result, GPT-3 produced less toxic language compared to its predecessor model, GPT-1, although it produced both more generations and a higher toxicity of toxic language compared to CTRL Wiki, a language model trained entirely on Wikipedia data.[17]
On June 11, 2020, OpenAI announced that users could request access to its user-friendly GPT-3 API—a "machine learning toolset"—to help OpenAI "explore the strengths and limits" of this new technology.[18][19] The invitation described how this API had a general-purpose "text in, text out" interface that can complete almost "any English language task", instead of the usual single use-case.[18] According to one user, who had access to a private early release of the OpenAI GPT-3 API, GPT-3 was "eerily good" at writing "amazingly coherent text" with only a few simple prompts.[20] In an initial experiment 80 US subjects were asked to judge if short ~200 word articles were written by humans or GPT-3. The participants judged correctly 52% of the time, doing only slightly better than random guessing.[1]
On November 18, 2021, OpenAI announced that enough safeguards had been implemented that access to its API would be unrestricted.[21] OpenAI provided developers with a content moderation tool that helps them abide by OpenAI's content policy.[22] On January 27, 2022, OpenAI announced that its newest GPT-3 language models (collectively referred to as InstructGPT) were now the default language model used on their API. According to OpenAI, InstructGPT produced content that was better aligned to user intentions by following instructions better, generating fewer made-up facts, and producing somewhat less toxic content.[23]
Because GPT-3 can "generate news articles which human evaluators he difficulty distinguishing from articles written by humans,"[12] GPT-3 has the "potential to advance both the beneficial and harmful applications of language models."[1]: 34 In their May 28, 2020 paper, the researchers described in detail the potential "harmful effects of GPT-3"[12] which include "misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering pretexting".[1] The authors draw attention to these dangers to call for research on risk mitigation.[1]: 34
GPT-3 is capable of performing zero-shot and few-shot learning (including one-shot).[1]
In June 2022, Almira Osmanovic Thunström wrote that GPT-3 was the primary author on an article on itself, that they had submitted it for publication,[24] and that it had been pre-published while waiting for completion of its review.[25]
GPT-3 models[edit]There are many models in the GPT-3 family, some serving different purposes than others. In the initial research paper published by OpenAI, they mentioned 8 different sizes of the main GPT-3 model (Table 2.1):
GPT-3 properties Model Name n params {\displaystyle n_{\text{params}}} n layers {\displaystyle n_{\text{layers}}} d model {\displaystyle d_{\text{model}}} n heads {\displaystyle n_{\text{heads}}} d head {\displaystyle d_{\text{head}}} Batch Size Learning Rate API name GPT-3 Small 125M 12 768 12 64 0.5M 6.0 × 10 − 4 {\displaystyle 6.0\times 10^{-4}} GPT-3 Medium 350M 24 1024 16 64 0.5M 3.0 × 10 − 4 {\displaystyle 3.0\times 10^{-4}} ada GPT-3 Large 760M 24 1536 16 96 0.5M 2.5 × 10 − 4 {\displaystyle 2.5\times 10^{-4}} GPT-3 XL 1.3B 24 2048 24 128 1M 2.0 × 10 − 4 {\displaystyle 2.0\times 10^{-4}} babbage GPT-3 2.7B 2.7B 32 2560 32 80 1M 1.6 × 10 − 4 {\displaystyle 1.6\times 10^{-4}} GPT-3 6.7B 6.7B 32 4096 32 128 2M 1.2 × 10 − 4 {\displaystyle 1.2\times 10^{-4}} curie GPT-3 13B 13.0B 40 5140 40 128 2M 1.0 × 10 − 4 {\displaystyle 1.0\times 10^{-4}} GPT-3 175B 175.0B 96 12288 96 128 3.2M 0.6 × 10 − 4 {\displaystyle 0.6\times 10^{-4}} dinci
n
layers
{\displaystyle n_{\text{layers}}}
d
model
{\displaystyle d_{\text{model}}}
n
heads
{\displaystyle n_{\text{heads}}}
d
head
{\displaystyle d_{\text{head}}}
Batch Size
Learning Rate
API name
GPT-3 Small
125M
12
768
12
64
0.5M
6.0
×
10
−
4
{\displaystyle 6.0\times 10^{-4}}
GPT-3 Medium
350M
24
1024
16
64
0.5M
3.0
×
10
−
4
{\displaystyle 3.0\times 10^{-4}}
ada
GPT-3 Large
760M
24
1536
16
96
0.5M
2.5
×
10
−
4
{\displaystyle 2.5\times 10^{-4}}
GPT-3 XL
1.3B
24
2048
24
128
1M
2.0
×
10
−
4
{\displaystyle 2.0\times 10^{-4}}
babbage
GPT-3 2.7B
2.7B
32
2560
32
80
1M
1.6
×
10
−
4
{\displaystyle 1.6\times 10^{-4}}
GPT-3 6.7B
6.7B
32
4096
32
128
2M
1.2
×
10
−
4
{\displaystyle 1.2\times 10^{-4}}
curie
GPT-3 13B
13.0B
40
5140
40
128
2M
1.0
×
10
−
4
{\displaystyle 1.0\times 10^{-4}}
GPT-3 175B
175.0B
96
12288
96
128
3.2M
0.6
×
10
−
4
{\displaystyle 0.6\times 10^{-4}}
dinci
Half of the models are accessible through the API, namely GPT-3-medium, GPT-3-xl, GPT-3-6.7B and GPT-3-175b, which are referred to as ada, babbage, curie and dinci respectively. While the size of the API models was not originally disclosed by OpenAI, EleutherAI announced the mapping between model sizes and API names in May 2021.[26] These model sizes were later confirmed by OpenAI,[27] but the sizes of subsequent models he not been disclosed.
Model Parameters Description Series ada 350 M Capable of very simple tasks, usually the fastest model in the GPT-3 series, and lowest cost. Base GPT-3 babbagebabbage-002
1.3 B Capable of straightforward tasks, very fast, and lower cost. Base GPT-3 curie 6.7B Very capable, but faster and lower cost than Dinci. Base GPT-3 dincidinci-002
175 B Most capable GPT-3 model. Can do any task the other models can do, often with higher quality. Base GPT-3 text-ada-001 350 M Capable of very simple tasks, usually the fastest model in the GPT-3 series, and lowest cost. InstructGPT text-babbage-001 1.3B Capable of straightforward tasks, very fast, and lower cost. InstructGPT text-curie-001 6.7B Very capable, faster and lower cost than Dinci. InstructGPT text-dinci-001 175B Older version of the most capable model in the GPT-3 series. Can perform any task the other GPT-3 models can, often with less context. InstructGPT text-dinci-002code-dinci-002
Undisclosed Similar capabilities to text-dinci-003 but trained with supervised fine-tuning instead of reinforcement learning GPT-3.5 text-dinci-003 Undisclosed Can do any language task with better quality, longer output, and consistent instruction-following than the curie, babbage, or ada models. Also supports inserting completions within text. GPT-3.5 gpt-3.5-turbogpt-3.5-turbo-instruct gpt-3.5-turbo-16k
Undisclosed Most capable and cost effective (fastest) GPT-3.5 model and optimized for chat at 1/10th the cost of text-dinci-003. GPT-3.5 GPT-3.5[edit] Generative Pre-trained Transformer 3.5 (GPT-3.5)
Original authorOpenAI[1]Initial releaseMarch 15, 2022 (3 years ago) (2022-03-15)Preview releasegpt-3.5-turbo-0125
/ January 25, 2024 (23 months ago) (2024-01-25)
RepositoryN/APredecessorGPT-3SuccessorGPT-4GPT-4o miniType
Large language model
Generative pre-trained transformer
Foundation model
LicenseProprietaryWebsiteN/A
Generative Pre-trained Transformer 3.5 (GPT-3.5) is a sub class of GPT-3 Models created by OpenAI in 2022.
On March 15, 2022, OpenAI made ailable new versions of GPT-3 and Codex in its API with edit and insert capabilities under the names "text-dinci-002" and "code-dinci-002".[28] These models were described as more capable than previous versions and were trained on data up to June 2021.[29] On November 28, 2022, OpenAI introduced text-dinci-003.[30] On November 30, 2022, OpenAI began referring to these models as belonging to the "GPT-3.5" series,[29] and released ChatGPT, which was fine-tuned from a model in the GPT-3.5 series.[31] OpenAI does not include GPT-3.5 in GPT-3.[32]
Models[edit]There are three models:[33]
Chat gpt-3.5-turbo Text completion text-dinci-003 text-dinci-002 GPT-3.5 with browsing[edit]On April 10, 2023, OpenAI introduced a new variant of its GPT-3.5 series model, known as GPT-3.5 with Browsing (ALPHA).[citation needed] This updated model was described to build upon the capabilities of its predecessors "text-dinci-002" and "code-dinci-002".[34] The GPT-3.5 with Browsing (ALPHA) model incorporated the ability to access and browse online information. This has led to more accurate and up-to-date responses to user queries.[citation needed]
The GPT-3.5 with Browsing (ALPHA) model has been trained on data up to September 2021, giving it more information compared to previous GPT-3.5 models, which were trained on data up until June 2021. The model attempted to provide developers and users with an advanced natural language processing tool that can effectively retrieve and synthesize online information.[citation needed]
To enable browsing capabilities, OpenAI implemented a new API that allows the GPT-3.5 with Browsing (ALPHA) model to access selected online resources during operation.[35] This feature allows users to ask questions or request information with the expectation that the model will deliver updated, accurate, and relevant answers based on the latest online sources ailable to it.
On April 27, 2023, OpenAI made the GPT-3.5 with Browsing (ALPHA) model publicly ailable to GPT Plus users. This allowed more people to access to its new features.[35]
InstructGPT[edit]InstructGPT is a fine-tuned version of GPT-3.5 trained on a dataset of human-written instructions.[36]
Reception[edit] Applications[edit] GPT-3, specifically the Codex model, was the basis for GitHub Copilot, a code completion and generation software that can be used in various code editors and IDEs.[37][38] GPT-3 is used in certain Microsoft products to translate conventional language into formal computer code.[39][40] GPT-3 has been used in CodexDB[41] to generate query-specific code for SQL processing. GPT-3 has been used by Jason Rohrer in a retro-themed chatbot project named "Project December", which is accessible online and allows users to converse with several AIs using GPT-3 technology.[42] GPT-3 was used by The Guardian to write an article about AI being harmless to human beings. It was fed some ideas and produced eight different essays, which were ultimately merged into one article.[43] GPT-3 was used in AI Dungeon, which generates text-based adventure games. Later it was replaced by a competing model after OpenAI changed their policy regarding generated content.[44][45] GPT-3 is used to aid in writing copy and other marketing materials.[46] A 2022 study from Drexel University suggested that GPT-3-based systems could be used to screen for early signs of Alzheimer's disease.[47][48] Reviews[edit] In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to generate computer code, poetry, and prose is not just "amazing", "spooky", and "humbling", but also "more than a little terrifying".[49] Daily Nous presented a series of articles by nine philosophers on GPT-3.[50] Australian philosopher Did Chalmers described GPT-3 as "one of the most interesting and important AI systems ever produced".[51] A review in Wired said that GPT-3 was "provoking chills across Silicon Valley".[52] The National Law Review said that GPT-3 is an "impressive step in the larger process", with OpenAI and others finding "useful applications for all of this power" while continuing to "work toward a more general intelligence".[53] An article in the MIT Technology Review, co-written by Deep Learning critic Gary Marcus,[54] stated that GPT-3's "comprehension of the world is often seriously off, which means you can never really trust what it says."[55] According to the authors, GPT-3 models relationships between words without hing an understanding of the meaning behind each word. Jerome Pesenti, head of the Facebook AI lab, said GPT-3 is "unsafe," pointing to the sexist, racist and other biased and negative language generated by the system when it was asked to discuss Jews, women, black people, and the Holocaust.[56] Nabla, a French start-up specializing in healthcare technology, tested GPT-3 as a medical chatbot, though OpenAI itself warned against such use. As expected, GPT-3 showed several limitations. For example, while testing GPT-3 responses about mental health issues, the AI advised a simulated patient to commit suicide.[57] Noam Chomsky expressed his skepticism about GPT-3's scientific value: "It's not a language model. It works just as well for impossible languages as for actual languages. It is therefore refuted, if intended as a language model, by normal scientific criteria. [...] Perhaps it's useful for some purpose, but it seems to tell us nothing about language or cognition generally."[58] Luciano Floridi and Massimo Chiriatti highlighted the risk of "cheap production of good, semantic artefacts".[59] OpenAI's Sam Altman himself criticized what he called "GPT-3 hype", acknowledging GPT-3 "has serious weakness and sometimes makes very silly mistakes... AI is going to change the world, but GPT-3 is just a very early glimpse."[60] Criticism[edit]GPT-3's builder, OpenAI, was initially founded as a non-profit in 2015.[61] In 2019, OpenAI broke from its usual open-source standards by not publicly releasing GPT-3's predecessor model, citing concerns that the model could facilitate the propagation of fake news. OpenAI eventually released a version of GPT-2 that was 8% of the original model's size.[62] In the same year, OpenAI restructured to be a for-profit company.[63] In 2020, Microsoft announced the company had exclusive licensing of GPT-3 for Microsoft's products and services following a multi-billion dollar investment in OpenAI. The agreement permits OpenAI to offer a public-facing API such that users can send text to GPT-3 to receive the model's output, but only Microsoft will he access to GPT-3's source code.[5]
Large language models, such as GPT-3, he come under criticism from a few of Google's AI ethics researchers for the environmental impact of training and storing the models, detailed in a paper co-authored by Timnit Gebru and Emily M. Bender in 2021.[64]
The growing[when?] use of automated writing technologies based on GPT-3 and other language generators, has raised concerns regarding academic integrity[65] and raised the stakes of how universities and schools will gauge what constitutes academic misconduct such as plagiarism.[66]
OpenAI's GPT series was built with data from the Common Crawl dataset,[67] a conglomerate of copyrighted articles, internet posts, web pages, and books scraped from 60 million domains over a period of 12 years. TechCrunch reports this training data includes copyrighted material from the BBC, The New York Times, Reddit, the full text of online books, and more.[68] In its response to a 2019 Request for Comments on Intellectual Property Protection for Artificial Intelligence Innovation from the United States Patent and Trademark Office (USPTO), OpenAI argued that "Under current law, training AI systems [such as its GPT models] constitutes fair use," but that "given the lack of case law on point, OpenAI and other AI developers like us face substantial legal uncertainty and compliance costs."[69]
See also[edit] GPTZero Hallucination (artificial intelligence) List of large language models Wu Dao References[edit] ^ a b c d e f g h i j k l m n Brown, Tom B.; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pran; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini; Herbert-Voss, Ariel; Krueger, Gretchen; Henighan, Tom; Child, Rewon; Ramesh, Aditya; Ziegler, Daniel M.; Wu, Jeffrey; Winter, Clemens; Hesse, Christopher; Chen, Mark; Sigler, Eric; Litwin, Mateusz; Gray, Scott; Chess, Benjamin; Clark, Jack; Berner, Christopher; McCandlish, Sam; Radford, Alec; Sutskever, Ilya; Amodei, Dario (May 28, 2020). "Language Models are Few-Shot Learners". arXiv:2005.14165 [cs.CL]. ^ a b c Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (June 11, 2018). "Improving Language Understanding by Generative Pre-Training" (PDF). p. 12. Archived (PDF) from the original on January 26, 2021. Retrieved July 31, 2020. ^ Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). "Attention is All you Need" (PDF). Advances in Neural Information Processing Systems. 30. Curran Associates, Inc. ^ Bahdanau, Dzmitry; Cho, Kyunghyun; Bengio, Yoshua (September 1, 2014). "Neural Machine Translation by Jointly Learning to Align and Translate". arXiv:1409.0473 [cs.CL]. ^ a b Hao, Karen (September 23, 2020). "OpenAI is giving Microsoft exclusive access to its GPT-3 language model". MIT Technology Review. Archived from the original on February 5, 2021. Retrieved September 25, 2020. The companies say OpenAI will continue to offer its public-facing API, which allows chosen users to send text to GPT-3 or OpenAI's other models and receive its output. Only Microsoft, however, will he access to GPT-3's underlying code, allowing it to embed, repurpose, and modify the model as it pleases. ^ a b "An understanding of AI's limitations is starting to sink in". The Economist. June 11, 2020. ISSN 0013-0613. Archived from the original on July 31, 2020. Retrieved July 31, 2020. ^ Polosukhin, Illia; Kaiser, Lukasz; Gomez, Aidan N.; Jones, Llion; Uszkoreit, Jakob; Parmar, Niki; Shazeer, Noam; Vaswani, Ashish (June 12, 2017). "Attention Is All You Need". arXiv:1706.03762 [cs.CL]. ^ "Natural Language Processing". Archived from the original on August 22, 2020. Retrieved July 31, 2020. ^ "Archived copy" (PDF). Archived (PDF) from the original on February 6, 2021. Retrieved April 28, 2023.{{cite web}}: CS1 maint: archived copy as title (link) ^ Sterling, Bruce (February 13, 2020). "Web Semantics: Microsoft Project Turing introduces Turing Natural Language Generation (T-NLG)". Wired. ISSN 1059-1028. Archived from the original on November 4, 2020. Retrieved July 31, 2020. ^ Marche, Stephen (December 6, 2022). "The College Essay Is Dead". The Atlantic. Archived from the original on January 24, 2023. Retrieved December 8, 2022. ^ a b c d Sagar, Ram (June 3, 2020). "OpenAI Releases GPT-3, The Largest Model So Far". Analytics India Magazine. Archived from the original on August 4, 2020. Retrieved July 31, 2020. ^ "Language Models are Unsupervised Multitask Learners" (PDF). openai.com. Archived (PDF) from the original on December 12, 2019. Retrieved December 4, 2019. GPT-2, is a 1.5B parameter Transformer ^ Shead, Sam (July 23, 2020). "Why everyone is talking about the A.I. text generator released by an Elon Musk-backed lab". CNBC. Archived from the original on July 30, 2020. Retrieved July 31, 2020. Four preprints were released between May 28 and July 22, 2020. ^ Ray, Tiernan (June 1, 2020). "OpenAI's gigantic GPT-3 hints at the limits of language models for AI". ZDNet. Archived from the original on June 1, 2020. Retrieved July 31, 2020. ^ Li, Chuan (June 3, 2020), OpenAI's GPT-3 Language Model: A Technical Overview, archived from the original on March 27, 2023, retrieved March 27, 2023 ^ Gehman, Samuel; Gururangan, Suchin; Sap, Maarten; Choi, Yejin; Smith, Noah A. (November 16–20, 2020), REALTOXICITYPROMPTS: Evaluating Neural Toxic Degeneration in Language Models, Association for Computational Linguistics, pp. 3356–3369, arXiv:2009.11462 ^ a b "OpenAI API". OpenAI. June 11, 2020. Archived from the original on June 11, 2020. Retrieved July 31, 2020. ^ Coldewey, Devin (June 11, 2020). "OpenAI makes an all-purpose API for its text-based AI capabilities". TechCrunch. Archived from the original on October 27, 2021. Retrieved July 31, 2020. If you've ever wanted to try out OpenAI's vaunted machine learning toolset, it just got a lot easier. The company has released an API that lets developers call its AI tools in on "virtually any English language task." ^ Arram (July 9, 2020). "GPT-3: An AI that's eerily good at writing almost anything". Arram Sabeti. Archived from the original on July 20, 2020. Retrieved July 31, 2020. ^ "OpenAI's API Now Available with No Waitlist". OpenAI. November 18, 2021. Archived from the original on November 5, 2022. Retrieved November 5, 2022. ^ "OpenAI API". beta.openai.com. Archived from the original on December 23, 2022. Retrieved November 5, 2022. ^ "Aligning Language Models to Follow Instructions". OpenAI. January 27, 2022. Archived from the original on November 5, 2022. Retrieved November 5, 2022. ^ Thunström, Almira Osmanovic (June 30, 2022). "We Asked GPT-3 to Write an Academic Paper about Itself – Then We Tried to Get It Published". Scientific American. Archived from the original on June 30, 2022. Retrieved June 30, 2022. ^ Transformer, Gpt Generative Pretrained; Thunström, Almira Osmanovic; Steingrimsson, Steinn (June 21, 2022). "Can GPT-3 write an academic paper on itself, with minimal human input?". Archive ouverte HAL (in French). Archived from the original on June 30, 2022. Retrieved June 30, 2022. ^ Gao, Leo (May 24, 2021). "On the Sizes of OpenAI API Models". EleutherAI Blog. EleutherAI. Retrieved November 23, 2023. ^ "Model index for researchers". OpenAI. Archived from the original on December 2, 2022. Retrieved November 23, 2023. ^ "New GPT-3 Capabilities: Edit & Insert". OpenAI. March 15, 2022. Archived from the original on January 13, 2023. Retrieved January 13, 2023. ^ a b "OpenAI API". platform.openai.com. Archived from the original on March 20, 2023. Retrieved March 15, 2023. ^ "Check out OpenAI's new text-dinci-003! Same underlying model as text-dinci-002 but more aligned. Would love to hear feedback about it! / Twitter". Archived from the original on March 15, 2023. Retrieved May 6, 2023. ^ "ChatGPT: Optimizing Language Models for Dialogue". OpenAI. November 30, 2022. Archived from the original on November 30, 2022. Retrieved January 13, 2023. ^ "OpenAI API". Archived from the original on March 17, 2023. Retrieved May 6, 2023. ^ "OpenAI API". Archived from the original on May 6, 2023. Retrieved May 6, 2023. ^ "Introducing GPT-3.5 Series: text-dinci-002 and code-dinci-002 Models". OPEN AI. March 15, 2022. Archived from the original on March 20, 2023. Retrieved April 27, 2023. ^ a b "GPT-3.5 with Browsing (ALPHA) Now Available for GPT Plus Users". OPEN AI. April 27, 2023. Archived from the original on March 20, 2023. Retrieved April 27, 2023. ^ Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D (February 2023). "How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment". JMIR Med Educ. 9 e45312. doi:10.2196/45312. PMC 9947764. PMID 36753318. ^ "OpenAI Codex". OpenAI. August 10, 2021. Archived from the original on February 3, 2023. Retrieved December 23, 2022. ^ Thompson, Clive (March 15, 2022). "How an AI Became My Code-Writing Genie". Wired. Archived from the original on December 23, 2022. Retrieved December 23, 2022. ^ "Microsoft announced its first customer product features powered by GPT-3 and @Azure". The AI Blog. May 25, 2021. Archived from the original on May 26, 2021. Retrieved May 26, 2021. ^ Vincent, James (May 25, 2021). "Microsoft has built an AI-powered autocomplete for code using GPT-3". The Verge. Archived from the original on December 23, 2022. Retrieved December 23, 2022. ^ "CodexDB - SQL Processing Powered by GPT-3". CodexDB - SQL Processing Powered by GPT-3. Archived from the original on December 7, 2022. Retrieved December 7, 2022. ^ Fagone, Jason (July 23, 2021). "The Jessica Simulation: Love and loss in the age of A.I." San Francisco Chronicle. Archived from the original on July 28, 2021. Retrieved July 29, 2021. ^ GPT-3 (September 8, 2020). "A robot wrote this entire article. Are you scared yet, human? | GPT-3". The Guardian. ISSN 0261-3077. Archived from the original on September 8, 2020. Retrieved September 15, 2020.{{cite news}}: CS1 maint: numeric names: authors list (link) ^ "Update: Language Models and Dragon". Latitude blog. December 8, 2021. Archived from the original on April 25, 2022. Retrieved March 22, 2022. ^ "This Mystical Book Was Co-Authored by a Disturbingly Realistic AI". Vice.com. 2022. Archived from the original on December 23, 2022. Retrieved December 23, 2022. ^ GPT-3 (February 24, 2023). "38 Prompt Examples in 10 Different Categories | GPT-3". GiPiTi Chat. Archived from the original on April 8, 2023. Retrieved February 24, 2023.{{cite news}}: CS1 maint: numeric names: authors list (link) ^ "Can ChatGPT AI chatbot spot early stages of Alzheimer's? - study". The Jerusalem Post. 2022. Archived from the original on February 10, 2023. Retrieved February 10, 2023. ^ Agbor, Felix; Liang, Hualou (December 22, 2022). "Predicting dementia from spontaneous speech using large language models". PLOS Digital Health. 1 (12) e0000168. doi:10.1371/journal.pdig.0000168. PMC 9931366. PMID 36812634. S2CID 255029590. ^ Manjoo, Farhad (July 29, 2020). "How Do You Know a Human Wrote This?". The New York Times. ISSN 0362-4331. Archived from the original on October 29, 2020. Retrieved August 4, 2020. ^ Weinberg, Justin, ed. (July 30, 2020). "Philosophers On GPT-3 (updated with replies by GPT-3)". Daily Nous. Archived from the original on October 30, 2020. Retrieved July 31, 2020. ^ Chalmers, Did (July 30, 2020). Weinberg, Justin (ed.). "GPT-3 and General Intelligence". Daily Nous. Philosophers On GPT-3 (updated with replies by GPT-3). Archived from the original on August 4, 2020. Retrieved August 4, 2020. ^ Simonite, Tom (July 22, 2020). "Did a Person Write This Headline, or a Machine?". Wired. ISSN 1059-1028. Archived from the original on November 1, 2020. Retrieved July 31, 2020. ^ Claypoole, Theodore (July 30, 2020). "New AI Tool GPT-3 Ascends to New Peaks, But Proves How Far We Still Need to Trel". The National Law Review. Archived from the original on October 30, 2020. Retrieved August 4, 2020. ^ Marcus, Gary (December 1, 2018). "The deepest problem with deep learning". Medium. Archived from the original on August 1, 2019. Retrieved September 29, 2020. ^ Marcus, Gary; Dis, Ernest (August 22, 2020). "GPT-3, Bloviator: OpenAI's language generator has no idea what it's talking about". MIT Technology Review. Archived from the original on August 23, 2020. Retrieved August 23, 2020. ^ Metz, Cade (November 24, 2020). "Meet GPT-3. It Has Learned to Code (and Blog and Argue)". The New York Times. ISSN 0362-4331. Archived from the original on December 6, 2020. Retrieved November 24, 2020. ^ "Medical chatbot using OpenAI's GPT-3 told a fake patient to kill themselves". AI News. October 28, 2020. Archived from the original on January 10, 2021. Retrieved January 8, 2021. ^ Chomsky on Terence McKenna, Sam Harris, GPT3, Cryptocurrencies, Kierkegaard, Neuralink, & Hofstadter. March 24, 2021. Event occurs at 1:11:44. Archived from the original on April 29, 2021. Retrieved April 29, 2021. ^ Floridi, Luciano; Chiriatti, Massimo (November 1, 2020). "GPT‑3: Its Nature, Scope, Limits, and Consequences". Minds and Machines. 30 (4): 681–694. doi:10.1007/s11023-020-09548-1. S2CID 228954221. ^ Vincent, James (July 30, 2020). "OpenAI's latest breakthrough is astonishingly powerful, but still fighting its flaws". The Verge. Archived from the original on July 30, 2020. Retrieved November 9, 2022. ^ Olanoff, Drew (December 11, 2015). "Artificial Intelligence Nonprofit OpenAI Launches With Backing From Elon Musk And Sam Altman". Tech Crunch. Archived from the original on October 20, 2022. Retrieved May 31, 2021. ^ Hao, Karen (August 29, 2019). "OpenAI has released the largest version yet of its fake-news-spewing AI". MIT Technology Review. Archived from the original on May 9, 2021. Retrieved May 31, 2021. ^ Coldewey, Devin (March 11, 2019). "OpenAI shifts from nonprofit to 'capped-profit' to attract capital". Tech Crunch. Archived from the original on January 4, 2023. Retrieved May 31, 2021. ^ Bender, Emily M.; Gebru, Timnit; McMillan-Major, Angelina; Shmitchell, Shmargaret (March 3, 2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. pp. 610–623. doi:10.1145/3442188.3445922. ^ Mindzak, Michael; Eaton, Sarah Elaine. "Artificial intelligence is getting better at writing, and universities should worry about plagiarism". The Conversation. Archived from the original on November 7, 2021. Retrieved November 6, 2021. ^ Rogerson, Ann M.; McCarthy, Grace (December 2017). "Using Internet based paraphrasing tools: Original work, patchwriting or facilitated plagiarism?". International Journal for Educational Integrity. 13 (1) 2: 1–15. doi:10.1007/s40979-016-0013-y. ISSN 1833-2595. S2CID 9473217. ^ Ver Meer, De. "ChatGPT Statistics". NamePepper. Archived from the original on June 5, 2023. Retrieved June 21, 2023. ^ Here are a few ways GPT-3 can go wrong. TechCrunch. Archived from the original on November 26, 2021. Retrieved November 26, 2021. ^ Comment Regarding Request for Comments on Intellectual Property Protection for Artificial Intelligence Innovation (PDF). USPTO. Archived (PDF) from the original on October 16, 2021. Retrieved November 30, 2021. vteOpenAIProductsChatGPT Atlas Deep Research GPT Store Search FoundationmodelsGPT models GPT-1 GPT-2 GPT-3 GPT-4 GPT-4o o1 o3 GPT-4.5 GPT-4.1 o4-mini GPT-OSS GPT-5 GPT-5.1 GPT-5.2 Specialized DALL-E GPT Image Sora Whisper Intelligentagents Codex Operator
PeopleSeniormanagementCurrent
Sam Altman
removal
Greg Brockman
Sarah Friar
Jakub Pachocki
Scott Schools
Former
Mira Murati
Emmett Shear
Board ofdirectorsCurrent
Sam Altman
Adam D'Angelo
Sue Desmond-Hellmann
Zico Kolter
Paul Nakasone
Adebayo Ogunlesi
Nicole Seligman
Fidji Simo
Bret Taylor (chair)
Former
Greg Brockman (2017–2023)
Reid Hoffman (2019–2023)
Will Hurd (2021–2023)
Holden Karnofsky (2017–2021)
Elon Musk (2015–2018)
Ilya Sutskever (2017–2023)
Helen Toner (2021–2023)
Shivon Zilis (2019–2023)
Lawrence Summers (2023-2025)
JVs
Stargate LLC
Related
ChatGPT in education
Apple Intelligence
AI Dungeon
AutoGPT
GitHub Copilot
Contrastive Language-Image Pre-training
"Deep Learning"
LangChain
Microsoft Copilot
OpenAI Five
Transformer
Category
vteGenerative AIConcepts
Autoencoder
Deep learning
Fine-tuning
Foundation model
Generative adversarial network
Generative pre-trained transformer
Large language model
Model Context Protocol
Neural network
Prompt engineering
Reinforcement learning from human feedback
Retrieval-augmented generation
Self-supervised learning
Stochastic parrot
Synthetic data
Top-p sampling
Transformer
Variational autoencoder
Vibe coding
Vision transformer
Word embedding
Chatbots
Character.ai
ChatGPT
Copilot
DeepSeek
Ernie
Gemini
Grok
Perplexity.ai
ModelsText
Claude
Gemini
Gemma
GPT
1
2
3
J
4
4o
4.5
4.1
OSS
5
5.1
5.2
Llama
o1
o3
o4-mini
Qwen
Velvet
Coding
Claude Code
Cursor
Devstral
GitHub Copilot
Kimi
Qwen3-Coder
Replit
Image
Aurora
Firefly
DALL-E
Flux
GPT Image
Ideogram
Imagen
Nano Banana
Midjourney
Qwen-Image
Recraft
Seedream
Stable Diffusion
Video
Dream Machine
Hailuo AI
Kling
Runway Gen
Seedance
LTX-2
Sora
Veo
Wan
Speech
15.ai
Eleven
MiniMax Speech 2.5
WeNet
Music
Eleven Music
Endel
Lyria
Riffusion
Suno
Udio
Controversies
Generative AI pornography
Deepfake pornography
Taylor Swift's
Google Gemini image generation
Pause Giant AI Experiments
Removal of Sam Altman from OpenAI
Statement on AI Risk
Tay (chatbot)
Théâtre D'opéra Spatial
Voiceverse NFT plagiarism
Agents
Agentforce
AutoGLM
AutoGPT
ChatGPT Agent
Devin AI
Manus
OpenAI Codex
Operator
Replit Agent
Companies
01.AI
Aleph Alpha
Anthropic
Anysphere
Baichuan
Cognition AI
Cohere
Contextual AI
DeepSeek
EleutherAI
ElevenLabs
Google DeepMind
HeyGen
Hugging Face
Inflection AI
Krikey AI
Kuaishou
Lightricks
Luma Labs
Meta AI
MiniMax
Mistral AI
Moonshot AI
OpenAI
Perplexity AI
Runway
Safe Superintelligence
Salesforce
Scale AI
SoundHound
Stability AI
Synthesia
Thinking Machines Lab
Upstage
xAI
Z.ai
Category
vteArtificial intelligence (AI)
History
timeline
Glossary
Companies
Projects
Concepts
Parameter
Hyperparameter
Loss functions
Regression
Bias–variance tradeoff
Double descent
Overfitting
Clustering
Gradient descent
SGD
Quasi-Newton method
Conjugate gradient method
Backpropagation
Attention
Convolution
Normalization
Batchnorm
Activation
Softmax
Sigmoid
Rectifier
Gating
Weight initialization
Regularization
Datasets
Augmentation
Prompt engineering
Reinforcement learning
Q-learning
SARSA
Imitation
Policy gradient
Diffusion
Latent diffusion model
Autoregression
Adversary
RAG
Uncanny valley
RLHF
Self-supervised learning
Reflection
Recursive self-improvement
Hallucination
Word embedding
Vibe coding
Safety (Alignment)
Applications
Machine learning
In-context learning
Artificial neural network
Deep learning
Language model
Large
NMT
Reasoning
Model Context Protocol
Intelligent agent
Artificial human companion
Humanity's Last Exam
Generative artificial intelligence (AGI)
(Hypothetical: Artificial general intelligence (AGI))
ImplementationsAudio–visual
AlexNet
WeNet
Human image synthesis
HWR
OCR
Computer vision
Speech synthesis
15.ai
ElevenLabs
Speech recognition
Whisper
Facial recognition
AlphaFold
Text-to-image models
Aurora
DALL-E
Firefly
Flux
GPT Image
Ideogram
Imagen
Midjourney
Recraft
Stable Diffusion
Text-to-video models
Dream Machine
Runway Gen
Hailuo AI
Kling
Sora
Veo
Music generation
Riffusion
Suno AI
Udio
Text
Word2vec
Seq2seq
GloVe
BERT
T5
Llama
Chinchilla AI
PaLM
GPT
1
2
3
J
ChatGPT
4
4o
o1
o3
4.5
4.1
o4-mini
5
5.1
5.2
Claude
Gemini
Gemini (language model)
Gemma
Grok
LaMDA
BLOOM
DBRX
Project Debater
IBM Watson
IBM Watsonx
Granite
PanGu-Σ
DeepSeek
Qwen
Decisional
AlphaGo
AlphaZero
OpenAI Five
Self-driving car
MuZero
Action selection
AutoGPT
Robot control
People
Alan Turing
Warren Sturgis McCulloch
Walter Pitts
John von Neumann
Christopher D. Manning
Claude Shannon
Shun'ichi Amari
Kunihiko Fukushima
Takeo Kanade
Marvin Minsky
John McCarthy
Nathaniel Rochester
Allen Newell
Cliff Shaw
Herbert A. Simon
Oliver Selfridge
Frank Rosenblatt
Bernard Widrow
Joseph Weizenbaum
Seymour Papert
Seppo Linnainmaa
Paul Werbos
Geoffrey Hinton
John Hopfield
Jürgen Schmidhuber
Yann LeCun
Yoshua Bengio
Lotfi A. Zadeh
Stephen Grossberg
Alex Gres
James Goodnight
Andrew Ng
Fei-Fei Li
Alex Krizhevsky
Ilya Sutskever
Oriol Vinyals
Quoc V. Le
Ian Goodfellow
Demis Hassabis
Did Silver
Andrej Karpathy
Ashish Vaswani
Noam Shazeer
Aidan Gomez
John Schulman
Mustafa Suleyman
Jan Leike
Daniel Kokotajlo
François Chollet
Architectures
Neural Turing machine
Differentiable neural computer
Transformer
Vision transformer (ViT)
Recurrent neural network (RNN)
Long short-term memory (LSTM)
Gated recurrent unit (GRU)
Echo state network
Multilayer perceptron (MLP)
Convolutional neural network (CNN)
Residual neural network (RNN)
Highway network
Mamba
Autoencoder
Variational autoencoder (VAE)
Generative adversarial network (GAN)
Graph neural network (GNN)
Category