Document · Research paper

Improving Language Understanding by Generative Pre-Training

Work-level record. Each recorded version is listed as a source with its own date. A citation to the work does not identify which version the citing author read.

Issued by

No issuing record yet. A missing record is a gap in coverage.

Authors credited

Alec Radford

Dates

No date claim recorded yet. A missing record is a gap in coverage.

Exact source

Improving Language Understanding by Generative Pre-Training OpenAI · publication date unknown · Technical paper PDF served by OpenAI; the PDF prints no date. OpenAI’s release page of 11 June 2018 links it (cross-family check, item A005). Retrieved for the candidate pack on 2026-10-28 (SHA-256 eb81a65e0856bd38e855e1f1de6ee12e7c3eb92ae05c8270d345a475beb631d0). The author list is the byline as printed on the first page.

Appears in

No collection holds a record about this document yet.

Versions

No other versions of this document are recorded. The evidence applies to the source version listed above. Other versions, such as a preprint, working paper or journal publication, are not identified or substituted.

Issue and authorship records

Author: Alec RadfordAlec Radford is a credited author of Improving Language Understanding by Generative Pre-Training.Date unknown · Stated in source · Supported Inspect claim & evidence ↗Alec Radford: person page →

Documentary relations

Each relation is listed separately. Citing, crediting and reporting the use of a method are different acts, and none of them shows agreement or influence by itself.

Cited by · 10

Recorded documents that explicitly reference this one. A citation to a work does not identify which version was read.

Reports using a method from · 1

Methods this document reports using in its own study. Reported use is not independently verified implementation.

Attention Is All You NeedRadford, Narasimhan, Salimans and Sutskever (2018) report using the Transformer, citing Vaswani et al., as their model architecture.2018 · event · Stated in source · Supported Inspect claim & evidence ↗Attention Is All You Need: document page →

Discusses · 1

Concepts explicitly discussed. No earliest use, invention or transmission is claimed.

Transformers and large language modelsRadford et al. (2018) describe generative pre-training of a language model on unlabeled text followed by discriminative fine-tuning on each task.2018 · event · Stated in source · Supported Inspect claim & evidence ↗

Not recorded for this document: cites · explicitly credits · method reported as used by · applies parts of · parts applied by · reports evaluation of. A missing record does not show that the relation does not exist.

What the document says it adds

The document’s own description of its contribution, quoted from the version each row names. A claim to be the first is the document’s claim; Selekoa does not assess it.

In its own wordsThe authors state that large gains on natural language understanding tasks can be achieved by generative pre-training of a language model on unlabelled text followed by discriminative fine-tuning on each task.“We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task.” PDF p. 1, abstractDate unknown · Stated in source · Supported Inspect claim & evidence ↗