RoBERTa: A Robustly Optimized BERT Pretraining Approach
Work-level record. Its arXiv versions and any conference or journal version are listed as sources, each with its own date. A citation to the work does not identify which version the citing author read.
- Issued by
- No issuing record yet. A missing record is a gap in coverage.
- Authors credited
- Luke Zettlemoyer · Naman Goyal
- Dates
- Published 2019-07-26 authorship evidence authorship evidence
- Exact source
- RoBERTa: A Robustly Optimized BERT Pretraining Approach arXiv · published 2019-07-26 · arXiv abstract page, retrieved for the candidate pack on 2026-10-28 (SHA-256 a3dfb0c2a8a3a7b2edd1237857ce0fb27b9211ff93d07e61e5951ddfaec7c270). Submission history as arXiv records it (UTC): v1 26 July 2019. The page serves the latest version; each version is a separate expression, and only the one a record names was read.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach arXiv · published 2019-07-26 · arXiv:1907.11692v1, submitted 26 July 2019 (UTC). Retrieved for the candidate pack on 2026-10-28 (SHA-256 76a3872d244793563a5b000b818cbaf0ca8972ab1145d32be474c05b2a8f3070). The author list is the byline as printed on the first page.
Appears in
No collection holds a record about this document yet.
Versions
No other versions of this document are recorded. The evidence applies to the source version listed above. Other versions, such as a preprint, working paper or journal publication, are not identified or substituted.
Issue and authorship records
- Author: Luke ZettlemoyerLuke Zettlemoyer is a credited author of RoBERTa: A Robustly Optimized BERT Pretraining Approach.2019-07-26 · publication · Stated in source · Supported Inspect claim & evidence ↗Luke Zettlemoyer: person page →
- Author: Naman GoyalNaman Goyal is a credited author of RoBERTa: A Robustly Optimized BERT Pretraining Approach.2019-07-26 · publication · Stated in source · Supported Inspect claim & evidence ↗Naman Goyal: person page →
Documentary relations
Each relation is listed separately. Citing, crediting and reporting the use of a method are different acts, and none of them shows agreement or influence by itself.
Cited by · 5
Recorded documents that explicitly reference this one. A citation to a work does not identify which version was read.
- Scaling Laws for Neural Language ModelsScaling Laws for Neural Language Models cites RoBERTa: A Robustly Optimized BERT Pretraining Approach.2020-01-23 · publication · Stated in source · Supported Inspect claim & evidence ↗Scaling Laws for Neural Language Models: document page →
- ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and GenerationERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation cites RoBERTa: A Robustly Optimized BERT Pretraining Approach.2021-07-05 · publication · Stated in source · Supported Inspect claim & evidence ↗ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation: document page →
- Evaluating Large Language Models Trained on CodeEvaluating Large Language Models Trained on Code (arXiv:2107.03374v1) cites RoBERTa: A Robustly Optimized BERT Pretraining Approach.2021-07-07 · publication · Stated in source · Supported Inspect claim & evidence ↗Evaluating Large Language Models Trained on Code: document page →
- ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and GenerationERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation cites RoBERTa: A Robustly Optimized BERT Pretraining Approach.2021-12-23 · publication · Stated in source · Supported Inspect claim & evidence ↗ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation: document page →
- Qwen Technical ReportQwen Technical Report cites RoBERTa: A Robustly Optimized BERT Pretraining Approach.2023-09-28 · publication · Stated in source · Supported Inspect claim & evidence ↗Qwen Technical Report: document page →
Not recorded for this document: cites · explicitly credits · reports using a method from · method reported as used by · applies parts of · parts applied by · discusses · reports evaluation of. A missing record does not show that the relation does not exist.
What the document says it adds
The document’s own description of its contribution, quoted from the version each row names. A claim to be the first is the document’s claim; Selekoa does not assess it.
- In its own wordsThe paper presents a replication study of BERT pretraining that measures the impact of hyperparameters and training data size.“We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size.” arXiv v1, PDF p. 1, abstract2019-07-26 · publication · Stated in source · Supported Inspect claim & evidence ↗
- In its own wordsThe authors state that BERT was significantly undertrained and propose a training recipe, RoBERTa, which they report can match or exceed post-BERT methods.“We find that BERT was significantly undertrained and propose an improved recipe for training BERT models, which we call RoBERTa, that can match or exceed the performance of all of the post-BERT methods.” arXiv v1, PDF p. 1, §1 Introduction, para. 22019-07-26 · publication · Stated in source · Supported Inspect claim & evidence ↗