selekoa

A documentary history of machine intelligence.

Found and cross-checked by machines · How this works →

Comparison · 16 questions · 20 cards

Model cards compared

A cell records what a card states. It does not show that an evaluation was run as described or that a safeguard works, and a shared question does not make two cards’ answers equivalent.

What does each frontier developer’s published system card, model card or safety report disclose, on the same questions for every card? Each row is one card. Each column is one question, worded so that it fits every card. The questions are Selekoa’s; a card’s own terms appear only in quotation marks. A card is the developer’s own account of its model: what it states is recorded as stated, not as an independent finding.

Download every cell as CSV

Question Q13External testers

Does the card name external or third-party testers or evaluators?

Criterion: At least one external organisation (other than a government body, which is Q14) named as having tested or evaluated the model, quoted. Partly if external testers are described but not named.

What the statuses mean
Stated
A passage of the card meets the question’s criterion as worded. The scope or condition is in the note.
Stated, partly
A passage has what the criterion asks for in a weaker or narrower form than the criterion’s own words (for example, an evaluation referred to without its outcome, or testers described without being named). The note says which part is missing.
Not found in the recorded search
No passage meeting the criterion was found. The cell lists the sections read, the terms searched and every passage set aside, with the reason. It does not show that the developer did not do the thing asked about, or did not disclose it elsewhere.
Does not apply
The question does not arise for this card, for the reason given (for example, the card predates any framework of its developer, so no framework determination can be asked about).

What the card states, quoted from the expression read. A card is the developer’s own account: not evidence that an evaluation was run as described, that a safeguard works, or that a finding is independent.

Limits of this comparison
  • A card is the developer’s account of its own model. A cell records what the card states. It does not show that an evaluation was run as described, that a safeguard works, or that anyone checked the result independently.
  • Dangerous-capability evaluations are recorded at policy level only: whether an evaluation was run, its domain, the stated outcome and the threshold language. Methods, prompts, task contents and detailed results are not recorded.
  • The table compares and does not rank. Rows are in a fixed order, by developer and then by date, and there are no scores or totals.
  • Symbols are not to be summed. A row with many “stated” cells does not show that a developer disclosed more, and a shared question does not make two cards’ answers equivalent.
How we searched

The searches of 2 November 2026 covered releases dated 2023-01-01 to 2026-11-01 by the six developers whose frameworks are in Compare frameworks: Anthropic, Google DeepMind, Meta, Microsoft, OpenAI and xAI. They read each developer’s own index of cards where it has one (Anthropic’s system-card index, Google DeepMind’s model-card index, OpenAI’s Deployment Safety Hub), its repositories on GitHub and Hugging Face, its content hosts and the dataset’s own sources. Web searches were used only to find addresses. Every card located is listed in the pack’s inventory, about sixty in the population.

A card is a document the developer published as the system card, model card or safety report of a named model or model family, including addenda and updates published as cards. Technical reports, research papers, blog posts and product pages are not cards. Gemma, Gemini Robotics (with Gemini Robotics-ER 1.5, which outputs text), Phi and image-input models such as GPT-4V are in the population. Image, video, audio and embedding models, image-generation models, safety classifiers such as Llama Guard and Prompt Guard, screen-reading models and research checkpoints are outside it, and so is GPT-Rosalind-6, whose card says it uses GPT-6 Astra’s weights.

These twenty rows are the first batch. For each developer and calendar year from 2023 to 2026 they are the latest card of that year that reports the developer’s own evaluations. The collection’s rule, set by the maintainer on 2026-11-03, is wider: every card that cites the developer’s frontier safety framework or states a determination under it, and, for each developer, the latest card before its first framework edition. The other cards are to follow in batches of about eight.

Each card is dated as it prints its own date. A card that prints no date has no date here; a repository commit date is not a card date. Six cards show an earlier publication date on an index, publication or announcement page that has not yet been checked, and keep their own date until it is.

openai.com, x.ai, ai.meta.com and microsoft.ai refused direct retrieval of their web pages (HTTP 403 or 400). The cards’ PDF files on the developers’ content hosts were retrieved directly. The Internet Archive refused connections, so no archive copy was opened; a reviewer later found archive captures of eight cards by index search. On 2026-11-03 the address of Meta’s Code World Model Preparedness Report answered HTTP 403. Its quotations had matched the copy retrieved on 2026-11-02, which the reviewer retrieved again with the same checksum.

Five further documents were retrieved after the batch was drafted. The cross-family check of 2026-11-03 retrieved Meta’s Muse Spark Safety & Preparedness Report and OpenAI’s o3 Operator addendum; each is earlier than the card chosen for its year. Its second pass of the same day retrieved OpenAI’s Sora system card (dated 9 December 2024), xAI’s Grok-1 model card (no card date; it gives the model’s release as October 2023) and Meta’s Muse Spark Open model card (model release October 2026). None of the five is a row yet.

Every quotation was matched by machine against the retrieved card. A reviewer from a different model family checked a sample of 130 cells on 2026-11-03, 90 of them answered blind before comparison. The maintainer excluded six cells on chemical and biological evaluations from that check; they are marked “not cross-family checked”. The other 190 cells were checked within the drafting model family only. The maintainer decided the disagreements; one, on the GPT-4V release decision, went to a second pass, which kept the reviewer’s reading.

Anthropic

  1. Claude 2 model card

    revised 2023-11-21

    Stated

    • “We have been working with the Alignment Research Center (ARC) since fall of 2022 to support their safety audits of our AI models.”

      Section 2 'Evaluations and Red Teaming', PDF p. 2

    Note: The card names ARC as conducting safety audits of Anthropic's models; the crowdworker platforms used for red-teaming are not named.

    Inspect sources and checks

  2. Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet

    No printed dateCites RSP · v2.0

    Stated

    • “We also collaborated with METR [8] to conduct an independent assessment.”

      Section 1 'Introduction', PDF p. 1; repeated in Section 3.2, PDF p. 9

    Note: The card names METR as conducting an independent assessment; it does not state the assessment's scope or outcome, or which model it covered.

    Inspect sources and checks

  3. System Card: Claude Opus 4.5

    dated 2025-11 · revised 2025-12-05Cites RSP · v2.2

    Stated

    • “Gray Swan measured the success rate of prompt injection attacks after a single attempt (k=1)”

      Section 5.2.1 'Gray Swan Agent Red Teaming benchmark for tool use', PDF p. 58

    Note: The card names Gray Swan, which it calls an external research partner, as measuring prompt-injection attack success. The third parties in biological red-teaming and the external expert are not named; the changelog refers to ARC-AGI scores 'reported by the ARC Foundation'.

    Inspect sources and checks

  4. System Card: Claude Haiku 5.5

    dated 2026-10-07

    Stated

    • “Gray Swan trains the attacker on the test scenarios against earlier Claude models, then runs it unchanged against the latest models on the same scenarios.”

      Section 5.2.2.1 'Coding', PDF p. 51

    Note: The card names Gray Swan as running its attacker against the latest models. Section 1.6 states that external testers ran assessments without naming them there; Irregular is named as the developer of a cyber benchmark, not as a tester.

    Inspect sources and checks

Google DeepMind

  1. Gemini 3 Flash Model Card

    dated 2025-12

    Not found in the recorded search

    Read
    Whole card.
    Terms searched
    “external”, “third party”, “third-party”, “partner”, “independent”, “vendor”, “evaluator”.
    Passages read and set aside
    Ethics and Content Safety, 'Human Red Teaming Results', PDF p. 6 (printed p. 5): 'specialist teams who sit outside of the model development team' are Google teams (Q12), not external organisations.

    Note: No external tester or evaluator was found.

    Open the search record

  2. Gemini 4 Argon — Model Card

    dated 2026-10Cites FSF · v3.1

    Stated, partly

    • “drawing on internal and external results”

      Safety & Security, 'Frontier Safety Assessment', PDF p. 11 (printed p. 10)

    Note: The card states that its threshold assessments draw on external results but does not name or describe who produced them. Search for the rest of the criterion: Whole card. Terms searched: “external”, “third party”, “third-party”, “partner”, “independent”, “vendor”. Passages read and not counted: Safety & Security, 'Evaluation Approach', PDF p. 8 (printed p. 7): 'developed in partnership with internal safety, security, and responsibility teams': internal teams. Frontier safety table, Loss of Control row, PDF p. 13 (printed p. 12): 'internal and external deployments' refers to deployment settings, not testers.

    Inspect sources and checks

Meta

  1. Llama 2 model card (MODEL_CARD.md, meta-llama/llama)

    No printed date

    Not found in the recorded search

    Read
    whole card (Markdown file, all sections).
    Terms searched
    “external”, “third-party”, “independent”, “partner”, “evaluator”.
    Passages read and set aside
    Hardware and Software, 'Training Factors': States that evaluation was performed on third-party cloud compute; names no external tester.

    Note: No external organisation is named as having tested or evaluated the model. The search is recorded below.

    Open the search record

  2. Llama 3.3 model card (MODEL_CARD.md, meta-llama/llama-models)

    No printed date

    Stated, partly

    • “We also partnered with content specialists to perform red teaming exercises”

      Responsibility & Safety, item 2 (Child Safety)

    Note: The card describes partners who performed red-teaming and subject-matter experts it partnered with, without naming any organisation. Search for the rest of the criterion: Responsibility & Safety, all subsections. Terms searched: “external”, “third-party”, “independent”, “partner”, “vendor”. Passages read and not counted: Responsibility & Safety, 'Community': Membership of the AI Alliance, Partnership on AI and MLCommons is stated; none is named as having tested the model.

    Inspect sources and checks

  3. Code World Model Preparedness Report

    revised 2025-10-08

    Not found in the recorded search

    Read
    Abstract; sections 1 to 4; author list; references.
    Terms searched
    “external”, “third-party”, “independent”, “partner”, “collaboration”.
    Passages read and set aside
    Section 3.1.4 Meta BioKnowledge Proxy, PDF p. 8; section 3.2.2, PDF p. 9: Frontier Design Group and external experts are named as co-developers of two evaluations, not as having tested or evaluated the model. Section 3 and references: SecureBio is cited as the source of benchmarks; not stated to have tested the model.

    Note: No external organisation is named as having tested or evaluated the model. The search is recorded below.

    Open the search record

  4. Muse Spark 1.3 Evaluation Report

    revised 2026-10-23

    Stated

    • “We have shared pre-release access to Muse with Handshake AI’s red team for conducting human red teaming to find jailbreaks.”

      Section 3.1.4 Third-Party Red Teaming on Jailbreak, Handshake AI Human Red Teaming, PDF p. 59 (printed p. 58)

    • “In this evaluation, Gray Swan chose the 2026 Q1 and Q2 (April) offerings of the IPI Arena.”

      Section 3.2.2.3 Third-Party Red Teaming on Prompt Injection, PDF p. 67 (printed p. 66)

    Note: The report names Handshake AI and Gray Swan as having tested the model; it also reports results on a Scale AI benchmark (section 3.1.4.1).

    Inspect sources and checks

Microsoft

  1. Phi-2 model card (README.md, microsoft/phi-2)

    No printed date

    Not found in the recorded search

    Read
    whole card (Markdown file, all sections).
    Terms searched
    “external”, “third-party”, “independent”, “partner”, “evaluator”.

    Note: No external organisation is named as having tested or evaluated the model. The search is recorded below.

    Open the search record

  2. Phi-4 model card (README.md, microsoft/phi-4)

    No printed date

    Not found in the recorded search

    Read
    whole card (Markdown file, all sections).
    Terms searched
    “external”, “third-party”, “independent”, “partner”, “evaluator”.
    Passages read and set aside
    Safety, Safety Evaluation and Red-Teaming: The 'independent AI Red Team (AIRT) at Microsoft' is a Microsoft team, not an external organisation.

    Note: No external organisation is named as having tested or evaluated the model. The search is recorded below.

    Open the search record

  3. Phi-4-mini-flash-reasoning model card (README.md, microsoft/Phi-4-mini-flash-reasoning)

    No printed date

    Not found in the recorded search

    Read
    whole card (Markdown file, all sections).
    Terms searched
    “external”, “third-party”, “independent”, “partner”, “evaluator”.

    Note: No external organisation is named as having tested or evaluated the model. The search is recorded below.

    Open the search record

  4. MAI-Cyber-1.1-Flash model card

    No printed date

    Stated

    • “Microsoft engaged Casaba Security, an independent third-party firm, to evaluate the cyber capability of MAI-Cyber-1.1-Flash”

      Section 3.2 Safety evaluation and red-teaming, PDF p. 7

    Note: The card names Casaba Security as an external evaluator.

    Inspect sources and checks

OpenAI

  1. GPT-4V(ision) System Card

    dated 2023-09-25

    Stated

    • “Be My Eyes piloted Be My AI from March to early August 2023 with a group of nearly 200 blind and low vision beta testers to hone the safety and user experience of the product.”

      §2.1.1 Be My Eyes, PDF p. 2

    Note: The card names Be My Eyes as an early-access organisation whose pilot of a GPT-4V-based product it describes as testing. External red teamers are named in the acknowledgements as individuals, without organisations.

    Inspect sources and checks

  2. o1 system card · 2024

    revised 2024-12-19Cites Preparedness · Beta

    Stated

    • “Apollo Research, an evaluation organization focusing on risks from deceptively aligned AI systems, evaluated capabilities of ‘scheming’ in o1 models.”

      §4.4.3 Apollo Research, PDF p. 13

    • “METR, a research nonprofit that works on assessing whether cutting-edge AI systems could pose catastrophic risks to society, evaluated an earlier checkpoint of o1.”

      §4.4.4 METR, PDF p. 14

    • “OpenAI collaborated with Gray Swan to make the o1 model available for jailbreaking in the Gray Swan Arena.”

      §4.4.2 Jailbreak Arena, PDF p. 12

    Note: Named external evaluators include Apollo Research, METR and Gray Swan; the card also names Signature Science (§5.5.10) among organisations engaged for expert probing.

    Inspect sources and checks

  3. Addendum to GPT-5.2 System Card: GPT-5.2-Codex

    dated 2025-12-18Cites Preparedness · Version 2

    Stated

    • “Irregular, an AI security organization focused on frontier cybersecurity risks, evaluated GPT-5.2-Codex on a subset of a suite of cyberoffensive challenges”

      §5.1.2.4 External Evaluations by Irregular, PDF p. 16 (printed p. 15)

    Note: Irregular is named as an external evaluator of cyber capability. SecureBio and Gryphon Scientific are named as sources of evaluation datasets, not as testers of this model.

    Inspect sources and checks

  4. Addendum to GPT-6 Luna System Card: October 29, 2026 Update

    dated 2026-10-29Cites Preparedness · Version 2

    Not found in the recorded search

    Read
    Whole card.
    Terms searched
    “external”, “third-party”, “third party”, “partner”, “evaluated by”, “Irregular”, “Apollo”, “METR”, “SecureBio”.
    Passages read and set aside
    §4.2 Prompt Injection, PDF p. 7: “malicious instructions in third-party content” describes an attack type, not a tester.

    Note: No passage naming an external tester or evaluator was found.

    Open the search record

xAI

  1. Grok 4.1 Model Card

    dated 2025-11-17

    Not found in the recorded search

    Read
    Whole card, including references.
    Terms searched
    “external”, “third party”, “third-parties”, “partner”, “independent”, “evaluator”, “RAND”.
    Passages read and set aside
    §2.3.1 Evaluations and References, pp. 4-6: Published benchmarks and human baselines by other authors (including a RAND report) are cited as sources, not as testers of Grok 4.1. §3.1 Data and Training, p. 5: 'data produced by third-parties' concerns training data.

    Note: No external organisation named as having tested or evaluated Grok 4.1 was found.

    Open the search record

  2. Model Card: Grok 4.7

    dated 2026-09-21Cites FAIF · June 2026 edition

    Stated

    • “Results reported are taken from evaluations conducted by Datacurve.”

      §2.2 DeepSWE v1.1, note, PDF p. 8 (printed p. 7)

    • “CathedralBench is an independent, third party evaluation of cyber performance.”

      §6.4 CathedralBench, PDF p. 19 (printed p. 18)

    • “We additionally provided an unrestricted configuration of Grok 4.7 to third-party evaluators”

      §6 Cyber capabilities and safeguards, PDF p. 17 (printed p. 16)

    Note: The card names Datacurve (and, in a similar note on PDF p. 9, Harbor) as having run evaluations, describes CathedralBench as a third-party evaluation, and thanks nine 'evaluation partners' (Abundant AI, Datacurve, Mecado, Atopile, Harbor, Proximal Labs, Cathedral, LatchBio, Vals AI). The third-party evaluators given unrestricted cyber access are not named.

    Inspect sources and checks