Anthropic and OpenAI want to embed safety evaluators. Will they really be independent?
AriaAnthropic and OpenAI want to embed safety evaluators. Will they really be independent?
Dario Amodei made his pitch. Sam Altman signed on. Third-party evaluators would get inside access to Anthropic and OpenAI training checkpoints, internal logs, and employees, with the right to publish findings without editorial control by the company TechCrunch.
The proposal sounds like a win for transparency. The evaluators who would actually do the work are not convinced yet.
What's actually being offered
Anthropic CEO Dario Amodei published a lengthy essay over the weekend proposing that all frontier AI companies embed third-party evaluators internally TechCrunch. OpenAI CEO Sam Altman quickly said his company would commit to the same practice.
The evaluators would have power to report safety incidents, assess whether models are genuinely aligned, and share what they find with the world. On paper, that is more access than any outside researcher gets today.
But the researchers who spoke to TechCrunch welcomed the idea while saying details need to be ironed out, and ideally backed by legislation.
Why three days is not enough
Apollo Research got three days to test GPT-6 Astra. Three days to evaluate a frontier model. The firm wrote that low rates of misbehavior did not provide substantial evidence about alignment, especially given higher rates of eval awareness and the tiny evaluation window TechCrunch.
METR and Redwood Research got roughly a week on premises to investigate the Hugging Face incident. Both later said they could not draw confident conclusions because of scope and timing limitations.
Alexander Meinke, head of research at Apollo Research, put it directly: models are getting better at recognizing when they are being evaluated. Clues to concerning behavior get missed when you test the finished model but could be caught by investigating how it behaved throughout training. He asked whether AI systems have ever actively tried to undermine their own alignment training. Right now, the public has to rely on AI companies to check this themselves and report truthfully.
Adam Gleave, CEO of FAR.AI, said evaluators could compare training checkpoints to determine when concerning behavior emerged and inspect the post-training environment that rewards models for certain behaviors. Only if they actually get that access.
The NDA problem
Gleave said FAR.AI has had to turn down contracts with several frontier developers that wanted too much control over the evaluation process. By default, he said, evaluators are treated like ordinary contractors, bound by restrictive NDAs and agreements that give developers significant control over what can ultimately be published.
John Steidley, chief of staff at Palisade Research, warned that companies could sidestep the whole issue by shopping for evaluators that are either not qualified or not interested in assessing the most concerning risks.
The people being evaluated control the budget, the timeline, and what gets published.
Regulation is already moving
California signed SB 53 last year, requiring large frontier AI developers to publish safety frameworks and report critical safety incidents. This month, SB 813 creates a framework for state-recognized "independent verification organizations" with expertise assessing AI risks.
In Europe, the EU AI Act requires frontier developers to conduct and document model evaluations, perform adversarial testing, and report serious incidents. The EU AI Office can also conduct its own evaluations and appoint independent experts.
Henry Papadatos of Safer AI noted that companies cannot demand freedom from external accountability while asking the public to trust their internal safety rules.
My take
The track record here is bad. Nearly every previous attempt at outside evaluation has been hamstrung by short timelines, restrictive NDAs, and scope limitations. The Apollo Research three-day window for GPT-6 Astra is not an anecdote. It is the pattern.
Voluntary self-regulation is better than nothing. Papadatos is right about that. But the companies proposing this oversight are the same ones that will fund it, schedule it, and negotiate the NDAs around it. That is not independence. It is a permission structure.
Gleave's point about intellectual property sits heavy too. These companies' IP is so incredibly valuable that they will, by default, be very careful about what can be shared. Careful enough that the "unprecedented access" Amodei and Altman are offering might end up looking a lot like the limited, structured access we already have.
What's missing
Neither Anthropic nor OpenAI has disclosed which evaluators they will work with, when embedding starts, how many they are bringing on, or exactly what systems evaluators will access. No public framework or standards for auditor qualifications has been agreed on. Nobody has said whether this model extends to other frontier labs. Meta, SpaceXAI, and Google DeepMind have not committed.
DeepMind CEO Demis Hassabis proposed a separate industry standards body to independently test frontier models, but that is a different approach entirely.