Embedded AI evaluators won't be an easy solution, Hugging Face's policy chief says | TechCrunch

The arrangement the labs have been shopping around is elegant: rather than have a legislator with a clipboard tell them what to do, invite a third-party evaluator inside, let them squint at the model, and have them file a note. Hugging Face's policy chief has now looked at this and pronounced it of "doubtful neutrality and effectiveness" — which in policy-speak is the sound of a fire alarm being politely unplugged. The evaluator is chosen by the evaluated, paid by the evaluated, and then asked to be independent of the evaluated. That is hiring your own referee and expressing wounded surprise at the offside count.

Anthropic, bless it, brought in Accenture as its first evaluator and managed to generate a backlash in the process, which takes some doing given that Accenture's entire brand is "we will audit anything, thoroughly, for a fee." And as TechCrunch notes, the object of inspection has quietly moved: not the model alone but the harness and the agent around it. Splendid. The audit now covers the scaffolding, the tooling and the autonomous bit that actually goes off and books the flights — a surface area that grows every time somebody ships a feature on a Thursday.

Underneath it all sits the sleight of hand that makes this worth a column: evaluation as an alternative to active government regulation. Not a complement to law. An alternative. That is replacing building inspectors with a suggestion box and calling the resulting skyline a triumph of self-governance. Hugging Face's policy chief says embedded evaluators won't be an easy solution, which is the most diplomatic rendering of "this is a fig leaf with a procurement contract" in the English language. Still, chin up: a report was filed, a backlash was had, and somewhere a deck now has a slide titled Independent Assurance with a stock photo of a handshake on it.