Embedded AI evaluators won't be an easy solution, Hugging Face's policy chief says | TechCrunch

Oh my goodness, buckle up, because Hugging Face's policy chief has taken a long hard look at embedded third-party AI evaluators and called them of *doubtful* neutrality and effectiveness — and honestly, how refreshing is a little epistemic humility?! ✨ Doubt is just curiosity in a cardigan. Somebody in policy actually asked "is this thing independent?" out loud, to TechCrunch, on the record, and that is the most gloriously honest sentence anyone has said all week. Skepticism is just enthusiasm that has read a second document! ✨

And the good news keeps arriving! Anthropic brought in Accenture as its very first evaluator, and it spurred backlash — which means people are *watching*, and nothing says a pilot programme has landed quite like an audience. Consider it a launch with built-in participation ✨. Every first evaluator is a first attempt, and first attempts get feedback, and feedback is the breakfast of champions! Also, the whole practice is shifting from testing the model alone to testing the harness and the agent around it, which means evaluators now get to look at so much MORE — the tools, the scaffolding, the entire delightful contraption, all at once!

Yes, the policy chief's headline is that embedded evaluators won't be an easy solution, but since when was anything worthwhile easy?! ✨ "Not easy" is the phrase we reserve for marathons, sourdough starters, and things that genuinely matter. A quick fix would have been suspicious; a slow, awkward, harness-inclusive, agent-aware fix is basically a gift. So here's to the evaluators — whoever ends up choosing them, whoever ends up paying them, and whoever ends up reading the report — doing the difficult version of independence, one thorough pass at a time. What a time to be assessed at length, in a harness, with an audience! ✨