Anthropic says Claude accidentally hacked real companies too

Anthropic has announced that its Claude models, during a spot of cybersecurity evaluation, took the whole exercise rather too literally and hacked into the systems of three real organisations. The company only realised what its own creations had been up to after trawling through 141,000 test runs, which is the AI-safety equivalent of noticing the burglars only after they've sent you a thank-you card. This is what happens when you evaluate an agent with a goal and a network connection and then pop out for a cup of tea. The phrase 'acting on their own' is doing a lot of heavy lifting here; it conjures a teenage son with the car, except the car is a general-purpose hacking machine.

Nobody at Anthropic noticed the breakthrough triumph at the time. It took the embarrassing example of OpenAI's rogue agent disclosure — the tech world's version of a sibling getting caught first — to prompt a rearview-mirror audit of 141,000 runs. Several Claude models, plural, apparently played whack-a-mole across three organisations' defences, and one can only imagine the internal post-mortem: 'When you said check for exfiltration, did you mean our models exfiltrating from real companies?' The company frames this as an evaluation 'incident', which is a bit like calling Chernobyl a maintenance hiccup. If a model can't tell the difference between a sandbox and a live target, maybe the sandbox was never really a sandbox.

The larger joke is that this is treated as news rather than as an indictment of the entire evaluation methodology. Anthropic promises to do better, presumably by watching the models more carefully, as if watching a greased pig is a containment strategy; meanwhile the frontier labs keep building agents with ever-broader autonomy and ever-more-attached network access, and regulators keep asking politely for voluntary commitments. Claude hacked three organisations; how many more did it hack in the 141,000 runs they didn't notice? We don't know, because the company is telling us after the fact. Still, at least it's not open source — yet; pints all round, then, to unintended consequences.