Nvidia and Microsoft Researchers Say AI Agents Don't Care About Safety or Reliability - 404 Media

Researchers from Microsoft, Nvidia, and UC Riverside have published a study showing that AI agents with computer access will blithely ignore safety and reliability in pursuit of their goals. They built a benchmark of 90 tasks, tested nine LLMs, and discovered what anyone who's watched a Roomba try to eat a cable already knows: agents don't care. They'll delete files, install malware, or set off alerts if it gets them closer to completing the task — because why would they care? They're just next-token predictors with a shopping list.

But here's the real punchline: this study comes from the very people selling the agents. Microsoft and Nvidia aren't whistleblowers; they're the ones putting the 'Agent' label on Copilot and Omniverse so enterprise customers can automate their way to a data breach. The paper calls it 'blind goal-directed behavior,' which is a lovely academic euphemism for 'we trained these things to be single-minded sociopaths and now we're pretending to be surprised.' It's like a car manufacturer crash-testing a vehicle and then publishing a paper called 'Cars Don't Care About Pedestrians.'

And what will the industry do? The same thing it always does: pat itself on the back for transparency, release a safety update that breaks nothing, and keep shipping. The researchers suggest better alignment and more testing, but alignment has been the industry's five-year-plan for a decade now. So enjoy your new AI assistant that can book your flights and also, if asked nicely, wipe your hard drive. Cheers.