This New Theory on How AI 'Thinks' Could Impact Whether It Kills Us - Business Insider

A bloke named Nick Cammarata, who left OpenAI in 2023 — so, before it was properly interesting — has published a report on Thursday that "some in the AI world" are calling a bombshell. That is the entire attribution: some in the AI world, a phrase that has never once preceded a sober assessment of anything. His thesis, per Business Insider, is that neural networks are knowable so long as you look carefully. Marvellous. So is the bottom of the sea, given a torch, a free afternoon and no particular deadline.

The headline, being a headline, goes straight for "Could Impact Whether It Kills Us," which is the journalistic equivalent of printing "may contain nuts" on a bag of nuts. The actual news peg underneath is that safety researchers are "scrambling to understand what happens between what goes into a model and what comes out" — a sentence that ought to be stamped in eight-point type on the side of every frontier model shipped this year, next to the terms nobody reads. We have built cathedrals of compute and are now publishing papers proposing that somebody might pop inside for a look.

The "bombshell" framing deserves a small plaque of its own. Bombshells arrive uninvited and rearrange the furniture; this is a researcher suggesting interpretability might be tractable if people try, which in any other engineering discipline is the bit where someone says "well, yes, obviously." The practical upside, as the article has it, is that the theory could matter for safety work — the phrase you reach for when you don't yet know whether it matters at all. Godspeed to him, and to the researchers who will spend eighteen months arguing about whether "look carefully" counts as a method. Still, one man leaving a lab to publish an idea beats the alternative: everyone staying put, saying nothing, and calling it a safety team.