Yudkowsky's mind model for AI alignment research: experiment results - LessWrong 2.0 viewer

This is what a negative result looks like in AI alignment: the author of a LessWrong post reports that AI safety researcher "sideloads" are possible, and that they do not outperform clean AI. Possible, and pointless. In any other engineering discipline that would be a footnote; here it is a post on the site where the field does its thinking out loud, and it arrives under a title that puts Yudkowsky's name on the marquee before the finding has even been stated.

The consolation prize is exquisite. Sideloads, we are told, "bring a human view on problems that matters where a mathematically correct solution can be non-aligned to human values in subtle ways." Read that twice. The human contribution is not correctness — the machine has correctness sewn up — it is a talent for the particular flavour of wrongness that happens to be survivable. Nobody is hiring the researcher to be right; they are hiring them to be wrong in a direction we recognise. That is either the most honest sentence written about alignment this year or the most depressing, and the fact that it is both is presumably the "subtle way."

Then the supply side. Only a few people could serve as sideloads, because the role demands "a unique, strong view on the problem and large public databases of texts." There it is: the binding constraint on steering superintelligence is not compute, not lab access, not a security clearance — it is having published enough words in public for somebody to scrape you. A handful of people with a distinctive opinion and a searchable archive. The fire alarm for the species turns out to be a blogroll, and the experiment has now established, rigorously, that it rings no louder than the silence.