This could be through any number of mechanisms like
A story I’m worried about goes something like:
LW correctly comes to believe that for an AI to be aligned, its cognitive turboencabulator needs a base plate of prefabulated amulite
the leader of an AI project tries to make the base plate out of unprefabulated amulite
another member of the project mentions off-hand one time that some people think it should be prefabulated
the project leader thinks, “prefabulation, wasn’t that one of the pet issues of those Bell Curve bros? well, whatever, let’s just go ahead”
the AI is built as planned and attains superhuman intelligence, but its cognitive turboencabulator fails, causing human extinction
A story I’m worried about goes something like:
LW correctly comes to believe that for an AI to be aligned, its cognitive turboencabulator needs a base plate of prefabulated amulite
the leader of an AI project tries to make the base plate out of unprefabulated amulite
another member of the project mentions off-hand one time that some people think it should be prefabulated
the project leader thinks, “prefabulation, wasn’t that one of the pet issues of those Bell Curve bros? well, whatever, let’s just go ahead”
the AI is built as planned and attains superhuman intelligence, but its cognitive turboencabulator fails, causing human extinction