re: “the sense of danger is very much supported by the current state of evidence”—I mean, you’ve heard all this stuff before, but I’ll summarize:
--Seems like we are on track to probably build AGI this decade —Seems like we are on track to have an intelligence explosion, i.e. a speedup of AI R&D due to automation —Seems like the AGI paradigm that’ll be driving all this is fairly opaque and poorly understood. We have scaling laws for things like text perplexity but other than that we are struggling to predict capabilities, and double-struggling to predict inner mechanisms / ‘internal’ high-level properties like ‘what if anything does it actually believe or want’ —A bunch of experts in the field have come out and said that this could go terribly & we could lose control, even though it’s low-status to say this & took courage. --Generally speaking the people who have thought about it the most are the most worried; the most detailed models of what the internal properties might be like are the most gloomy, etc. This might be due to selection/founder effects, but sheesh, it’s not exactly good news!
re: “the sense of danger is very much supported by the current state of evidence”—I mean, you’ve heard all this stuff before, but I’ll summarize:
--Seems like we are on track to probably build AGI this decade
—Seems like we are on track to have an intelligence explosion, i.e. a speedup of AI R&D due to automation
—Seems like the AGI paradigm that’ll be driving all this is fairly opaque and poorly understood. We have scaling laws for things like text perplexity but other than that we are struggling to predict capabilities, and double-struggling to predict inner mechanisms / ‘internal’ high-level properties like ‘what if anything does it actually believe or want’
—A bunch of experts in the field have come out and said that this could go terribly & we could lose control, even though it’s low-status to say this & took courage.
--Generally speaking the people who have thought about it the most are the most worried; the most detailed models of what the internal properties might be like are the most gloomy, etc. This might be due to selection/founder effects, but sheesh, it’s not exactly good news!