It’s physically possible to build STEM+ AI (though it’s OK if we collectively decide not to build it.)
STEM+ AI will exist by the year 2035 but not by 2100 (human scientists will improve significantly thanks to cohering blending volition, aka ⿻Plurality).
If STEM+AI is built, within 10 years AIs will be able to disempower humanity (but won’t do it.)
The future will be better if AI does not wipe out humanity.
Given sufficient technical knowledge, humanity could in principle build vastly superhuman AIs that reliably produce very good outcomes.
Researchers will produce good enough processes for continuous alignment.
Technical research and policy work are ~equally useful.
Governments will generally be reasonable in how they handle AI risk.
Thanks! You seem really confident that enough of the alignment problem will be solved in time and that governments will be generally reasonable. I’d love to hear more elaboration on those points; this seems like the biggest disagreement between us.
Based on recent conversations with policymakers, labs and journalists, I see increased coordination around societal evaluation & risk mitigation — (cyber)security mindset is now mainstream.
Also, imminent society-scale harm (e.g. contextual integrity harms caused by over-reliance & precision persuasion since ~a decade ago) has shown to be effective in getting governments to consider risk reasonably.
I definitely agree that policymakers, labs, and journalists seem to be “waking up” to AGI risk recently. However the wakeup is not a binary thing & there’s still a lot of additional wakeup that needs to happen before people behave responsibly enough to keep the risk below, say, 10%. And my timelines are short enough that I don’t currently expect that to happen in time.
Based on my personal experience in pandemic resilience, additional wakeups can proceed swiftly as soon as a specific society-scale harm is realized.
Specifically, as we are waking up to over-reliance harms and addressing them (esp. within security OODA loops), it would buy time for good enough continuous alignment.
Dang, assuming you’re who I think you are, great to see you here. If you aren’t, great to see you here anyway, and you should look up people you have a name collision with, as one of them is inspiring; and nice to meet you either way!
I’m impressed that Ben Goertzel got a politician to adopt his obscure alternative to CEV as their model of the future! This after the Saudis made his robot Sophia into the first AI citizen anywhere. He must have some political skills.
Hi, it’s kind of an honor! We’ve had at least one billionaire and one celebrity academic comment here, but I don’t remember ever seeing a government minister before. :-)
Is there a story to how a Taiwanese digital civil society forum, ended up drawing inspiration from CBV?
I just mean he must know how to liaise credibly and effectively with politicians (although Minister Tang has clarified that she knew about his alignment ideas even before she went into government). And I find that impressive, given his ability to also liaise with people from weird corners of AI and futurology. He was one of the very first people in AI to engage with Eliezer. He’s had a highly unusual career.
STEM+ AI will exist by the year 2035 but not by year 2100 (human scientists will improve significantly thanks to cohering blending volition, aka ⿻Plurality).
Are you saying that STEM+ AI won’t exist in 2100 because by then human scientists will have become super good, such that the bar for STEM+ AI (“better at STEM research than the best human scientists”) will have gone up?
If this is your view it sounds extremely wild to me, it seems like humans would basically just slow the AIs down. This seems maybe plausible if this is mandated by law, i.e. “You aren’t allowed to build powerful STEM+ AI, although you are allowed to do human/AI cyborgs”.
Why do you think the pedantic medical issues (brain swelling, increased risks of various forms of early death) from brain implants will be solved pre- (ASI driven) singularity? Gene hacks exhibit the same issues. To me these problems look unsolvable in that getting from “it’s safe 90-99 percent of the time, 1% uh oh” to “it’s always safe, no matter what goes wrong we can fix it” requires superintelligent medicine, because you’re dealing with billions or more permutations of patient genetics and rare cascading events.
Safer than implants is to connect at scale “telepathically” leveraging only full sensory bandwidth and much better coordination arrangements. That is the ↗️ direction of the depth-breadth spectrum here.
How do you propose the hardware that does this works? I thought you needed wires to the outer regions of the brain with enough resolution to send/receive from ~1-10 target axons at a time.
Something like a lightweight version of the off-the-shelf Vision Pro will do. Just as nonverbal cues can transmit more effectively with codec avatars, post-symbolic communication can approach telepathy with good enough mental models facilitated by AI (not necessarily ASI.)
That sounds kinda like in person meetings. And you have the issue, same with those, of revealing information you didn’t intend to disclose, and the issues that happen when the parties incentives aren’t aligned.
Yes. The basic assumption (of my current day job) is that good-enough contextual integrity and continuous incentive alignment are solvable well within the slow takeoff we are currently in.
Fair. You think the takeoff is rate limited by compute, which is being produced at a slowly accelerating rate? (Nvidia has increased h100 run rate several fold, amd dropped their competitor today, etc)
Thank you for this. Here’s mine:
It’s physically possible to build STEM+ AI (though it’s OK if we collectively decide not to build it.)
STEM+ AI will exist by the year 2035 but not by 2100 (human scientists will improve significantly thanks to cohering blending volition, aka ⿻Plurality).
If STEM+AI is built, within 10 years AIs will be able to disempower humanity (but won’t do it.)
The future will be better if AI does not wipe out humanity.
Given sufficient technical knowledge, humanity could in principle build vastly superhuman AIs that reliably produce very good outcomes.
Researchers will produce good enough processes for continuous alignment.
Technical research and policy work are ~equally useful.
Governments will generally be reasonable in how they handle AI risk.
Always glad to update as new evidence arrives.
Thanks! You seem really confident that enough of the alignment problem will be solved in time and that governments will be generally reasonable. I’d love to hear more elaboration on those points; this seems like the biggest disagreement between us.
Based on recent conversations with policymakers, labs and journalists, I see increased coordination around societal evaluation & risk mitigation — (cyber)security mindset is now mainstream.
Also, imminent society-scale harm (e.g. contextual integrity harms caused by over-reliance & precision persuasion since ~a decade ago) has shown to be effective in getting governments to consider risk reasonably.
I definitely agree that policymakers, labs, and journalists seem to be “waking up” to AGI risk recently. However the wakeup is not a binary thing & there’s still a lot of additional wakeup that needs to happen before people behave responsibly enough to keep the risk below, say, 10%. And my timelines are short enough that I don’t currently expect that to happen in time.
What about the technical alignment problem crux?
Based on my personal experience in pandemic resilience, additional wakeups can proceed swiftly as soon as a specific society-scale harm is realized.
Specifically, as we are waking up to over-reliance harms and addressing them (esp. within security OODA loops), it would buy time for good enough continuous alignment.
Dang, assuming you’re who I think you are, great to see you here. If you aren’t, great to see you here anyway, and you should look up people you have a name collision with, as one of them is inspiring; and nice to meet you either way!
I’m impressed that Ben Goertzel got a politician to adopt his obscure alternative to CEV as their model of the future! This after the Saudis made his robot Sophia into the first AI citizen anywhere. He must have some political skills.
Well, before 2016, I had no idea I’d serve in the public sector...
(The vTaiwan process was already modeled after CBV in 2015.)
Hi, it’s kind of an honor! We’ve had at least one billionaire and one celebrity academic comment here, but I don’t remember ever seeing a government minister before. :-)
Is there a story to how a Taiwanese digital civil society forum, ended up drawing inspiration from CBV?
Also one of the Godfathers of AI.
Can you give detail about what you mean?
I just mean he must know how to liaise credibly and effectively with politicians (although Minister Tang has clarified that she knew about his alignment ideas even before she went into government). And I find that impressive, given his ability to also liaise with people from weird corners of AI and futurology. He was one of the very first people in AI to engage with Eliezer. He’s had a highly unusual career.
Nice to meet you too & thank you for the kind words. Yes, same person as AudreyTang. (I posted the map at audreyt.org as a proof of sorts.)
Are you saying that STEM+ AI won’t exist in 2100 because by then human scientists will have become super good, such that the bar for STEM+ AI (“better at STEM research than the best human scientists”) will have gone up?
If this is your view it sounds extremely wild to me, it seems like humans would basically just slow the AIs down. This seems maybe plausible if this is mandated by law, i.e. “You aren’t allowed to build powerful STEM+ AI, although you are allowed to do human/AI cyborgs”.
Yes, that, and a further focus on assistive AI systems that excel at connecting humans — I believe this is a natural outcome of the original CBV idea.
Why do you think the pedantic medical issues (brain swelling, increased risks of various forms of early death) from brain implants will be solved pre- (ASI driven) singularity? Gene hacks exhibit the same issues. To me these problems look unsolvable in that getting from “it’s safe 90-99 percent of the time, 1% uh oh” to “it’s always safe, no matter what goes wrong we can fix it” requires superintelligent medicine, because you’re dealing with billions or more permutations of patient genetics and rare cascading events.
Safer than implants is to connect at scale “telepathically” leveraging only full sensory bandwidth and much better coordination arrangements. That is the ↗️ direction of the depth-breadth spectrum here.
How do you propose the hardware that does this works? I thought you needed wires to the outer regions of the brain with enough resolution to send/receive from ~1-10 target axons at a time.
Something like a lightweight version of the off-the-shelf Vision Pro will do. Just as nonverbal cues can transmit more effectively with codec avatars, post-symbolic communication can approach telepathy with good enough mental models facilitated by AI (not necessarily ASI.)
That sounds kinda like in person meetings. And you have the issue, same with those, of revealing information you didn’t intend to disclose, and the issues that happen when the parties incentives aren’t aligned.
Yes. The basic assumption (of my current day job) is that good-enough contextual integrity and continuous incentive alignment are solvable well within the slow takeoff we are currently in.
Fair. You think the takeoff is rate limited by compute, which is being produced at a slowly accelerating rate? (Nvidia has increased h100 run rate several fold, amd dropped their competitor today, etc)
Wait, how is it that you think 99% STEM+AI will exist by 2035, but only 1% that it’ll exist by 2100? Isn’t that a contradiction?