Which things I said made you worry I was conflating these? If hard and soft takeoff were equally likely I’d still think we should put most of our energy into worrying about hard takeoff; but hard seems significantly likelier to me.
At least from my readings, points 11, 12, and 13 are the big focus points on AGI risks, and they’re defaulting to genie-level capabilities: the only earlier machine is purely instruction-set blue-minimizing robot.
Hard takeoff being significantly more likely means that your concerns are, naturally and reasonably, going to gravitate toward discussing AGI risks and hungry AGI in the context of FOOMing AGI. That makes sense for people who can jump the inferential difference into explosive recursive improvement. If you’re writing a work to help /others understand/ the concept of AGI risks, though, discussing how a FOOMing AGI could start taking apart Jupiter in order to make more smiley faces, due next Tuesday, requires that they accept a more complex scenario than that of a general machine intelligence to begin with. This makes sense from a risk analysis viewpoint, where Bayesian multiplication is vital for comparing relative risks—very important to the values of SIRI, targeting folk who know what a Singularity is. It’s unnecessary for the purpose of risk awareness, where showing the simplest threshold risk gets folk to pay attention—which is more important to the MIRI, targeting folk who want to know what machine intelligence could be (and are narrative thinkers, with the resulting logical biases).
If the possibility of strong AGI occurring is P1, the probability of strong AGI going FOOM is P2, and probability of any strong AGI being destructive is P3, the necessary understanding to grasp P1xP2xP3 is unavoidably going to be higher than P1xP3, even if P2 is very close to 1. You can always introduce P2 later, in order to show why the results would be much worse than everyone’s already expecting—and that has a stronger effect on avoidance-heavy human neurology than letting people think that machine intelligence can be made safe by just preventing the AGI from reaching high levels of self-improvement.
If there are serious existential risks to soft and takeoff and even no takeoff AGI, then discussing a general risk first not only appears more serious, but also makes later discussion of hard takeoff hit even harder.
Do you think non-hungry AGIs are likely? And if we build one, do you think it’s likely to be dangerous?
Hungry AGIs occur when the utility of additional resources exceeds the costs of additional resources, as amortorized by whatever time discounting function you’re using. That’s very likely as the AGI calculates a sufficiently long-duration event, even with heavy time discounting, but that’s not the full set of possible minds. It’s quite easy to imagine a non-hungry AGI that causes civilization-level risks, or even a non-hungry non-FOOM AGI that causes continent-level risks. ((I don’t think it’s terribly likely, since barring exceptional information control or unlikely design constraints, it’d be bypassed by a copy turned intentionally-hungry AGI, but as above, this is a risk awareness matter rather than risk analysis one.))
More importantly, you don’t need to FOOM to have a hungry AGI. A ‘stupid’ tool AI, even a ‘stupid’ tool AI that gets only small benefits from additional resources, could still go hungry with the wrong question or the wrong discount on future time—or even if it merely made a bad time estimation on a normal question. It’s bad to have a few kilotons of computronium pave over the galaxy with smiley faces; it’s /embarrassing/ to have the solar system paved over with inefficient transistors trying to find a short answer to Fermat’s Last Theorem. Or if I’m wrong, and a machine intelligent slightly smarter than the chess team at MIT can crack the protein folding problem in a year, a blue-minimizing AGI becomes /very/ frightening even with a small total intelligence.
Ever? For how long?
The strict version of the protein folding prediction problem was defined about half a century ago, and has been a fairly well-known and well-studied problem enough that I’m willing to wager we’ve had several-dozen intelligent people working on it for most of that time period (and, recently, several-dozen intelligent people working on just software implementations). An AGI built today has the advantage of their research, along with a different neurological design, but in turn it may have additional limitations. Predictions are hard, especially about the future, but for the purposes of a thought experiment it’s not obvious that another fifty years without an AGI would change the matter so dramatically. I suspect /That Alien Message/ discusses a boxed AI with the sum computational power of the entire planet across long periods of time precisely because I’m not the only one to give that estimate.
And, honestly, once you have an AGI in the field, fifty years is a very long event horizon for even the slow takeoff scenarios.
I suspect non-MLP fans will miss most of what makes such a future inspiring or fun. And inspiring and fun is what I’m shooting for.
Not as much as you’d expect. It’s more calling on the sort of things that get folk interested in The Sims or in World of Warcraft, and iceman seemed to intentionally write it to be accessible to the general audience in preference to pony fans. The big benefit about ponies is that they’re strange enough that it’s someone /else’s/ wish fulfillment. ((Conversely, it doesn’t really benefit from knowledge of the show, since it doesn’t use the main cast or default setting: Celest-AI shares very little overlap with the character Princess Celestia, excepting that they can control a sun.)) The full work is probably not useful for this, but chapter six alone might be a useful contrast to / Just Another Day in Utopia/.
But for ‘Why is it hard to give AGI known or predictable goals?’ what I had in mind is a popularization of the problems with evolutionary algorithms, or the relevance of Löb’s Theorem to self-modifying AI, or some combination of these and other concerns.
Hm… that would be a tricky requirement to fill: there are very few good layperson’s versions of Löb’s Problem as it is, and the question does not easily reduce from the mathematic analysis. (EDIT: Or rather, it goes from being formal logic Deep Magic to obvious truism in attempts to demonstrate it… still, space to improve on the matter after that cartoon.)
At least from my readings, points 11, 12, and 13 are the big focus points on AGI risks, and they’re defaulting to genie-level capabilities: the only earlier machine is purely instruction-set blue-minimizing robot.
Hard takeoff being significantly more likely means that your concerns are, naturally and reasonably, going to gravitate toward discussing AGI risks and hungry AGI in the context of FOOMing AGI. That makes sense for people who can jump the inferential difference into explosive recursive improvement. If you’re writing a work to help /others understand/ the concept of AGI risks, though, discussing how a FOOMing AGI could start taking apart Jupiter in order to make more smiley faces, due next Tuesday, requires that they accept a more complex scenario than that of a general machine intelligence to begin with. This makes sense from a risk analysis viewpoint, where Bayesian multiplication is vital for comparing relative risks—very important to the values of SIRI, targeting folk who know what a Singularity is. It’s unnecessary for the purpose of risk awareness, where showing the simplest threshold risk gets folk to pay attention—which is more important to the MIRI, targeting folk who want to know what machine intelligence could be (and are narrative thinkers, with the resulting logical biases).
If the possibility of strong AGI occurring is P1, the probability of strong AGI going FOOM is P2, and probability of any strong AGI being destructive is P3, the necessary understanding to grasp P1xP2xP3 is unavoidably going to be higher than P1xP3, even if P2 is very close to 1. You can always introduce P2 later, in order to show why the results would be much worse than everyone’s already expecting—and that has a stronger effect on avoidance-heavy human neurology than letting people think that machine intelligence can be made safe by just preventing the AGI from reaching high levels of self-improvement.
If there are serious existential risks to soft and takeoff and even no takeoff AGI, then discussing a general risk first not only appears more serious, but also makes later discussion of hard takeoff hit even harder.
Hungry AGIs occur when the utility of additional resources exceeds the costs of additional resources, as amortorized by whatever time discounting function you’re using. That’s very likely as the AGI calculates a sufficiently long-duration event, even with heavy time discounting, but that’s not the full set of possible minds. It’s quite easy to imagine a non-hungry AGI that causes civilization-level risks, or even a non-hungry non-FOOM AGI that causes continent-level risks. ((I don’t think it’s terribly likely, since barring exceptional information control or unlikely design constraints, it’d be bypassed by a copy turned intentionally-hungry AGI, but as above, this is a risk awareness matter rather than risk analysis one.))
More importantly, you don’t need to FOOM to have a hungry AGI. A ‘stupid’ tool AI, even a ‘stupid’ tool AI that gets only small benefits from additional resources, could still go hungry with the wrong question or the wrong discount on future time—or even if it merely made a bad time estimation on a normal question. It’s bad to have a few kilotons of computronium pave over the galaxy with smiley faces; it’s /embarrassing/ to have the solar system paved over with inefficient transistors trying to find a short answer to Fermat’s Last Theorem. Or if I’m wrong, and a machine intelligent slightly smarter than the chess team at MIT can crack the protein folding problem in a year, a blue-minimizing AGI becomes /very/ frightening even with a small total intelligence.
The strict version of the protein folding prediction problem was defined about half a century ago, and has been a fairly well-known and well-studied problem enough that I’m willing to wager we’ve had several-dozen intelligent people working on it for most of that time period (and, recently, several-dozen intelligent people working on just software implementations). An AGI built today has the advantage of their research, along with a different neurological design, but in turn it may have additional limitations. Predictions are hard, especially about the future, but for the purposes of a thought experiment it’s not obvious that another fifty years without an AGI would change the matter so dramatically. I suspect /That Alien Message/ discusses a boxed AI with the sum computational power of the entire planet across long periods of time precisely because I’m not the only one to give that estimate.
And, honestly, once you have an AGI in the field, fifty years is a very long event horizon for even the slow takeoff scenarios.
Not as much as you’d expect. It’s more calling on the sort of things that get folk interested in The Sims or in World of Warcraft, and iceman seemed to intentionally write it to be accessible to the general audience in preference to pony fans. The big benefit about ponies is that they’re strange enough that it’s someone /else’s/ wish fulfillment. ((Conversely, it doesn’t really benefit from knowledge of the show, since it doesn’t use the main cast or default setting: Celest-AI shares very little overlap with the character Princess Celestia, excepting that they can control a sun.)) The full work is probably not useful for this, but chapter six alone might be a useful contrast to / Just Another Day in Utopia/.
Hm… that would be a tricky requirement to fill: there are very few good layperson’s versions of Löb’s Problem as it is, and the question does not easily reduce from the mathematic analysis. (EDIT: Or rather, it goes from being formal logic Deep Magic to obvious truism in attempts to demonstrate it… still, space to improve on the matter after that cartoon.)