Contrary to singularity b.s., the AI you invent isn’t going to rewrite the laws of physics and destroy the universe before you can hit control-C. Basic space, time, and energy limitations will likely confound your laptop’s ambitions to take over the world for quite some time—plenty of time for those who best understand it to toy with what it really takes to make it friendly
Which is really just an assertion that you won’t get FOOM (I mean, no one thinks it’ll take less time than it takes you to hit Ctrl-C, but that’s just hyperbole for writing style). He doesn’t argue for that claim, he doesn’t address any of the arguments for FOOM (most notably and recently: IEM).
Ah, thanks, better understand your position now. I will endeavor to read IEM (if it isn’t too stocked with false presuppositions from the get go).
I agree the essay did not endeavor to disprove FOOM, but let’s say it’s just wrong on that claim, and that FOOM is really a possibility—then are you saying you’d rather let the military AI go FOOM than something homebrewed? Or are you claiming that it’s possible to reign in military efforts in this direction (world round)? Or give me a third option if neither of those applies.
AS to the ‘third option’, most work that I’m aware of falls into either educating people about AI risk, or trying to solve the problem before someone else builds an AGI. Most people advocate both.
FAI proponents (Of which I am one, yes) tend to say that, ceteris paribus, an AGI which is constructed without first ‘solving FAI’* will be ‘Unfriendly’, military or otherwise. This would be very very bad for humans and human values.
*There is significant disagreement over what exactly this consists of and how hard it will be.
I have lots of uncertainty around these sorts of question, especially considering the timeline-dependency and the how-likely-are-MIRI(and others)-to-succed-at-avoiding-UFAI. Suffice to say that it’s not an obvious win for ‘old age’, (which for me is hopefully >50 years away).
Let’s imagine you solve FAI tomorrow, but not AGI. (I see it as highly improbable that anyone will meaningfully solve FAI before solving AGI, but let’s explore that optimistic scenario.) Meanwhile, various folks and institutions out there are ahead of you in AGI research by however much time you’ve spent on FAI. At least one of them won’t care about FAI.
I have a hard time imagining any outcome from that scenario that doesn’t involve you wishing you’d been working on AGI and gotten there first. How do you imagine the outcome?
(In addition, MIRI claims that a FAI could be easier to implement than an AGI in general- i.e. that if you solve the philosophical difficulties regarding FAI, this also makes it easier to create an AGI in general. For example, MIRI’s specific most-likely scenario for the creation of an AGI is a sub-human AI that self-modifies to become smarter very quickly; MIRI’s research on modeling self-modification, while aimed at solving one specific problem that stands in the way of Friendliness, also has potential applications towards understanding self-modification in general.)
drethlin nailed it—If I counterfactually-had spent that time working on AGI, I wouldn’t have solved Friendliness, and (unless someone else had solved FAI without me) my AGI would be just as Unfriendly in expectation as the competitors.
If FAI is solved first, however, it increases the probability that the first AGI will be Friendly. Depending on the nature of the solution (how much of it is something that can be published so others can use it with their AGIs?), this could happen through AGI development by people already convinced of the problem, or it could be ‘added on’ to existing AGi projects.
Ok, I see the problem with this discussion, and I see no solution. If you understood AGI better, you would understand why your reply is like telling me I shouldn’t play with electricity because Zeus will get angry and punish the village. But that very concern prevents you from understanding AGI better, so we are at an impasse.
It makes me sad, because with the pervasiveness of this superstition, we’ve lost enough minds from our side that the military will probably beat us to it.
The other thing is: “Our Side” is not losing minds. People are going to try to make AGI regardless of friendliness. but almost no one anywhere has ever heard of AI friendliness and even fewer give a shit. That means that the marginal person working on friendliness is HUGELY more valuable. And if someone discovers friendliness, guess what? The military are going to want it too! Maybe someone actually insane would not, but any organization that has goals and cares about humans at all will be better off with a friendly AI than not.
You shouldn’t play with radiation because you don’t understand it and trying to build a bomb might get you and everyone else killed. This isn’t a question of superstition, you fool, it’s a question of NOT just throwing a bunch of radioactive material in a pile to see what happens, only when you fuck it up you don’t just kill yourself like Marie Curie or possibly blow up a few square miles like if the Manhattan Project hadn’t been careful enough. You fuck over EVERYTHING.
Appologies for the provacative phrasing—I was (inadvertently) asking for a heated reply...
But to clarify the point in light of your response (which no doubt will get another heated reply, though honestly trying to convey the point w/out provoking...):
Piles of radioactive material is not a good analogy here. But I think it’s appearance here is a good illustration of the very thing I’m hoping to convey: There are a lot of (vague, wrong) theories of AGI which map well to the radioactive pile analogy. Just put enough of the ingredients together in a pile, and FOOM. But the more you actually work on AGI, the more you realize how heuristic, incremental, and data bound it is; how a fantastic solution to monkey problems (vision, planning, etc) confers only the weakest ability in symbolic domains, and that, for instance, NP problems most likely remain NP hard regardless of intelligence, and their solutions are limited by time, space, and energy constraints—not cleverness. Can a hyper-intelligent AI improve upon hardware design, etc, etc? Sure! But the whole system (of progress) we’re speaking of is a large complex system of differential equations with many bottlenecks, at least some of which aren’t readily amendable to hyper-exponential change. Will there be a point where things are out of our (human) hands? Yes. Will it happen over night? No.
The radioactive pile analogy fails because AGI will not be had by heaping a bunch of stuff in a pile—it will be had through extensive engineering and design. It will progress incrementally, and it will be bounded by resource constraints—for a long long time.
A better analogy might be to building a fusion reactor. Sure, you have to be careful, especially in the final design, construction, and execution of the full scale device, but there’s a huge amount of engineering to be done before you get anywhere near having to worry about that, and tons of smaller experiments that need to be proven first, and so on. And you learn as you go, and after years of work you start getting closer and you know a shitload about the technology and what it’s quirks and hazards are and what’s easy to control, and so on.
And when you’re there—well along the way, and you understand the technology to a deep level even if you haven’t quite figured out how to make the sustainable fusion reactor yet—it’s pretty insulting/annoying when someone who doesn’t have any practical(!) grasp of the matter comes along and tells you you shouldn’t be working on this because they haven’t figured out how to make it safe yet! And because they think that if you put too much stuff in a pile, it will go boom! (Sigh!) You’re on your way to making clean energy before the peak oil appocolypse or whatever, and they’re working against you (if only they knew what their efforts were costing the world).
What do you do there? They’re fearful because they don’t understand, and the particulars of their fear are, really, superstition (in the sense that they are not founded on a solid understanding, but quite specifically on a lack thereof). You want to say: get up to speed, and you can help us make this work and make it safe (and when you understand it better, you’ll start to actually understand how that might be done—and how much less of an explosive problem it is than you think). But GTF out of my way if you’re just going to pontificate from ignorance and try to dictate how I do my job from there. No matter how long you sofa-think about how to keep the pile-o-stuff from going bad-FOOM, your answers are never going to mesh with reality because you’ve got way too many false premises that need to be sifted out first (through actual experience in the topic). [And, sorry, but no matter how big Eliezer’s cloud of self-citations is, that’s just someone else’s sofa-think, not actual experience.]
Personally, I do not think FAI is a hard problem (a highly educated opinion, not offhand dismissal). But I also know that UAI is going to happen eventually (intentionally), no matter how many conferences y’all have. And I also know the odds are highest of all we’ll all die of old age because AI didn’t happen well enough soon enough. But I understand if you disagree.
I think it will be incidental to AGI. That is, by the time you are approaching human-level AGI it will be essentially obvious (to the sort of person who groks human-level AGI in the first place). Motivation (as a component of the process of thinking) is integral to AGI, not some extra thing only humans and animals happen to have. Motivation needs be grokked before you will have AGI in the first place. Human motivational structure is quite complex, with far more alterior motives (clan affiliation, reproduction, etc) than straightforward ones. AGIs needn’t be so-burdened, which in many ways makes the FAI problem easier in fact than our human-based intuition might surmise. On the other hand, simple random variation is a huge risk—that is, no matter the intentional agenda, there is always the possibility that a simple error will put that very abstract coefficient of feedback over unity, and then you have a problem. If AGI weren’t going to happen regardless, I might say it’s worthy of a debate now what the nature of that problem would be (but in that debate, I still say it’s not a huge problem—it’s not instantaneous FOOM, it’s time-to-uplug FOOM; and you have the advantage of other FAIs by then with full ability to analyze each other so you actually have a lot of tools available to put out fires long before they’re raging); but AGI is going to happen regardless, so the race is not FAI vs. AGI, but whether the first to solve AGI wants FAI or something else. And like I say, there is also the race against our own inevitable demise of old age (talk to anybody who’s been in the longevity community for > 20 years and you will learn they once had your optimism about progress).
Don’t get me wrong, FAI is not an uninteresting problem. My claim is quite simply that for the goals of the FAI community (which I have to assume includes your own long-term survival), y’all would do far better to be working (hard and seriously) on AGI than not. All of this sofa-think today will be replicated in short order by better-informed consideration down the road. And I aint sayin’ don’t think about it today—I’m saying find a realistic balance between FAI and AGI research that doesn’t leave you so far behind the game that your goals never get to matter, and I’m sayin’ that’s 99% AGI research and 1% FAI (for now). (And no, that doesn’t mean 99 people doing AGI and 1 doing FAI. My point is the 1 doing FAI is useless if they aren’t 99% steeped in AGI from which to think about FAI in the first place.)
Just to follow up, I’m seeing nothing new in IEM (or if it’s there it’s too burried in “hear me think” to find—Eliezer really would benefit from pruning down to essentials). Most of it concerns the point where AGI approaches or exceeds human intelligence. There’s very little to support concern for the long ramp up to that point (other than some matter of genetic programming, which I haven’t the time to address here). I could go on rather at length in rebuttal of the post-human-intelligence FOOM theory (not discounting it entirely, but putting certain qualitative bounds on it that justify the claim that FAI will be most fruitfully pursued during that transition, not before it), but for the reasons implied in the original essay and in my other comments here, it seems moot against the overriding truth that AGI is going to happen without FAI regardless—which means our best hope is to see AGI+FAI happen first. If it’s really not obvious that that has to lead with AGI, then tell me why.
Does anybody really think they are going to create an AGI that will get out of their hands before they can stop it? That they will somehow bypass ant, mouse, dog, monkey, and human and go straight to superhuman? Do you really think that you can solve FAI faster or better than someone who’s invented monkey-level AI first?
I feel most of this fear is risidual leftovers from the self-modifying symbolic-program singularity FOOM theories that I hope are mostly left behind by now. But this is just the point—people who don’t understand real AGI don’t understand what the real risks are and aren’t (and certainly can’t mediate them).
I feel most of this fear is risidual leftovers from the self-modifying symbolic-program singularity FOOM theories that I hope are mostly left behind by now. But this is just the point—people who don’t understand real AGI don’t understand what the real risks are and aren’t (and certainly can’t mediate them).
Self-modifying AI is the point behind FOOM. I’m not sure why you’re connecting self-modification/FOOM/singularity with symbolic programming (I assume you mean GOFAI), but everyone I’m aware of who thinks FOOM is plausible thinks it will be because of self-modification.
Yes, I understand that. But it matters a lot what premises underlie AGI how self-modification is going to impact it. The stronger fast-FOOM arguments spring from older conceptions of AGI. Imo, a better understanding of AGI does not support it.
Thanks much for the interesting conversation, I think I am expired.
I don’t think anyone is saying that an ‘ant-level’ AGI is a problem. The issue is with ‘relatively-near-human-level’ AGI. I also don’t think there’s much disagreement about whether a better understanding of AGI would make FAI work easier. People aren’t concerned about AI work being done today, except inasmuch as it hastens better AGI work done in the future.
“I mean, no one thinks it’ll take less time than it takes you to hit Ctrl-C”—by the way, are you sure about this? Would it be more accurate to say “before you realize you should hit control-C”? Because it seems to me, if it aint goin’ FOOM before you realize you should hit control-C (and do so) then.… it aint goin’ FOOM.
More importantly: If someone who KNOWS how important stopping it is sitting at the button, then they’re more likely to stop it, but if someone is like “it’s getting more powerful and better optimized! Let’s see how it looks in a week!” is in charge, then problems.
Well, then, I hope it’s someone like you or me that’s at the button. But that’s not going to be the case if we’re working on FAI instead of AGI, is it...
“The Intelligence Explosion Thesis says that an AI can potentially grow in capability on a timescale that seems fast relative to human experience due to recursive self-improvement. This in turn implies that strategies which rely on humans reacting to and restraining or punishing AIs are unlikely to be successful in the long run, and that what the first strongly self-improving AI prefers can end up mostly determining the final outcomes for Earth-originating intelligent life. ”—Eliezer Yudkowsky, IEM.
I.e., Eliezer thinks it’ll take less time than it takes you to hit Ctrl-C. (Granted it takes Eliezer a whole paragraph to say what the essay captures in a phrase, but I digress.)
Eliezer’s position is somewhat more nuanced than that. He admits a possibility of a FOOM timescale on the order of seconds, but a timescale on the order of weeks/months/years is also in line with the IE thesis.
FOOM on the order of seconds can be strongly argued against (Eli does a fair job of it himself, but likes to leave everything open so he can cite himself later no matter what happens), and if it’s weeks/months/years, then Hit Control C. Seriously. If your computer is trying to take over the world and is likely to succeed in the next few weeks, then kill −9 the thing. I realize that at that point you’ve likely got other AIs to worry about, but at least you’re in a position to understand it well enough to have some hope at making yours friendly and re-activating it before Skynet goes live. (I know Eli has counters to this, and counter-counters, and counter-counter-counters, but so do I—I just don’t assume you’ll be interested in hearing them. The main point here really is that the original statement wasn’t so much hyperbole as refreshingly concise—whether or not you agree with it.)
Trying to understand here. What’s the strawman in this case?
Can you point me to an essay that addresses the points in this one?
He doesn’t really make any relevant points.
The closest is this:
Which is really just an assertion that you won’t get FOOM (I mean, no one thinks it’ll take less time than it takes you to hit Ctrl-C, but that’s just hyperbole for writing style). He doesn’t argue for that claim, he doesn’t address any of the arguments for FOOM (most notably and recently: IEM).
Ah, thanks, better understand your position now. I will endeavor to read IEM (if it isn’t too stocked with false presuppositions from the get go).
I agree the essay did not endeavor to disprove FOOM, but let’s say it’s just wrong on that claim, and that FOOM is really a possibility—then are you saying you’d rather let the military AI go FOOM than something homebrewed? Or are you claiming that it’s possible to reign in military efforts in this direction (world round)? Or give me a third option if neither of those applies.
AS to the ‘third option’, most work that I’m aware of falls into either educating people about AI risk, or trying to solve the problem before someone else builds an AGI. Most people advocate both.
FAI proponents (Of which I am one, yes) tend to say that, ceteris paribus, an AGI which is constructed without first ‘solving FAI’* will be ‘Unfriendly’, military or otherwise. This would be very very bad for humans and human values.
*There is significant disagreement over what exactly this consists of and how hard it will be.
Do you think people who can’t implement AGI can solve FAI?
The problem of Friendliness can be worked on before the problem of AGI has been solved, yes
Which do you think is more likely: That you will die of old age, or of unfriendly-AI? (Serious question, genuinely curious.)
I have lots of uncertainty around these sorts of question, especially considering the timeline-dependency and the how-likely-are-MIRI(and others)-to-succed-at-avoiding-UFAI. Suffice to say that it’s not an obvious win for ‘old age’, (which for me is hopefully >50 years away).
Let’s imagine you solve FAI tomorrow, but not AGI. (I see it as highly improbable that anyone will meaningfully solve FAI before solving AGI, but let’s explore that optimistic scenario.) Meanwhile, various folks and institutions out there are ahead of you in AGI research by however much time you’ve spent on FAI. At least one of them won’t care about FAI.
I have a hard time imagining any outcome from that scenario that doesn’t involve you wishing you’d been working on AGI and gotten there first. How do you imagine the outcome?
“worked on” != “solved”
(In addition, MIRI claims that a FAI could be easier to implement than an AGI in general- i.e. that if you solve the philosophical difficulties regarding FAI, this also makes it easier to create an AGI in general. For example, MIRI’s specific most-likely scenario for the creation of an AGI is a sub-human AI that self-modifies to become smarter very quickly; MIRI’s research on modeling self-modification, while aimed at solving one specific problem that stands in the way of Friendliness, also has potential applications towards understanding self-modification in general.)
drethlin nailed it—If I counterfactually-had spent that time working on AGI, I wouldn’t have solved Friendliness, and (unless someone else had solved FAI without me) my AGI would be just as Unfriendly in expectation as the competitors.
If FAI is solved first, however, it increases the probability that the first AGI will be Friendly. Depending on the nature of the solution (how much of it is something that can be published so others can use it with their AGIs?), this could happen through AGI development by people already convinced of the problem, or it could be ‘added on’ to existing AGi projects.
See reply below to drethlin.
why would he wish that? His unfriendly AI that he’d been working on will probably just kill him.
Sigh.
Ok, I see the problem with this discussion, and I see no solution. If you understood AGI better, you would understand why your reply is like telling me I shouldn’t play with electricity because Zeus will get angry and punish the village. But that very concern prevents you from understanding AGI better, so we are at an impasse.
It makes me sad, because with the pervasiveness of this superstition, we’ve lost enough minds from our side that the military will probably beat us to it.
The other thing is: “Our Side” is not losing minds. People are going to try to make AGI regardless of friendliness. but almost no one anywhere has ever heard of AI friendliness and even fewer give a shit. That means that the marginal person working on friendliness is HUGELY more valuable. And if someone discovers friendliness, guess what? The military are going to want it too! Maybe someone actually insane would not, but any organization that has goals and cares about humans at all will be better off with a friendly AI than not.
“but almost no one anywhere has ever heard of AI friendliness”
Ok, if this is your vantage point, I understand better. I must hang in the wrong circles ’cause I meet far more FAI than AGI folks.
You shouldn’t play with radiation because you don’t understand it and trying to build a bomb might get you and everyone else killed. This isn’t a question of superstition, you fool, it’s a question of NOT just throwing a bunch of radioactive material in a pile to see what happens, only when you fuck it up you don’t just kill yourself like Marie Curie or possibly blow up a few square miles like if the Manhattan Project hadn’t been careful enough. You fuck over EVERYTHING.
Appologies for the provacative phrasing—I was (inadvertently) asking for a heated reply...
But to clarify the point in light of your response (which no doubt will get another heated reply, though honestly trying to convey the point w/out provoking...):
Piles of radioactive material is not a good analogy here. But I think it’s appearance here is a good illustration of the very thing I’m hoping to convey: There are a lot of (vague, wrong) theories of AGI which map well to the radioactive pile analogy. Just put enough of the ingredients together in a pile, and FOOM. But the more you actually work on AGI, the more you realize how heuristic, incremental, and data bound it is; how a fantastic solution to monkey problems (vision, planning, etc) confers only the weakest ability in symbolic domains, and that, for instance, NP problems most likely remain NP hard regardless of intelligence, and their solutions are limited by time, space, and energy constraints—not cleverness. Can a hyper-intelligent AI improve upon hardware design, etc, etc? Sure! But the whole system (of progress) we’re speaking of is a large complex system of differential equations with many bottlenecks, at least some of which aren’t readily amendable to hyper-exponential change. Will there be a point where things are out of our (human) hands? Yes. Will it happen over night? No.
The radioactive pile analogy fails because AGI will not be had by heaping a bunch of stuff in a pile—it will be had through extensive engineering and design. It will progress incrementally, and it will be bounded by resource constraints—for a long long time.
A better analogy might be to building a fusion reactor. Sure, you have to be careful, especially in the final design, construction, and execution of the full scale device, but there’s a huge amount of engineering to be done before you get anywhere near having to worry about that, and tons of smaller experiments that need to be proven first, and so on. And you learn as you go, and after years of work you start getting closer and you know a shitload about the technology and what it’s quirks and hazards are and what’s easy to control, and so on.
And when you’re there—well along the way, and you understand the technology to a deep level even if you haven’t quite figured out how to make the sustainable fusion reactor yet—it’s pretty insulting/annoying when someone who doesn’t have any practical(!) grasp of the matter comes along and tells you you shouldn’t be working on this because they haven’t figured out how to make it safe yet! And because they think that if you put too much stuff in a pile, it will go boom! (Sigh!) You’re on your way to making clean energy before the peak oil appocolypse or whatever, and they’re working against you (if only they knew what their efforts were costing the world).
What do you do there? They’re fearful because they don’t understand, and the particulars of their fear are, really, superstition (in the sense that they are not founded on a solid understanding, but quite specifically on a lack thereof). You want to say: get up to speed, and you can help us make this work and make it safe (and when you understand it better, you’ll start to actually understand how that might be done—and how much less of an explosive problem it is than you think). But GTF out of my way if you’re just going to pontificate from ignorance and try to dictate how I do my job from there. No matter how long you sofa-think about how to keep the pile-o-stuff from going bad-FOOM, your answers are never going to mesh with reality because you’ve got way too many false premises that need to be sifted out first (through actual experience in the topic). [And, sorry, but no matter how big Eliezer’s cloud of self-citations is, that’s just someone else’s sofa-think, not actual experience.]
Personally, I do not think FAI is a hard problem (a highly educated opinion, not offhand dismissal). But I also know that UAI is going to happen eventually (intentionally), no matter how many conferences y’all have. And I also know the odds are highest of all we’ll all die of old age because AI didn’t happen well enough soon enough. But I understand if you disagree.
How not hard is it? How long do you think it would take you to solve it?
I think it will be incidental to AGI. That is, by the time you are approaching human-level AGI it will be essentially obvious (to the sort of person who groks human-level AGI in the first place). Motivation (as a component of the process of thinking) is integral to AGI, not some extra thing only humans and animals happen to have. Motivation needs be grokked before you will have AGI in the first place. Human motivational structure is quite complex, with far more alterior motives (clan affiliation, reproduction, etc) than straightforward ones. AGIs needn’t be so-burdened, which in many ways makes the FAI problem easier in fact than our human-based intuition might surmise. On the other hand, simple random variation is a huge risk—that is, no matter the intentional agenda, there is always the possibility that a simple error will put that very abstract coefficient of feedback over unity, and then you have a problem. If AGI weren’t going to happen regardless, I might say it’s worthy of a debate now what the nature of that problem would be (but in that debate, I still say it’s not a huge problem—it’s not instantaneous FOOM, it’s time-to-uplug FOOM; and you have the advantage of other FAIs by then with full ability to analyze each other so you actually have a lot of tools available to put out fires long before they’re raging); but AGI is going to happen regardless, so the race is not FAI vs. AGI, but whether the first to solve AGI wants FAI or something else. And like I say, there is also the race against our own inevitable demise of old age (talk to anybody who’s been in the longevity community for > 20 years and you will learn they once had your optimism about progress).
Don’t get me wrong, FAI is not an uninteresting problem. My claim is quite simply that for the goals of the FAI community (which I have to assume includes your own long-term survival), y’all would do far better to be working (hard and seriously) on AGI than not. All of this sofa-think today will be replicated in short order by better-informed consideration down the road. And I aint sayin’ don’t think about it today—I’m saying find a realistic balance between FAI and AGI research that doesn’t leave you so far behind the game that your goals never get to matter, and I’m sayin’ that’s 99% AGI research and 1% FAI (for now). (And no, that doesn’t mean 99 people doing AGI and 1 doing FAI. My point is the 1 doing FAI is useless if they aren’t 99% steeped in AGI from which to think about FAI in the first place.)
Just to follow up, I’m seeing nothing new in IEM (or if it’s there it’s too burried in “hear me think” to find—Eliezer really would benefit from pruning down to essentials). Most of it concerns the point where AGI approaches or exceeds human intelligence. There’s very little to support concern for the long ramp up to that point (other than some matter of genetic programming, which I haven’t the time to address here). I could go on rather at length in rebuttal of the post-human-intelligence FOOM theory (not discounting it entirely, but putting certain qualitative bounds on it that justify the claim that FAI will be most fruitfully pursued during that transition, not before it), but for the reasons implied in the original essay and in my other comments here, it seems moot against the overriding truth that AGI is going to happen without FAI regardless—which means our best hope is to see AGI+FAI happen first. If it’s really not obvious that that has to lead with AGI, then tell me why.
Does anybody really think they are going to create an AGI that will get out of their hands before they can stop it? That they will somehow bypass ant, mouse, dog, monkey, and human and go straight to superhuman? Do you really think that you can solve FAI faster or better than someone who’s invented monkey-level AI first?
I feel most of this fear is risidual leftovers from the self-modifying symbolic-program singularity FOOM theories that I hope are mostly left behind by now. But this is just the point—people who don’t understand real AGI don’t understand what the real risks are and aren’t (and certainly can’t mediate them).
Self-modifying AI is the point behind FOOM. I’m not sure why you’re connecting self-modification/FOOM/singularity with symbolic programming (I assume you mean GOFAI), but everyone I’m aware of who thinks FOOM is plausible thinks it will be because of self-modification.
Yes, I understand that. But it matters a lot what premises underlie AGI how self-modification is going to impact it. The stronger fast-FOOM arguments spring from older conceptions of AGI. Imo, a better understanding of AGI does not support it.
Thanks much for the interesting conversation, I think I am expired.
I don’t think anyone is saying that an ‘ant-level’ AGI is a problem. The issue is with ‘relatively-near-human-level’ AGI. I also don’t think there’s much disagreement about whether a better understanding of AGI would make FAI work easier. People aren’t concerned about AI work being done today, except inasmuch as it hastens better AGI work done in the future.
“I mean, no one thinks it’ll take less time than it takes you to hit Ctrl-C”—by the way, are you sure about this? Would it be more accurate to say “before you realize you should hit control-C”? Because it seems to me, if it aint goin’ FOOM before you realize you should hit control-C (and do so) then.… it aint goin’ FOOM.
More importantly: If someone who KNOWS how important stopping it is sitting at the button, then they’re more likely to stop it, but if someone is like “it’s getting more powerful and better optimized! Let’s see how it looks in a week!” is in charge, then problems.
Well, then, I hope it’s someone like you or me that’s at the button. But that’s not going to be the case if we’re working on FAI instead of AGI, is it...
What do you think Ctrl-C does in this?
“The Intelligence Explosion Thesis says that an AI can potentially grow in capability on a timescale that seems fast relative to human experience due to recursive self-improvement. This in turn implies that strategies which rely on humans reacting to and restraining or punishing AIs are unlikely to be successful in the long run, and that what the first strongly self-improving AI prefers can end up mostly determining the final outcomes for Earth-originating intelligent life. ”—Eliezer Yudkowsky, IEM.
I.e., Eliezer thinks it’ll take less time than it takes you to hit Ctrl-C. (Granted it takes Eliezer a whole paragraph to say what the essay captures in a phrase, but I digress.)
Eliezer’s position is somewhat more nuanced than that. He admits a possibility of a FOOM timescale on the order of seconds, but a timescale on the order of weeks/months/years is also in line with the IE thesis.
FOOM on the order of seconds can be strongly argued against (Eli does a fair job of it himself, but likes to leave everything open so he can cite himself later no matter what happens), and if it’s weeks/months/years, then Hit Control C. Seriously. If your computer is trying to take over the world and is likely to succeed in the next few weeks, then kill −9 the thing. I realize that at that point you’ve likely got other AIs to worry about, but at least you’re in a position to understand it well enough to have some hope at making yours friendly and re-activating it before Skynet goes live. (I know Eli has counters to this, and counter-counters, and counter-counter-counters, but so do I—I just don’t assume you’ll be interested in hearing them. The main point here really is that the original statement wasn’t so much hyperbole as refreshingly concise—whether or not you agree with it.)
How would you know that you have to press Crtl-C? What observation would you need to make?