As far as I understand, the mission of SIAI (the people who host this site) is to prevent the rise of un-Friendly AGI, not to actually build one.
I think they are kind of keen on the idea of not dying too. Improving the chances that a Friendly AI will be created by someone is probably up there as a goal too.
I think they are kind of keen on the idea of not dying too.
Imagine that ! :-)
Improving the chances that a Friendly AI will be created by someone is probably up there as a goal too.
That’s a different goal, though. As far as I understand, olalonde’s master plan looks something like this:
1). Figure out how to build AGI. 2). Build a reasonably smart one as a proof of concept. 3). Figure out where to go from there, and how to make AGI safe. 4). Eventually, build a transhuman AGI once we know it’s safe.
Whereas the SIAI master plan looks something like this:
1). Make sure that an un-Friendly AGI does not get built. 2). Figure out how to build a Friendly AGI. 3). Build one. 4). Now that we know it’s safe, build a transhuman AGI (or simply wait long enough, since the AGI from step (3) will boost itself to transhuman levels).
One key difference between olalonde’s plan and SIAI’s plan is the assumption SIAI is making: they are assuming that any AGI will inevitably (plus or minus epsilon) self-improve itself to transhuman levels. Thus, from their perspective, olalonde’s step (2) above might as well say, “build a machine that’s guaranteed to eat us all”, which would clearly be a bad thing.
One key difference between olalonde’s plan and SIAI’s plan is the assumption SIAI is making: they are assuming that any AGI will inevitably (plus or minus epsilon) self-improve itself to transhuman levels. Thus, from their perspective, olalonde’s step (2) above might as well say, “build a machine that’s guaranteed to eat us all”, which would clearly be a bad thing.
A good summary. I’d slightly modify it in as much as they would allow the possibility that a really weak AGI may not do much in the way of FOOMing but they pretty much ignore those ones and expect they would just be a stepping stone for the developers who would go on to make better ones. (This is just my reasoning but I assume they would think similarly.)
Good point. Though I guess we could still say that the weak AI is recursively self-improving in this scenario—it’s just using the developers’ brains as its platform, as opposed to digital hardware. I don’t know whether the SIAI folks would endorse this view, though.
Good point. Though I guess we could still say that the weak AI is recursively self-improving in this scenario—it’s just using the developers’ brains as its platform, as opposed to digital hardware.
Can’t we limit the meaning of “self-improving” to at least stuff that the AI actually does? We can already say more precisely that the AI is being iteratively improved by the creators. We don’t have to go around removing the distinction between what an agent does and what the creator of the agent happens to do to it.
[SIAI] are assuming that any AGI will inevitably (plus or minus epsilon) self-improve itself to transhuman levels.
Can you clarify your reasons for believing this, as distinct from ”...any AGI has a non-negligible chance of self-improving itself to transhuman levels, and the cost of that happening is so vast that it’s worth devoting effort to avoid even if the chance is relatively low”?
That’s a good point, but, from reading what Eliezer and Luke are writing, I formed the impression that my interpretation is correct. In addition, the SIAI FAQ seems to be saying that intelligence explosion is a natural consequence of Moore’s Law; thus, if Moore’s Law continues to hold, intelligence explosion is inevitable.
FWIW, I personally disagree with both statements, but that’s probably a separate topic.
You’re right, I just re-read it and it doesn’t mention Moore’s Law; either it did at some point and then changed, or I saw that argument somewhere else. Still, the FAQ does seem to suggest that the only thing that can stop the Singularity is total human extinction (well, that, or the existence of souls, which IMO we can safely discount); that’s pretty close to inevitability as far as I’m concerned.
Note that the section you’re quoting is no longer talking about the inevitable ascension of any given AGI, but rather the inevitability of some AGI ascending.
I thought they were talking specifically about an AGI that is capable of recursive self-improvement. This does not encompass all possible AGIs, but the non-self-improving ones are not likely to be very smart, as far as I understand, and thus aren’t a concern.
[SIAI] are assuming that any AGI will inevitably (plus or minus epsilon) self-improve itself to transhuman levels.
and I asked why you believed that, as distinct from ”...any AGI has a non-negligible chance of self-improving itself to transhuman levels, and the cost of that happening is so vast that it’s worth devoting effort to avoid even if the chance is relatively low”?
Now you seem to be saying that SI doesn’t believe that any AGI will inevitably (plus or minus epsilon) self-improve itself to transhuman levels, but it is primarily concerned with those who do.
I agree with that entirely; it was my point in the first place.
Were we in agreement all along, have you changed your mind in the course of this exchange, or am I really really confused about what’s going on?
Sorry, I think I am guilty of misusing terminology. I have been using AI and AGI interchangeably, but that’s obviously not right. As far as I understand, “AGI” refers to a general intelligence who can solve (or, at least, attempt to solve) any problem, whereas “AI” refers to any kind of an artificial intelligence, including the specialized kind. There are many AIs that already exist in the world—for example, Google’s AdSense algorithm—but SIAI is not concerned about them (as far as I know), because they lack the capacity to self-improve.
My own hidden assumption, which I should’ve recognized and voiced earlier, is that an AGI (as contrasted with non-general AI) would most likely be produced through a process of recursive self-improvement; it is highly unlikely that an AGI could be created from scratch by humans writing lines of code. As far as I understand, the SIAI agrees with this statement, but again, I could be wrong.
Thus, it is unlikely that a non-general AI will ever be smart enough to warrant concern. It could still do some damage, of course, but then, so could a busted water main. On the other hand, an AGI will most likely arise as the result of recursive self-improvement, and thus will be capable of further self-improvement, thus boosting itself to transhuman levels very quickly unless its self-improvement is arrested by some mechanism.
Yeah, I’ve been talking throughout about what you’re labeling “AI” here. We agree that these won’t necessarily self-improve. Awesome.
With respect to what you’re labeling “AGI” here, you’re saying the following: 1) given that X is an AGI developed by humans, the probability that X has thus far been capable of recursive self-improvement is very high, and 2) given that X has thus far been capable of recursive self-improvement, the probability that X will continue to be capable of recursive self-improvement in the future is very high. 3) SIAI believes 1) and 2).
Yes, with the caveats that a). as far as I know, no such X currently exists, and b). my confidence in (3) is much lower than my confidence in (1) and (2).
I think they are kind of keen on the idea of not dying too. Improving the chances that a Friendly AI will be created by someone is probably up there as a goal too.
Imagine that ! :-)
That’s a different goal, though. As far as I understand, olalonde’s master plan looks something like this:
1). Figure out how to build AGI.
2). Build a reasonably smart one as a proof of concept.
3). Figure out where to go from there, and how to make AGI safe.
4). Eventually, build a transhuman AGI once we know it’s safe.
Whereas the SIAI master plan looks something like this:
1). Make sure that an un-Friendly AGI does not get built.
2). Figure out how to build a Friendly AGI.
3). Build one.
4). Now that we know it’s safe, build a transhuman AGI (or simply wait long enough, since the AGI from step (3) will boost itself to transhuman levels).
One key difference between olalonde’s plan and SIAI’s plan is the assumption SIAI is making: they are assuming that any AGI will inevitably (plus or minus epsilon) self-improve itself to transhuman levels. Thus, from their perspective, olalonde’s step (2) above might as well say, “build a machine that’s guaranteed to eat us all”, which would clearly be a bad thing.
A good summary. I’d slightly modify it in as much as they would allow the possibility that a really weak AGI may not do much in the way of FOOMing but they pretty much ignore those ones and expect they would just be a stepping stone for the developers who would go on to make better ones. (This is just my reasoning but I assume they would think similarly.)
Good point. Though I guess we could still say that the weak AI is recursively self-improving in this scenario—it’s just using the developers’ brains as its platform, as opposed to digital hardware. I don’t know whether the SIAI folks would endorse this view, though.
Can’t we limit the meaning of “self-improving” to at least stuff that the AI actually does? We can already say more precisely that the AI is being iteratively improved by the creators. We don’t have to go around removing the distinction between what an agent does and what the creator of the agent happens to do to it.
Yeah, I am totally onboard with this suggestion.
Great. I hope I wasn’t being too pedantic there. I wasn’t trying to find technical fault with anything essential to your position.
Can you clarify your reasons for believing this, as distinct from ”...any AGI has a non-negligible chance of self-improving itself to transhuman levels, and the cost of that happening is so vast that it’s worth devoting effort to avoid even if the chance is relatively low”?
That’s a good point, but, from reading what Eliezer and Luke are writing, I formed the impression that my interpretation is correct. In addition, the SIAI FAQ seems to be saying that intelligence explosion is a natural consequence of Moore’s Law; thus, if Moore’s Law continues to hold, intelligence explosion is inevitable.
FWIW, I personally disagree with both statements, but that’s probably a separate topic.
Huh. The FAQ you cite doesn’t seem to be positing inevitability to me. (shrug)
You’re right, I just re-read it and it doesn’t mention Moore’s Law; either it did at some point and then changed, or I saw that argument somewhere else. Still, the FAQ does seem to suggest that the only thing that can stop the Singularity is total human extinction (well, that, or the existence of souls, which IMO we can safely discount); that’s pretty close to inevitability as far as I’m concerned.
Note that the section you’re quoting is no longer talking about the inevitable ascension of any given AGI, but rather the inevitability of some AGI ascending.
I thought they were talking specifically about an AGI that is capable of recursive self-improvement. This does not encompass all possible AGIs, but the non-self-improving ones are not likely to be very smart, as far as I understand, and thus aren’t a concern.
OK, now I am confused.
This whole thread started because you said:
and I asked why you believed that, as distinct from ”...any AGI has a non-negligible chance of self-improving itself to transhuman levels, and the cost of that happening is so vast that it’s worth devoting effort to avoid even if the chance is relatively low”?
Now you seem to be saying that SI doesn’t believe that any AGI will inevitably (plus or minus epsilon) self-improve itself to transhuman levels, but it is primarily concerned with those who do.
I agree with that entirely; it was my point in the first place.
Were we in agreement all along, have you changed your mind in the course of this exchange, or am I really really confused about what’s going on?
Sorry, I think I am guilty of misusing terminology. I have been using AI and AGI interchangeably, but that’s obviously not right. As far as I understand, “AGI” refers to a general intelligence who can solve (or, at least, attempt to solve) any problem, whereas “AI” refers to any kind of an artificial intelligence, including the specialized kind. There are many AIs that already exist in the world—for example, Google’s AdSense algorithm—but SIAI is not concerned about them (as far as I know), because they lack the capacity to self-improve.
My own hidden assumption, which I should’ve recognized and voiced earlier, is that an AGI (as contrasted with non-general AI) would most likely be produced through a process of recursive self-improvement; it is highly unlikely that an AGI could be created from scratch by humans writing lines of code. As far as I understand, the SIAI agrees with this statement, but again, I could be wrong.
Thus, it is unlikely that a non-general AI will ever be smart enough to warrant concern. It could still do some damage, of course, but then, so could a busted water main. On the other hand, an AGI will most likely arise as the result of recursive self-improvement, and thus will be capable of further self-improvement, thus boosting itself to transhuman levels very quickly unless its self-improvement is arrested by some mechanism.
OK, I think I understand better now.
Yeah, I’ve been talking throughout about what you’re labeling “AI” here. We agree that these won’t necessarily self-improve. Awesome.
With respect to what you’re labeling “AGI” here, you’re saying the following:
1) given that X is an AGI developed by humans, the probability that X has thus far been capable of recursive self-improvement is very high, and
2) given that X has thus far been capable of recursive self-improvement, the probability that X will continue to be capable of recursive self-improvement in the future is very high.
3) SIAI believes 1) and 2).
Yes? Have I understood you?
Yes, with the caveats that a). as far as I know, no such X currently exists, and b). my confidence in (3) is much lower than my confidence in (1) and (2).