What distinction are you trying to draw between “arbitrarily large” and “indefinitely large” that turns the concept into one which is applicable to the real world?
Arbitrarily large means just that in the mathematical sense. Indefinitely large is a term that would be used in other contexts. In the contexts that I’ve seen “indefinitely” used and the way I would mean it, it means so large as to not matter as the exact value for the purpose under discussion (as in “our troops can hold the fort indefinitely”).
Lots of factors indicate that “FOOM” is poorly defined—including the disagreement surrounding it,
Disagreement about something is not always a definitional issue. Indeed, when dealing with people on LW where people try to be rational as possible and have whole sequences about tabooing words and the like, one shouldn’t assign a very high probability to disagreements being due to definitions. Moreover, as one of the people who assigns a low probability to foom and have talked to people here about those issues, I’m pretty sure that we aren’t disagreeing on definitions. Our estimates for what the world will probably look like in 50 years disagree. That’s not simply a definitional issue.
Usually, step 1 in those kinds of discussions is to make sure that people are using the terms in the same way—and have a real disagreement—and not just a semantic one.
Ok. So why are you now doing step 1 years later? And moreover, how long should this step take as you’ve phrased it, given that we know that there’s substantial disagreement in terms of predicted observations about reality in the next few years? That can’t come from definitions. This is not a tree in a forest.
Recently I participated in this exchange—where a poster here gave p(FOOM) = 0.001 - and when pressed they agreed that they did not have a clear idea of what class of events they were referring to.
Yes! Empirical evidence. Unfortunately, it isn’t very strong evidence. I don’t know if he meant in that context that he didn’t have a precise definition or just that he didn’t feel that he understood things well enough to assign a probability estimate. Note that those aren’t the same thing.
I don’t see how the proposed word substitution is supposed to help. If FOOM means: “to quickly, recursively self-improve so as to influence our world with indefinitely large strength and subtlety”, we still face the same issues—of how fast is “quickly” and how big is “indefinitely large”. Those terms are uncalibrated. For the idea to be meaningful or useful, some kind of quantification is needed. Otherwise, we are into “how long is a piece of string?” territory.
So why are you now doing step 1 years later?
I did also raise the issue two years ago. No response, IIRC. I am not too worried if FOOM is a vague term. It isn’t a term I use very much. However, for the folks here—who like to throw their FOOMs around—the issue may merit some attention.
If indefinitely large is still too vague, you can replace it with “”to quickly, recursively self-improve so as to influence our world with sufficient strength and subtlety such that it can a) easily wipe out humans b) humans are not a major threat to it achieving almost any goal set and c) humans are sufficiently weak that it doesn’t gain resources by bothering to bargain with us.” Is that narrow enough?
What is supposed to have happened in the mean time?
You partly address the third question—and suggest that the clock is stopped “quickly” after it is started.
I don’t think that is any good. If we have “quickly” being the proposed-elsewhere “inside six weeks”, it is better—but there is still a problem, which is that there are no constraints being placed on the capabilities of the humans back when the clock was started. Maybe they were just as weak back then.
Since I am the one pointing out this mess, maybe I should also be proposing solutions:
I think the problem is that people want to turn the “FOOM” term into a binary categorisation—to FOOM or not to FOOM.
Yudkowsky’s original way of framing the issue doesn’t really allow for that. The idea is explicitly and deliberately not quantified in his post on the topic. I think the concept is challenging to quantify—and so there is some wisdom in not doing so. All that means is that you can’t really talk about: “to FOOM or not to FOOM”. Rather, there are degrees of FOOM. If you want to quantify or classify them, it’s your responsibility to say how you are measuring things.
It does look as though Yudkowsky has tried this elsewhere—and made an effort to say something a little bit more quantitative.
I’m puzzled a bit by your repeated questions about when to “start the clock” and this seems like it is possibly connected to the issue that people when discussing fooming are discussing a general intelligence going foom. They aren’t talking about little machine intelligences, whether neural networks or support vector machines or matchbox learning systems. They are talking about artificial general intelligence. The “clock” starts from when a a general intelligence with intelligence about as much as a bright human goes online.
I don’t think that is any good. If we have “quickly” being the proposed-elsewhere “inside six weeks”, it is better—but there is still a problem, which is that there are no constraints being placed on the capabilities of the humans back when the clock was started. Maybe they were just as weak back then.
Arbitrarily large means just that in the mathematical sense. Indefinitely large is a term that would be used in other contexts. In the contexts that I’ve seen “indefinitely” used and the way I would mean it, it means so large as to not matter as the exact value for the purpose under discussion (as in “our troops can hold the fort indefinitely”).
Disagreement about something is not always a definitional issue. Indeed, when dealing with people on LW where people try to be rational as possible and have whole sequences about tabooing words and the like, one shouldn’t assign a very high probability to disagreements being due to definitions. Moreover, as one of the people who assigns a low probability to foom and have talked to people here about those issues, I’m pretty sure that we aren’t disagreeing on definitions. Our estimates for what the world will probably look like in 50 years disagree. That’s not simply a definitional issue.
Ok. So why are you now doing step 1 years later? And moreover, how long should this step take as you’ve phrased it, given that we know that there’s substantial disagreement in terms of predicted observations about reality in the next few years? That can’t come from definitions. This is not a tree in a forest.
Yes! Empirical evidence. Unfortunately, it isn’t very strong evidence. I don’t know if he meant in that context that he didn’t have a precise definition or just that he didn’t feel that he understood things well enough to assign a probability estimate. Note that those aren’t the same thing.
I don’t see how the proposed word substitution is supposed to help. If FOOM means: “to quickly, recursively self-improve so as to influence our world with indefinitely large strength and subtlety”, we still face the same issues—of how fast is “quickly” and how big is “indefinitely large”. Those terms are uncalibrated. For the idea to be meaningful or useful, some kind of quantification is needed. Otherwise, we are into “how long is a piece of string?” territory.
I did also raise the issue two years ago. No response, IIRC. I am not too worried if FOOM is a vague term. It isn’t a term I use very much. However, for the folks here—who like to throw their FOOMs around—the issue may merit some attention.
If indefinitely large is still too vague, you can replace it with “”to quickly, recursively self-improve so as to influence our world with sufficient strength and subtlety such that it can a) easily wipe out humans b) humans are not a major threat to it achieving almost any goal set and c) humans are sufficiently weak that it doesn’t gain resources by bothering to bargain with us.” Is that narrow enough?
The original issues were:
When to start the clock?
When to stop the clock?
What is supposed to have happened in the mean time?
You partly address the third question—and suggest that the clock is stopped “quickly” after it is started.
I don’t think that is any good. If we have “quickly” being the proposed-elsewhere “inside six weeks”, it is better—but there is still a problem, which is that there are no constraints being placed on the capabilities of the humans back when the clock was started. Maybe they were just as weak back then.
Since I am the one pointing out this mess, maybe I should also be proposing solutions:
I think the problem is that people want to turn the “FOOM” term into a binary categorisation—to FOOM or not to FOOM.
Yudkowsky’s original way of framing the issue doesn’t really allow for that. The idea is explicitly and deliberately not quantified in his post on the topic. I think the concept is challenging to quantify—and so there is some wisdom in not doing so. All that means is that you can’t really talk about: “to FOOM or not to FOOM”. Rather, there are degrees of FOOM. If you want to quantify or classify them, it’s your responsibility to say how you are measuring things.
It does look as though Yudkowsky has tried this elsewhere—and made an effort to say something a little bit more quantitative.
I’m puzzled a bit by your repeated questions about when to “start the clock” and this seems like it is possibly connected to the issue that people when discussing fooming are discussing a general intelligence going foom. They aren’t talking about little machine intelligences, whether neural networks or support vector machines or matchbox learning systems. They are talking about artificial general intelligence. The “clock” starts from when a a general intelligence with intelligence about as much as a bright human goes online.
Huh? I don’t follow.