This doesn’t sound like a bad idea. Could someone give reasons to think that donations to SIAI now would be better than this?
utilitymonster
Karma: 339
I think the page makes a case that it is worth doing something about AI risk, and that SIAI is doing something. The page gives no one any reason to think that SIAI is doing better than anything else you could do about x-risk (there could be reasons elsewhere).
In this respect, the page is similar to other non-profit pages: (i) argue that there is a problem, (ii) argue that you’re doing something to solve the problem, but don’t (iii) try to show that you’re solving the problem better than others. Maybe that’s reasonable, since that rubs some donors the wrong way and is hard to establish that you’re the best; but it doesn’t advance our discussion about the best way to reduce x-risk.
Thanks.
So, you’re really interested in this question: what is the best decision algorithm? And then you’re interested, in a subsidiary way, in what you ought to do. You think the “action” sense is silly, since you can’t run one algorithm and make some other choice.
Your answer to my objection involving the parody argument is that you ought to do something else (not go with loss aversion) because there is some better decision algorithm (that you could, in some sense of “could”, use?) that tells you to do something else.
What do you do with cases where it is impossible for you to run a different algorithm? You can’t exactly use your algorithm to switch to some other algorithm, unless your original algorithm told you to do that all along, so these cases won’t be that rare. How do you avoid the result that you should just always use whatever algorithm you started with? However you answer this objection, why can’t two-boxers who care about the “action sense” of ought answer your objection analogously?
tl;dr Philsophers have been writing about what probabilities reduce to for a while. As far as I know, the only major reductionist view is David Lewis’s “best system” account of laws (of nature) and chances. You can look for “best system” in this article for an intro. Barry Loewer has developed this view in this paper.
I agree with all of this.
I agree that this fact [you can’t have a one-boxing disposition and then two box] could appear as premise in an argument, together with an alternative proposed decision theory, for the conclusion that one-boxing is a bad idea. If that was the implicit argument, then I now understand the point.
To be clear: I have not been trying to argue that you ought to take two boxes in Newcomb’s problem.
But I thought this fact [you can’t have a one-boxing disposition and then two box] was supposed to be a part of an argument that did not use a decision theory as a premise. Maybe I was misreading things, but I thought it was supposed to be clear that two-boxers were irrational, and that this should be pretty clear once we point out that you can’t have the one-boxing disposition and then take two boxes.
What is false is that you ought to have disposition a and do B.
OK. So the argument is this one:
According to two-boxers, you ought to (i) have the disposition to one-box, and (ii) take two boxes.
It is impossible to do (i) and (ii).
Ought implies can.
So two-boxers are wrong.
But, on your use of “disposition”, two-boxers reject 1. They do not believe that you should have a FAWS-disposition to one-box, since having a FAWS-disposition to one-box just means “actually taking one box, where this is not a result of randomness”. Two-boxers think you should non-randomly choose to take two boxes.
ETA: Some two-boxers may hesitate to agree that you “ought to have a disposition to one-box”, even in the philosopher’s sense of “disposition”. This is because they might want “ought” to only apply to actions; such people would, at most, agree that you ought to make yourself a one-boxer.
Everyone agrees about what the best disposition to have is. The disagreement is about what to do. I have uniformly meant “ought” in the action sense, not the dispositional sense. (FYI: this is always the sense in which philosophers (incl. Richard) mean “ought”, unless otherwise specified.)
BTW: I still don’t understand the relevance of the fact that it is impossible for people with one-boxing dispositions to two-box. If you don’t like the arguments that I formalized for you, could you tell me what other premises you are using to reach your conclusion?
Whatever you actually do (modulo randomness) at time t, that’s your one and only disposition vs X at time t.
Okay, I understand how you use the word “disposition” now. This is not the way I was using the word, but I don’t think that is relevant to our disagreement. I hereby resolve to use the phrase “disposition to A” in the same way as you for the rest of our conversation.
I still don’t understand how this point suggests that people with one-boxing dispositions ought not to two-box. I can only understand it in one way: as in the argument in my original reply to you. But that argument form leads to this absurd conclusion:
(a) whenever you have a disposition to A and you do A, it is false that you ought to have done something else
In particular, it backfires for the intended argumentative purpose, since it entails that two-boxers shouldn’t one-box.
As far as I can tell, it would only have that implication in situations where an outcome depended directly on one’s disposition (as opposed to one’s actions).
I don’t think so:
John has the loss-aversion disposition..
It is impossible to have the loss-aversion disposition and maximize expected utility in case C.
Ought implies can.
Therefore, it is false that John ought to maximize expected utility in case C.
Or, for Newcomb:
It is impossible for someone with the two-boxing disposition to one-box.
Ought implies can.
Therefore, it is false that someone with the two-boxing disposition ought to one box.
You can’t have a disposition to act in a certain way without counter-factually acting that way. You can’t counter-factually act a certain way without actually acting that way in a situation indistinguishable form the counter-factual.
What is the relevance of this? Are you using this argument? (See comment above.)*
It is impossible to have the one-boxing disposition and then two-box.
Ought implies can.
Therefore, it is false that someone with a one-boxing disposition ought to two-box.
If that isn’t your argument, what is the force of the quoted text?
At any rate, it seems like a bad argument, since analogous arguments will entail that whenever you have some decisive disposition, it is false that you ought to act differently. (It will entail, for instance, NOT[people who have a decisive loss aversion disposition should follow expected utility theory].)
Notice that an analogous argument also cuts the other way:
It is impossible for someone with the two-boxing disposition to one-box.
Ought implies can.
Therefore, it is false that someone with the two-boxing disposition ought to one box.
*I made a similar comment above, but I don’t know how to link to it. Help appreciated.
Is this the argument?
It is impossible to have the one-boxing disposition and then two-box.
Ought implies can.
Therefore, it is false that someone with a one-boxing disposition ought to two-box.
Or are you agreeing that you ought to two-box, but claiming that this fact isn’t interesting because of premise 1?
At any rate, it seems like a bad argument, since analogous arguments will entail that whenever you have some decisive disposition, it is false that you ought to act differently. (It will entail, for instance, NOT[people who have a decisive loss aversion disposition should follow expected utility theory].)
Runaway AI/AI going FOOM—This is potentially the most interesting of your points simply because it is so much more unique to the SIAI and Eliezer. So what can one do to figure out if this is correct? One thing to do is to examine the arguments and claims being made in detail. And see what other experts think on the subject. In this context, most AI people seem to consider this to be an unlikely problem, so maybe look at what they have to say? Note also that Robert Hanson of Overcoming Bias has discussed these issues extensively with Eliezer and has not been at all convinced (they had a written debate a while ago but I can’t find the link right now. If someone else can track it down I’d appreciate it).
a) Something much smarter than us will do whatever it wants, and very thoroughly. (this doesn’t require godlike AI, just smarter than us. Self-improving helps, too.) b) The vast majority of possible “wants” done thoroughly will destroy us. (Any goal taken to extremes will use all available matter in accomplishing it.) Therefore, it will be dangerous if not VERY carefully designed. Humans are notably greedy and bad planners individually, and often worse in groups.
I’ve heard a lot of variations on this theme. They all seem to assume that the AI will be a maximizer rather than a satisficer. I agree the AI could be a maximizer, but don’t see that it must be. How much does this risk go away if we give the AI small ambitions?
Have you read Nick Bostrom’s paper, Astronomical Waste? You don’t have to be able to affect the probabilities by very much for existential risk to be the thing to worry about, especially if you have a decent dose of credence in utilitarianism.
Is there a decent chance, in your view, of decrease x-risk by 10^-18 if you put all of your resources into it? That could be enough. (I agree that this kind of argument is worrisome; maybe expected utility theory or utilitarianism breaks down with these huge numbers and tiny probabilities, but it is worth thinking about.)
If you’re sold on x-risk, are there some candidate other things that might have higher expectations of x-risk reductions on the margin (after due reflection)? (I’m not saying SIAI clearly wins, I just want to know what else you’re thinking about.)
I feel some of the force of this...I do think we should take the opinions of other experts seriously, even if their arguments don’t seem good.
I sort of think that many of these criticisms of SIAI turn on not being Bayesian enough. Lots of people only want to act on things they know, where knowing requires really solid evidence, the kind of evidence you get through conventional experimental science, with low p-values and all. It is just impossible to have that kind of robust confidence about the far future. So you’re going to have people just more or less ignore speculative issues about the far future, even if those issues are by far the most important. Once you adopt a Bayesian perspective, and you’re just interested in maximizing expected utility, the complaint that we don’t have a lot of evidence about what will be best for the future, or the complaint that we just don’t really know whether SIAI’s mission and methodology are going to work seems to lose a lot of force.
Given your answers to 1-3, you should spend all of your altruistic efforts on mitigating x-risk (unless you’re just trying to feel good, entertain yourself, etc.).
For 4, I shouldn’t have asked you whether you “think” something beats negotiating a positive singularity in terms of x-risk reduction. Better: Is there some other fairly natural class of interventions (or list of potential examples) such that, given your credences, has a higher expected value? What might such things be?
For 5-6, perhaps you should think about what such organizations might be. Those interested in convincing XiXiDu might try listing some alternative best x-risk mitigating groups and provide arguments that they don’t do as well. As for me, my credences are highly unstable in this area, so info is appreciated on my part as well.
This assumes countable additivity. Otherwise, each basic event could have probability zero, though the whole has probability 1.
I’m inclined to use countable additivity, but I couldn’t give you arguments for it, other than by sketching models that violate it and pointing out that they are weird.
Question about Solomonoff induction: does anyone have anything good to say about how to associate programs with basic events/propositions/possible worlds?