Regarding your question, I don’t see theoretical reasons why one shouldn’t be making deals like that (assuming one can and would stick to them etc). I’m not sure which decision theory to apply to them though.
Toby_Ord(Toby Ord)
Karma: 203
The Moral Parliament idea generally has a problem regarding time. If it is thought of as making decisions for the next action (or other bounded time period), with new distribution of votes etc when the next choice comes up, then there are intertemporal swaps (and thus pareto improvements according to each theory) that it won’t be able to achieve. This is pretty bad, as it at least appears to be getting pareto dominated by another method. However, if it is making one decision for all time over all policies for resolving future decisions, then (1) it is even harder to apply in real life than it looked, and (2) it doesn’t seem to be able to deal with cases where you learn more about ethics (i.e. update your credence function over moral theories) -- at least not without quite a bit of extra explanation about how that works. I suppose the best answer may well be that the policies over which the representatives are arguing include branches dealing with all ways the credences could change, weighted by their probabilities. This is even more messy.
My guess is that of these two broad options (decide one bounded decision vs decide everything all at once) the latter is better. But either way it is a bit less intuitive than it first appears.
This is a good idea, though not a new one. Others have abandoned the idea of a formal system for this on the grounds that:
1) It may be illegal 2) Quite a few people think it is illegal or morally dubious (whether or not it is actually illegal or immoral)
It would be insane to proceed with this without confirming (1). If illegal, it would open you up to criminal prosecution, and more importantly, seriously hurt the movements you are trying to help. I think that whether or not it turns out to be illegal, (2) is sufficient reason to not pursue it. It may cause serious reputational damage to the movement which I’d expect to easily outweigh the financial benefits.
I also think that the 10% to 20% boost is extremely optimistic. That would only be achieved if almost everyone was using it and they all wanted to spend most of their money funding charities that don’t operate in their countries. I’d expect something more like a boost of a few percent.
Note that there are also very good alternatives. One example is a large effort to encourage people to informally do this in a non-matched way by donating to the subset of effective charities that are tax deductable in their country. This could get most of the benefits for none of the costs.
This is a really nice and useful article. I particularly like the list of problems AI experts assumed would be AI-complete, but turned out not to be.
I’d add that if we are trying to reach the conclusion that “we should be more worried about non-general intelligences than we currently are”, then you don’t need it to be true that general intelligences are really difficult. It would be enough that “there is a reasonable chance we will encounter a dangerous non-general one before a dangerous general one”. I’d be inclined to believe that even without any of the theorising about possibility.
I think one reason for the focus on ‘general’ in the AI Safety community is that it is a stand in for the observation that we are not worried about path planners or chess programs or self-driving cars etc. One way to say this is that these are specialised systems, not general ones. But you rightly point out that it doesn’t follow that we should only be worried about completely general systems.
Thanks for bringing this up Luke. I think the term ‘friendly AI’ has become something of an albatross around our necks as it can’t be taken seriously by people who take themselves seriously. This leaves people studying this area without a usable name for what they are doing. For example, I talk with parts of the UK government about the risks of AGI. I could never use the term ‘friendly AI’ in such contexts—at least without seriously undermining my own points. As far as I recall, the term was not originally selected with the purpose of getting traction with policy makers or academics, so we shouldn’t be too surprised if we can see something that looks superior for such purposes. I’m glad to hear from your post that ‘AGI safety’ hasn’t rubbed people up the wrong way, as feared.
It seems from the poll that there is a front runner, which is what I tend to use already. It is not too late to change which term is promoted by MIRI / FHI etc. I think we should.
This is quite possibly the best LW comment I’ve ever read. An excellent point with a really concise explanation. In fact it is one of the most interesting points I’ve seen within Kolmogorov complexity too. Well done on independently deriving the result!
Without good ways to overcome selection bias, it is unclear that data like this can provide any evidence of outsized impact of unconventional approaches. I would expect a list of achievements as impressive as the above whether or not there was any correlation between the two.
Carl,
You are completely right that there is a somewhat illicit factor-of-1000 intuition pump in a certain direction in the normal problem specification, which makes it a bit one-sided. Will McAskill and I had half-written a paper on this and related points regarding decision-theoretic uncertainty and Newcomb’s problem before discovering that Nozick had already considered it (even if very few people have read or remembered his commentary on this).
We did still work out though that you can use this idea to create compound problems where for any reasonable distribution of credences in the types of decision theory, you should one-box on one of them and two-box on the other: something that all the (first order) decision theories agree is wrong. So much the worse for them, we think. I’ve stopped looking into this, but I think Will has a draft paper where he talks about this alongside some other issues.
Regarding (2), this is a particularly clean way to do it (with some results of my old simulations too).
http://www.amirrorclear.net/academic/papers/sipd.pdf http://www.amirrorclear.net/academic/ideas/dilemma/index.html
We can’t use the universal prior in practice unless physics contains harnessable non-recursive processes. However, this is exactly the situation in which the universal prior doesn’t always work. Thus, one source of the ‘magic’ is through allowing us to have access to higher levels of computation than the phenomena we are predicting (and to be certain of this).
Also, the constants involved could be terrible and there are no guarantees about this (not even probabilistic ones). It is nice to reach some ratio in the limit, but if your first Graham’s number of guesses are bad, then that is very bad for (almost) all purposes.
Phil,
It’s not actually that hard to make a commitment to give away a large fraction of your income. I’ve done it, my wife has done it, several of my friends have done it etc. Even for yourself, the benefits of peace of mind and lack of cognitive dissonance will be worth the price, and by my calculations you can make the benefits for others at least 10,000 times as big as the costs for yourself. The trick is to do some big thinking and decision making about how to live very rarely (say once a year) then limit your salary through regular giving. That way you don’t have to agonise at the hairdresser’s etc, you just live within your reduced means. Check out my site on this, http://www.givingwhatwecan.org—if you haven’t already.
I didn’t watch the video, but I don’t see how that could be true. Occam’s razor is about complexity, while the conjunction fallacy is about logical strength.
Sure ‘P & Q’ is more complex than ‘P’, but ‘P’ is simpler than ‘(P or ~Q)’ despite it being stronger in the same way (P is equivalent to (P or ~Q) & (P or Q)).
(Another way to see this is that violating Occam’s razor does not make things fallacies).
This certainly doesn’t work in all cases:
There is a hidden object which is either green, red or blue. Three people have conflicting opinions about its colour, based on different pieces of reasoning. If you are the one who believes it is green, you have to add up the opponents who say not-green, despite the fact that there is no single not-green position (think of the symmetry—otherwise everyone could have too great confidence). The same holds true if these are expert opinions.
The above example is basically as general as possible, so in order for your argument to work it will need to add specifics of some sort.
Also, the Koran/Bible case doesn’t work. By symmetry, the Koran readers can say that they don’t need to add up the Bible readers and the atheists, since they are heterogeneous, so they can keep their belief in the Koran...
I don’t think I can persuaded.
I have many good responses to the comments here, and I suppose I could sketch out some of the main arguments against anti-realism, but there are also many serious demands on my time and sadly this doesn’t look like a productive discussion. There seems to be very little real interest in finding out more (with a couple of notable exceptions). Instead the focus is on how to justify what is already believed without finding out any thing else about what the opponents are saying (which is particularly alarming given that many commenters are pointing out that they don’t understand what the opponents are saying!).
Given all of this, I fear that writing a post would not be a good use of my time.
You are correct that it is reasonable to assign high confidence to atheism even if it doesn’t have 80% support, but we must be very careful here. Atheism is presumably the strongest example of such a claim here on Less Wrong (i.e. one which you can tell a great story why so many intelligent people would disagree etc and hold a high confidence in the face of disagreement). However, this does not mean that we can say that any other given view is just like atheism in this respect and thus hold beliefs in the face of expert disagreement, that would be far too convenient.
Roko, you make a good point that it can be quite murky just what realism and anti-realism mean (in ethics or in anything else). However, I don’t agree with what you write after that. Your Strong Moral Realism is a claim that is outside the domain of philosophy, as it is an empirical claim in the domain of exo-biology or exo-sociology or something. No matter what the truth of a meta-ethical claim, smart entities might refuse to believe it (the same goes for other philosophical claims or mathematical claims).
Pick your favourite philosophical claim. I’m sure there are very smart possible entities that don’t believe this and very smart ones that do. There are probably also very smart entities without the concepts needed to consider it.
I understand why you introduced Strong Moral Realism: you want to be able to see why the truth of realism would matter and so you came up with truth conditions. However, reducing a philosophical claim to an empirical one never quite captures it.
For what its worth, I think that the empirical claim Strong Moral Realism is false, but I wouldn’t be surprised if there was considerable agreement among radically different entities on how to transform the world.
Thanks for looking that up Carl—I didn’t know they had the break-downs. This is the more relevant result for this discussion, but it doesn’t change my point much. Unless it was 80% or so in favour of anti-realism, I think holding something like 95% credence in anti-realism this is far too high for non-experts.
You are entirely right that the 56% would split up into many subgroups, but I don’t really see how this weakens my point: more philosophers support realist positions than anti-realist ones. For what its worth, the anti-realists are also fragmented in a similar way.
In metaethics, there are typically very good arguments against all known views, and only relatively weak arguments for each of them. For anything in philosophy, a good first stop is the Stanford Encyclopedia of Philosophy. Here are some articles on the topic at SEP:
I think the best book to read on metaethics is:
This is an interesting theorem which helps illuminate the relationship between unbounded utilities and St Petersburg gambles. I particularly appreciate that you don’t make an explicit assumption that the values of gambles must be representable by real numbers which is very common, but unhelpful in a setting like this. However, I do worry a bit about the argument structure.
The St Petersburg gamble is a famously paradox-riddled case. That is, it is a very difficult case where it isn’t clear what to say, and many theories seem to produce outlandish results. When this happens, it isn’t so impressive to say that we can rule out an opposing theory because in that paradox-riddled situation it would lead to strange results. It strikes me as similar to saying that a rival theory leads to strange result in variable population-size cases so we can reject it (when actually, all theories do), or that it leads to strange results in infinite population cases (when again, all theories do).
Even if one had a proof that an alternative theory doesn’t lead to strange conclusions in the St Petersburg gamble, I don’t think this would count all that much in its favour. As it seems plausible to me that various rules of decision theory that were developed in the cleaner cases of finite possibility spaces (or well-behaved infinite spaces) need to be tweaked to account for more pathological possibility spaces. For a simple example, I’m sympathetic to the sure thing principle, but it directly implies that the St Petersburg Gamble is better than itself, because an unresolved gamble is better than a resolved one, no matter how the latter was resolved. My guess is that this means the sure thing principle needs to have its scope limited to exclude gambles whose value is higher than that of any of their resolutions.