Did you actually read through the MWI sequence before deciding that you still can’t tell whether MWI is true because of (as I understand your post correctly) the state of the social evidence? If so, do you know what pluralistic ignorance is, and Asch’s conformity experiment?
If you know all these things and you still can’t tell that MWI is obviously true—a proposition far simpler than the argument for supporting SIAI—then we have here a question that is actually quite different from the one you seem to try to be presenting:
I do not have sufficient g-factor to follow the detailed arguments on Less Wrong. What epistemic state is it rational for me to be in with respect to SIAI?
If you haven’t read through the MWI sequence, read it. Then try to talk with your smart friends about it. You will soon learn that your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don’t.
If you know all these things and you still can’t tell that MWI is obviously true—a proposition far simpler than the argument for supporting SIAI—then we have here a question that is actually quite different from the one you seem to try to be presenting:
I do not have sufficient g-factor to follow the detailed arguments on Less Wrong. What epistemic state is it rational for me to be in with respect to SIAI?
I respectfully disagree. I am someone who was convinced by your MWI explanations but even so I am not comfortable with outright associating reserved judgement with lack of g.
This is a subject that relies on an awful lot of crystalized knowledge about physics. For someone to come to a blog knowing only what they can recall of high school physics and be persuaded to accept a contrarian position on what is colloquially considered the most difficult part of science is a huge step.
The trickiest part is correctly accounting for meta-uncertainty. There are a lot of things that seem extremely obvious but turn out to be wrong. I would even suggest that the trustworthiness of someone’s own thoughts is not always proportionate to g-factor. That leaves people with some situations where they need to trust social processes more than their own g. That may prompt them to go and explore the topic from various other sources until such time that they can trust that their confidence is not just naivety.
On a subject like physics and MWI, I wouldn’t take the explanation of any non-professional as enough to establish that a contrarian position is “obviously correct”. Even if they genuinely believed in what they said, they’ll still only be presenting the evidence from their own point of view. Or they might be missing something essential and I wouldn’t have the expertise to realize that. Heck, I wouldn’t even go on the word of a full-time researcher in the field before I’d heard what their opponents had to say.
On a subject matter like cryonics I was relatively convinced from simply hearing what the cryonics advocates had to say, because it meshed with my understanding of human anatomy and biology, and it seemed like nobody was very actively arguing the opposite. But to the best of my knowledge, people are arguing against MWI, and I simply wouldn’t have enough domain knowledge to evaluate either sort of claim. You could argue your case of “this is obviously true” with completely made-up claims, and I’d have no way to tell.
But whose domain knowledge are we talking about in the first place? Eliezer argues that MWI is a question of probability theory rather than physics per se. In general, I don’t see much evidence that physicists who argue against MWI actually have the kind of understanding of probability theory necessary to make their arguments worth anything. (Though of course it’s worth emphasizing here that “MWI” in this context means only “the traditional collapse postulate can’t be right” and not “the Schroedinger equation is a complete theory of physics”.)
In general, I don’t see much evidence that physicists who argue against MWI actually have the kind of understanding of probability theory necessary to make their arguments worth anything.
Physicists have something else, however, and that is domain expertise. As far as I am concerned, MWI is completely at odds with the spirit of relativity. There is no model of the world-splitting process that is relativistically invariant. Either you reexpress MWI in a form where there is no splitting, just self-contained histories each of which is internally relativistic, or you have locally propagating splitting at every point of spacetime in every branch, in which case you don’t have “worlds” any more, you just have infinitely many copies of infinitely many infinitesimal patches of space-time which are glued together in some complicated way. You can’t even talk about extended objects in this picture, because the ends are spacelike separated and there’s no inherent connection between the state at one end and the state at the other end. It’s a complete muddle, even before we try to recover the Born probabilities.
Rather than seeing MWI as the simple and elegant way to understand QM, I see it as an idea which in a way turns out to be too simple—which is another way of saying, naive or uninformed. Like Bohmian mechanics, conceptually it relies on a preferred frame.
The combination of quantum mechanics with special relativity yields quantum field theory. In quantum field theory, everything empirically meaningful is conceptually relativistic. In your calculations, you may employ entities (like wavefunctions evolving in time) which are dependent on a particular reference frame, but you can always do such calculations in a different frame. An example of a calculational output which is frame-independent would be the correlation function between two field operators at different points in space-time. By the time we reach the point of making predictions, that correlation function should only depend on the (relativistically invariant) space-time separation. But in order to calculate it, we may adopt a particular division into space and time, write down wavefunctions defined to exist on the constant-time hypersurfaces in that reference frame, and evolve them according to a Hamiltonian. These wavefunctions are only defined with respect to a particular reference frame and a particular set of hypersurfaces. Therefore, they are somehow an artefact of a particular coordinate system. But they are the sorts of objects in terms of which MWI is constructed.
The truly relativistic approach to QFT is the path integral, the sum over all field histories interpolating between conditions on an initial and a final hypersurface. These histories are objects which are defined independently of any particular coordinate system, because they are histories and not just instantaneous spacelike states. But then we no longer have an evolving superposition, we just have a “superposition” of histories which do not “split” or “join”.
At any time, theoretical physics contains many ideas and research programs, and there are always a lot of them that are going nowhere. MWI has all the signs of an idea going nowhere. It doesn’t advance the field in any way. Instead, as with Bohmian mechanics, what happens is that specific quantum theories are proposed (field theories, string theories), and then the Everettians, the Bohmians, and so on wheel out their interpretive apparatus, which they then “apply” to the latest theoretical advance. It’s a parasitic relationship and it’s a sign that in the long run this is a dead end.
I will provide an example of an idea which is more like what I would look for in an explanation of quantum theory. The real problem with quantum theory is the peculiar way its probabilities are obtained. You have complex numbers and negative quasiprobabilities and histories that cancel each other. The cancellation of possibilities makes no sense from the perspective of orthodox probability. If an outcome can come about in one way, the existence of a second way can only increase the probability of the outcome—according to probability theory and common sense. Yet in the double-slit experiment we have outcomes that are reduced in probability through “destructive interference”. That is what we need to explain.
There is a long history of speculation that maybe the peculiar structure of quantum probabilities can be obtained by somehow conditioning on the future as well as on the past, or by having causality working backwards as well as forwards in time. No-one has ever managed to derive QM this way, but many people have talked about it.
In string theory, there are light degrees of freedom, and heavy degrees of freedom. The latter correspond to the higher (more energetic) excitations of the string, though we should not expect that strings are fundamental in the full theory. In any case, these heavy excitations should cause space to be very strongly curved. So, what if the heavy degrees of freedom create a non-time-orientable topology on the Planck scale, giving rise to temporally bidirectional constraints on causality, and then the light strings interact (lightly) with that background, and quantum-probability effects are the indirect manifestation of that deeper causal structure, which has nonlocal correlations in space and time?
That’s an idea I had during my string studies. It is not likely to be right, because it’s just an idea. But it is an explanation which is intrinsically connected to the developing edge of theoretical physics, rather than a prefabricated explanation which is then applied in a one-size-fits-all fashion to any quantum theory. It would be an intrinsically string-theoretic derivation of QM. That is the sort of explanation for QM that I find plausible, for the reason that everything deep in physics is deeply connected to every other deep thing.
Either you reexpress MWI in a form where there is no splitting, just self-contained histories each of which is internally relativistic
Huh? This is what I’ve always¹ taken MWI in a relativistic context...
Just kidding. More like, since the first time I thought about the issue after graduating (and hence having an understanding of SR and QM devoid of the misconceptions found in certain popularizations).
Anyway, I’ll have to read the works by ’t Hooft when I have time. They look quite interesting.
In 1204.4926 the idea is that a quantum oscillator is actually a discrete deterministic system that cycles through a finite number of states. Then in 1205.4107 he maps a cellular automaton onto a free field theory made out of coupled quantum oscillators. Then in 1207.3612 he adds boolean variables to his CA (previously the cells were integer-valued) in order to add fermionic fields. At this point his CA is looking a little like a superstring, which from a “worldsheet” perspective is a line with bosonic and fermionic quantum fields on it. But there are still many issues whose resolution needs to be worked out.
I wasn’t convinced of MWI by the quantum mechanics sequence when I read it. I came to the conclusion that it’s probably true later, after thinking intensively about the anthropic trilemma (my preferred resolution is incompatible with single-world interpretations); but my probability estimate is still only at 0.8.
I’ll write more about this, but it will take some time. I posted the basic idea on the Anthropic Trilemma thread when I first had it, but the explanation is too pithy to follow easily, and I don’t think many people saw it. Revisiting it now has brought to mind an intuition pump to use.
I do not have sufficient g-factor to follow the detailed arguments on Less Wrong. What epistemic state is it rational for me to be in with respect to SIAI?
This is rude (although I realize there is now name-calling and gratuitous insult being mustered on both sides) , and high g-factor does not make those MWI arguments automatically convincing. High g-factor combined with bullet-biting, a lack of what David Lewis called the argument of the incredulous stare, does seem to drive MWI pretty strongly. I happen to think that weighting the incredulous stare as an epistemic factor independent of its connections with evolution, knowledge in society, etc, is pretty mistaken, but bullet-dodgers often don’t. Accusing someone of being low-g rather than a non-bullet-biter is the insulting possibility.
Just recently I encountered someone very high IQ/SAT/GRE scores who bought partial quantitative parsimony/Speed Prior type views, and biases against the unseen. This person claimed that the power of parsimony was not enough to defeat the evidence for galaxies and quarks, but was sufficient to defeat a Big World much beyond our Hubble Bubble, and to favor Bohm’s interpretation over MWI. I think that view isn’t quite consistent without a lot of additional jury-rigging, but it isn’t reliably prevented by high g and exposure to the arguments from theoretical simplicity, non-FTL, etc.
It seems to me that a sufficiently cunning arguer can come up with what appears to be a slam-dunk argument for just about anything. As far as I can tell, I follow the arguments in the MWI sequence perfectly, and the conclusion does pretty much follow from the premises. I just don’t know if those premises are actually true. Is MWI what you get if you take the Schrodinger equation literally? (Never mind that the basic Schrodinger equation is non-relativistic; I know that there are relativistic formulations of QM.) I can’t tell you, because I don’t know the underlying math. And, indeed, the “Copenhagen interpretation” seems like patent nonsense, but what about all the others? I don’t know enough to answer the question, and I’m not going to bother doing much more research because I just don’t really care what the answer is.
I’m not an expert on relativistic QM (anyone who is, correct me if I misspeak), but I know enough to tell that Mitchell Porter is confused by what splitting means. In relativistic QM, the wavefunction evolves in a local manner in configuration space, as opposed to the Schrödinger equation’s instantaneous (but exponentially small) assignment of mass to distant configurations. Decoherence happens in this picture just as it happened in the regular one.
The reason that Mitchell (and others) are confused about the EPR experiment is that, although the two entangled particles are separated in space, the configurations which will decohere are very close to one another in configuration space. Locality is therefore not violated by the decoherence.
Sorry, but no. The Dirac equation is invariant under Lorentz transformations, so what’s local in one inertial reference frame is local in any other as well.
The Dirac equation is invariant, but there are a lot of problems with the concept of locality. For example, if you want to create localised one-particle states that remain local in any reference frame and form an orthonormal eigenbasis of the (one-particle subspace of) the Hilbert space, you will find it impossible.
The canonical solution in axiomatic QFT is to begin with local operations instead of localised particles. However, to see the problem, one has to question the notion of measurement and define it in a covariant manner, which may be a mess. See e.g.
I am sympathetic to the approach which is used by quantum gravitists, which uses extended Hamiltonian formalism and the Wheeler—DeWitt equation instead of the Schrödinger one. This approach doesn’t use time as special, however the phase space isn’t isomorphic to the state space on the classical level and similar thing holds on the quantum level for the Hilbert space, which makes the interpretation less obvious.
You’re missing my point. To make sense of the Dirac equation, you have to interpret it as a statement about field operators, so locality means (e.g.) that spacelike-separated operators commute. But that’s just a statement about expectation values of observables. MWI is supposed to be a comprehensive ontological interpretation, i.e. a theory of what is actually there in reality.
You seem to be saying that configurations (field configurations, particle configurations, it makes no difference for this argument) are what is actually there. But a “configuration” is spatially extended. Therefore, it requires a universal time coordinate. Everett worlds are always defined with respect to a particular time-slicing—a particular set of spacelike hypersurfaces. From a relativistic perspective, it looks as arbitrary as any “objective collapse” theory.
It looks to me as though you’ve focused in on one of the weaker points in XiXiDu’s post rather than engaging with the (logically independent) stronger points.
XiXiDu wants to know why he can trust SIAI instead of Charles Stross. Reading the MWI sequence is supposed to tell him that far more effectively than any cute little sentence I could write. The first thing I need to know is whether he read the sequence and something went wrong, or if he didn’t read the sequence.
Well, you’ve picked the weakest of his points to answer, and I put it to you that it was clearly the weakest.
You are right of course that what does or doesn’t show up in Charles Stross’s writing doesn’t constitute evidence in either direction—he’s a professional fiction author, he has to write for entertainment value regardless of what he may or may not know or believe about what’s actually likely or unlikely to happen.
A better example would be e.g. Peter Norvig, whose credentials are vastly more impressive than yours (or, granted, than mine), and who thinks we need to get at least another couple of decades of progress under our belts before there will be any point in resuming attempts to work on AGI. (Even I’m not that pessimistic.)
If you want to argue from authority, the result of that isn’t just tilted against the SIAI, it’s flat out no contest.
A better example would be e.g. Peter Norvig, whose credentials are vastly more impressive than yours (or, granted, than mine), and who thinks we need to get at least another couple of decades of progress under our belts before there will be any point in resuming attempts to work on AGI. (Even I’m not that pessimistic.)
If this means “until the theory and practice of machine learning is better developed, if you try to build an AGI using existing tools you will very probably fail” it’s not unusually pessimistic at all. “An investment of $X in developing AI theory will do more to reduce the mean time to AI than $X on AGI projects using existing theory now” isn’t so outlandish either. What was the context/cite?
I don’t have the reference handy, but he wasn’t saying let’s spend 20 years of armchair thought developing AGI theory before we start writing any code (I’m sure he knows better than that), he was saying forget about AGI completely until we’ve got another 20 years of general technological progress under our belts.
Those would seem likely to be helpful indeed. Better programming tools might also help, as would additional computing power (not so much because computing power is actually a limiting factor today, as because we tend to scale our intuition about available computing power to what we physically deal with on an everyday basis—which for most of us, is a cheap desktop PC—and we tend to flinch away from designs whose projected requirements would exceed such a cheap PC; increasing the baseline makes us less likely to flinch away from good designs).
Here too, it looks like you’re focusing on a weak aspect of his post rather than engaging him. Nobody who’s smart and has read your writing carefully doubts that you’re uncommonly brilliant and that this gives you more credibility than the other singulatarians. But there are more substantive aspects of XiXiDu’s post which you’re not addressing.
Like what? Why he should believe in exponential growth? When by “exponential” he actually means “fast” and no one at SIAI actually advocates for exponentials, those being a strictly Kurzweilian obsession and not even very dangerous by our standards? When he picks MWI, of all things, to accuse us of overconfidence (not “I didn’t understand that” but “I know something you don’t about how to integrate the evidence on MWI, clearly you folks are overconfident”)? When there’s lots of little things scattered through the post like that (“I’m engaging in pluralistic ignorance based on Charles Stross’s nonreaction”) it doesn’t make me want to plunge into engaging the many different little “substantive” parts, get back more replies along the same line, and recapitulate half of Less Wrong in the process. The first thing I need to know is whether XiXiDu did the reading and the reading failed, or did he not do the reading? If he didn’t do the reading, then my answer is simply, “If you haven’t done enough reading to notice that Stross isn’t in our league, then of course you don’t trust SIAI”. That looks to me like the real issue. For substantive arguments, pick a single point and point out where the existing argument fails on it—don’t throw a huge handful of small “huh?”s at me.
Castles in the air. Your claims are based on long chains of reasoning that you do not write down in a formal style. Is the probability of correctness of each link in that chain of reasoning so close to 1, that their product is also close to 1?
I can think of a couple of ways you could respond:
Yes, you are that confident in your reasoning. In that case you could explain why XiXiDu should be similarly confident, or why it’s not of interest to you whether he is similarly confident.
It’s not a chain of reasoning, it’s a web of reasoning, and robust against certain arguments being off. If that’s the case, then we lay readers might benefit if you would make more specific and relevant references to your writings depending on context, instead of encouraging people to read the whole thing before bringing criticisms.
Most of the long arguments are concerned with refuting fallacies and defeating counterarguments, which flawed reasoning will always be able to supply in infinite quantity. The key predictions, when you look at them, generally turn out to be antipredictions, and the long arguments just defeat the flawed priors that concentrate probability into anthropomorphic areas. The positive arguments are simple, only defeating complicated counterarguments is complicated.
“Fast AI” is simply “Most possible artificial minds are unlikely to run at human speed, the slow ones that never speed up will drop out of consideration, and the fast ones are what we’re worried about.”
“UnFriendly AI” is simply “Most possible artificial minds are unFriendly, most intuitive methods you can think of for constructing one run into flaws in your intuitions and fail.”
MWI is simply “Schrodinger’s equation is the simplest fit to the evidence”; there are people who think that you should do something with this equation other than taking it at face value, like arguing that gravity can’t be real and so needs to be interpreted differently, and the long arguments are just there to defeat them.
The only argument I can think of that actually approaches complication is about recursive self-improvement, and even there you can say “we’ve got a complex web of recursive effects and they’re unlikely to turn out exactly exponential with a human-sized exponent”, the long arguments being devoted mainly to defeating the likes of Robin Hanson’s argument for why it should be exponential with an exponent that smoothly couples to the global economy.
One problem I have with your argument here is that you appear to be saying that if XiXiDu doesn’t agree with you, he must be stupid (the stuff about low g etc.). Do you think Robin Hanson is stupid too, since he wasn’t convinced?
I haven’t found the text during a two minute search or so, but I think I remember Robin assigning a substantial probability, say, 30% or so, to the possibility that MWI is false, even if he thinks most likely (i.e. the remaining 70%) that it’s true.
Much as you argued in the post about Einstein’s arrogance, there seems to be a small enough difference between a 30% chance of being false, and a 90% chance of being false, if the latter would imply that Robin was stupid, the former would imply it too.
Right: in fact he would act as though MWI is certainly false… or at least as though Quantum Immortality is certainly false, which has a good chance of being true given MWI.
Quantum Immortality is certainly false, which has a good chance of being true given MWI.
No! He will act as if Quantum Immortality is a bad choice, which is true even if QI works exactly as described. ‘True’ isn’t the right kind word to use unless you include a normative conclusion in the description of QI.
Suppose that being shot with the gun cannot possibly have intermediate results: either the gun fails, or he is killed instantly and painlessly.
Also suppose that given that there are possible worlds where he exists, each copy of him only cares about its anticipated experiences, not about the other copies, and that this is morally the right thing to do… in other words, if he expects to continue to exist, he doesn’t care about other copies that cease to exist. This is certainly the attitude some people would have, and we could suppose (for the LCPW) that it is the correct attitude.
Even so, given these two suppositions, I suspect it would not affect his behavior in the slightest, showing that he would be acting as though QI is certainly false, and therefore as though there is a good chance that MWI is false.
each copy of him only cares about its anticipated experiences, not about the other copies, and that this is morally the right thing to do… in other words, if he expects to continue to exist, he doesn’t care about other copies that cease to exist.
But that is crazy and false, and uses ‘copies’ to in a misleading way. Why would I assume that?
Even so, given these two suppositions, I suspect it would not affect his behavior in the slightest, showing that he would be acting as though QI is certainly false,
This ‘least convenient possible world’ is one in which Robin’s values are changed according to your prescription but his behaviour is not, ensuring that your conclusion is true. That isn’t the purpose of inconvenient worlds (kind of the opposite...)
and therefore as though there is a good chance that MWI is false.
Not at all. You are conflating “MWI is false” with a whole different set of propositions. MWI != QS.
Many people in fact have those values and opinions, and nonetheless act in the way I mention (and there is no one who does not so act) so it is quite reasonable to suppose that even if Robin’s values were so changed, his behavior would remain unchanged.
The very reason Robin was brought up (by you I might add) was to serve as an ad absurdum with respect to intellectual disrespect.
One problem I have with your argument here is that you appear to be saying that if XiXiDu doesn’t agree with you, he must be stupid (the stuff about low g etc.). Do you think Robin Hanson is stupid too, since he wasn’t convinced?
In the Convenient World where Robin is, in fact, too stupid to correctly tackle the concept of QS, understand the difference between MWI and QI or form a sophisticated understanding of his moral intuitions with respect to quantum uncertainty this Counterfactual-Stupid-Robin is a completely useless example.
I can imagine two different meanings for “not convinced about MWI”
It refers to someone who is not convinced that MWI is as good as any other model of reality, and better than most.
It refers to someone who is not convinced that MWI describes the structure of reality.
If we are meant to understand the meaning as #1, then it may well indicate that someone is stupid. Though, more charitably, it might more likely indicate that he is ignorant.
If we are meant to understand the meaning as #2, then I think that it indicates someone who is not entrapped by the Mind Projection Fallacy.
What do you mean by belief in MWI? What sort of experiment could settle whether MWI is true or not?
I suspect that a lot of people object to the stuff including copies of humans and other worlds we should care about and hypotheses about consciousness tacitly build on MWI, rather than MWI itself.
First, the links say that MWI needs a linear quantum theory, and lists therefore the linearity among its predictions. However, linearity is a part of the quantum theory and its mathematical formalism, and nothing specific to MWI. Also, weak non-linearity would be explicable using the language of MWI saying that the different worlds interact a little. I don’t see how testing the superposition principle establishes MWI. A very weak evidence at best.
Second, there is a very confused paragraph about quantum gravity, which, apart from linking to itself, states only that MWI requires gravity to be quantised (without supporting argument) and therefore if gravity is successfully quantised, it forms evidence for MWI. However, nobody doubts that gravity has to be quantised somehow, even hardcore Copenhageners.
The most interesting part is that about the reversible measurement done by an artificial intelligence. As I understand it, it supposes that we construct a machine which could perform measurements in reversed direction of time, for which it has to be immune to quantum decoherence. It sounds interesting, but is also suspicious. I see no way how can we get the information into our brains without decoherence. The argument apparently tries to circumvent this objection by postulating an AI, which is reversible and decoherence-immune, but the AI will still face the same problem when trying to tell us the results. In fact, postulating the need of an AI here seems to be only a tool to make the proposed experiment more obscure and difficult to analyse. We will have a “reversible AI”, therefore miraculously we will detect differences between Copenhagen and MWI.
However, at least there is a link to Deutsch’s article which hopefully explains the experiment in greater detail, so I will read it and edit the comment later.
“Many-worlds is often referred to as a theory, rather than just an interpretation, by those who propose that many-worlds can make testable predictions (such as David Deutsch) or is falsifiable (such as Everett) or by those who propose that all the other, non-MW interpretations, are inconsistent, illogical or unscientific in their handling of measurements”
None of the tests in that FAQ look to me like they could distinguish MWI from MWI+worldeater. The closest thing to an experimental test I’ve come up with is the following:
Flip a quantum coin. If heads, copy yourself once, advance both copies enough to observe the result, then kill one of the copies. If tails, do nothing.
In a many-worlds interpretation of QM, from the perspective of the experimenter, the coin will be heads with probability 2⁄3, since there are two observers in that case and only one if the coin was tails. In the single-world case, the coin will be heads with probability 1⁄2. So each time you repeat the experiment, you get 0.4 bits of evidence for or against MWI. Unfortunately, this evidence is also non-transferrable; someone else can’t use your observation as evidence the same way you can. And getting enough evidence for a firm conclusion involves a very high chance of subjective death (though it is guaranteed that exactly one copy will be left behind). And various quantum immortality hypotheses screw up the experiment, too.
So it is testable in principle, but the experiment involved more odious than one would imagine possible.
The math works the same in all interpretations, but some experiments are difficult to understand intuitively without the MWI. I usually give people the example of the Elitzur-Vaidman bomb tester where the easy MWI explanation says “we know the bomb works because it exploded in another world”, but other interpretations must resort to clever intellectual gymnastics.
If all interpretations are equivalent with respect to testable outcomes, what makes the belief in any particular interpretation so important? Ease of intuitive understanding is a dangerous criterion to rely on, and a relative thing too. Some people are more ready to accept mental gymnastic than existence of another worlds.
Well, that depends. Have you actually tried to do the mental gymnastics and explain the linked experiment using the Copenhagen interpretation? I suspect that going through with that may influence your final opinion.
Have you actually tried to do the mental gymnastics and explain the linked experiment [the Elitzur-Vaidman bomb tester] using the Copenhagen interpretation?
Maybe I’m missing something, but how exactly does this experiment challenge the Copenhagen interpretation more than the standard double-slit stuff? Copenhagen treats “measurement” as a fundamental and irreducible process and measurement devices as special components in each experiment—and in this case it simply says that a dud bomb doesn’t represent a measurement device, whereas a functioning one does, so that they interact with the photon wavefunction differently. The former leaves it unchanged, while the latter collapses it to one arm of the interferometer—eiher its own, in which case it explodes, or the other one, in which case it reveals itself as a measurement device just by the act of collapsing.
As far as I understand, this would be similar to the standard variations on the double-slit experiment where one destroys the interference pattern by placing a particle detector at the exit from one of the holes. One could presumably do a similar experiment with a detector that might be faulty, and conclude that an interference-destroying detector works even if it doesn’t flash when several particles are let through (in cases where they all happen to go through the other hole). Unless I’m misunderstanding something, this would be a close equivalent of the bomb test.
The final conclusion in the bomb test is surely more spectacular, but I don’t see how it produces any extra confusion for Copenhageners compared to the most basic QM experiments.
Frankly, I don’t know what you consider an explanation here. I am quite comfortable with the prediction which the theory gives, and accept that as an explanation. So I never needed mental gymnastics here. The experiment is weird, but it doesn’t seem to me less weird by saying that the information about the bomb’s functionality came from its explosion in the other world.
Your claims are only anti-predictions relative to science-fiction notions of robots as metal men.
Most possible artificial minds are neither Friendly nor unFriendly (unless you adopt such a stringent definition of mind that artificial minds are not going to exist in my lifetime or yours).
Fast AI (along with most of the other wild claims about what future technology will do, really) falls afoul of the general version of Amdahl’s law. (On which topic, did you ever update your world model when you found out you were mistaken about the role of computers in chip design?)
About MWI, I agree with you completely, though I am more hesitant to berate early quantum physicists for not having found it obvious. For a possible analogy: what do you think of my resolution of the Anthropic Trilemma?
This is quite helpful, and suggests that what I wanted is not a lay-reader summary, but an executive summary.
I brought this up elsewhere in this thread, but the fact that quantum mechanics and gravity are not reconciled suggests that even Schrodinger’s equation does not fit the evidence. The “low-energy” disclaimer one has to add is very weird, maybe weirder than any counterintuitive consequences of quantum mechanics.
I brought this up elsewhere in this thread, but the fact that quantum mechanics and gravity are not reconciled suggests to be that even Schrodinger’s equation does not fit the evidence. The “low-energy” disclaimer one has to add is very weird, maybe weirder than any counterintuitive consequences of quantum mechanics.
It’s not the Schrödinger equation alone that gives rise to decoherence and thus many-worlds. (Read Good and Real for another toy model, the “quantish” system.) The EPR experiment and Bell’s inequality can be made to work on macroscopic scales, so we know that whatever mathematical object the universe will turn out to be, it’s not going to go un-quantum on us again: it has the same relevant behavior as the Schrödinger equation, and accordingly MWI will be the best interpretation there as well.
“There is no intangible stuff of goodness that you can divorce from life and love and happiness in order to ask why things like that are good. They are simply what you are talking about in the first place when you talk about goodness.”
And then the long arguments are about why your brain makes you think anything different.
This is less startling than your more scientific pronouncements. Are there any atheists reading this that find this (or at first found this) very counterintuitive or objectionable?
I would go further, and had the impression from somewhere that you did not go that far. Is that accurate?
I’m a cognitivist. Sentences about goodness have truth values after you translate them into being about life and happiness etc. As a general strategy, I make the queerness go away, rather than taking the queerness as a property of a thing and using it to deduce that thing does not exist; it’s a confusion to resolve, not an existence to argue over.
No, nothing, and because while religion does contain some confusion, after you eliminate the confusion you are left with claims that are coherent but false.
Morality is a specific set of values (Or, more precisely, a specific algorithm/dynamic for judging values). Humans happen to be (for various reasons) the sort of beings that value morality as opposed to valuing, say, maximizing paperclip production. It is indeed objectively better (by which we really mean “more moral”/”the sort of thing we should do”) to be moral than to be paperclipish. And indeed we should be moral, where by “should” we mean, “more moral”.
(And moral, when we actually cash out what we actually mean by it seems to translate to a complicated blob of values like happiness, love, creativity, novelty, self determination, fairness, life (as in protecting theirof), etc...)
It may appear that paperclip beings and moral beings disagree about something, but not really. The paperclippers would, once they’ve analyzed what humans actually mean by “moral”, would agree “yep, humans are more moral than us. But who cares about this morality stuff, it doesn’t maximize paperclips!”
Of course, screw the desires of the paperclippers, after all, they’re not actually moral. We really are objectively better (once we think carefully by what we mean by “better”) than them.
(note, “does something or does something not actually do a good job of fulfilling a certain value?” is an objective question. ie, “does a particular action tend to increase the expected number of paperclips?” (on the paperclipper side) or, on our side, stuff like “does a particular action tend to save more lives, increase happiness, increase fairness, add novelty...” etc etc etc is an objective question in that we can extract specific meaning from that question and can objectively (in a way the paperclippers would agree with) judge that. It simply happens to be that we’re the sorts of beings that actually care about the answer to that (as we should be), while the screwy hypothetical paperclippers are immoral and only care about paperclips.
How’s that, that make sense? Or, to summarize the summary, “Morality is objective, and we humans happen to be the sorts of beings that value morality, as opposed to valuing something else instead”
a specific algorithm/dynamic for judging values, or
a complicated blob of values like happiness, love, creativity, novelty, self determination, fairness, life (as in protecting theirof), etc.?
If it’s 1, can we say something interesting and non-trivial about the algorithm, besides the fact that it’s an algorithm? In other words, everything can be viewed as an algorithm, but what’s the point of viewing morality as an algorithm?
If it’s 2, why do we think that two people on opposite sides of the Earth are referring to the same complicated blob of values when they say “morality”? I know the argument about the psychological unity of humankind (not enough time for significant genetic divergence), but what about cultural/memetic evolution?
I’m guessing the answer to my first question is something like, morality is an algorithm whose current “state” is a complicated blob of values like happiness, love, … so both of my other questions ought to apply.
If it’s 2, why do we think that two people on opposite sides of the Earth are referring to the same complicated blob of values when they say “morality”? I know the argument about the psychological unity of humankind (not enough time for significant genetic divergence), but what about cultural/memetic evolution?
You don’t even have to do any cross-cultural comparisons to make such an argument. Considering the insights from modern behavioral genetics, individual differences within any single culture will suffice.
There is no reason to be at all tentative about this. There’s tons of cog sci data about what people mean when they talk about morality. It varies hugely (but predictably) across cultures.
Why are you using algorithm/dynamic here instead of function or partial function? (On what space, I will ignore that issue, just as you have...) Is it supposed to be stateful? I’m not even clear what that would mean. Or is function what you mean by #2? I’m not even really clear on how these differ.
You might have gotten confused because I quoted Psy-Kosh’s phrase “specific algorithm/dynamic for judging values” whereas Eliezer’s original idea I think was more like an algorithm for changing one’s values in response to moral arguments. Here are Eliezer’s own words:
I would say, by the way, that the huge blob of a computation is not just my present terminal values (which I don’t really have - I am not a consistent expected utility maximizers); the huge blob of a computation includes the specification of those moral arguments, those justifications, that would sway me if I heard them.
Others have pointed out that this definition is actually quite unlikely to be coherent: people would be likely to be ultimately persuaded by different moral arguments and justifications if they had different experiences and heard arguments in different orders etc.
Others have pointed out that this definition is actually quite unlikely to be coherent
Yes, see here for an argument to that effect by Marcello and subsequent discussion about it between Eliezer and myself.
I think the metaethics sequence is probably the weakest of Eliezer’s sequences on LW. I wonder if he agrees with that, and if so, what he plans to do about this subject for his rationality book.
I think the metaethics sequence is probably the weakest of Eliezer’s sequences on LW. I wonder if he agrees with that, and if so, what he plans to do about this subject for his rationality book.
This is somewhat of a concern given Eliezer’s interest in Friendliness!
As far as I can understand, Eliezer has promoted two separate ideas about ethics: defining personal morality as a computation in the person’s brain rather than something mysterious and external, and extrapolating that computation into smarter creatures. The former idea is self-evident, but the latter (and, by extension, CEV) has received a number of very serious blows recently. IMO it’s time to go back to the drawing board. We must find some attack on the problem of preference, latch onto some small corner, that will allow us to make precise statements. Then build from there.
defining personal morality as a computation in the person’s brain rather than something mysterious and external
But I don’t see how that, by itself, is a significant advance. Suppose I tell you, “mathematics is a computation in a person’s brain rather than something mysterious and external”, or “philosophy is a computation in a person’s brain rather than something mysterious and external”, or “decision making is a computation in a person’s brain rather than something mysterious and external” how much have I actually told you about the nature of math, or philosophy, or decision making?
This makes sense in that it is coherent, but it is not obvious to me what arguments would be marshaled in its favor. (Yudkowsky’s short formulations do point in the direction of their justifications.) Moreover, the very first line, “morality is a specific set of values,” and even its parenthetical expansion (algorithm for judging values), seems utterly preposterous to me. The controversies between human beings about which specific sets of values are moral, at every scale large and small, are legendary beyond cliche.
The controversies between human beings about which specific sets of values are moral, at every scale large and small, are legendary beyond cliche.
It is a common thesis here that most humans would ultimately have the same moral judgments if they were in full agreement about all factual questions and were better at reasoning. In other words, human brains have a common moral architecture, and disagreements are at the level of instrumental, rather than terminal, values and result from mistaken factual beliefs and reasoning errors.
You may or may not find that convincing (you’ll get to the arguments regarding that if you’re reading the sequences), but assuming that is true, then “morality is a specific set of values” is correct, though vague: more precisely, it is a very complicated set of terminal values, which, in this world, happens to be embedded solely in a species of minds who are not naturally very good at rationality, leading to massive disagreement about instrumental values (though most people do not notice that it’s about instrumental values).
It is a common thesis here that most humans would ultimately have the same moral judgments if they were in full agreement about all factual questions and were better at reasoning. In other words, human brains have a common moral architecture, and disagreements are at the level of instrumental, rather than terminal, values and result from mistaken factual beliefs and reasoning errors.
It is? That’s a worry. Consider this a +1 for “That thesis is totally false and only serves signalling purposes!”
I… think it is. Maybe I’ve gotten something terribly wrong, but I got the impression that this is one of the points of the complexity of value and metaethics sequences, and I seem to recall that it’s the basis for expecting humanity’s extrapolated volition to actually cohere.
I seem to recall that it’s the basis for expecting humanity’s extrapolated volition to actually cohere.
This whole area isn’t covered all that well (as Wei noted). I assumed that CEV would rely on solving an implicit cooperation problem between conflicting moral systems. It doesn’t appear at all unlikely to me that some people are intrinsically selfish to some degree and their extrapolated volitions would be quite different.
Note that I’m not denying that some people present (or usually just assume) the thesis you present. I’m just glad that there are usually others who argue against it!
It is a common thesis here that most humans would ultimately have the same moral judgments if they were in full agreement about all factual questions and were better at reasoning.
Maybe it’s true if you also specify “if they were fully capable of modifying their own moral intuitions.” I have an intuition (an unexamined belief? a hope? a sci-fi trope?) that humanity as a whole will continue to evolve morally and roughly converge on a morality that resembles current first-world liberal values more than, say, Old Testament values. That is, it would converge, in the limit of global prosperity and peace and dialogue, and assuming no singularity occurs and the average lifespan stays constant. You can call this naive if you want to; I don’t know whether it’s true. It’s what I imagine Eliezer means when he talks about “humanity growing up together”.
This growing-up process currently involves raising children, which can be viewed as a crude way of rewriting your personality from scratch, and excising vestiges of values you no longer endorse. It’s been an integral part of every culture’s moral evolution, and something like it needs to be part of CEV if it’s going to actually converge.
It is a common thesis here that most humans would ultimately have the same moral judgments if they were in full agreement about all factual questions and were better at reasoning.
That’s not plausible. That would be some sort of objective morality, and there is no such thing. Humans have brains, and brains are complicated. You can’t have them imply exactly the same preference.
Now, the non-crazy version of what you suggest is that preferences of most people are roughly similar, that they won’t differ substantially in major aspects. But when you focus on detail, everyone is bound to want their own thing.
It makes sense in its own terms, but it leaves the unpleasant implication that morality differs greatly between humans, at both individual and group level—and if this leads to a conflict, asking who is right is meaningless (except insofar as everyone can reach an answer that’s valid only for himself, in terms of his own morality).
So if I live in the same society with people whose morality differs from mine, and the good-fences-make-good-neighbors solution is not an option, as it often isn’t, then who gets to decide whose morality gets imposed on the other side? As far as I see, the position espoused in the above comment leaves no other answer than “might is right.” (Where “might” also includes more subtle ways of exercising power than sheer physical coercion, of course.)
...and if this leads to a conflict, asking who is right is meaningless (except insofar as everyone can reach an answer that’s valid only for himself, in terms of his own morality).
So if I live in the same society with people whose morality differs from mine, and the good-fences-make-good-neighbors solution is not an option, as it often isn’t, then who gets to decide whose morality gets imposed on the other side?
That two people mean different things by the same word doesn’t make all questions asked using that word meaningless, or even hard to answer.
If by “castle” you mean “a fortified structure”, while I mean “a fortified structure surrounded by a moat”, who will be right if we’re asked if the Chateau de Gisors is a castle? Any confusion here is purely semantic in nature. If you answer yes and I answer no, we won’t have given two answers to the same question, we’ll have given two answers to two different questions. If Psy-Kosh says that the Chateau de Gisors is a fortified structure but it is not surrounded by a moat, he’ll have answered both our questions.
Now, once this has been clarified, what would it mean to ask who gets to decide whose definition of ‘castle’ gets imposed on the other side? Do we need a kind of meta-definition of castle to somehow figure out what the one true definition is? If I could settle this issue by exercising power over you, would it change the fact that the Chateau de Gisors is not surrounded by a moat? If I killed everyone who doesn’t mean the same thing by the word ‘castle’ than I do, would the sentence “a fortified structure” become logically equivalent to the sentence “a fortified structure surrounded by a moat”?
In short, substituting the meaning of a word for the word tends to make lots of seemingly difficult problems become laughably easy to solve. Try it.
*blinks* how did I imply that morality varies? I thought (was trying to imply) that morality is an absolute standard and that humans simply happen to be the sort of beings that care about the particular standard we call “morality”. (Well, with various caveats like not being sufficiently reflective to be able to fully explicitly state our “morality algorithm”, nor do we fully know all its consequences)
However, when humans and paperclippers interact, well, there will probably be some sort of fight if one doesn’t end up with some sort PD cooperation or whatever. It’s not that paperclippers and humans disagree on anything, it’s simply, well, they value paperclips a whole lot more than lives. We’re sort of stuck with having to act in a way to prevent the hypothetical them from acting on that.
(of course, the notion that most humans seem to have the same underlying core “morality algorithm”, just disagreeing on the implications or such, is something to discuss, but that gets us out of executive summary territory, no?)
(of course, the notion that most humans seem to have the same underlying core “morality algorithm”, just disagreeing on the implications or such, is something to discuss, but that gets us out of executive summary territory, no?)
I would say that it’s a crucial assumption, which should be emphasized clearly even in the briefest summary of this viewpoint. It is certainly not obvious, to say the least. (And, for full disclosure, I don’t believe that it’s a sufficiently close approximation of reality to avoid the problem I emphasized above.)
Hrm, fair enough. I thought I’d effectively implied it, but apparently not sufficiently.
(Incidentally… you don’t think it’s a close approximation to reality? Most humans seem to value (to various extents) happiness, love, (at least some) lives, etc… right?)
Different people (and cultures) seem to put very different weights on these things.
Here’s an example:
You’re a government minister who has to decide who to hire to do a specific task. There are two applicants. One is your brother, who is marginally competent at the task. The other is a stranger with better qualifications who will probably be much better at the task.
The answer is “obvious.”
In some places, “obviously” you hire your brother. What kind of heartless bastard won’t help out his own brother by giving him a job?
In others, “obviously” you should hire the stranger. What kind of corrupt scoundrel abuses his position by hiring his good-for-nothing brother instead of the obviously superior candidate?
Okay, I can see how XiXiDu’s post might come across that way. I think I can clarify what I think that XiXiDu is trying to get at by asking some better questions of my own.
What evidence has SIAI presented that the Singularity is near?
If the Singularity is near then why has the scientific community missed this fact?
What evidence has SIAI presented for the existence of grey goo technology?
If grey goo technology is feasible then why has the scientific community missed this fact?
Assuming that the Singularity is near, what evidence is there that SIAI has a chance to lower global catastrophic risk in a nontrivial way?
What evidence is there that SIAI has room for more funding?
“Near”? Where’d we say that? What’s “near”? XiXiDu thinks we’re Kurzweil?
What kind of evidence would you want aside from a demonstrated Singularity?
Grey goo? Huh? What’s that got to do with us? Read Nanosystems by Eric Drexler or Freitas on “global ecophagy”. XiXiDu thinks we’re Foresight?
If this business about “evidence” isn’t a demand for particular proof, then what are you looking for besides not-further-confirmed straight-line extrapolations from inductive generalizations supported by evidence?
“Near”? Where’d we say that? What’s “near”? XiXiDu thinks we’re Kurzweil?
You’ve claimed that in your blogging heads divlog with Scott Aaronson that you think that it’s pretty obvious that there will be an AGI within the next century. As far as I know you have not offered a detailed description of the reasoning that led you to this conclusion that can be checked by others.
I see this as significant for the reasons given in my comment here.
Grey goo? Huh? What’s that got to do with us? Read Nanosystems by Eric Drexler or Freitas on “global ecophagy”. XiXiDu thinks we’re Foresight?
I don’t know what the situation is with SIAI’s position on grey goo—I’ve heard people say the SIAI staff believe in nanotechnology having capabilities out of line with the beliefs of the scientific community, but they may have been misinformed. So let’s forget about about questions 3 and 4.
You’ve claimed that in your blogging heads divlog with Scott Aaronson that you think that it’s pretty obvious that there will be an AGI within the next century.
You’ve shifted the question from “is SIAI on balance worth donating to” to “should I believe everything Eliezer has ever said”.
I don’t know what the situation is with SIAI’s position on grey goo—I’ve heard people say the SIAI staff believe in nanotechnology having capabilities out of line with the beliefs of the scientific community, but they may have been misinformed.
The point is that grey goo is not relevant to SIAI’s mission (apart from being yet another background existential risk that FAI can dissolve). “Scientific community” doesn’t normally professionally study (far) future technological capabilities.
My whole point about grey goo has been, as stated, that a possible superhuman AI could use it to do really bad things. That is, I do not see how an encapsulated AI, even a superhuman AI, could pose the stated risks without the use of advanced nanotechnology. Is it going to use nukes, like Skynet? Another question related to the SIAI, regarding advanced nanotechnology, is that if without advanced nanotechnology superhuman AI is at all possible.
I’m shocked how you people misintepreted my intentions there.
Grey goo is only a potential danger in its own right because it’s a way dumb machinery can grow in destructive power (you don’t need to assume AI controlling it for it to be dangerous, at least so goes the story). AGI is not dumb, so it can use something more fitting to precise control than grey goo (and correspondingly more destructive and feasible).
The grey goo example was named to exemplify the speed and sophistication of nanotechnology that would have to be around to either allow an AI to be build in the first place or be of considerable danger.
I consider your comment an expression of personal disgust. No way you could possible misinterpret my original point and subsequent explanation to this extent.
The grey goo example was named to exemplify the speed and sophistication of nanotechnology that would have to be around to either allow an AI to be build in the first place or be of considerable danger.
As katydee pointed out, if for some strange reason grey goo is what AI would want, AI will invent grey goo. If you used “grey goo” to refer to the rough level of technological development necessary to produce grey goo, then my comments missed that point.
I consider your comment an expression of personal disgust. No way you could possible misinterpret my original point and subsequent explanation to this extent.
Illusion of transparency. Since the general point about nanotech seems equally wrong to me, I couldn’t distinguish between the error of making it and making a similarly wrong point about the relevance of grey goo in particular. In general, I don’t plot, so take my words literally. If I don’t like something, I just say so, or keep silent.
If it seems equally wrong, why haven’t you pointed me to some further reasoning on the topic regarding the feasibility of AGI without advanced (grey goo level) nanotechnology? Why haven’t you argued about the dangers of AGI which is unable to make use of advanced nanotechnology? I was inquiring about these issues in my original post and not trying to argue against the scenarios in question.
Yes, I’ve seen the comment regarding the possible invention of advanced nanotechnology by AGI. If AGI needs something that isn’t there it will just pull it out of its hat. Well, I have my doubts that even a superhuman AGI can steer the development of advanced nanotechnology so that it can gain control of it. Sure, it might solve the problems associated with it and send the solutions to some researcher. Then it could buy the stocks of the subsequent company involved with the new technology and somehow gain control...well, at this point we are already deep into subsequent reasoning about something shaky that at the same time is used as evidence of the very reasoning involving it.
To the point: if AGI can’t pose a danger, because its hands are tied, that’s wonderful! Then we have more time to work of FAI. FAI is not about superpowerful robots, it’s about technically understanding what we want, and using that understanding to automate the manufacturing of goodness. The power is expected to come from unbounded automatic goal-directed behavior, something that happens without humans in the system to ever stop the process if it goes wrong.
Overall I’d feel a lot more comfortable if you just said “there’s a huge amount of uncertainty as to when existential risks will strike and which ones will strike, I don’t know whether or not I’m on the right track in focusing on Friendly AI or whether I’m right about when the Singularity will occur, I’m just doing the best that I can.”
This is largely because of the issue that I raise here
I should emphasize that I don’t think that you’d ever knowingly do something that raised existential risk, I think that you’re a kind and noble spirit. But I do think I’m raising a serious issue which you’ve missed.
If this business about “evidence” isn’t a demand for particular proof, then what are you looking for besides not-further-confirmed straight-line extrapolations from inductive generalizations supported by evidence?
I am looking for the evidence in “supported by evidence”. I am further trying to figure how you anticipate your beliefs to pay rent, what you anticipate to see if explosive recursive self-improvement is possible, and how that belief could be surprised by data.
If you just say, “I predict we will likely be wiped out by badly done AI.”, how do you expect to update on evidence? What would constitute such evidence?
To put my own spin on XiXiDu’s questions: What quality or position does Charles Stross possess that should cause us to leave him out of this conversation (other than the quality ‘Eliezer doesn’t think he should be mentioned’)?
What stronger points are you referring to? It seems to me XiXiDu’s post has only 2 points, both of which Eliezer addressed:
“Given my current educational background and knowledge I cannot differentiate LW between a consistent internal logic, i.e. imagination or fiction and something which is sufficiently based on empirical criticism to provide a firm substantiation of the strong arguments for action that are proclaimed on this site.”
His smart friends/favorite SF writers/other AI researchers/other Bayesians don’t support SIAI.
My point is that your evidence has to stand up to whatever estimations you come up with. My point is the missing transparency in your decision making regarding the possibility of danger posed by superhuman AI. My point is that any form of external peer review is missing and that therefore I either have to believe you or learn enough to judge all of your claims myself after reading hundreds of posts and thousands of documents to find some pieces of evidence hidden beneath. My point is that competition is necessary, that not just the SIAI should work on the relevant problems. There are many other points you seem to be missing entirely.
That one’s easy: We’re doing complex multi-step extrapolations argued to be from inductive generalizations themselves supported by the evidence, which can’t be expected to come with experimental confirmation of the “Yes, we built an unFriendly AI and it went foom and destroyed the world” sort. This sort of thing is dangerous, but a lot of our predictions are really antipredictions and so the negations of the claims are even more questionable once you examine them.
If you have nothing valuable to say, why don’t you stay away from commenting at all? Otherwise you could simply ask me what I meant to say, if something isn’t clear. But those empty statements coming from you recently make me question if you’ve been the person that I thought you are. You cannot even guess what I am trying to ask here? Oh come on...
I was inquiring about the supportive evidence at the origin of your complex multi-step extrapolations argued to be from inductive generalizations. If there isn’t any, what difference is there between writing fiction and complex multi-step extrapolations argued to be from inductive generalizations?
Agreed, and I think there’s a pattern here. XiXiDu is asking the right questions about why SIAI doesn’t have wider support. It is because there are genuine holes in its reasoning about the singularity, and SIAI chooses not to engage with serious criticism that gets at those holes. Example (one of many): I recall Shane Legg commenting that it’s not practical to formalize friendliness before anyone builds any form of AGI (or something to that effect). I haven’t seen SIAI give a good argument to the contrary.
Example (one of many): I recall Shane Legg commenting that it’s not practical to formalize friendliness before anyone builds any form of AGI (or something to that effect). I haven’t seen SIAI give a good argument to the contrary.
Gahhh! The hoard of arguments against that idea that instantly sprang to my mind (with warning bells screeching) perhaps hints at why a good argument hasn’t been given to the contrary (if, in fact, it hasn’t). It just seems so obvious. And I don’t mean that as a criticism of you or Shane at all. Most things that we already understand well seem like they should be obvious to others. I agree that there should be a post making the arguments on that topic either here on LessWrong or on the SIAI website somewhere. (Are you sure there isn’t?)
Edit: And you demonstrate here just why Eliezer (or others) should bother to answer XiXiDu’s questions even if there are some weaknesses in his reasoning.
My point is that Shane’s conclusion strikes me as the obvious one, and I believe many smart, rational, informed people would agree. It may be the case that, for the majority of smart, rational, informed people, there exists an issue X for which they think “obviously X” and SIAI thinks “obviously not X.” To be taken seriously, SIAI needs to engage with the X’s.
I understand your point, and agree that your conclusion is one that many smart, rational people with good general knowledge would share. Once again I concur that engaging with those X’s is important, including that ‘X’ we’re discussing here.
Sounds like we mostly agree. However, I don’t think it’s a question of general knowledge. I’m talking about smart, rational people who have studied AI enough to have strongly-held opinions about it. Those are the people who need to be convinced; their opinions propagate to smart, rational people who haven’t personally investigated AI in depth.
I’d love to hear your take on X here. What are your reasons for believing that friendliness can be formalized practically, and an AGI based on that formalization built before any other sort of AGI?
If I was SIAI my reasoning would be the following. First stop with the believes- believes not dichotomy and move to probabilities.
So what is the probability of a good outcome if you can’t formalize friendliness before AGI? Some of them would argue infinitesimal. This is based on fast take-off winner take all type scenarios (I have a problem with this stage, but I would like it to be properly argued and that is hard).
So looking at the decision tree (under these assumptions) the only chance of a good outcome is to try to formalise FAI before AGI becomes well known. All the other options lead to extinction.
So to attack the “formalise Friendliness before AGI” position you would need to argue that the first AGIs are very unlikely to kill us all. That is the major battleground as far as I am concerned.
Agreed about what the “battleground” is, modulo one important nit: not the first AGI, but the first AGI that recursively self-improves at a high speed. (I’m pretty sure that’s what you meant, but it’s important to keep in mind that, e.g., a roughly human-level AGI as such is not what we need to worry about—the point is not that intelligent computers are magically superpowerful, but that it seems dangerously likely that quickly self-improving intelligences, if they arrive, will be non-magically superpowerful.)
I don’t think formalize-don’t formalize should be a simple dichotomy either; friendliness can be formalized in various levels of detail, and the more details are formalized, the fewer unconstrained details there are which could be wrong in a way that kills us all.
I’d look at it the other way: I’d take it as practically certain that any superintelligence built without explicit regard to Friendliness will be unFriendly, and ask what the probability is that through sufficiently slow growth in intelligence and other mere safeguards, we manage to survive building it.
My best hope currently rests on the AGI problem being hard enough that we get uploads first.
(This is essentially the Open Thread about everything Eliezer or SIAI have ever said now, right?)
Uploading would have quite a few benefits, but I get the impression it would make us more vulnerable to whatever tools a hostile AI may possess, not less.
“So what is the probability of a good outcome if you can’t formalize friendliness before AGI? Some of them would argue infinitesimal.”
One problem here is the use of a circular definition of “friendliness”—that defines the concept it in terms of whether it leads to a favourable outcome. If you think “friendly” is defined in terms of whether or not the machine destroys humanity, then clearly you will think that an “unfriendly” machine would destroy the world. However, this is just a word game—which doesn’t tell us anything about the actual chances of such destruction happening.
Let’s say “we” are the good guys in the race for AI. Define
W = we win the race to create an AI powerful enough to protect humanity from any subsequent AIs
G = our AI can be used to achieve a good outcome
F = we go the “formalize friendliness” route
O = we go a promising route other than formalizing friendliness
At issue is which of the following is higher:
P(G|WF)P(W|F) or P(G|WO)P(W|O)
From what I know of SIAI’s approach to F, I estimate P(W|F) to be many orders of magnitude smaller than P(W|O). I estimate P(G|WO) to be more than 1% for a good choice of O (this is a lower bound; my actual estimate of P(G|WO) is much higher, but you needn’t agree with that to agree with my conclusion). Therefore the right side wins.
There are two points here that one could conceivably dispute, but it sounds like the “SIAI logic” is to dispute my estimate of P(G|WO) and say that P(G|WO) is in fact tiny. I haven’t seen SIAI give a convincing argument for that.
My summary would be: there are huge numbers of types of minds and motivations, so if we pick one at random from the space of minds then it likely to be contrary to our values because it will have a different sense of what is good or worthwhile. This moderately relies on the speed/singleton issue, because evolution pressure between AI might force them in the same direction as us. We would likely be out-competed before this happens though, if we rely on competition between AIs.
I think various people associated with SIAI mean different things by formalizing friendliness. I remember Vladimir Nesov means getting better than 50% probability for providing a good outcome.
It doesn’t matter what happens when we sample a mind at random. We only care about the sorts of minds we might build, whether by designing them or evolving them. Either way, they’ll be far from random.
Consider my “at random” short hand for “at random from the space of possible minds built by humans”.
The Eliezer approved example of humans not getting a simple system to do what they want is the classic Machine Learning example where a Neural Net was trained on two different sorts of tanks. It had happened that the photographs of the different types of tanks had been taken at different times of day. So the classifier just worked on that rather than actually looking at the types of tank. So we didn’t build a tank classifier but a day/night classifier. More here.
While I may not agree with Eliezer on everything, I do agree with him it is damn hard to get a computer to do what you want when you stop programming them explicitly .
Obviously AI is hard, and obviously software has bugs.
To counter my argument, you need to make a case that the bugs will be so fundamental and severe, and go undetected for so long, that despite any safeguards we take, they will lead to catastrophic results with probability greater than 99%.
Things like AI boxing or “emergency stop buttons” would be instances of safeguards. Basically any form of human supervision that can keep the AI in check even if it’s not safe to let it roam free.
Are you really suggesting a trial and error approach where we stick evolved and human created AIs in boxes and then eyeball them to see what they are like? Then pick the nicest looking one, on a hunch, to have control over our light cone?
This is why we need to create friendliness before AGI → A lot of people who are loosely familiar with the subject think those options will work!
A goal directed intelligence will work around any obstacles in front of it. It’ll make damn sure that it prevents anyone from pressing emergency stop buttons.
The first AI will be determined by the first programmer, sure. But I wasn’t talking about that level; the biases and concern for the ethics of the AI of that programmer will be random from the space of humans. Or at least I can’t see any reason why I should expect people who care about ethics to be more likely to make AI than those that think economics will constrain AI to be nice,
That is now a completely different argument to the original “there are huge numbers of types of minds and motivations, so if we pick one at random from the space of minds”.
Re: “the biases and concern for the ethics of the AI of that programmer will be random from the space of humans”
Those concerned probably have to be an expert programmers, able to build a company or research group, and attract talented assistance, as well as probably customers. They will probably be far-from what you would get if you chose at “random”.
Do we pick a side of a coin “at random” from the two possibilities when we flip it?
Epistemically, yes, we don’t have sufficient information to predict it*. However if we do the same thing twice it has the same outcome so it is not physically random.
So while the process that decides what the first AI is like is not physically random, it is epistemically random until we have a good idea of what AIs produce good outcomes and get humans to follow those theories. For this we need something that looks like a theory of friendliness, to some degree.
Considering we might use evolutionary methods for part of the AI creation process, randomness doesn’t look like too bad a model.
*With a few caveats. I think it is biased to land the same way up as it was when flipped, due to the chance of making it spin and not flip.
We do have an extensive body of knowledge about how to write computer programs that do useful things. The word “random” seems like a terrible mis-summary of that body of information to me.
As for “evolution” being equated to “randomnness”—isn’t that one of the points that creationists make all the time? Evolution has two motors—variation and selection. The first of these may have some random elements, but it is only one part of the overall process.
I think we have a disconnect on how much we believe proper scary AIs will be like previous computer programs.
My conception of current computer programs is that they are crystallised thoughts plucked from our own minds and easily controllable and unchangeable. When we get interesting AI the programs will morphing and be far less controllable without a good theory of how to control the change.
I shudder every time people say the “AI’s source code” as if it is some unchangeable and informative thing about the AI’s behaviour after the first few days of the AI’s existence.
You have correctly identified the area in which we do not agree.
The most relevant knowledge needed in this case is knowledge of game theory and human behaviour. They also need to know ‘friendliness is a very hard problem’. They then need to ask themselves the following question:
What is likely to happen if people have the ability to create an AGI but do not have a proven mechanism for implementing friendliness? Is it:
Shelve the AGI, don’t share the research and set to work on creating a framework for friendliness. Don’t rush the research—act as if the groundbreaking AGI work that you just created was a mere toy problem and the only real challenge is the friendliness. Spend an even longer period of time verifying the friendliness design and never let on that you have AGI capabilities.
Something else.
What are your reasons for believing that friendliness can be formalized practically, and an AGI based on that formalization built before any other sort of AGI?
I don’t (with that phrasing). I actually suspect that the problem is too difficult to get right and far too easy to get wrong. We’re probably all going to die. However, I think we’re even more likely to die if some fool goes and invents a AGI before they have a proven theory of friendliness.
Those are the people, indeed. But where do the donations come from? EY seems to be using this argument against me as well. I’m just not educated, well-read or intelligent enough for any criticism. Maybe so, I acknowledged that in my post. But have I seen any pointers to how people arrive at their estimations yet? No, just the demand to read all of LW, which according to EY doesn’t even deal with what I’m trying to figure out, but rather the dissolving of biases. A contradiction?
I’m inquiring about the strong claims made by the SIAI, which includes EY and LW. Why? Because they ask for my money and resources. Because they gather fanatic followers who believe into the possibility of literally going to hell. If you follow the discussion surrounding Roko’s posts you’ll see what I mean. And because I’m simply curious and like to discuss, besides becoming less wrong.
But if EY or someone else is going to tell me that I’m just too dumb and it doesn’t matter what I do, think or donate, I can accept that. I don’t expect Richard Dawkins to enlighten me about evolution either. But don’t expect me to stay quiet about my insignificant personal opinion and epistemic state (as you like to call it) either! Although since I’m conveniently not neurotypical (I guess), you won’t have to worry me turning into an antagonist either, simply because EY is being impolite.
SIAI position does dot require “obviously X” from a decision perspective, the opposite one does. To be so sure of something as complicated as the timeline of FAI math vs AGI development seems seriously foolish to me.
It is not a matter about being sure of it but to weigh it against what is asked for in return, other possible events of equal probability and the utility payoff from spending the resources on something else entirely.
I’m not asking the SIAI to prove “obviously X” but rather to prove the very probability of X that they are claiming it has within the larger context of possibilities.
Capa: It’s the problem right there. Between the boosters and the gravity of the sun the velocity of the payload will get so great that space and time will become smeared together and everything will distort. Everything will be unquantifiable.
Kaneda: You have to come down on one side or the other. I need a decision.
Capa: It’s not a decision, it’s a guess. It’s like flipping a coin and asking me to decide whether it will be heads or tails.
Kaneda: And?
Capa: Heads… We harvested all Earth’s resources to make this payload. This is humanity’s last chance… our last, best chance… Searle’s argument is sound. Two last chances are better than one.
Not being able to calculate chances does not excuse one from using their best de-biased neural machinery to make a guess at a range. IMO 50 years is reasonable (I happen to know something about the state of AI research outside of the FAI framework). I would not roll over in surprise if it’s 5 years given state of certain technologies.
(I happen to know something about the state of AI research outside of the FAI framework). I would not roll over in surprise if it’s 5 years given state of certain technologies.
I’m curious, because I like to collect this sort of data: what is your median estimate?
(If you don’t want to say because you don’t want to defend a specific number or list off a thousand disclaimers I completely understand.)
Well it’s clear to me now that formalizing Friendliness with pen and paper is as naively impossible as it would have been for the people of ancient Babylon to actually build a tower that reached the heavens; so if resources are to be spent attempting it, then it’s something that does need to be explicitly argued for.
“By focusing on excessively challenging engineering projects it seems possible that those interested in creating a positive future might actually create future problems – by delaying their projects to the point where less scrupulous rivals beat them to the prize”
[This comment is a response to the original post, but seemed to fit here most.]
I upvoted the OP for raising interesting questions that will arise often and deserve an accessible answer. If someone can maybe put out or point to a reading guide with references.
On the crackpot index the claim that everyone else got it wrong deserves to raise a red flag, but that does not mean it is wrong. There are way to many examples on that in the world. (To quote Eliezer:‘yes, people really are that stupid’)
Read “The Checklist Manifesto” by Atul Gawande for a real life example that is ridiculously simple to understand. (Really read that. It is also entertaining!)
Look at the history of science. Consider the treatment that Semmelweis got for suggesting that doctors wash their hands before operations. You find lots of samples were plain simple ideas where ridiculed. So yes it can happen that a whole profession goes blind on one spot and for every change there has to be someone trying it out in the first place.
The degree on which research is not done well is subject to judgment
.
Now it might be helpful to start out with more applicable ideas, like improving the tool set for real life problems.
You don’t have to care about the singularity to care about other LW content like self-debiasing, or winning.
Regarding the donation aspect, it seems like rationalist are particularly bad at supporting their own causes. You might estimate how much effort you spend in checking out any charity you do support, and then try to not demand higher standards of this one.
Yes, but there are also many examples that show people coming up with the same idea or conclusion at the same time. Take for example A. N. Kolmogorov and Gregory Chaitin who proposed the same definition of randomness independently.
The circumstances regarding Eliezer Yudkowsky are however different. Other people came up with the ideas he is using as supportive fortification and pronunciamento. Some of those people even made similar inferences, yet they do not ask for donations to stop the otherwise inevitable apocalypse.
Your argument does not seem to work. I pointed out how there is stupidity in professionals, but I made no claim that there is only stupidity. So your samples do not disprove the point.
It is nice when people come up with similar things, especially if they happen to be correct, but it is by no means to be expected in every case.
Would you be interested in taking specific pieces apart and/or arguing them?
The argument was that Eliezer Yudkowsky, to my knowledge, has not come up with anything unique. The ideas on which the SIAI is based and asks for donations are not new. Given the basic idea of superhuman AI and widespread awareness of it I thought it was not unreasonable to inquire about the state of activists trying to prevent it.
Are you trying to disprove an argument I made? I asked for an explanation and wasn’t stating some insight about why the SIAI is wrong.
Is Robin Hanson donating most of his income to the SIAI?
You might estimate how much effort you spend in checking out any charity you do support, and then try to not demand higher standards of this one.
While it is silly to selectively apply efficacy standards to charity (giving to inefficient charities without thinking, and then rejecting much more efficient ones on the grounds that they are not maximal [compared to what better choice?]), far better to apply the same high standards across the board than low ones.
If you haven’t read through the MWI sequence, read it. Then try to talk with your smart friends about it. You will soon learn that your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don’t.
I’m curious what evidence you actually have that “You will soon learn that your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong.” As far as I can tell, LWians are on a whole more rational than the general populace, and probably more rational than most smart people. But I’d be very curious as to what evidence you have that leads to conclude that the rationality standards of LW massively exceed those of a random individual’s “smart friends.” Empirically, people on LW have trouble telling when they have sufficient knowledge base about topics and repeat claims that aren’t true that support their pre-existing worldview (I have examples of both of these which I’ll link to if asked). LWians seem to be better than general smart people at updating views when confronted with evidence and somewhat better about not falling into certain common cognitive ruts.
That said, I agree that XiXi should read the MWI sequence and am annoyed that XiXi apparently has not read the sequence before making this posting.
Well, I could try to rephrase as “Below the standards of promoted, highly rated LW posts”, i.e., below the standards of the LW corpus, but what I actually meant there (though indeed I failed to say it) was “the standards I hold myself to when writing posts on LW”, i.e., what XiXiDu is trying to compare to Charles Stross.
Below the standards of promoted, highly rated LW posts
seems to be different than
our smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong
because we do a lot of armchair speculating in comment threads about things on which the “rational” position to take is far from clear—and, furthermore, just because someone isn’t trying to present a rational argument for their position at any given moment doesn’t mean that they can’t.
Pfft, it was an example whose truth value is circumstantial as it was merely an analogy used to convey the gist what I was trying to say, namely to subsequently base conclusions and actions on other conclusions which themselves do not bear evidence. And I won’t read the MWI sequence before learning the required math.
Check my comment here. More details would hint at the banned content.
I never said EY or the SIAI based any conclusions on it. It was, as I frequently said, an example to elucidate what I’m talking about when saying that I cannot fathom the origin of some of the assertions made here as they appear to me to be based on other conclusions that are not yet tested themselves.
What the hell? That link doesn’t contain any conclusions based on MWI—in fact, it doesn’t seem to contain any conclusions at all, just a bunch of questions. If you mean that MWI is based on unfounded conclusions (rather than that other conclusions are based on MWI), then that’s a claim that you really shouldn’t be making if you haven’t read the MWI sequence.
I see no connection whatsoever to the banned content, either in the topic of MWI or in the comment you linked to. This is a bizarre non-sequitur, and as someone who wants to avoid thinking about that topic, I do not appreciate it. (If you do see a connection, explain only by private message, please. But I’d rather you just let it drop.)
My post was intended to be asking questions, not making arguments. Obviously you haven’t read the banned content.
You seem not to understand my primary question that I tried to highlight by the MWI analogy. MWI is a founded conclusion but you shouldn’t use it to make further conclusions based on it. That is, a conclusion first has to yield a new hypothesis that makes predictions. Once you got new data, something that makes a difference, you can go from there and hypothesize that you can influence causally disconnected parts of the multiverse or that it would be a good idea tossing a quantum coin to make key decisions.
After all it was probably a bad decisions to use that example. All you have to do is to substitute MWI with AGI. AGI is, though I’m not sure, a founded conclusion. But taking that conclusion and running with it building a huge framework of further conclusions around it is in my opinion questionable. First this conclusion has to yield marginal evidence of its feasibility, then you are able to create a further hypothesis engaged with further consequences.
I do not appreciate being told that I “obviously” have not read something that I have, in fact, read. And if you were keeping track, I have previously sent you private messages correcting your misconceptions on that topic, so you should have known that. And now that I’ve hinted at why you think it’s connected to MWI, I can see that that’s just another misconception.
Your tone is antagonistic and I had to restrain myself from saying some very hurtful things that I would’ve regretted. You need to take a step back and think about what you’re doing here, before you burn any more social bridges.
EDIT: Argh, restraint fail. That’s what the two deleted comments below this are.
Is it, and that of EY? Are you telling him the same? Check this comment and tell me again that I am antagonistic.
If I come over as such, I’m sorry. I’m a bit stressed writing so many comments accusing me of trying to damage this movement or making false arguments when all I did was indeed trying to inquire about some problems I have, asking questions.
I think part of the reason this went over badly is that in the US, there is a well-known and widely hated talk show host named Glenn Beck whose favorite rhetorical trick is to disguise attacks as questions, saying things like “Is it really true that so-and-so eats babies?”, repeating it enough times that his audience comes to believe that person eats babies, and then defending his accusations by saying “I’m just asking questions”. So some of us, having been exposed to that in the past, see questions and rhetoric mixed a certain way, subconsciously pattern-match against that, and get angry.
If I come over as such, I’m sorry. I’m a bit stressed writing so many comments accusing me of trying to damage this movement or making false arguments when all I did was indeed trying to inquire about some problems I have, asking questions.
I did get the impression that some took your questions as purely rhetorical, soldiers fighting against the credibility of SIAI. I took you as someone hoping to be convinced but with a responsible level of wariness.
I did get the impression that some took your questions as purely rhetorical, soldiers fighting against the credibility of SIAI. I took you as someone hoping to be convinced but with a responsible level of wariness.
That was my impression, also. As a result, I found many elements of the responses to XiXiDu to be disappointing. While there were a few errors in his post (e.g. attributing Kurweil views to SIAI), in general it should have been taken as an opportunity to clarify and throw down some useful links, rather than treat XiXiDu (who is also an SIAI donor!) as a low-g interloper.
Oh, you are the guy who’s spreading all the misinformation about it just so nobody is going to ask more specific questions regarding that topic. Hah, I remember you know. Thanks, but no thanks.
You wrote this and afterwards sending me a private message on how you are telling me this so that I shut up.
Why would I expect honest argumentation from someone who makes use of such tactics? Especially when I talked about the very same topic with you before just to find out that you do this deliberately?
I apologize for my previous comment—I felt provoked, but regardless of the context, it was way out of line.
The thing with the banned topic is, I’m really trying to avoid thinking about it, and seeing it mentioned makes it hard to do that, so I feel annoyed whenever it’s brought up. That’s not something I’m used to dealing with, and it’s a corner case that the usual rules of discourse don’t really cover, so I may not have handled it correctly.
It was my fault all the way to the OP. I was intrigued about the deletion incident and couldn’t shut up and now I thought it was a good idea to inquire about questions that trouble me for so long and to to steer some debate by provoking strong emotions.
I actually understand that you do not want to think about it. It was a dumb idea to steer further debate into that direction. But how could I know before finding out about it? I’m not the personality type who’s going to follow someone telling me not to read about something, to not even think about it.
Rest of the argument… given relativistic issues in QM as described, QM is just approximation which does not work at the relevant scale, and so concluding existence of multiple worlds from it is very silly.
… a proposition far simpler than the argument for supporting SIAI …
Indeed.
If you know all these things and you still can’t tell that MWI is obviously true—a proposition far simpler than the argument for supporting SIAI—then we have here a question that is actually quite different from the one you seem to try to be presenting:
I do not have sufficient g-factor to follow the detailed arguments on Less Wrong. What epistemic state is it rational for me to be in with respect to SIAI?
If you haven’t read through the MWI sequence, read it. Then try to talk with your smart friends about it. You will soon learn that your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don’t.
Ghahahahaha. “A community blog devoted to refining the art of human rationality”… or in other world an online equivalent of a green-ink letter.
Right in the beginning of the sequence you managed to get phases wrong.
Hopefully this mistake will be fixed one day, so the sequence will be judged on the merits of the argument it presents, and not by the presence of a wrong factor of “i”.
given relativistic issues in QM as described, QM is just approximation which does not work at the relevant scale, and so concluding existence of multiple worlds from it is very silly.
Nonrelativistic QM is an approximation to relativistic QFT, and while relativity certainly introduces a new problem for MWI, it remains true that QFT employs the superposition principle just as much as QM. It’s a formalism of “many histories” rather than “many worlds”, but the phenomenon of superposition, and therefore the possibility of parallel coexisting realities, is still there.
I would agree that it was foolish for Eliezer to flaunt his dogmatism about MWI as if that was evidence of superior rationality. What I would say is that he wasn’t worse than physicists in general. Professional physicists who know far more about the subject than Eliezer still manage to say equally foolish things about the implications of quantum mechanics.
What the evidence suggests to me is that to discover the explanation of QM, you need deep technical knowledge, not just of QM but also QFT, and probably of quantum gravity, at least to the level of the holographic principle, and you also need a very powerful imagination. Possibly the correct answer is a variation on a concept we already possess: many worlds, Bohmian mechanics, loops in time, a ’t Hooft cellular automaton. If so, then the big imaginative leap was already carried out, but the technicalities are still hard enough that we don’t even know that it’s the right type of answer. Eliezer-style dogmatism would be wrong for all the available explanations: we do not know which if any is right; at this stage there is no better strategy than pluralistic investigation, including hybridization of these supposedly distinct concepts. But it’s also possible that the correct answer hasn’t yet been conceived, even in outline, which is why imagination remains important, as well as technical knowledge.
If you accept this analysis, then it’s easier to understand why interpretations of quantum mechanics present such a chaotic scene. The radical ontological differences between the candidate explanations create a lot of conceptual tension, and the essential role of subtle technicalities, and mathematical facts not yet known, in pointing the way to the right answer, mean that this conceptual tension can’t be resolved by a simple adjudication like “non-collapse is simpler than collapse”. The possibility that the answer is something we haven’t even imagined yet, makes life even more difficult for people who can’t bear to settle for Copenhagen positivism—should they just insist “there must be an answer, even if we don’t know anything about how it works”?
It’s therefore difficult to avoid both dogmatic rationalization and passive agnosticism. It’s the sort of problem in which the difficulties are such that a return to basics—a review of “what I actually know, rather than what I habitually assume or say”—can take you all the way back to the phenomenological level—“under these circumstances, this is observed to occur”.
For people who don’t want to devote their lives to solving the problem, but who at least want to have a “rational” perspective on it, what I recommend is that you understand the phenomenological Copenhagen interpretation—not the one which says wavefunctions are real and they collapse when observed, just the one which says that wavefunctions are like probability distributions and describe the statistics of observable quantities—and that you also develop some idea of what’s involved in all the major known candidate ontologies.
For readers of this site who believe that questions like this should be resolved by a quantified Occam’s razor like Solomonoff induction: in principle, your first challenge is just to make the different theories commensurable—to find a common language precise enough that you can compare their complexity. In practice, that is a difficult enough task (on account of all these ideas being a little bit underspecified) that it couldn’t be done without a level of technical engagement which meant you had joined the ranks of “people trying to solve the problem, rather than just pontificating about it”.
Hopefully this mistake will be fixed one day, so the sequence will be judged on the merits of the argument it presents, and not by the presence of a wrong factor of “i”.
The argument is pure incompetent self important rambling about nothing. The mistakes only make this easier to demonstrate to people who do not know QM, who assume it must have some merit because someone wasted time writing it up. Removal of mistakes would constitute deception.
Nonrelativistic QM is an approximation to relativistic QFT, and while relativity certainly introduces a new problem for MWI, it remains true that QFT employs the superposition principle just as much as QM.
Nonetheless, there is no satisfactory quantum gravity. It is still only an approximation to reality, and subsequently the mathematical artifacts it has (multiple realities) mean nothing. Even if it was exact it is questionable what is the meaning of such artifacts.
Professional physicists who know far more about the subject than Eliezer still manage to say equally foolish things about the implications of quantum mechanics.
They did not have the stupidity of not even learning it before trying to say something smart about it.
The muckiness surrounding the interferometer is well-known; in fact, the PSE question was written by a LWer.
Rest of the argument… given relativistic issues in QM as described, QM is just approximation which does not work at the relevant scale, and so concluding existence of multiple worlds from it is very silly.
The conclusion isn’t “MWI is true.” The conclusion is “MWI is a simpler explanation than collapse (or straw-Copenhagen, as we in the Contrarian Conspiracy like to call it) for quantum phenomena, and therefore a priori more likely to be true.”
And yes, it is also well-known that this quote is not Yudkowsky at his most charming. Try not to conflate him with either rationalism or the community (which are also distinct things!).
I have not read the MWI sequence yet, but if the argument is that MWI is simpler than collapse, isn’t Bohm even simpler than MWI?
(The best argument against Bohm I can find on LW is a brief comment that claims it implies MWI, but I don’t understand how and there doesn’t seem to be much else on the Web making that case.)
Good point. I’d say that it doesn’t have any calculation of the probability. But some people hope that the probabilities can be derived from just MW. If they achieve this then it would be the simplest theory. But if they need extra hypotheses then it will gain complexity, and may well come out worse than Bohm.
Mitchell_Porter makes the case, but reading him makes my brain shut down for lack of coherence. I assume Yudkowsky doesn’t favor Bohm because it requires non-local hidden variables. Non-local theories are unexpected in physics, and local hidden variables don’t exist.
There’s more to Bohmian mechanics than you may think. There are actually observables whose expectation values correspond to the Bohmian trajectories—“weak-valued” position measurements. This is a mathematical fact that ought to mean something, but I don’t know what. Also, you can eliminate the pilot wave from Bohmian mechanics. If you start with a particular choice of universal wavefunction, that will be equivalent to adding a particular nonlocal potential to a classical equation of motion. That nonlocal potential might be the product of a holographic transformation away from the true fundamental degrees of freedom, or it might approximate the nonlocal correlations induced by planck-scale time loops in the spacetime manifold.
I have never found the time or the energy to do my own quantum sequence, so perhaps it’s my fault if I’m hard to understand. The impression of incoherence may also arise from the fact that I put out lots and lots of ideas. There are a lot of possibilities. But if you want an overall opinion on QM which you wish to be able to attribute to me, here it is:
The explanation of QM might be “Bohm”, “Everett”, “Cramer”, “’t Hooft”, or “None of the Above”. By “Bohm”, I don’t just mean Bohmian mechanics, I mean lines of investigation arising from Bohmian mechanics, like the ones I just described. The other names in quotes should be interpreted similarly.
Also, we are not in a position to say that one of these five approaches is clearly favored over the others. The first four are all lines of investigation with fundamental questions unanswered and fundamental issues unresolved, and yet they are the best specific proposals that we have (unless I missed one). It’s reasonable for a person to prefer one type of model, but in the current state of knowledge any such preference is necessarily superficial, and very liable to be changed by new information.
I have never found the time or the energy to do my own quantum sequence, so perhaps it’s my fault if I’m hard to understand. The impression of incoherence may also arise from the fact that I put out lots and lots of ideas.
Well, that’s understandable. Not everyone has all the free time in the world to write sequences.
It’s reasonable for a person to prefer one type of model, but in the current state of knowledge any such preference is necessarily superficial, and very liable to be changed by new information.
That’s exactly what I wish Yudkowsky’s argument in the QM sequence would have been, but for some reason he felt the need to forever crush the hopes and dreams of the people clinging to alternative interpretations, in a highly insulting manner. What ever happened to leaving a line of retreat?
That’s exactly what I wish Yudkowsky’s argument in the QM sequence would have been, but for some reason he felt the need to forever crush the hopes and dreams of the people clinging to alternative interpretations, in a highly insulting manner. What ever happened to leaving a line of retreat?
Something feels very wrong about this sentence… I get a nagging feeling that you believe he has a valid argument, but he should have been nice to people who are irrationally clinging to alternative interpretations, via such irrational ways as nitpicking on the unimportant details.
Meanwhile, a coherent hypothesis: the guy does not know QM, thinks he knows QM, proceeds to explain whatever simplistic nonsense he thinks is the understanding of QM, getting almost everything wrong. Then interprets the discrepancies in his favour, and feels incredibly intelligent.
Something feels very wrong about this sentence… I get a nagging feeling that you believe he has a valid argument, but he should have been nice to people who are irrationally clinging to alternative interpretations, via such irrational ways as nitpicking on the unimportant details.
I believe he has a valid argument for a substantially weaker claim of the sort I described earlier.
He “should have been nice to people” (without qualification) by not trying to draw (without a shred of credible evidence) a link between rationality/intelligence/g-factor and (even a justified amount of) MWI-skepticism. It’s hard to imagine a worse way to immediately put your audience on the defensive. It’s all there in the manual.
I believe he has a valid argument for a substantially weaker claim of the sort I described earlier.
Why do you think so? Quantum mechanics is complicated, and questions of what is a ‘better’ theory are very subtle.
On the other hand, figuring out what claim your arguments actually support, is rather simple. You have an argument which: gets wrong elementary facts, gets wrong terminology, gets wrong the very claim. All the easy stuff is wrong. You still believe that it gets right some hard stuff. Why?
It’s all there in the manual.
He should have left a line of retreat for himself.
For the reasons outlined above. Occam’s razor + locality.
On the other hand, figuring out what claim your arguments actually support, is rather simple.
My argument is distinct from Yudkowsky’s in that our claims are radically different. If you disagree that MWI is more probable than straw-Copenhagen, I’d like to know why.
You have an argument which: gets wrong elementary facts, gets wrong terminology, gets wrong the very claim. All the easy stuff is wrong. You still believe that it gets right some hard stuff. Why?
None of the “easy stuff” is pertinent to the argument that MWI is more probable than straw-Copenhagen. For example, the interferometer calculation is neither used as evidence that MWI is local, nor that MWI is less complicated. The calculation is independent of any interpretation, after all.
For the reasons outlined above. Occam’s razor + locality.
if I stand a needle on it’s tip on a glass plate, will needle remain standing indefinitely? No it probably won’t even though by Occam’s razor, zero deviation from vertical is (arguably) more probable than any other specific deviation from vertical. MWI seems to require exact linearity, and QM and QFT don’t do gravity, i.e. are approximate. Linear is a first order approximation to nearly anything.
None of the “easy stuff” is pertinent to the argument that MWI is more probable than straw-Copenhagen.
Intelligence and careful thinking --> getting easy stuff right and maybe (very rarely) getting hard stuff right.
Lack of intelligence and/or careful thinking --> getting easy stuff wrong and getting hard stuff certainly wrong.
What is straw Copenhagen anyway? Objective collapse caused by consciousness? Copenhagen is not objective collapse. It is a theory for predicting and modelling the observations. With the MWI you still need to single out one observer, because something happens in real world that does single out one observer, as anyone can readily attest, and so there’s no actual difference here in any math, it’s only a difference in how you look at this math.
edit: ghahahahaha, wait, you literally think it has higher probability? (i seen another of the Yudkowsky’s comments where he said something about his better understanding of probability theory) Well, here’s the bullet: the probability of our reality being quantum mechanics or quantum field theory, within platonic space, is 0 (basically, vanishingly small, predicated on the experiments confirming general relativity all failing), because gravity exists and works so and so but that’s not part of QFT. 0 times anything is still 0. (That doesn’t mean the probability of alternate realities is 0, if there can be such a thing)
From the one comment on Bohm I can find, it seems that he actually dislikes Bohm because the particles are “epiphenomena” to the pilot wave. Meaning the particles don’t actually do anything except follow the pilot wave, and it’s actually the the pilot wave itself that does all the computation (of minds and hence observers).
This assumption is made by every other interpretation of quantum mechanics I know. On the other hand, I’m not a physicist; I’m clearly not up to date on things.
Local HV’s do exist.
I meant the classical HV theories that were ruled out by actual experiments detecting violations of Bell’s inequality.
Well, you didn’t link to his view of qualia, but to a link where he explains why MWI is not the “winner” or “preferred” as EY claimed so confidently in his series on QM.
You might disagree with him on his stance on qualia ( I do too ) but it would be a logical fallacy to state that therefore all his other opinions are also incoherent.
Mitchell Porter’s view on qualia is not non-sense either, it is highly controversial and speculative, no doubt. But his motivation is sound, he think that it is the only way to avoid some sort of dualism, so in that sense his view is even more reductionist than that of Dennett etc.
He is also in good company with people like David Deutsch (another famous many world fundamentlist).
As for local hidden variables, obviously there does not exist a local HV that has been ruled out ;p but you claimed there was none in existence in general.
The muckiness surrounding the interferometer is well-known; in fact, the PSE question was written by a LWer.
Ahh, that would explain why a non-answer is the accepted one. Was this non-answer written by LWer by chance?
Rest of sequence is no better. Photon going in particular way is not really ‘configuration’ with a complex amplitude, I am not even sure the guy actually understands how interferometer works or what happens if length of one path is modified a little. Someone who can’t correctly solve even a simplest QM problem has no business ‘explaining’ anything about QM by retelling popular books.
The conclusion isn’t “MWI is true.”
You clearly do not have enough g-factor:
If you know all these things and you still can’t tell that MWI is obviously true
And yes, it is also well-known that this quote is not Yudkowsky at his most charming.
When people are at their most charming, they are pretending.
Try not to conflate him with either rationalism or the community (which are also distinct things!).
Rationalism? I see. This would explain why the community would take that seriously instead of pointing and laughing.
Are you sure? I’ve seen posts speaking of ‘aspiring rationalists’. It does make sense that rationalists would see themselves as rational, but it does not make sense for rational people to call themselves rationalists. Rationalism is sort of like a belief in power of rationality. It’s to rationality as communism is to community.
Believing that the alternate realities must exist if they are a part of a theory, even if the same theory says that the worlds are unreachable, that’s rationalism. Speaking of which, even a slightest non-linearity is incompatible with many worlds.
There is something that makes me feel confused about MWI. Maybe it is its reliance on anthropic principle (probability of finding myself in a world where recorded history have probability P (according to Born’s rule) must be equal to P). This condition depends on every existing universe, not just on ours. Thus it seems that to justify Born’s rule we should leave observable evidence behind and trail along after unprovable philosophical ideas.
If the map is not the territory how is it that the maths and logic can assign some worlds, or infinite many, but not others with the attribute of being real and instantiated beyond the model?
Can selections for “realness” be justified or explained logically, is it a matter of deduction?
Say, what makes something a real thing versus an abstract matter. When does the map become the territory?
As far as I know the uniformity or different states of the universe are not claimed to be factual beyond the observable because we can deduce that it is logical to think one way or the other?
You say that once I read the relevant sequence I will understand. That might be so, as acknowledged in my post. But given my partial knowledge I’m skeptic that it is sound enough to allow for ideas to be taken serious enough such as that you can influence causally disconnected parts of the multiverse. That it would be a good idea tossing a quantum coin to make key decisions and so on.
It was however just one example to illustrate some further speculations that are based on the interpretation of a incomplete view of the world.
I think the idea of a Mathematical Universe is very appealing. Yet I’m not going to base decisions on this idea, not given the current state of knowledge.
If you claim that sufficient logical consistency can be used as a fundament for further argumentation about the real world, I’ll take note. I have to think about it. People say, “no conclusions can be drawn if you fail to build a contradiction”. They also say you have to make strong, falsifiable predictions. Further, people say that picking a given interpretation of the world has to have practical value to be capable of being differentiated from that which isn’t useful.
It seems that you mainly study and observe nature with emphasis on an exclusively abstract approach rather than the empirical. As I said, I do not claim there’s anything wrong with it. But so far I have my doubts.
MWI may be a logical correct and reasonable deduction. But does it provide guidance or increase confidence? Is it justified to be taken for granted, to be perceived as part of the territory simply because it makes sense? It is not a necessity.
Your skepticism is aimed in the wrong direction and MWI does not say what you think it does. Read the sequence. When you’re done you’ll have a much better gut sense of the gap between SIAI and Charles Stross.
Did you actually read through the MWI sequence before deciding that you still can’t tell whether MWI is true because of (as I understand your post correctly) the state of the social evidence? If so, do you know what pluralistic ignorance is, and Asch’s conformity experiment?
If you know all these things and you still can’t tell that MWI is obviously true—a proposition far simpler than the argument for supporting SIAI—then we have here a question that is actually quite different from the one you seem to try to be presenting:
I do not have sufficient g-factor to follow the detailed arguments on Less Wrong. What epistemic state is it rational for me to be in with respect to SIAI?
If you haven’t read through the MWI sequence, read it. Then try to talk with your smart friends about it. You will soon learn that your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong, and you will no longer think it anywhere near as plausible that their differing opinion is because they know some incredible secret knowledge you don’t.
I respectfully disagree. I am someone who was convinced by your MWI explanations but even so I am not comfortable with outright associating reserved judgement with lack of g.
This is a subject that relies on an awful lot of crystalized knowledge about physics. For someone to come to a blog knowing only what they can recall of high school physics and be persuaded to accept a contrarian position on what is colloquially considered the most difficult part of science is a huge step.
The trickiest part is correctly accounting for meta-uncertainty. There are a lot of things that seem extremely obvious but turn out to be wrong. I would even suggest that the trustworthiness of someone’s own thoughts is not always proportionate to g-factor. That leaves people with some situations where they need to trust social processes more than their own g. That may prompt them to go and explore the topic from various other sources until such time that they can trust that their confidence is not just naivety.
On a subject like physics and MWI, I wouldn’t take the explanation of any non-professional as enough to establish that a contrarian position is “obviously correct”. Even if they genuinely believed in what they said, they’ll still only be presenting the evidence from their own point of view. Or they might be missing something essential and I wouldn’t have the expertise to realize that. Heck, I wouldn’t even go on the word of a full-time researcher in the field before I’d heard what their opponents had to say.
On a subject matter like cryonics I was relatively convinced from simply hearing what the cryonics advocates had to say, because it meshed with my understanding of human anatomy and biology, and it seemed like nobody was very actively arguing the opposite. But to the best of my knowledge, people are arguing against MWI, and I simply wouldn’t have enough domain knowledge to evaluate either sort of claim. You could argue your case of “this is obviously true” with completely made-up claims, and I’d have no way to tell.
This is probably the best comment so far:
Rounds it up pretty well. Thank you.
I’ve said that before, but apparently not quite so well.
But whose domain knowledge are we talking about in the first place? Eliezer argues that MWI is a question of probability theory rather than physics per se. In general, I don’t see much evidence that physicists who argue against MWI actually have the kind of understanding of probability theory necessary to make their arguments worth anything. (Though of course it’s worth emphasizing here that “MWI” in this context means only “the traditional collapse postulate can’t be right” and not “the Schroedinger equation is a complete theory of physics”.)
Physicists have something else, however, and that is domain expertise. As far as I am concerned, MWI is completely at odds with the spirit of relativity. There is no model of the world-splitting process that is relativistically invariant. Either you reexpress MWI in a form where there is no splitting, just self-contained histories each of which is internally relativistic, or you have locally propagating splitting at every point of spacetime in every branch, in which case you don’t have “worlds” any more, you just have infinitely many copies of infinitely many infinitesimal patches of space-time which are glued together in some complicated way. You can’t even talk about extended objects in this picture, because the ends are spacelike separated and there’s no inherent connection between the state at one end and the state at the other end. It’s a complete muddle, even before we try to recover the Born probabilities.
Rather than seeing MWI as the simple and elegant way to understand QM, I see it as an idea which in a way turns out to be too simple—which is another way of saying, naive or uninformed. Like Bohmian mechanics, conceptually it relies on a preferred frame.
The combination of quantum mechanics with special relativity yields quantum field theory. In quantum field theory, everything empirically meaningful is conceptually relativistic. In your calculations, you may employ entities (like wavefunctions evolving in time) which are dependent on a particular reference frame, but you can always do such calculations in a different frame. An example of a calculational output which is frame-independent would be the correlation function between two field operators at different points in space-time. By the time we reach the point of making predictions, that correlation function should only depend on the (relativistically invariant) space-time separation. But in order to calculate it, we may adopt a particular division into space and time, write down wavefunctions defined to exist on the constant-time hypersurfaces in that reference frame, and evolve them according to a Hamiltonian. These wavefunctions are only defined with respect to a particular reference frame and a particular set of hypersurfaces. Therefore, they are somehow an artefact of a particular coordinate system. But they are the sorts of objects in terms of which MWI is constructed.
The truly relativistic approach to QFT is the path integral, the sum over all field histories interpolating between conditions on an initial and a final hypersurface. These histories are objects which are defined independently of any particular coordinate system, because they are histories and not just instantaneous spacelike states. But then we no longer have an evolving superposition, we just have a “superposition” of histories which do not “split” or “join”.
At any time, theoretical physics contains many ideas and research programs, and there are always a lot of them that are going nowhere. MWI has all the signs of an idea going nowhere. It doesn’t advance the field in any way. Instead, as with Bohmian mechanics, what happens is that specific quantum theories are proposed (field theories, string theories), and then the Everettians, the Bohmians, and so on wheel out their interpretive apparatus, which they then “apply” to the latest theoretical advance. It’s a parasitic relationship and it’s a sign that in the long run this is a dead end.
I will provide an example of an idea which is more like what I would look for in an explanation of quantum theory. The real problem with quantum theory is the peculiar way its probabilities are obtained. You have complex numbers and negative quasiprobabilities and histories that cancel each other. The cancellation of possibilities makes no sense from the perspective of orthodox probability. If an outcome can come about in one way, the existence of a second way can only increase the probability of the outcome—according to probability theory and common sense. Yet in the double-slit experiment we have outcomes that are reduced in probability through “destructive interference”. That is what we need to explain.
There is a long history of speculation that maybe the peculiar structure of quantum probabilities can be obtained by somehow conditioning on the future as well as on the past, or by having causality working backwards as well as forwards in time. No-one has ever managed to derive QM this way, but many people have talked about it.
In string theory, there are light degrees of freedom, and heavy degrees of freedom. The latter correspond to the higher (more energetic) excitations of the string, though we should not expect that strings are fundamental in the full theory. In any case, these heavy excitations should cause space to be very strongly curved. So, what if the heavy degrees of freedom create a non-time-orientable topology on the Planck scale, giving rise to temporally bidirectional constraints on causality, and then the light strings interact (lightly) with that background, and quantum-probability effects are the indirect manifestation of that deeper causal structure, which has nonlocal correlations in space and time?
That’s an idea I had during my string studies. It is not likely to be right, because it’s just an idea. But it is an explanation which is intrinsically connected to the developing edge of theoretical physics, rather than a prefabricated explanation which is then applied in a one-size-fits-all fashion to any quantum theory. It would be an intrinsically string-theoretic derivation of QM. That is the sort of explanation for QM that I find plausible, for the reason that everything deep in physics is deeply connected to every other deep thing.
Huh? This is what I’ve always¹ taken MWI in a relativistic context...
Just kidding. More like, since the first time I thought about the issue after graduating (and hence having an understanding of SR and QM devoid of the misconceptions found in certain popularizations).
Anyway, I’ll have to read the works by ’t Hooft when I have time. They look quite interesting.
In 1204.4926 the idea is that a quantum oscillator is actually a discrete deterministic system that cycles through a finite number of states. Then in 1205.4107 he maps a cellular automaton onto a free field theory made out of coupled quantum oscillators. Then in 1207.3612 he adds boolean variables to his CA (previously the cells were integer-valued) in order to add fermionic fields. At this point his CA is looking a little like a superstring, which from a “worldsheet” perspective is a line with bosonic and fermionic quantum fields on it. But there are still many issues whose resolution needs to be worked out.
I wasn’t convinced of MWI by the quantum mechanics sequence when I read it. I came to the conclusion that it’s probably true later, after thinking intensively about the anthropic trilemma (my preferred resolution is incompatible with single-world interpretations); but my probability estimate is still only at 0.8.
Ooh, tell us more!
I’ll write more about this, but it will take some time. I posted the basic idea on the Anthropic Trilemma thread when I first had it, but the explanation is too pithy to follow easily, and I don’t think many people saw it. Revisiting it now has brought to mind an intuition pump to use.
This is rude (although I realize there is now name-calling and gratuitous insult being mustered on both sides) , and high g-factor does not make those MWI arguments automatically convincing. High g-factor combined with bullet-biting, a lack of what David Lewis called the argument of the incredulous stare, does seem to drive MWI pretty strongly. I happen to think that weighting the incredulous stare as an epistemic factor independent of its connections with evolution, knowledge in society, etc, is pretty mistaken, but bullet-dodgers often don’t. Accusing someone of being low-g rather than a non-bullet-biter is the insulting possibility.
Just recently I encountered someone very high IQ/SAT/GRE scores who bought partial quantitative parsimony/Speed Prior type views, and biases against the unseen. This person claimed that the power of parsimony was not enough to defeat the evidence for galaxies and quarks, but was sufficient to defeat a Big World much beyond our Hubble Bubble, and to favor Bohm’s interpretation over MWI. I think that view isn’t quite consistent without a lot of additional jury-rigging, but it isn’t reliably prevented by high g and exposure to the arguments from theoretical simplicity, non-FTL, etc.
It seems to me that a sufficiently cunning arguer can come up with what appears to be a slam-dunk argument for just about anything. As far as I can tell, I follow the arguments in the MWI sequence perfectly, and the conclusion does pretty much follow from the premises. I just don’t know if those premises are actually true. Is MWI what you get if you take the Schrodinger equation literally? (Never mind that the basic Schrodinger equation is non-relativistic; I know that there are relativistic formulations of QM.) I can’t tell you, because I don’t know the underlying math. And, indeed, the “Copenhagen interpretation” seems like patent nonsense, but what about all the others? I don’t know enough to answer the question, and I’m not going to bother doing much more research because I just don’t really care what the answer is.
Yes. This is agreed on even by even those who don’t subscribe to MWI.
Do you still get MWI if you start with the Dirac equation (which I understand to be the version of the Schrodinger equation that’s consistent with special relativity) instead? Mitchell Porter commented that MWI has issues with special relativity, so I wonder...
I’m not an expert on relativistic QM (anyone who is, correct me if I misspeak), but I know enough to tell that Mitchell Porter is confused by what splitting means. In relativistic QM, the wavefunction evolves in a local manner in configuration space, as opposed to the Schrödinger equation’s instantaneous (but exponentially small) assignment of mass to distant configurations. Decoherence happens in this picture just as it happened in the regular one.
The reason that Mitchell (and others) are confused about the EPR experiment is that, although the two entangled particles are separated in space, the configurations which will decohere are very close to one another in configuration space. Locality is therefore not violated by the decoherence.
The moment you speak about configuration space as real, you have already adopted a preferred frame. That is the problem.
Sorry, but no. The Dirac equation is invariant under Lorentz transformations, so what’s local in one inertial reference frame is local in any other as well.
The Dirac equation is invariant, but there are a lot of problems with the concept of locality. For example, if you want to create localised one-particle states that remain local in any reference frame and form an orthonormal eigenbasis of the (one-particle subspace of) the Hilbert space, you will find it impossible.
The canonical solution in axiomatic QFT is to begin with local operations instead of localised particles. However, to see the problem, one has to question the notion of measurement and define it in a covariant manner, which may be a mess. See e.g.
http://prd.aps.org/abstract/PRD/v66/i2/e023510
I am sympathetic to the approach which is used by quantum gravitists, which uses extended Hamiltonian formalism and the Wheeler—DeWitt equation instead of the Schrödinger one. This approach doesn’t use time as special, however the phase space isn’t isomorphic to the state space on the classical level and similar thing holds on the quantum level for the Hilbert space, which makes the interpretation less obvious.
You’re missing my point. To make sense of the Dirac equation, you have to interpret it as a statement about field operators, so locality means (e.g.) that spacelike-separated operators commute. But that’s just a statement about expectation values of observables. MWI is supposed to be a comprehensive ontological interpretation, i.e. a theory of what is actually there in reality.
You seem to be saying that configurations (field configurations, particle configurations, it makes no difference for this argument) are what is actually there. But a “configuration” is spatially extended. Therefore, it requires a universal time coordinate. Everett worlds are always defined with respect to a particular time-slicing—a particular set of spacelike hypersurfaces. From a relativistic perspective, it looks as arbitrary as any “objective collapse” theory.
Not that accurately model the motions of planets, there aren’t.
It looks to me as though you’ve focused in on one of the weaker points in XiXiDu’s post rather than engaging with the (logically independent) stronger points.
XiXiDu wants to know why he can trust SIAI instead of Charles Stross. Reading the MWI sequence is supposed to tell him that far more effectively than any cute little sentence I could write. The first thing I need to know is whether he read the sequence and something went wrong, or if he didn’t read the sequence.
Well, you’ve picked the weakest of his points to answer, and I put it to you that it was clearly the weakest.
You are right of course that what does or doesn’t show up in Charles Stross’s writing doesn’t constitute evidence in either direction—he’s a professional fiction author, he has to write for entertainment value regardless of what he may or may not know or believe about what’s actually likely or unlikely to happen.
A better example would be e.g. Peter Norvig, whose credentials are vastly more impressive than yours (or, granted, than mine), and who thinks we need to get at least another couple of decades of progress under our belts before there will be any point in resuming attempts to work on AGI. (Even I’m not that pessimistic.)
If you want to argue from authority, the result of that isn’t just tilted against the SIAI, it’s flat out no contest.
If this means “until the theory and practice of machine learning is better developed, if you try to build an AGI using existing tools you will very probably fail” it’s not unusually pessimistic at all. “An investment of $X in developing AI theory will do more to reduce the mean time to AI than $X on AGI projects using existing theory now” isn’t so outlandish either. What was the context/cite?
I don’t have the reference handy, but he wasn’t saying let’s spend 20 years of armchair thought developing AGI theory before we start writing any code (I’m sure he knows better than that), he was saying forget about AGI completely until we’ve got another 20 years of general technological progress under our belts.
Not general technological progress surely, but the theory and tools developed by working on particular machine learning problems and methodologies?
Those would seem likely to be helpful indeed. Better programming tools might also help, as would additional computing power (not so much because computing power is actually a limiting factor today, as because we tend to scale our intuition about available computing power to what we physically deal with on an everyday basis—which for most of us, is a cheap desktop PC—and we tend to flinch away from designs whose projected requirements would exceed such a cheap PC; increasing the baseline makes us less likely to flinch away from good designs).
Here too, it looks like you’re focusing on a weak aspect of his post rather than engaging him. Nobody who’s smart and has read your writing carefully doubts that you’re uncommonly brilliant and that this gives you more credibility than the other singulatarians. But there are more substantive aspects of XiXiDu’s post which you’re not addressing.
Like what? Why he should believe in exponential growth? When by “exponential” he actually means “fast” and no one at SIAI actually advocates for exponentials, those being a strictly Kurzweilian obsession and not even very dangerous by our standards? When he picks MWI, of all things, to accuse us of overconfidence (not “I didn’t understand that” but “I know something you don’t about how to integrate the evidence on MWI, clearly you folks are overconfident”)? When there’s lots of little things scattered through the post like that (“I’m engaging in pluralistic ignorance based on Charles Stross’s nonreaction”) it doesn’t make me want to plunge into engaging the many different little “substantive” parts, get back more replies along the same line, and recapitulate half of Less Wrong in the process. The first thing I need to know is whether XiXiDu did the reading and the reading failed, or did he not do the reading? If he didn’t do the reading, then my answer is simply, “If you haven’t done enough reading to notice that Stross isn’t in our league, then of course you don’t trust SIAI”. That looks to me like the real issue. For substantive arguments, pick a single point and point out where the existing argument fails on it—don’t throw a huge handful of small “huh?”s at me.
Castles in the air. Your claims are based on long chains of reasoning that you do not write down in a formal style. Is the probability of correctness of each link in that chain of reasoning so close to 1, that their product is also close to 1?
I can think of a couple of ways you could respond:
Yes, you are that confident in your reasoning. In that case you could explain why XiXiDu should be similarly confident, or why it’s not of interest to you whether he is similarly confident.
It’s not a chain of reasoning, it’s a web of reasoning, and robust against certain arguments being off. If that’s the case, then we lay readers might benefit if you would make more specific and relevant references to your writings depending on context, instead of encouraging people to read the whole thing before bringing criticisms.
Most of the long arguments are concerned with refuting fallacies and defeating counterarguments, which flawed reasoning will always be able to supply in infinite quantity. The key predictions, when you look at them, generally turn out to be antipredictions, and the long arguments just defeat the flawed priors that concentrate probability into anthropomorphic areas. The positive arguments are simple, only defeating complicated counterarguments is complicated.
“Fast AI” is simply “Most possible artificial minds are unlikely to run at human speed, the slow ones that never speed up will drop out of consideration, and the fast ones are what we’re worried about.”
“UnFriendly AI” is simply “Most possible artificial minds are unFriendly, most intuitive methods you can think of for constructing one run into flaws in your intuitions and fail.”
MWI is simply “Schrodinger’s equation is the simplest fit to the evidence”; there are people who think that you should do something with this equation other than taking it at face value, like arguing that gravity can’t be real and so needs to be interpreted differently, and the long arguments are just there to defeat them.
The only argument I can think of that actually approaches complication is about recursive self-improvement, and even there you can say “we’ve got a complex web of recursive effects and they’re unlikely to turn out exactly exponential with a human-sized exponent”, the long arguments being devoted mainly to defeating the likes of Robin Hanson’s argument for why it should be exponential with an exponent that smoothly couples to the global economy.
One problem I have with your argument here is that you appear to be saying that if XiXiDu doesn’t agree with you, he must be stupid (the stuff about low g etc.). Do you think Robin Hanson is stupid too, since he wasn’t convinced?
If he wasn’t convinced about MWI it would start to become a serious possibility.
I haven’t found the text during a two minute search or so, but I think I remember Robin assigning a substantial probability, say, 30% or so, to the possibility that MWI is false, even if he thinks most likely (i.e. the remaining 70%) that it’s true.
Much as you argued in the post about Einstein’s arrogance, there seems to be a small enough difference between a 30% chance of being false, and a 90% chance of being false, if the latter would imply that Robin was stupid, the former would imply it too.
I suspect that Robin would not actually act-as-if those odds with a gun to his head, and he is being conveniently modest.
Right: in fact he would act as though MWI is certainly false… or at least as though Quantum Immortality is certainly false, which has a good chance of being true given MWI.
No! He will act as if Quantum Immortality is a bad choice, which is true even if QI works exactly as described. ‘True’ isn’t the right kind word to use unless you include a normative conclusion in the description of QI.
Consider the Least Convenient Possible World...
Suppose that being shot with the gun cannot possibly have intermediate results: either the gun fails, or he is killed instantly and painlessly.
Also suppose that given that there are possible worlds where he exists, each copy of him only cares about its anticipated experiences, not about the other copies, and that this is morally the right thing to do… in other words, if he expects to continue to exist, he doesn’t care about other copies that cease to exist. This is certainly the attitude some people would have, and we could suppose (for the LCPW) that it is the correct attitude.
Even so, given these two suppositions, I suspect it would not affect his behavior in the slightest, showing that he would be acting as though QI is certainly false, and therefore as though there is a good chance that MWI is false.
But that is crazy and false, and uses ‘copies’ to in a misleading way. Why would I assume that?
This ‘least convenient possible world’ is one in which Robin’s values are changed according to your prescription but his behaviour is not, ensuring that your conclusion is true. That isn’t the purpose of inconvenient worlds (kind of the opposite...)
Not at all. You are conflating “MWI is false” with a whole different set of propositions. MWI != QS.
Many people in fact have those values and opinions, and nonetheless act in the way I mention (and there is no one who does not so act) so it is quite reasonable to suppose that even if Robin’s values were so changed, his behavior would remain unchanged.
The very reason Robin was brought up (by you I might add) was to serve as an ad absurdum with respect to intellectual disrespect.
In the Convenient World where Robin is, in fact, too stupid to correctly tackle the concept of QS, understand the difference between MWI and QI or form a sophisticated understanding of his moral intuitions with respect to quantum uncertainty this Counterfactual-Stupid-Robin is a completely useless example.
I can imagine two different meanings for “not convinced about MWI”
It refers to someone who is not convinced that MWI is as good as any other model of reality, and better than most.
It refers to someone who is not convinced that MWI describes the structure of reality.
If we are meant to understand the meaning as #1, then it may well indicate that someone is stupid. Though, more charitably, it might more likely indicate that he is ignorant.
If we are meant to understand the meaning as #2, then I think that it indicates someone who is not entrapped by the Mind Projection Fallacy.
What do you mean by belief in MWI? What sort of experiment could settle whether MWI is true or not?
I suspect that a lot of people object to the stuff including copies of humans and other worlds we should care about and hypotheses about consciousness tacitly build on MWI, rather than MWI itself.
From THE EVERETT FAQ:
“Is many-worlds (just) an interpretation?”
http://www.hedweb.com/manworld.htm#interpretation
“What unique predictions does many-worlds make?”
http://www.hedweb.com/manworld.htm#unique
“Could we detect other Everett-worlds?”
http://www.hedweb.com/manworld.htm#detect
I’m (yet) not convinced.
First, the links say that MWI needs a linear quantum theory, and lists therefore the linearity among its predictions. However, linearity is a part of the quantum theory and its mathematical formalism, and nothing specific to MWI. Also, weak non-linearity would be explicable using the language of MWI saying that the different worlds interact a little. I don’t see how testing the superposition principle establishes MWI. A very weak evidence at best.
Second, there is a very confused paragraph about quantum gravity, which, apart from linking to itself, states only that MWI requires gravity to be quantised (without supporting argument) and therefore if gravity is successfully quantised, it forms evidence for MWI. However, nobody doubts that gravity has to be quantised somehow, even hardcore Copenhageners.
The most interesting part is that about the reversible measurement done by an artificial intelligence. As I understand it, it supposes that we construct a machine which could perform measurements in reversed direction of time, for which it has to be immune to quantum decoherence. It sounds interesting, but is also suspicious. I see no way how can we get the information into our brains without decoherence. The argument apparently tries to circumvent this objection by postulating an AI, which is reversible and decoherence-immune, but the AI will still face the same problem when trying to tell us the results. In fact, postulating the need of an AI here seems to be only a tool to make the proposed experiment more obscure and difficult to analyse. We will have a “reversible AI”, therefore miraculously we will detect differences between Copenhagen and MWI.
However, at least there is a link to Deutsch’s article which hopefully explains the experiment in greater detail, so I will read it and edit the comment later.
“Many-worlds is often referred to as a theory, rather than just an interpretation, by those who propose that many-worlds can make testable predictions (such as David Deutsch) or is falsifiable (such as Everett) or by those who propose that all the other, non-MW interpretations, are inconsistent, illogical or unscientific in their handling of measurements”
http://en.wikipedia.org/wiki/Many-worlds_interpretation
None of the tests in that FAQ look to me like they could distinguish MWI from MWI+worldeater. The closest thing to an experimental test I’ve come up with is the following:
Flip a quantum coin. If heads, copy yourself once, advance both copies enough to observe the result, then kill one of the copies. If tails, do nothing.
In a many-worlds interpretation of QM, from the perspective of the experimenter, the coin will be heads with probability 2⁄3, since there are two observers in that case and only one if the coin was tails. In the single-world case, the coin will be heads with probability 1⁄2. So each time you repeat the experiment, you get 0.4 bits of evidence for or against MWI. Unfortunately, this evidence is also non-transferrable; someone else can’t use your observation as evidence the same way you can. And getting enough evidence for a firm conclusion involves a very high chance of subjective death (though it is guaranteed that exactly one copy will be left behind). And various quantum immortality hypotheses screw up the experiment, too.
So it is testable in principle, but the experiment involved more odious than one would imagine possible.
The math works the same in all interpretations, but some experiments are difficult to understand intuitively without the MWI. I usually give people the example of the Elitzur-Vaidman bomb tester where the easy MWI explanation says “we know the bomb works because it exploded in another world”, but other interpretations must resort to clever intellectual gymnastics.
If all interpretations are equivalent with respect to testable outcomes, what makes the belief in any particular interpretation so important? Ease of intuitive understanding is a dangerous criterion to rely on, and a relative thing too. Some people are more ready to accept mental gymnastic than existence of another worlds.
Well, that depends. Have you actually tried to do the mental gymnastics and explain the linked experiment using the Copenhagen interpretation? I suspect that going through with that may influence your final opinion.
cousin_it:
Maybe I’m missing something, but how exactly does this experiment challenge the Copenhagen interpretation more than the standard double-slit stuff? Copenhagen treats “measurement” as a fundamental and irreducible process and measurement devices as special components in each experiment—and in this case it simply says that a dud bomb doesn’t represent a measurement device, whereas a functioning one does, so that they interact with the photon wavefunction differently. The former leaves it unchanged, while the latter collapses it to one arm of the interferometer—eiher its own, in which case it explodes, or the other one, in which case it reveals itself as a measurement device just by the act of collapsing.
As far as I understand, this would be similar to the standard variations on the double-slit experiment where one destroys the interference pattern by placing a particle detector at the exit from one of the holes. One could presumably do a similar experiment with a detector that might be faulty, and conclude that an interference-destroying detector works even if it doesn’t flash when several particles are let through (in cases where they all happen to go through the other hole). Unless I’m misunderstanding something, this would be a close equivalent of the bomb test.
The final conclusion in the bomb test is surely more spectacular, but I don’t see how it produces any extra confusion for Copenhageners compared to the most basic QM experiments.
Frankly, I don’t know what you consider an explanation here. I am quite comfortable with the prediction which the theory gives, and accept that as an explanation. So I never needed mental gymnastics here. The experiment is weird, but it doesn’t seem to me less weird by saying that the information about the bomb’s functionality came from its explosion in the other world.
Fair enough.
This should be revamped into a document introducing the sequences.
Your claims are only anti-predictions relative to science-fiction notions of robots as metal men.
Most possible artificial minds are neither Friendly nor unFriendly (unless you adopt such a stringent definition of mind that artificial minds are not going to exist in my lifetime or yours).
Fast AI (along with most of the other wild claims about what future technology will do, really) falls afoul of the general version of Amdahl’s law. (On which topic, did you ever update your world model when you found out you were mistaken about the role of computers in chip design?)
About MWI, I agree with you completely, though I am more hesitant to berate early quantum physicists for not having found it obvious. For a possible analogy: what do you think of my resolution of the Anthropic Trilemma?
This is quite helpful, and suggests that what I wanted is not a lay-reader summary, but an executive summary.
I brought this up elsewhere in this thread, but the fact that quantum mechanics and gravity are not reconciled suggests that even Schrodinger’s equation does not fit the evidence. The “low-energy” disclaimer one has to add is very weird, maybe weirder than any counterintuitive consequences of quantum mechanics.
It’s not the Schrödinger equation alone that gives rise to decoherence and thus many-worlds. (Read Good and Real for another toy model, the “quantish” system.) The EPR experiment and Bell’s inequality can be made to work on macroscopic scales, so we know that whatever mathematical object the universe will turn out to be, it’s not going to go un-quantum on us again: it has the same relevant behavior as the Schrödinger equation, and accordingly MWI will be the best interpretation there as well.
Speaking of executive summaries, will you offer one for your metaethics?
“There is no intangible stuff of goodness that you can divorce from life and love and happiness in order to ask why things like that are good. They are simply what you are talking about in the first place when you talk about goodness.”
And then the long arguments are about why your brain makes you think anything different.
This is less startling than your more scientific pronouncements. Are there any atheists reading this that find this (or at first found this) very counterintuitive or objectionable?
I would go further, and had the impression from somewhere that you did not go that far. Is that accurate?
I’m a cognitivist. Sentences about goodness have truth values after you translate them into being about life and happiness etc. As a general strategy, I make the queerness go away, rather than taking the queerness as a property of a thing and using it to deduce that thing does not exist; it’s a confusion to resolve, not an existence to argue over.
To be clear, if sentence X about goodness is translated into sentence Y about life and happiness etc., does sentence Y contain the word “good”?
Edit: What’s left of religion after you make the queerness go away? Why does there seem to be more left of morality?
No, nothing, and because while religion does contain some confusion, after you eliminate the confusion you are left with claims that are coherent but false.
I can do that:
Morality is a specific set of values (Or, more precisely, a specific algorithm/dynamic for judging values). Humans happen to be (for various reasons) the sort of beings that value morality as opposed to valuing, say, maximizing paperclip production. It is indeed objectively better (by which we really mean “more moral”/”the sort of thing we should do”) to be moral than to be paperclipish. And indeed we should be moral, where by “should” we mean, “more moral”.
(And moral, when we actually cash out what we actually mean by it seems to translate to a complicated blob of values like happiness, love, creativity, novelty, self determination, fairness, life (as in protecting theirof), etc...)
It may appear that paperclip beings and moral beings disagree about something, but not really. The paperclippers would, once they’ve analyzed what humans actually mean by “moral”, would agree “yep, humans are more moral than us. But who cares about this morality stuff, it doesn’t maximize paperclips!”
Of course, screw the desires of the paperclippers, after all, they’re not actually moral. We really are objectively better (once we think carefully by what we mean by “better”) than them.
(note, “does something or does something not actually do a good job of fulfilling a certain value?” is an objective question. ie, “does a particular action tend to increase the expected number of paperclips?” (on the paperclipper side) or, on our side, stuff like “does a particular action tend to save more lives, increase happiness, increase fairness, add novelty...” etc etc etc is an objective question in that we can extract specific meaning from that question and can objectively (in a way the paperclippers would agree with) judge that. It simply happens to be that we’re the sorts of beings that actually care about the answer to that (as we should be), while the screwy hypothetical paperclippers are immoral and only care about paperclips.
How’s that, that make sense? Or, to summarize the summary, “Morality is objective, and we humans happen to be the sorts of beings that value morality, as opposed to valuing something else instead”
Is morality actually:
a specific algorithm/dynamic for judging values, or
a complicated blob of values like happiness, love, creativity, novelty, self determination, fairness, life (as in protecting theirof), etc.?
If it’s 1, can we say something interesting and non-trivial about the algorithm, besides the fact that it’s an algorithm? In other words, everything can be viewed as an algorithm, but what’s the point of viewing morality as an algorithm?
If it’s 2, why do we think that two people on opposite sides of the Earth are referring to the same complicated blob of values when they say “morality”? I know the argument about the psychological unity of humankind (not enough time for significant genetic divergence), but what about cultural/memetic evolution?
I’m guessing the answer to my first question is something like, morality is an algorithm whose current “state” is a complicated blob of values like happiness, love, … so both of my other questions ought to apply.
Wei_Dai:
You don’t even have to do any cross-cultural comparisons to make such an argument. Considering the insights from modern behavioral genetics, individual differences within any single culture will suffice.
There is no reason to be at all tentative about this. There’s tons of cog sci data about what people mean when they talk about morality. It varies hugely (but predictably) across cultures.
Why are you using algorithm/dynamic here instead of function or partial function? (On what space, I will ignore that issue, just as you have...) Is it supposed to be stateful? I’m not even clear what that would mean. Or is function what you mean by #2? I’m not even really clear on how these differ.
You might have gotten confused because I quoted Psy-Kosh’s phrase “specific algorithm/dynamic for judging values” whereas Eliezer’s original idea I think was more like an algorithm for changing one’s values in response to moral arguments. Here are Eliezer’s own words:
Others have pointed out that this definition is actually quite unlikely to be coherent: people would be likely to be ultimately persuaded by different moral arguments and justifications if they had different experiences and heard arguments in different orders etc.
Yes, see here for an argument to that effect by Marcello and subsequent discussion about it between Eliezer and myself.
I think the metaethics sequence is probably the weakest of Eliezer’s sequences on LW. I wonder if he agrees with that, and if so, what he plans to do about this subject for his rationality book.
This is somewhat of a concern given Eliezer’s interest in Friendliness!
As far as I can understand, Eliezer has promoted two separate ideas about ethics: defining personal morality as a computation in the person’s brain rather than something mysterious and external, and extrapolating that computation into smarter creatures. The former idea is self-evident, but the latter (and, by extension, CEV) has received a number of very serious blows recently. IMO it’s time to go back to the drawing board. We must find some attack on the problem of preference, latch onto some small corner, that will allow us to make precise statements. Then build from there.
But I don’t see how that, by itself, is a significant advance. Suppose I tell you, “mathematics is a computation in a person’s brain rather than something mysterious and external”, or “philosophy is a computation in a person’s brain rather than something mysterious and external”, or “decision making is a computation in a person’s brain rather than something mysterious and external” how much have I actually told you about the nature of math, or philosophy, or decision making?
The linked discussion is very nice.
This is currently at +1. Is that from Yudkowsky?
(Edit: +2 after I vote it up.)
This makes sense in that it is coherent, but it is not obvious to me what arguments would be marshaled in its favor. (Yudkowsky’s short formulations do point in the direction of their justifications.) Moreover, the very first line, “morality is a specific set of values,” and even its parenthetical expansion (algorithm for judging values), seems utterly preposterous to me. The controversies between human beings about which specific sets of values are moral, at every scale large and small, are legendary beyond cliche.
It is a common thesis here that most humans would ultimately have the same moral judgments if they were in full agreement about all factual questions and were better at reasoning. In other words, human brains have a common moral architecture, and disagreements are at the level of instrumental, rather than terminal, values and result from mistaken factual beliefs and reasoning errors.
You may or may not find that convincing (you’ll get to the arguments regarding that if you’re reading the sequences), but assuming that is true, then “morality is a specific set of values” is correct, though vague: more precisely, it is a very complicated set of terminal values, which, in this world, happens to be embedded solely in a species of minds who are not naturally very good at rationality, leading to massive disagreement about instrumental values (though most people do not notice that it’s about instrumental values).
It is? That’s a worry. Consider this a +1 for “That thesis is totally false and only serves signalling purposes!”
I… think it is. Maybe I’ve gotten something terribly wrong, but I got the impression that this is one of the points of the complexity of value and metaethics sequences, and I seem to recall that it’s the basis for expecting humanity’s extrapolated volition to actually cohere.
This whole area isn’t covered all that well (as Wei noted). I assumed that CEV would rely on solving an implicit cooperation problem between conflicting moral systems. It doesn’t appear at all unlikely to me that some people are intrinsically selfish to some degree and their extrapolated volitions would be quite different.
Note that I’m not denying that some people present (or usually just assume) the thesis you present. I’m just glad that there are usually others who argue against it!
That’s exactly what I took CEV to entail.
Now this is a startling claim.
Be more specific!
Maybe it’s true if you also specify “if they were fully capable of modifying their own moral intuitions.” I have an intuition (an unexamined belief? a hope? a sci-fi trope?) that humanity as a whole will continue to evolve morally and roughly converge on a morality that resembles current first-world liberal values more than, say, Old Testament values. That is, it would converge, in the limit of global prosperity and peace and dialogue, and assuming no singularity occurs and the average lifespan stays constant. You can call this naive if you want to; I don’t know whether it’s true. It’s what I imagine Eliezer means when he talks about “humanity growing up together”.
This growing-up process currently involves raising children, which can be viewed as a crude way of rewriting your personality from scratch, and excising vestiges of values you no longer endorse. It’s been an integral part of every culture’s moral evolution, and something like it needs to be part of CEV if it’s going to actually converge.
That’s not plausible. That would be some sort of objective morality, and there is no such thing. Humans have brains, and brains are complicated. You can’t have them imply exactly the same preference.
Now, the non-crazy version of what you suggest is that preferences of most people are roughly similar, that they won’t differ substantially in major aspects. But when you focus on detail, everyone is bound to want their own thing.
Psy-Kosh:
It makes sense in its own terms, but it leaves the unpleasant implication that morality differs greatly between humans, at both individual and group level—and if this leads to a conflict, asking who is right is meaningless (except insofar as everyone can reach an answer that’s valid only for himself, in terms of his own morality).
So if I live in the same society with people whose morality differs from mine, and the good-fences-make-good-neighbors solution is not an option, as it often isn’t, then who gets to decide whose morality gets imposed on the other side? As far as I see, the position espoused in the above comment leaves no other answer than “might is right.” (Where “might” also includes more subtle ways of exercising power than sheer physical coercion, of course.)
That two people mean different things by the same word doesn’t make all questions asked using that word meaningless, or even hard to answer.
If by “castle” you mean “a fortified structure”, while I mean “a fortified structure surrounded by a moat”, who will be right if we’re asked if the Chateau de Gisors is a castle? Any confusion here is purely semantic in nature. If you answer yes and I answer no, we won’t have given two answers to the same question, we’ll have given two answers to two different questions. If Psy-Kosh says that the Chateau de Gisors is a fortified structure but it is not surrounded by a moat, he’ll have answered both our questions.
Now, once this has been clarified, what would it mean to ask who gets to decide whose definition of ‘castle’ gets imposed on the other side? Do we need a kind of meta-definition of castle to somehow figure out what the one true definition is? If I could settle this issue by exercising power over you, would it change the fact that the Chateau de Gisors is not surrounded by a moat? If I killed everyone who doesn’t mean the same thing by the word ‘castle’ than I do, would the sentence “a fortified structure” become logically equivalent to the sentence “a fortified structure surrounded by a moat”?
In short, substituting the meaning of a word for the word tends to make lots of seemingly difficult problems become laughably easy to solve. Try it.
*blinks* how did I imply that morality varies? I thought (was trying to imply) that morality is an absolute standard and that humans simply happen to be the sort of beings that care about the particular standard we call “morality”. (Well, with various caveats like not being sufficiently reflective to be able to fully explicitly state our “morality algorithm”, nor do we fully know all its consequences)
However, when humans and paperclippers interact, well, there will probably be some sort of fight if one doesn’t end up with some sort PD cooperation or whatever. It’s not that paperclippers and humans disagree on anything, it’s simply, well, they value paperclips a whole lot more than lives. We’re sort of stuck with having to act in a way to prevent the hypothetical them from acting on that.
(of course, the notion that most humans seem to have the same underlying core “morality algorithm”, just disagreeing on the implications or such, is something to discuss, but that gets us out of executive summary territory, no?)
Psy-Kosh:
I would say that it’s a crucial assumption, which should be emphasized clearly even in the briefest summary of this viewpoint. It is certainly not obvious, to say the least. (And, for full disclosure, I don’t believe that it’s a sufficiently close approximation of reality to avoid the problem I emphasized above.)
Hrm, fair enough. I thought I’d effectively implied it, but apparently not sufficiently.
(Incidentally… you don’t think it’s a close approximation to reality? Most humans seem to value (to various extents) happiness, love, (at least some) lives, etc… right?)
Different people (and cultures) seem to put very different weights on these things.
Here’s an example:
You’re a government minister who has to decide who to hire to do a specific task. There are two applicants. One is your brother, who is marginally competent at the task. The other is a stranger with better qualifications who will probably be much better at the task.
The answer is “obvious.”
In some places, “obviously” you hire your brother. What kind of heartless bastard won’t help out his own brother by giving him a job?
In others, “obviously” you should hire the stranger. What kind of corrupt scoundrel abuses his position by hiring his good-for-nothing brother instead of the obviously superior candidate?
Okay, I can see how XiXiDu’s post might come across that way. I think I can clarify what I think that XiXiDu is trying to get at by asking some better questions of my own.
What evidence has SIAI presented that the Singularity is near?
If the Singularity is near then why has the scientific community missed this fact?
What evidence has SIAI presented for the existence of grey goo technology?
If grey goo technology is feasible then why has the scientific community missed this fact?
Assuming that the Singularity is near, what evidence is there that SIAI has a chance to lower global catastrophic risk in a nontrivial way?
What evidence is there that SIAI has room for more funding?
“Near”? Where’d we say that? What’s “near”? XiXiDu thinks we’re Kurzweil?
What kind of evidence would you want aside from a demonstrated Singularity?
Grey goo? Huh? What’s that got to do with us? Read Nanosystems by Eric Drexler or Freitas on “global ecophagy”. XiXiDu thinks we’re Foresight?
If this business about “evidence” isn’t a demand for particular proof, then what are you looking for besides not-further-confirmed straight-line extrapolations from inductive generalizations supported by evidence?
You’ve claimed that in your blogging heads divlog with Scott Aaronson that you think that it’s pretty obvious that there will be an AGI within the next century. As far as I know you have not offered a detailed description of the reasoning that led you to this conclusion that can be checked by others.
I see this as significant for the reasons given in my comment here.
I don’t know what the situation is with SIAI’s position on grey goo—I’ve heard people say the SIAI staff believe in nanotechnology having capabilities out of line with the beliefs of the scientific community, but they may have been misinformed. So let’s forget about about questions 3 and 4.
Questions 1, 2, 5 and 6 remain.
You’ve shifted the question from “is SIAI on balance worth donating to” to “should I believe everything Eliezer has ever said”.
The point is that grey goo is not relevant to SIAI’s mission (apart from being yet another background existential risk that FAI can dissolve). “Scientific community” doesn’t normally professionally study (far) future technological capabilities.
My whole point about grey goo has been, as stated, that a possible superhuman AI could use it to do really bad things. That is, I do not see how an encapsulated AI, even a superhuman AI, could pose the stated risks without the use of advanced nanotechnology. Is it going to use nukes, like Skynet? Another question related to the SIAI, regarding advanced nanotechnology, is that if without advanced nanotechnology superhuman AI is at all possible.
I’m shocked how you people misintepreted my intentions there.
If a superhuman AI is possible without advanced nanotechnology, a superhuman AI could just invent advanced nanotechnology and implement it.
Grey goo is only a potential danger in its own right because it’s a way dumb machinery can grow in destructive power (you don’t need to assume AI controlling it for it to be dangerous, at least so goes the story). AGI is not dumb, so it can use something more fitting to precise control than grey goo (and correspondingly more destructive and feasible).
The grey goo example was named to exemplify the speed and sophistication of nanotechnology that would have to be around to either allow an AI to be build in the first place or be of considerable danger.
I consider your comment an expression of personal disgust. No way you could possible misinterpret my original point and subsequent explanation to this extent.
As katydee pointed out, if for some strange reason grey goo is what AI would want, AI will invent grey goo. If you used “grey goo” to refer to the rough level of technological development necessary to produce grey goo, then my comments missed that point.
Illusion of transparency. Since the general point about nanotech seems equally wrong to me, I couldn’t distinguish between the error of making it and making a similarly wrong point about the relevance of grey goo in particular. In general, I don’t plot, so take my words literally. If I don’t like something, I just say so, or keep silent.
If it seems equally wrong, why haven’t you pointed me to some further reasoning on the topic regarding the feasibility of AGI without advanced (grey goo level) nanotechnology? Why haven’t you argued about the dangers of AGI which is unable to make use of advanced nanotechnology? I was inquiring about these issues in my original post and not trying to argue against the scenarios in question.
Yes, I’ve seen the comment regarding the possible invention of advanced nanotechnology by AGI. If AGI needs something that isn’t there it will just pull it out of its hat. Well, I have my doubts that even a superhuman AGI can steer the development of advanced nanotechnology so that it can gain control of it. Sure, it might solve the problems associated with it and send the solutions to some researcher. Then it could buy the stocks of the subsequent company involved with the new technology and somehow gain control...well, at this point we are already deep into subsequent reasoning about something shaky that at the same time is used as evidence of the very reasoning involving it.
To the point: if AGI can’t pose a danger, because its hands are tied, that’s wonderful! Then we have more time to work of FAI. FAI is not about superpowerful robots, it’s about technically understanding what we want, and using that understanding to automate the manufacturing of goodness. The power is expected to come from unbounded automatic goal-directed behavior, something that happens without humans in the system to ever stop the process if it goes wrong.
To the point: if AI can’t pose a danger, because its hands are tied, that’s wonderful! Then we have more time to work of FAI.
Overall I’d feel a lot more comfortable if you just said “there’s a huge amount of uncertainty as to when existential risks will strike and which ones will strike, I don’t know whether or not I’m on the right track in focusing on Friendly AI or whether I’m right about when the Singularity will occur, I’m just doing the best that I can.”
This is largely because of the issue that I raise here
I should emphasize that I don’t think that you’d ever knowingly do something that raised existential risk, I think that you’re a kind and noble spirit. But I do think I’m raising a serious issue which you’ve missed.
Edit: See also these comments
I am looking for the evidence in “supported by evidence”. I am further trying to figure how you anticipate your beliefs to pay rent, what you anticipate to see if explosive recursive self-improvement is possible, and how that belief could be surprised by data.
If you just say, “I predict we will likely be wiped out by badly done AI.”, how do you expect to update on evidence? What would constitute such evidence?
I haven’t done the reading. For further explanation read this comment.
Why do you always and exclusively mention Charles Stross? I need to know if you actually read all of my post.
Because the fact that you’re mentioning Charles Stross means that you need to do basic reading, not complicated reading.
To put my own spin on XiXiDu’s questions: What quality or position does Charles Stross possess that should cause us to leave him out of this conversation (other than the quality ‘Eliezer doesn’t think he should be mentioned’)?
Another vacuous statement. I expected more.
What stronger points are you referring to? It seems to me XiXiDu’s post has only 2 points, both of which Eliezer addressed:
“Given my current educational background and knowledge I cannot differentiate LW between a consistent internal logic, i.e. imagination or fiction and something which is sufficiently based on empirical criticism to provide a firm substantiation of the strong arguments for action that are proclaimed on this site.”
His smart friends/favorite SF writers/other AI researchers/other Bayesians don’t support SIAI.
My point is that your evidence has to stand up to whatever estimations you come up with. My point is the missing transparency in your decision making regarding the possibility of danger posed by superhuman AI. My point is that any form of external peer review is missing and that therefore I either have to believe you or learn enough to judge all of your claims myself after reading hundreds of posts and thousands of documents to find some pieces of evidence hidden beneath. My point is that competition is necessary, that not just the SIAI should work on the relevant problems. There are many other points you seem to be missing entirely.
“Is the SIAI evidence-based, or merely following a certain philosophy?”
Oh, is that the substantive point? How the heck was I supposed to know you were singling that out?
That one’s easy: We’re doing complex multi-step extrapolations argued to be from inductive generalizations themselves supported by the evidence, which can’t be expected to come with experimental confirmation of the “Yes, we built an unFriendly AI and it went foom and destroyed the world” sort. This sort of thing is dangerous, but a lot of our predictions are really antipredictions and so the negations of the claims are even more questionable once you examine them.
If you have nothing valuable to say, why don’t you stay away from commenting at all? Otherwise you could simply ask me what I meant to say, if something isn’t clear. But those empty statements coming from you recently make me question if you’ve been the person that I thought you are. You cannot even guess what I am trying to ask here? Oh come on...
I was inquiring about the supportive evidence at the origin of your complex multi-step extrapolations argued to be from inductive generalizations. If there isn’t any, what difference is there between writing fiction and complex multi-step extrapolations argued to be from inductive generalizations?
What you say here makes sense, sorry for not being more clear earlier. See my list of questions in my response to another one of your comments.
How was Eliezer supposed to answer that, given that XiXiDu stated that he didn’t have enough background knowledge to evaluate what’s already on LW?
Agreed, and I think there’s a pattern here. XiXiDu is asking the right questions about why SIAI doesn’t have wider support. It is because there are genuine holes in its reasoning about the singularity, and SIAI chooses not to engage with serious criticism that gets at those holes. Example (one of many): I recall Shane Legg commenting that it’s not practical to formalize friendliness before anyone builds any form of AGI (or something to that effect). I haven’t seen SIAI give a good argument to the contrary.
Gahhh! The hoard of arguments against that idea that instantly sprang to my mind (with warning bells screeching) perhaps hints at why a good argument hasn’t been given to the contrary (if, in fact, it hasn’t). It just seems so obvious. And I don’t mean that as a criticism of you or Shane at all. Most things that we already understand well seem like they should be obvious to others. I agree that there should be a post making the arguments on that topic either here on LessWrong or on the SIAI website somewhere. (Are you sure there isn’t?)
Edit: And you demonstrate here just why Eliezer (or others) should bother to answer XiXiDu’s questions even if there are some weaknesses in his reasoning.
My point is that Shane’s conclusion strikes me as the obvious one, and I believe many smart, rational, informed people would agree. It may be the case that, for the majority of smart, rational, informed people, there exists an issue X for which they think “obviously X” and SIAI thinks “obviously not X.” To be taken seriously, SIAI needs to engage with the X’s.
I understand your point, and agree that your conclusion is one that many smart, rational people with good general knowledge would share. Once again I concur that engaging with those X’s is important, including that ‘X’ we’re discussing here.
Sounds like we mostly agree. However, I don’t think it’s a question of general knowledge. I’m talking about smart, rational people who have studied AI enough to have strongly-held opinions about it. Those are the people who need to be convinced; their opinions propagate to smart, rational people who haven’t personally investigated AI in depth.
I’d love to hear your take on X here. What are your reasons for believing that friendliness can be formalized practically, and an AGI based on that formalization built before any other sort of AGI?
If I was SIAI my reasoning would be the following. First stop with the believes- believes not dichotomy and move to probabilities.
So what is the probability of a good outcome if you can’t formalize friendliness before AGI? Some of them would argue infinitesimal. This is based on fast take-off winner take all type scenarios (I have a problem with this stage, but I would like it to be properly argued and that is hard).
So looking at the decision tree (under these assumptions) the only chance of a good outcome is to try to formalise FAI before AGI becomes well known. All the other options lead to extinction.
So to attack the “formalise Friendliness before AGI” position you would need to argue that the first AGIs are very unlikely to kill us all. That is the major battleground as far as I am concerned.
Agreed about what the “battleground” is, modulo one important nit: not the first AGI, but the first AGI that recursively self-improves at a high speed. (I’m pretty sure that’s what you meant, but it’s important to keep in mind that, e.g., a roughly human-level AGI as such is not what we need to worry about—the point is not that intelligent computers are magically superpowerful, but that it seems dangerously likely that quickly self-improving intelligences, if they arrive, will be non-magically superpowerful.)
I don’t think formalize-don’t formalize should be a simple dichotomy either; friendliness can be formalized in various levels of detail, and the more details are formalized, the fewer unconstrained details there are which could be wrong in a way that kills us all.
I’d look at it the other way: I’d take it as practically certain that any superintelligence built without explicit regard to Friendliness will be unFriendly, and ask what the probability is that through sufficiently slow growth in intelligence and other mere safeguards, we manage to survive building it.
My best hope currently rests on the AGI problem being hard enough that we get uploads first.
(This is essentially the Open Thread about everything Eliezer or SIAI have ever said now, right?)
Uploading would have quite a few benefits, but I get the impression it would make us more vulnerable to whatever tools a hostile AI may possess, not less.
Re: “My best hope currently rests on the AGI problem being hard enough that we get uploads first.”
Surely a miniscule chance. It would be like Boeing booting up a scanned bird.
“So what is the probability of a good outcome if you can’t formalize friendliness before AGI? Some of them would argue infinitesimal.”
One problem here is the use of a circular definition of “friendliness”—that defines the concept it in terms of whether it leads to a favourable outcome. If you think “friendly” is defined in terms of whether or not the machine destroys humanity, then clearly you will think that an “unfriendly” machine would destroy the world. However, this is just a word game—which doesn’t tell us anything about the actual chances of such destruction happening.
Let’s say “we” are the good guys in the race for AI. Define
W = we win the race to create an AI powerful enough to protect humanity from any subsequent AIs
G = our AI can be used to achieve a good outcome
F = we go the “formalize friendliness” route
O = we go a promising route other than formalizing friendliness
At issue is which of the following is higher:
P(G|WF)P(W|F) or P(G|WO)P(W|O)
From what I know of SIAI’s approach to F, I estimate P(W|F) to be many orders of magnitude smaller than P(W|O). I estimate P(G|WO) to be more than 1% for a good choice of O (this is a lower bound; my actual estimate of P(G|WO) is much higher, but you needn’t agree with that to agree with my conclusion). Therefore the right side wins.
There are two points here that one could conceivably dispute, but it sounds like the “SIAI logic” is to dispute my estimate of P(G|WO) and say that P(G|WO) is in fact tiny. I haven’t seen SIAI give a convincing argument for that.
I’d start here to get an overview.
My summary would be: there are huge numbers of types of minds and motivations, so if we pick one at random from the space of minds then it likely to be contrary to our values because it will have a different sense of what is good or worthwhile. This moderately relies on the speed/singleton issue, because evolution pressure between AI might force them in the same direction as us. We would likely be out-competed before this happens though, if we rely on competition between AIs.
I think various people associated with SIAI mean different things by formalizing friendliness. I remember Vladimir Nesov means getting better than 50% probability for providing a good outcome.
Edited to add my own overview.
It doesn’t matter what happens when we sample a mind at random. We only care about the sorts of minds we might build, whether by designing them or evolving them. Either way, they’ll be far from random.
Consider my “at random” short hand for “at random from the space of possible minds built by humans”.
The Eliezer approved example of humans not getting a simple system to do what they want is the classic Machine Learning example where a Neural Net was trained on two different sorts of tanks. It had happened that the photographs of the different types of tanks had been taken at different times of day. So the classifier just worked on that rather than actually looking at the types of tank. So we didn’t build a tank classifier but a day/night classifier. More here.
While I may not agree with Eliezer on everything, I do agree with him it is damn hard to get a computer to do what you want when you stop programming them explicitly .
Obviously AI is hard, and obviously software has bugs.
To counter my argument, you need to make a case that the bugs will be so fundamental and severe, and go undetected for so long, that despite any safeguards we take, they will lead to catastrophic results with probability greater than 99%.
How do you consider “formalizing friendliness” to be different from “building safeguards”?
Things like AI boxing or “emergency stop buttons” would be instances of safeguards. Basically any form of human supervision that can keep the AI in check even if it’s not safe to let it roam free.
Are you really suggesting a trial and error approach where we stick evolved and human created AIs in boxes and then eyeball them to see what they are like? Then pick the nicest looking one, on a hunch, to have control over our light cone?
I’ve never seen the appeal of AI boxing.
This is why we need to create friendliness before AGI → A lot of people who are loosely familiar with the subject think those options will work!
A goal directed intelligence will work around any obstacles in front of it. It’ll make damn sure that it prevents anyone from pressing emergency stop buttons.
Better than chance? What chance?
Sorry, “Better than chance” is an english phrase than tends to mean more than 50%.
It assumes an even chance of each outcome. I.e. do better than selecting randomly.
Not appropriate in this context, my brain didn’t think of the wider implications as it wrote it.
It’s easy to do better than random. *Pours himself a cup of tea.*
Programmers do not operate by “picking programs at random”, though.
The idea that “picking programs at random” has anything to do with the issue seems just confused to me.
The first AI will be determined by the first programmer, sure. But I wasn’t talking about that level; the biases and concern for the ethics of the AI of that programmer will be random from the space of humans. Or at least I can’t see any reason why I should expect people who care about ethics to be more likely to make AI than those that think economics will constrain AI to be nice,
That is now a completely different argument to the original “there are huge numbers of types of minds and motivations, so if we pick one at random from the space of minds”.
Re: “the biases and concern for the ethics of the AI of that programmer will be random from the space of humans”
Those concerned probably have to be an expert programmers, able to build a company or research group, and attract talented assistance, as well as probably customers. They will probably be far-from what you would get if you chose at “random”.
Do we pick a side of a coin “at random” from the two possibilities when we flip it?
Epistemically, yes, we don’t have sufficient information to predict it*. However if we do the same thing twice it has the same outcome so it is not physically random.
So while the process that decides what the first AI is like is not physically random, it is epistemically random until we have a good idea of what AIs produce good outcomes and get humans to follow those theories. For this we need something that looks like a theory of friendliness, to some degree.
Considering we might use evolutionary methods for part of the AI creation process, randomness doesn’t look like too bad a model.
*With a few caveats. I think it is biased to land the same way up as it was when flipped, due to the chance of making it spin and not flip.
Edit: Oh and no open source AI then?
We do have an extensive body of knowledge about how to write computer programs that do useful things. The word “random” seems like a terrible mis-summary of that body of information to me.
As for “evolution” being equated to “randomnness”—isn’t that one of the points that creationists make all the time? Evolution has two motors—variation and selection. The first of these may have some random elements, but it is only one part of the overall process.
I think we have a disconnect on how much we believe proper scary AIs will be like previous computer programs.
My conception of current computer programs is that they are crystallised thoughts plucked from our own minds and easily controllable and unchangeable. When we get interesting AI the programs will morphing and be far less controllable without a good theory of how to control the change.
I shudder every time people say the “AI’s source code” as if it is some unchangeable and informative thing about the AI’s behaviour after the first few days of the AI’s existence.
I’m not sure how to resolve that difference.
You have correctly identified the area in which we do not agree.
The most relevant knowledge needed in this case is knowledge of game theory and human behaviour. They also need to know ‘friendliness is a very hard problem’. They then need to ask themselves the following question:
What is likely to happen if people have the ability to create an AGI but do not have a proven mechanism for implementing friendliness? Is it:
Shelve the AGI, don’t share the research and set to work on creating a framework for friendliness. Don’t rush the research—act as if the groundbreaking AGI work that you just created was a mere toy problem and the only real challenge is the friendliness. Spend an even longer period of time verifying the friendliness design and never let on that you have AGI capabilities.
Something else.
I don’t (with that phrasing). I actually suspect that the problem is too difficult to get right and far too easy to get wrong. We’re probably all going to die. However, I think we’re even more likely to die if some fool goes and invents a AGI before they have a proven theory of friendliness.
Those are the people, indeed. But where do the donations come from? EY seems to be using this argument against me as well. I’m just not educated, well-read or intelligent enough for any criticism. Maybe so, I acknowledged that in my post. But have I seen any pointers to how people arrive at their estimations yet? No, just the demand to read all of LW, which according to EY doesn’t even deal with what I’m trying to figure out, but rather the dissolving of biases. A contradiction?
I’m inquiring about the strong claims made by the SIAI, which includes EY and LW. Why? Because they ask for my money and resources. Because they gather fanatic followers who believe into the possibility of literally going to hell. If you follow the discussion surrounding Roko’s posts you’ll see what I mean. And because I’m simply curious and like to discuss, besides becoming less wrong.
But if EY or someone else is going to tell me that I’m just too dumb and it doesn’t matter what I do, think or donate, I can accept that. I don’t expect Richard Dawkins to enlighten me about evolution either. But don’t expect me to stay quiet about my insignificant personal opinion and epistemic state (as you like to call it) either! Although since I’m conveniently not neurotypical (I guess), you won’t have to worry me turning into an antagonist either, simply because EY is being impolite.
SIAI position does dot require “obviously X” from a decision perspective, the opposite one does. To be so sure of something as complicated as the timeline of FAI math vs AGI development seems seriously foolish to me.
It is not a matter about being sure of it but to weigh it against what is asked for in return, other possible events of equal probability and the utility payoff from spending the resources on something else entirely.
I’m not asking the SIAI to prove “obviously X” but rather to prove the very probability of X that they are claiming it has within the larger context of possibilities.
No such proof is possible with our machinery.
=======================================================
Capa: It’s the problem right there. Between the boosters and the gravity of the sun the velocity of the payload will get so great that space and time will become smeared together and everything will distort. Everything will be unquantifiable.
Kaneda: You have to come down on one side or the other. I need a decision.
Capa: It’s not a decision, it’s a guess. It’s like flipping a coin and asking me to decide whether it will be heads or tails.
Kaneda: And?
Capa: Heads… We harvested all Earth’s resources to make this payload. This is humanity’s last chance… our last, best chance… Searle’s argument is sound. Two last chances are better than one.
=====================================================
(Sunshine 2007)
Not being able to calculate chances does not excuse one from using their best de-biased neural machinery to make a guess at a range. IMO 50 years is reasonable (I happen to know something about the state of AI research outside of the FAI framework). I would not roll over in surprise if it’s 5 years given state of certain technologies.
I’m curious, because I like to collect this sort of data: what is your median estimate?
(If you don’t want to say because you don’t want to defend a specific number or list off a thousand disclaimers I completely understand.)
Median 15-20 years. I’m not really an expert, but certain technologies are coming really close to modeling cognition as I understand it.
Thanks!
Well it’s clear to me now that formalizing Friendliness with pen and paper is as naively impossible as it would have been for the people of ancient Babylon to actually build a tower that reached the heavens; so if resources are to be spent attempting it, then it’s something that does need to be explicitly argued for.
“By focusing on excessively challenging engineering projects it seems possible that those interested in creating a positive future might actually create future problems – by delaying their projects to the point where less scrupulous rivals beat them to the prize”
http://www.acceleratingfuture.com/michael/blog/2009/12/a-short-introduction-to-coherent-extrapolated-volition-cev/
[This comment is a response to the original post, but seemed to fit here most.] I upvoted the OP for raising interesting questions that will arise often and deserve an accessible answer. If someone can maybe put out or point to a reading guide with references.
On the crackpot index the claim that everyone else got it wrong deserves to raise a red flag, but that does not mean it is wrong. There are way to many examples on that in the world. (To quote Eliezer:‘yes, people really are that stupid’) Read “The Checklist Manifesto” by Atul Gawande for a real life example that is ridiculously simple to understand. (Really read that. It is also entertaining!) Look at the history of science. Consider the treatment that Semmelweis got for suggesting that doctors wash their hands before operations. You find lots of samples were plain simple ideas where ridiculed. So yes it can happen that a whole profession goes blind on one spot and for every change there has to be someone trying it out in the first place. The degree on which research is not done well is subject to judgment . Now it might be helpful to start out with more applicable ideas, like improving the tool set for real life problems. You don’t have to care about the singularity to care about other LW content like self-debiasing, or winning.
Regarding the donation aspect, it seems like rationalist are particularly bad at supporting their own causes. You might estimate how much effort you spend in checking out any charity you do support, and then try to not demand higher standards of this one.
Yes, but there are also many examples that show people coming up with the same idea or conclusion at the same time. Take for example A. N. Kolmogorov and Gregory Chaitin who proposed the same definition of randomness independently.
The circumstances regarding Eliezer Yudkowsky are however different. Other people came up with the ideas he is using as supportive fortification and pronunciamento. Some of those people even made similar inferences, yet they do not ask for donations to stop the otherwise inevitable apocalypse.
Your argument does not seem to work. I pointed out how there is stupidity in professionals, but I made no claim that there is only stupidity. So your samples do not disprove the point. It is nice when people come up with similar things, especially if they happen to be correct, but it is by no means to be expected in every case. Would you be interested in taking specific pieces apart and/or arguing them?
The argument was that Eliezer Yudkowsky, to my knowledge, has not come up with anything unique. The ideas on which the SIAI is based and asks for donations are not new. Given the basic idea of superhuman AI and widespread awareness of it I thought it was not unreasonable to inquire about the state of activists trying to prevent it.
Are you trying to disprove an argument I made? I asked for an explanation and wasn’t stating some insight about why the SIAI is wrong.
Is Robin Hanson donating most of his income to the SIAI?
While it is silly to selectively apply efficacy standards to charity (giving to inefficient charities without thinking, and then rejecting much more efficient ones on the grounds that they are not maximal [compared to what better choice?]), far better to apply the same high standards across the board than low ones.
I’m curious what evidence you actually have that “You will soon learn that your smart friends and favorite SF writers are not remotely close to the rationality standards of Less Wrong.” As far as I can tell, LWians are on a whole more rational than the general populace, and probably more rational than most smart people. But I’d be very curious as to what evidence you have that leads to conclude that the rationality standards of LW massively exceed those of a random individual’s “smart friends.” Empirically, people on LW have trouble telling when they have sufficient knowledge base about topics and repeat claims that aren’t true that support their pre-existing worldview (I have examples of both of these which I’ll link to if asked). LWians seem to be better than general smart people at updating views when confronted with evidence and somewhat better about not falling into certain common cognitive ruts.
That said, I agree that XiXi should read the MWI sequence and am annoyed that XiXi apparently has not read the sequence before making this posting.
Well, I could try to rephrase as “Below the standards of promoted, highly rated LW posts”, i.e., below the standards of the LW corpus, but what I actually meant there (though indeed I failed to say it) was “the standards I hold myself to when writing posts on LW”, i.e., what XiXiDu is trying to compare to Charles Stross.
seems to be different than
because we do a lot of armchair speculating in comment threads about things on which the “rational” position to take is far from clear—and, furthermore, just because someone isn’t trying to present a rational argument for their position at any given moment doesn’t mean that they can’t.
Pfft, it was an example whose truth value is circumstantial as it was merely an analogy used to convey the gist what I was trying to say, namely to subsequently base conclusions and actions on other conclusions which themselves do not bear evidence. And I won’t read the MWI sequence before learning the required math.
What subsequent conclusions are based on MWI?
Check my comment here. More details would hint at the banned content.
I never said EY or the SIAI based any conclusions on it. It was, as I frequently said, an example to elucidate what I’m talking about when saying that I cannot fathom the origin of some of the assertions made here as they appear to me to be based on other conclusions that are not yet tested themselves.
What the hell? That link doesn’t contain any conclusions based on MWI—in fact, it doesn’t seem to contain any conclusions at all, just a bunch of questions. If you mean that MWI is based on unfounded conclusions (rather than that other conclusions are based on MWI), then that’s a claim that you really shouldn’t be making if you haven’t read the MWI sequence.
I see no connection whatsoever to the banned content, either in the topic of MWI or in the comment you linked to. This is a bizarre non-sequitur, and as someone who wants to avoid thinking about that topic, I do not appreciate it. (If you do see a connection, explain only by private message, please. But I’d rather you just let it drop.)
My post was intended to be asking questions, not making arguments. Obviously you haven’t read the banned content.
You seem not to understand my primary question that I tried to highlight by the MWI analogy. MWI is a founded conclusion but you shouldn’t use it to make further conclusions based on it. That is, a conclusion first has to yield a new hypothesis that makes predictions. Once you got new data, something that makes a difference, you can go from there and hypothesize that you can influence causally disconnected parts of the multiverse or that it would be a good idea tossing a quantum coin to make key decisions.
After all it was probably a bad decisions to use that example. All you have to do is to substitute MWI with AGI. AGI is, though I’m not sure, a founded conclusion. But taking that conclusion and running with it building a huge framework of further conclusions around it is in my opinion questionable. First this conclusion has to yield marginal evidence of its feasibility, then you are able to create a further hypothesis engaged with further consequences.
I do not appreciate being told that I “obviously” have not read something that I have, in fact, read. And if you were keeping track, I have previously sent you private messages correcting your misconceptions on that topic, so you should have known that. And now that I’ve hinted at why you think it’s connected to MWI, I can see that that’s just another misconception.
Your tone is antagonistic and I had to restrain myself from saying some very hurtful things that I would’ve regretted. You need to take a step back and think about what you’re doing here, before you burn any more social bridges.
EDIT: Argh, restraint fail. That’s what the two deleted comments below this are.
Is it, and that of EY? Are you telling him the same? Check this comment and tell me again that I am antagonistic.
If I come over as such, I’m sorry. I’m a bit stressed writing so many comments accusing me of trying to damage this movement or making false arguments when all I did was indeed trying to inquire about some problems I have, asking questions.
I think part of the reason this went over badly is that in the US, there is a well-known and widely hated talk show host named Glenn Beck whose favorite rhetorical trick is to disguise attacks as questions, saying things like “Is it really true that so-and-so eats babies?”, repeating it enough times that his audience comes to believe that person eats babies, and then defending his accusations by saying “I’m just asking questions”. So some of us, having been exposed to that in the past, see questions and rhetoric mixed a certain way, subconsciously pattern-match against that, and get angry.
I did get the impression that some took your questions as purely rhetorical, soldiers fighting against the credibility of SIAI. I took you as someone hoping to be convinced but with a responsible level of wariness.
That was my impression, also. As a result, I found many elements of the responses to XiXiDu to be disappointing. While there were a few errors in his post (e.g. attributing Kurweil views to SIAI), in general it should have been taken as an opportunity to clarify and throw down some useful links, rather than treat XiXiDu (who is also an SIAI donor!) as a low-g interloper.
Oh, you are the guy who’s spreading all the misinformation about it just so nobody is going to ask more specific questions regarding that topic. Hah, I remember you know. Thanks, but no thanks.
Fuck you, and please leave. Is that the reaction you were hoping for, troll?
You wrote this and afterwards sending me a private message on how you are telling me this so that I shut up.
Why would I expect honest argumentation from someone who makes use of such tactics? Especially when I talked about the very same topic with you before just to find out that you do this deliberately?
Anyway, I herewith apologize unconditionally for any offence and deleted my previous comment.
Going to watch a movie now and eat ice cream. Have fun :-)
I apologize for my previous comment—I felt provoked, but regardless of the context, it was way out of line.
The thing with the banned topic is, I’m really trying to avoid thinking about it, and seeing it mentioned makes it hard to do that, so I feel annoyed whenever it’s brought up. That’s not something I’m used to dealing with, and it’s a corner case that the usual rules of discourse don’t really cover, so I may not have handled it correctly.
It was my fault all the way to the OP. I was intrigued about the deletion incident and couldn’t shut up and now I thought it was a good idea to inquire about questions that trouble me for so long and to to steer some debate by provoking strong emotions.
I actually understand that you do not want to think about it. It was a dumb idea to steer further debate into that direction. But how could I know before finding out about it? I’m not the personality type who’s going to follow someone telling me not to read about something, to not even think about it.
I deleted the other comment as well.
Right in the beginning of the sequence you managed to get phases wrong. Quick search turns up:
http://www.ex-parrot.com/~pete/quantum-wrong.html
http://www.poe-news.com/forums/spshort.php?pi=1002430803&ti=1002430709
http://physics.stackexchange.com/a/23833/4967
Ouch.
Rest of the argument… given relativistic issues in QM as described, QM is just approximation which does not work at the relevant scale, and so concluding existence of multiple worlds from it is very silly.
Indeed.
Ghahahahaha. “A community blog devoted to refining the art of human rationality”… or in other world an online equivalent of a green-ink letter.
Hopefully this mistake will be fixed one day, so the sequence will be judged on the merits of the argument it presents, and not by the presence of a wrong factor of “i”.
Nonrelativistic QM is an approximation to relativistic QFT, and while relativity certainly introduces a new problem for MWI, it remains true that QFT employs the superposition principle just as much as QM. It’s a formalism of “many histories” rather than “many worlds”, but the phenomenon of superposition, and therefore the possibility of parallel coexisting realities, is still there.
I would agree that it was foolish for Eliezer to flaunt his dogmatism about MWI as if that was evidence of superior rationality. What I would say is that he wasn’t worse than physicists in general. Professional physicists who know far more about the subject than Eliezer still manage to say equally foolish things about the implications of quantum mechanics.
What the evidence suggests to me is that to discover the explanation of QM, you need deep technical knowledge, not just of QM but also QFT, and probably of quantum gravity, at least to the level of the holographic principle, and you also need a very powerful imagination. Possibly the correct answer is a variation on a concept we already possess: many worlds, Bohmian mechanics, loops in time, a ’t Hooft cellular automaton. If so, then the big imaginative leap was already carried out, but the technicalities are still hard enough that we don’t even know that it’s the right type of answer. Eliezer-style dogmatism would be wrong for all the available explanations: we do not know which if any is right; at this stage there is no better strategy than pluralistic investigation, including hybridization of these supposedly distinct concepts. But it’s also possible that the correct answer hasn’t yet been conceived, even in outline, which is why imagination remains important, as well as technical knowledge.
If you accept this analysis, then it’s easier to understand why interpretations of quantum mechanics present such a chaotic scene. The radical ontological differences between the candidate explanations create a lot of conceptual tension, and the essential role of subtle technicalities, and mathematical facts not yet known, in pointing the way to the right answer, mean that this conceptual tension can’t be resolved by a simple adjudication like “non-collapse is simpler than collapse”. The possibility that the answer is something we haven’t even imagined yet, makes life even more difficult for people who can’t bear to settle for Copenhagen positivism—should they just insist “there must be an answer, even if we don’t know anything about how it works”?
It’s therefore difficult to avoid both dogmatic rationalization and passive agnosticism. It’s the sort of problem in which the difficulties are such that a return to basics—a review of “what I actually know, rather than what I habitually assume or say”—can take you all the way back to the phenomenological level—“under these circumstances, this is observed to occur”.
For people who don’t want to devote their lives to solving the problem, but who at least want to have a “rational” perspective on it, what I recommend is that you understand the phenomenological Copenhagen interpretation—not the one which says wavefunctions are real and they collapse when observed, just the one which says that wavefunctions are like probability distributions and describe the statistics of observable quantities—and that you also develop some idea of what’s involved in all the major known candidate ontologies.
For readers of this site who believe that questions like this should be resolved by a quantified Occam’s razor like Solomonoff induction: in principle, your first challenge is just to make the different theories commensurable—to find a common language precise enough that you can compare their complexity. In practice, that is a difficult enough task (on account of all these ideas being a little bit underspecified) that it couldn’t be done without a level of technical engagement which meant you had joined the ranks of “people trying to solve the problem, rather than just pontificating about it”.
The argument is pure incompetent self important rambling about nothing. The mistakes only make this easier to demonstrate to people who do not know QM, who assume it must have some merit because someone wasted time writing it up. Removal of mistakes would constitute deception.
Nonetheless, there is no satisfactory quantum gravity. It is still only an approximation to reality, and subsequently the mathematical artifacts it has (multiple realities) mean nothing. Even if it was exact it is questionable what is the meaning of such artifacts.
They did not have the stupidity of not even learning it before trying to say something smart about it.
The muckiness surrounding the interferometer is well-known; in fact, the PSE question was written by a LWer.
The conclusion isn’t “MWI is true.” The conclusion is “MWI is a simpler explanation than collapse (or straw-Copenhagen, as we in the Contrarian Conspiracy like to call it) for quantum phenomena, and therefore a priori more likely to be true.”
And yes, it is also well-known that this quote is not Yudkowsky at his most charming. Try not to conflate him with either rationalism or the community (which are also distinct things!).
I have not read the MWI sequence yet, but if the argument is that MWI is simpler than collapse, isn’t Bohm even simpler than MWI?
(The best argument against Bohm I can find on LW is a brief comment that claims it implies MWI, but I don’t understand how and there doesn’t seem to be much else on the Web making that case.)
MWI just calculates the wavefunction.
Copenhagen calculates the wavefunction but then has additional rules saying when some of the branches collapse.
Bohm calculates the wavefunction and then says that particles have single positions but are guided by the wavefunction.
But MWI doesn’t get the right calculation in terms of probability
Good point. I’d say that it doesn’t have any calculation of the probability. But some people hope that the probabilities can be derived from just MW. If they achieve this then it would be the simplest theory. But if they need extra hypotheses then it will gain complexity, and may well come out worse than Bohm.
Mitchell_Porter makes the case, but reading him makes my brain shut down for lack of coherence. I assume Yudkowsky doesn’t favor Bohm because it requires non-local hidden variables. Non-local theories are unexpected in physics, and local hidden variables don’t exist.
There’s more to Bohmian mechanics than you may think. There are actually observables whose expectation values correspond to the Bohmian trajectories—“weak-valued” position measurements. This is a mathematical fact that ought to mean something, but I don’t know what. Also, you can eliminate the pilot wave from Bohmian mechanics. If you start with a particular choice of universal wavefunction, that will be equivalent to adding a particular nonlocal potential to a classical equation of motion. That nonlocal potential might be the product of a holographic transformation away from the true fundamental degrees of freedom, or it might approximate the nonlocal correlations induced by planck-scale time loops in the spacetime manifold.
I have never found the time or the energy to do my own quantum sequence, so perhaps it’s my fault if I’m hard to understand. The impression of incoherence may also arise from the fact that I put out lots and lots of ideas. There are a lot of possibilities. But if you want an overall opinion on QM which you wish to be able to attribute to me, here it is:
The explanation of QM might be “Bohm”, “Everett”, “Cramer”, “’t Hooft”, or “None of the Above”. By “Bohm”, I don’t just mean Bohmian mechanics, I mean lines of investigation arising from Bohmian mechanics, like the ones I just described. The other names in quotes should be interpreted similarly.
Also, we are not in a position to say that one of these five approaches is clearly favored over the others. The first four are all lines of investigation with fundamental questions unanswered and fundamental issues unresolved, and yet they are the best specific proposals that we have (unless I missed one). It’s reasonable for a person to prefer one type of model, but in the current state of knowledge any such preference is necessarily superficial, and very liable to be changed by new information.
Well, that’s understandable. Not everyone has all the free time in the world to write sequences.
That’s exactly what I wish Yudkowsky’s argument in the QM sequence would have been, but for some reason he felt the need to forever crush the hopes and dreams of the people clinging to alternative interpretations, in a highly insulting manner. What ever happened to leaving a line of retreat?
Something feels very wrong about this sentence… I get a nagging feeling that you believe he has a valid argument, but he should have been nice to people who are irrationally clinging to alternative interpretations, via such irrational ways as nitpicking on the unimportant details.
Meanwhile, a coherent hypothesis: the guy does not know QM, thinks he knows QM, proceeds to explain whatever simplistic nonsense he thinks is the understanding of QM, getting almost everything wrong. Then interprets the discrepancies in his favour, and feels incredibly intelligent.
I believe he has a valid argument for a substantially weaker claim of the sort I described earlier.
He “should have been nice to people” (without qualification) by not trying to draw (without a shred of credible evidence) a link between rationality/intelligence/g-factor and (even a justified amount of) MWI-skepticism. It’s hard to imagine a worse way to immediately put your audience on the defensive. It’s all there in the manual.
Why do you think so? Quantum mechanics is complicated, and questions of what is a ‘better’ theory are very subtle.
On the other hand, figuring out what claim your arguments actually support, is rather simple. You have an argument which: gets wrong elementary facts, gets wrong terminology, gets wrong the very claim. All the easy stuff is wrong. You still believe that it gets right some hard stuff. Why?
He should have left a line of retreat for himself.
For the reasons outlined above. Occam’s razor + locality.
My argument is distinct from Yudkowsky’s in that our claims are radically different. If you disagree that MWI is more probable than straw-Copenhagen, I’d like to know why.
None of the “easy stuff” is pertinent to the argument that MWI is more probable than straw-Copenhagen. For example, the interferometer calculation is neither used as evidence that MWI is local, nor that MWI is less complicated. The calculation is independent of any interpretation, after all.
if I stand a needle on it’s tip on a glass plate, will needle remain standing indefinitely? No it probably won’t even though by Occam’s razor, zero deviation from vertical is (arguably) more probable than any other specific deviation from vertical. MWI seems to require exact linearity, and QM and QFT don’t do gravity, i.e. are approximate. Linear is a first order approximation to nearly anything.
Intelligence and careful thinking --> getting easy stuff right and maybe (very rarely) getting hard stuff right.
Lack of intelligence and/or careful thinking --> getting easy stuff wrong and getting hard stuff certainly wrong.
What is straw Copenhagen anyway? Objective collapse caused by consciousness? Copenhagen is not objective collapse. It is a theory for predicting and modelling the observations. With the MWI you still need to single out one observer, because something happens in real world that does single out one observer, as anyone can readily attest, and so there’s no actual difference here in any math, it’s only a difference in how you look at this math.
edit: ghahahahaha, wait, you literally think it has higher probability? (i seen another of the Yudkowsky’s comments where he said something about his better understanding of probability theory) Well, here’s the bullet: the probability of our reality being quantum mechanics or quantum field theory, within platonic space, is 0 (basically, vanishingly small, predicated on the experiments confirming general relativity all failing), because gravity exists and works so and so but that’s not part of QFT. 0 times anything is still 0. (That doesn’t mean the probability of alternate realities is 0, if there can be such a thing)
From the one comment on Bohm I can find, it seems that he actually dislikes Bohm because the particles are “epiphenomena” to the pilot wave. Meaning the particles don’t actually do anything except follow the pilot wave, and it’s actually the the pilot wave itself that does all the computation (of minds and hence observers).
Lack of coherence? where? It’s true that Bohm requires non-local HV’s, but there is a non-local flavor to MWI too. The states are still non-local. Local HV’s do exist. Gerard ’t Hooft is working on this as we speak: http://arxiv.org/find/all/1/all:+AND+hooft+AND+gerard+t/0/1/0/all/0/1
His monologue on color, for instance.
This assumption is made by every other interpretation of quantum mechanics I know. On the other hand, I’m not a physicist; I’m clearly not up to date on things.
I meant the classical HV theories that were ruled out by actual experiments detecting violations of Bell’s inequality.
Well, you didn’t link to his view of qualia, but to a link where he explains why MWI is not the “winner” or “preferred” as EY claimed so confidently in his series on QM. You might disagree with him on his stance on qualia ( I do too ) but it would be a logical fallacy to state that therefore all his other opinions are also incoherent.
Mitchell Porter’s view on qualia is not non-sense either, it is highly controversial and speculative, no doubt. But his motivation is sound, he think that it is the only way to avoid some sort of dualism, so in that sense his view is even more reductionist than that of Dennett etc. He is also in good company with people like David Deutsch (another famous many world fundamentlist).
As for local hidden variables, obviously there does not exist a local HV that has been ruled out ;p but you claimed there was none in existence in general.
Maybe I should have said “reading him in general...”
The rest is quibbling over definitions.
Ahh, that would explain why a non-answer is the accepted one. Was this non-answer written by LWer by chance?
Rest of sequence is no better. Photon going in particular way is not really ‘configuration’ with a complex amplitude, I am not even sure the guy actually understands how interferometer works or what happens if length of one path is modified a little. Someone who can’t correctly solve even a simplest QM problem has no business ‘explaining’ anything about QM by retelling popular books.
You clearly do not have enough g-factor:
When people are at their most charming, they are pretending.
Rationalism? I see. This would explain why the community would take that seriously instead of pointing and laughing.
Scott Aaronson is a not, as far as I know, a LWer, though he did an interview with Yudkowsky once on QM. He disagrees with him pretty substantially.
I don’t disagree?
It’s possible.
No, the other rationalism, rationality. My bad.
Was a joke.
Are you sure? I’ve seen posts speaking of ‘aspiring rationalists’. It does make sense that rationalists would see themselves as rational, but it does not make sense for rational people to call themselves rationalists. Rationalism is sort of like a belief in power of rationality. It’s to rationality as communism is to community.
Believing that the alternate realities must exist if they are a part of a theory, even if the same theory says that the worlds are unreachable, that’s rationalism. Speaking of which, even a slightest non-linearity is incompatible with many worlds.
There is something that makes me feel confused about MWI. Maybe it is its reliance on anthropic principle (probability of finding myself in a world where recorded history have probability P (according to Born’s rule) must be equal to P). This condition depends on every existing universe, not just on ours. Thus it seems that to justify Born’s rule we should leave observable evidence behind and trail along after unprovable philosophical ideas.
If the map is not the territory how is it that the maths and logic can assign some worlds, or infinite many, but not others with the attribute of being real and instantiated beyond the model?
Can selections for “realness” be justified or explained logically, is it a matter of deduction?
Say, what makes something a real thing versus an abstract matter. When does the map become the territory?
As far as I know the uniformity or different states of the universe are not claimed to be factual beyond the observable because we can deduce that it is logical to think one way or the other?
You say that once I read the relevant sequence I will understand. That might be so, as acknowledged in my post. But given my partial knowledge I’m skeptic that it is sound enough to allow for ideas to be taken serious enough such as that you can influence causally disconnected parts of the multiverse. That it would be a good idea tossing a quantum coin to make key decisions and so on.
It was however just one example to illustrate some further speculations that are based on the interpretation of a incomplete view of the world.
I think the idea of a Mathematical Universe is very appealing. Yet I’m not going to base decisions on this idea, not given the current state of knowledge.
If you claim that sufficient logical consistency can be used as a fundament for further argumentation about the real world, I’ll take note. I have to think about it. People say, “no conclusions can be drawn if you fail to build a contradiction”. They also say you have to make strong, falsifiable predictions. Further, people say that picking a given interpretation of the world has to have practical value to be capable of being differentiated from that which isn’t useful.
It seems that you mainly study and observe nature with emphasis on an exclusively abstract approach rather than the empirical. As I said, I do not claim there’s anything wrong with it. But so far I have my doubts.
MWI may be a logical correct and reasonable deduction. But does it provide guidance or increase confidence? Is it justified to be taken for granted, to be perceived as part of the territory simply because it makes sense? It is not a necessity.
Your skepticism is aimed in the wrong direction and MWI does not say what you think it does. Read the sequence. When you’re done you’ll have a much better gut sense of the gap between SIAI and Charles Stross.