Science as Attire

[Eliezer Yudkowsky]

The preview for the X-Men movie has a voice-over saying: “In every human being . . . there is the genetic code . . . for mutation.” Apparently you can acquire all sorts of neat abilities by mutation. The mutant Storm, for example, has the ability to throw lightning bolts.

I beg you, dear reader, to consider the biological machinery necessary to generate electricity; the biological adaptations necessary to avoid being harmed by electricity; and the cognitive circuitry required for finely tuned control of lightning bolts. If we actually observed any organism acquiring these abilities in one generation, as the result of mutation, it would outright falsify the neo-Darwinian model of natural selection. It would be worse than finding rabbit fossils in the pre-Cambrian. If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.

The X-Men comics use terms like “evolution,” “mutation,” and “genetic code,” purely to place themselves in what they conceive to be the literary genre of science. The part that scares me is wondering how many people, especially in the media, understand science only as a literary genre.

I encounter people who very definitely believe in evolution, who sneer at the folly of creationists. And yet they have no idea of what the theory of evolutionary biology permits and prohibits. They’ll talk about “the next step in the evolution of humanity,” as if natural selection got here by following a plan. Or even worse, they’ll talk about something completely outside the domain of evolutionary biology, like an improved design for computer chips, or corporations splitting, or humans uploading themselves into computers, and they’ll call that “evolution.” If evolutionary biology could cover that, it could cover anything.

Probably an actual majority of the people who believe in evolution use the phrase “because of evolution” because they want to be part of the scientific in-crowd—belief as scientific attire, like wearing a lab coat. If the scientific in-crowd instead used the phrase “because of intelligent design,” they would just as cheerfully use that instead—it would make no difference to their anticipation-controllers. Saying “because of evolution” instead of “because of intelligent design” does not, for them, prohibit Storm. Its only purpose, for them, is to identify with a tribe.

I encounter people who are quite willing to entertain the notion of dumber-than-human artificial intelligence, or even mildly smarter-than-human artificial intelligence. Introduce the notion of strongly superhuman artificial intelligence, and they’ll suddenly decide it’s “pseudoscience.” It’s not that they think they have a theory of intelligence which lets them calculate a theoretical upper bound on the power of an optimization process. Rather, they associate strongly superhuman AI to the literary genre of apocalyptic literature; whereas an AI running a small corporation associates to the literary genre of Wired magazine. They aren’t speaking from within a model of cognition. They don’t realize they need a model. They don’t realize that science is about models. Their devastating critiques consist purely of comparisons to apocalyptic literature, rather than, say, known laws which prohibit such an outcome. They understand science only as a literary genre, or in-group to belong to. The attire doesn’t look to them like a lab coat; this isn’t the football team they’re cheering for.

Is there any idea in science that you are proud of believing, though you do not use the belief professionally? You had best ask yourself which future experiences your belief prohibits from happening to you. That is the sum of what you have assimilated and made a true part of yourself. Anything else is probably passwords or attire.

[anonymous7]

Even if biological evolution would allow one-generation mutation for “Storm abilities”, it would not equal that evolution doesn’t explain anything. Even if everything “is possible” in evolution, there can still be different probabilities for different outcomes. The probability for “Storm ability mutation” is non-zero.

[Eliezer Yudkowsky]

The probability for anything is non-zero. But when we see something to which our hypothesis assigns a sufficiently infinitesimal probability, we call it falsified—because even maximum entropy does better.

[Ronny Fernandez]

What about the probability of “2+2=4 and not 2+2=4.” Pretty sure that’s zero, and if not we got some problems I believe.

[orthonormal]

Short answer: I can never assign probability exactly 0, because there’s a tiny chance that I’m currently hallucinating the “obvious” contradiction, etc.

Longer answer: How To Convince Me That 2+2=3, and my comment on it.

[Ronny Fernandez]

So is the conjunction of two contradictory phenomena not zero? I am confused. I believe if that is so the rest of bayes falls apart, no? Bayes requires that you give zero probabilities to contradictions, if you do not then you can be dutch booked, right? It also requires that you give a probability of one to logical tautologies, if you give more or less, then you can also be dutch booked. What am I messing up? Really, please help if you understand.

[dlthomas]

I expect the problem is not that you are wrong (that’s more or less open), but that there has been similar discussion in many places (one is here) on this site and building another tree with pretty much the same starting point doesn’t really make sense.

[Ronny Fernandez]

Ah i see. Thanks

(edit): shoud i just take my posts down?

[orthonormal]

(edit): shoud i just take my posts down?

No— it’s more helpful if you edit your first comment to say “OK, this is already discussed here”, so that if anyone else reads Eliezer’s comment and has the same objection as you, they know where to go to discuss it rather than opening another instance of the same comment thread...

[Odinn]

Even the simplest expression such as 2+2=4 should not be literally tautological. There is an infinitesimal possibility that the human brain has a fundamental flaw that causes us to read the expression incorrectly, or that every person or program that has ever attempted to calculate the sum of 2 and 2 has erroneously provided an incorrect answer, or that our universe is actually configured in a way that isn’t mathematically accurate (despite what those pictures of apples in first grade textbooks claim.) Conjugate all of these extremely strange possibilities and your odds of 2 and 2 not equaling 4 are so small, so imaginary, that we can confidently and adequately use probability 1 for 2+2=4, but a perfect Bayesian formula still has no room for entries of P=1 or 0, especially since the formula just doesn’t provide useful data that way.

[[deleted]]

“(2+2=4) and ~(2+2=4)” is a statement in arithmetic and propositional logic, which are quite distinct from Bayeasian Probability. Mathematics really does have separate magisteria, because it is not necessarily connected to practical reality.

[Ronny Fernandez]

Any (A & ~A) works just as well. How can you assign greater than zero probability to a contradiction? Doesn’t the whole system fall apart? Same thing for less than one probability to a tautology. If there are probability theories that do that, I would like to know of them.

[[deleted]]

This is the third time I try to reply to this, because it appears there are some serious inferential distances and I would rather not appear condescending.

Firstly, you must remember that a contradiction has no definite truth or falsehood value.

Now I have seen some interesting papers that make expanded probability theories that include 0 and 1 as logical falsehood and truth respectively. But that still does not include a special value for contradictions.

If we turn to Type theory (think programming) we might say the contradiction is an expression that has the bottom type (a type with no values, error in any case), but how do you put no value into probability theory?

All in all it is mathematically meaningless to talk about the probability of a logical contradiction, but don’t ask me for a proof of it, I am not that good.

So if you really want you can say that 0 is falsehood and 1 is truth and with a bit of sleight of hand you can use them as probabilities. But you will be stunted when you hit a contradiction.

Complete or Consistent, choose one.

[Ronny Fernandez]

Um, you can derive from Komolgorov that (A & ~A) has prob zero. Very easily.

If (A & B) = nullset, then P(A & B) = 0.
(A & ~A) = nullset
so: P(A & ~A) = 0.

In other words, if two sets are disjoint, the probability of their intersection is zero. Any set and its compliment are disjoint, so the probability of their conjunction is zero.

It also helps if you think of the boolean variables as fractions of a dart board, and the probability as the area of that fraction. The formalism is perfectly isomorphic. Obviously, the intersection of any fraction and its compliment will have an area of zero.

See, I believe I have a proof that the probability of any contradiction is zero. So I am going to have to ask for a proof to change my mind, or a problem with my proof (which I doubt, it’s very short, and I math often).

[[deleted]]

Oh I see, you are talking of exclusive outcomes, not contradictions. Yes you are entirely right, exclusive outcomes work exactly that way. The probability of both Heads and Tails occuring at the same time on a coinflip is zero.

Contradictions in a system isomorphic with Propositional Logic do not and are an entirely separate mathematical object.

[Ronny Fernandez]

I would like to read more on that, because I believed them to be exactly equivalent.

A set’s intersection with its compliment
has a perfect isomorphism with
A propositions conjunction with its negation.

[[deleted]]

The Principle of explosion tells us that a if one supposes both truth and falsehood as base premises, one can derive any conclusion, by means of Propositional Logic theorems.

The intersection of a set with it’s complement is the empty set.

A consistent isomorphism of ZF Set theory and Propositional Logic is achieved by letting truth be a non-empty set and falshood be an empty set. Then intersection becomes logical conjunction and complement with respect to the truth set becomes logical inversion.

Now, the reason I can argue for such things is actually backed up by Godel’s incompleteness theorem. Propositional Logic and ZF Set Theory can both implement Peano Arithmetic, Bayesian Probability cannot. Thus Propositional Logic and ZF Set Theory are Complete but not Consistent, while Bayes is Consistent but not Complete.

[Ronny Fernandez]

argue for what things? I have no clue what the POE or Gtheorem have to do with komologrov provably assigning zero probability to contradictions.

What is the difference between exclusive outcomes and contradictions? How are they not the same mathematical object? If two exclusive outcomes end up resulting, you can also explode the universe.

For A to be exclusive with B, means that if A happened B did not. So: If A and B, then ~B and B, then B or P, then ~B, so P. The POE is not something unique to contradictions which exclusive outcomes lack.

[[deleted]]

This comment explains where our communication runs skew

[Polymeron]

Now I have seen some interesting papers that make expanded probability theories that include 0 and 1 as logical falsehood and truth respectively. But that still does not include a special value for contradictions.

Except, contradictions really are the only way you can get to logical truth or falsehood; anything other than that necessarily relies on inductive reasoning at some point. So any probability theory employing those must use contradictions as a means for arriving at these values in the first place.

I do think that there’s not much room for contradictions in probability theories trying to actually work in the real world, in the sense that any argument of the form A->(B & ~B) also has to rely on induction at some point; but it’s still helpful to have an anchor where you can say that, if a certain relationship does exist, then a certain proposition is definitely true.

(This is not like saying that a proposition can have a probability of 0 or 1, because it must rely, at least somewhere down the line, on another proposition with a probability different from 0 and 1).

[A1987dM]

ISTM that you’re using the word contradiction in a non-standard way: in the usual sense they are logical falsehoods. What do you actually mean? (ETA: I guess paradoxes such as “This sentence is false”?)

[[deleted]]

I use contradiction in the completely ordinary sense as seen in propositional logic. (P & ~P)

[A1987dM]

How is that not a logical falsehood?

[[deleted]]

It is not a logical falsehood for several reasons. What I actually mean by using the traditional notation of propositional calculus is that the statement A is a true statement. Were it a false statement I would write ~A. Similarly i write (P & ~P) to mean “It is true that both P and not P” while I write ~(P & ~P) to mean “It is true that not both P and not P.”

Solving the latter as the equation ~(P & ~P) = TRUE for the variable P gives the trivial solution set of {TRUE, FALSE}, solving the former equation (P & ~P) = TRUE for the variable P gives the empty solution set {}

This is simple convention of notation, I am sorry if that wasn’t clear. Yes, evaluating the logic arithmetic statement (P & ~P) for any given boolean value of P yields false.

[Ronny Fernandez]

Logical falsehoods, and disjoint events, all reduce to contradictions in boole. There is no difference in propositional logic. Where are you getting this from?

[[deleted]]

See this comment

[orthonormal]

I think you and potato are talking about different things; potato’s criticism is close to what’s discussed in this post.

But for what it’s worth, Bayesians with finite (and flawed) computational powers can meaningfully assign probabilities to mathematical statements, and update as they prove more theorems.

[Ronny Fernandez]

More over, we would make little progress in prime analysis if we could not use probability theory to restrict expected experience in purely formal environments.

[Doug_S.]

::threadjack::

The probability for anything is non-zero.

I’d be willing to assign zero probability to mathematical falsehoods, such as “2+2=5”.

On topic:

Apparently, a lot of people really don’t understand biological evolution.

(Many people don’t really understand what a physicist means by a “wave”, either, but they tend to be familiar with examples.)

[handoflixue]

http://​​lesswrong.com/​​lw/​​jr/​​how_to_convince_me_that_2_2_3/​​

I’d be adverse to assigning zero probability even to mathematical falsehoods :)

(Edit: Okay, finally checked the math. A zero probability means “I absolutely refuse to update this belief regardless of the evidence.” I can see situations where I can’t imagine ever running in to evidence against, but not anything where I’d refuse to update my belief even in light of evidence...)

[thomblake]

I find it amusing that you responded to a comment from August 2007 by linking to a post from September 2007.

If only Doug_S. had bothered to read that post before making that comment, there wouldn’t have been any confusion!

[handoflixue]

laughs Cute, I hadn’t paid attention to either of the timestamps :)

[perry]

How about, “If this expectation of mine is ever violated, I’ll have to rebuild my world view from scratch anyway, discarding all major priors”? I’d say that’s a valid reason to assign zero. Fudging it with an infinitesimal nonzero probability feels like cheating in that case. (All your eggs are in that basket.)

Anyway, mathematical falsehoods are internal contradictions in a mathematical system. So reliable evidence of mathematical falsehood means either confusion (probable), broken axioms in the system used (possible), or a break in your core logic (reload firmware and reboot brain :-).

Cheers -- perry

[handoflixue]

“If this expectation of mine is ever violated, I’ll have to rebuild my world view from scratch anyway, discarding all major priors”?

I’ve had to rebuild huge chunks of my priors due to major system collapses a couple times now, so the idea doesn’t really bother me. I’ve also done maths where 2+2=3, thanks to a hobbyist interest in abstract algebra.

I’d rather seek Truth than Convenience, even if it does mean rebuilding everything again. Besides, a compelling proof that 2+2=3 probably means something fundamental has changed in the world, and that believing 2+2=4 will cause me some serious issues.

[Gray_Area]

Eliezer said: “I encounter people who are quite willing to entertain the notion of dumber-than-human Artificial Intelligence, or even mildly smarter-than-human Artificial Intelligence. Introduce the notion of strongly superhuman Artificial Intelligence, and they’ll suddenly decide it’s “pseudoscience”.”

It may be that the notion of strongly superhuman AI runs into people’s preconceptions they aren’t willing to give up (possibly of religious origins). But I wonder if the ‘Singularians’ aren’t suffering from a bias of their own. Our current understanding of science and intelligence is compatible with many non-Singularity outcomes:

(a) ‘human-level’ intelligence is, for various physical reasons, an approximate upper bound on intelligence (b) Scaling past ‘human-level’ intelligence is possible but difficult due to extremely poor returns (e.g., logarithmic rather than exponential growth past a certain point) (c) Scaling past ‘human-level’ intelligence is possible, is not difficult, but runs into an inherent ‘glass ceiling’ far below ‘incomprehensibility’ of the resulting intelligence

and so on

Many of these scenarios seem as interesting to me as a true Singularity outcome, but my perception is they aren’t being given equal time. Singularity is certainly more ‘vivid,’ but is it more likely?

[Kyle]

A small caveat: the word ‘evolution’ doesn’t have to refer to the scientific theory of biological evolution. The word existed long before the theory; otherwise, the theory would have become attached to a different word. Since the word itself means “incremental change over time,” then it is perfectly appropriate to refer to a new computer chip design, or a corporate reorganization as evolution. Make your own guesses about whether something totally different, such as uploading a personality, can be called “evolution.”

[JJ10DMAN]

This. I regularly refer to cultural trends, business models, and technology as undergoing evolution, without the slightest inkling of doubt or shame. Real Life is about compromises to that most obstinate debater Nature, and if one must deal with the pragmatic issue of conveying ideas in a conversation in a short period of time, “evolution” is perfectly acceptable shorthand for “process by which a system becomes incrementally more efficient via an ongoing process of simultaneous diversification and selection, similar to biological evolution if someone were to replace the concept of random genetic variation with human ideas and natural selection with artificial selection.” That simply takes way too long to say.

[jonvon]

i can’t help but see a few interesting ironies in this post.

the “mutants” in the world of the x-men are people who all have one and only one common “genetic” mutation. and that mutation is the ability to mutate, as you put it, “in one generation”. that is itself the essential mutation that is common to all “mutants”. the fact that they can move from normal human to super powered mutant in the space of one generation (their own lifetime) is exactly the point.

in other words, “control over lightning” is not the metaphor. shooting laser beams out of the eyes, or teleporting or flying or… whatever, none of those things are the metaphor. the metaphor is the ability to mutate in the space of one lifetime. and the stories they spin from that wellspring often have to do with the fallout, the blessings and curses that come from what the ability to mutate means for any particular individual.

it’s a metaphor that works quite well, as long as you don’t take it too literally. (pun not intended but i can’t help noting it.) everyone wants to be able to adapt to circumstances within the course of their own lifetime in a way that puts them on top of their situation. an entire religion—buddhism—seems to rotate around this central idea. mostly buddhists seem to be saying that only in rare circumstances does one soul “mutate” into a state in which that soul truly achieves enlightenment. most of us have to keep spinning on the great wheel and reincarnating until we get it right. i’m not a buddhist so i may be getting this wrong, at least in terms of emphasis.

for you, your “super power” or “mutation” is your understanding of scientific models or rational ways of thinking. it seems to me that you have achieved quite a bit along these lines. you started thinking about these things at a young age, coincidentally (or not?) about the same time stan lee’s mutants discover their powers, as a budding teenager. (hopefully i’m remembering your recent post correctly.) in my sparsely informed viewpoint, entirely based on the posts i’ve read here, you have been driven by something intrinsic to you to understand the world in just the way you do now. everything led you to the point you are at now. there were things that were obvious to you at a young age that were not, i assure you, obvious to most other people.

you might phrase that very differently, and with much greater precision, and without all the biases and faulty logic that i’m sure infect my own thinking.

if you think about it though, you’ll see that in some way you are a kind of mutant. you are the exception, not the rule. you started to think about things, at a young age, in a way that most people never get to. it may be that you come from a long line of such thinkers. the further out i push the metaphor, the shakier it gets. at the end of the day it’s about discovering your own inherent powers and using them in a way that is, hopefully, best for you and for the people around you.

[tlhonmey]

Such an ability isn’t even that ridiculous with proper grounding. Brain cells seem to use an RNA-like molecule to transfer programming from one neuron to another. Turning that into a way to make genetic changes at the speed of protein brains instead of at the speed of genetic evolution would be a big step, and very unlikely to come about without the assistance of some protein brains guiding the process, but once it was established it would be a large enough advantage that it would likely stick around.

So odds seem high that all the “mutants” are the descendants of some genetic engineering project somewhere.

[Armok_GoB]

So… the superpowers are a metaphor for general intelligence?

[Hopefully_Anonymous]

“I’d be willing to assign zero probability to mathematical falsehoods, such as “2+2=5″.”

You might be willing to, Doug S., but that doesn’t mean that it’s optimally rational for you to do so. I don’t know as much about bayesian reasoning as I’d like to, but my understanding is that would not be bayesian of you.

[Eliezer Yudkowsky]

Gray Area, the objections you list are objections from within a model. This is right and proper. A lot of people don’t reason the same way you do, though. Quick replies: (1) We know from sheerly physical considerations that you can build a brain at least a million times as fast as a human brain, which gives us many interesting results of itself. (2, 3) Barring a specific model of cognitive science one cannot disprove a magical upper bound which lies exactly above human intelligence, even though raw evolution encountered no apparent difficulty in accidentally building humans out of chimps using only a threefold increase in brain and a sixfold increase in prefrontal cortex. But the principle of mediocrity weighs heavily against such an arbitrary presumption, as well as the general notion that evolution doesn’t build optimal systems, plus all the known flaws we talk about on Overcoming Bias. Furthermore my own, specific model of cognitive science is already suggesting that we can go beyond human purely on the basis of writing better software, but this is too complex to justify in detail here—the common sense of this should be apparent, though.

Doug, it may help to think in terms of a log odds ratio, in which case a probability of zero corresponds to negative infinity. You could never get that much counterevidence—we are talking about a subjective probability, not an objective frequency. Perhaps you are hallucinating or deluded or a toy in a computer simulation of mathematical error. Infinity is not a number, and 0 is not a probability.

Jonvon, there is only one human superpower. It makes us what we are. It is our ability to think. Rationality trains this superpower, like martial arts trains a human body. It is not that some people are born with the power and others are not. Everyone has a brain. Not everyone tries to train it. Not everyone realizes that intelligence is the only superpower they will ever have, and so they seek other magics, spells and sorceries, as if any magic wand could ever be as powerful or as precious or as significant as a brain.

[Ronny Fernandez]

Do I not have a personal subjective probability of one that I just recalled the stimulus of writing this sentence?

[wedrifid]

Do I not have a personal subjective probability of one that I just recalled the stimulus of writing this sentence?

No, you don’t.

[lessdazed]

If you work hard at it, you can have a personal subjective probability of one that the Rangers will win the World Series next year. They might even win. In which case...nothing. You still shouldn’t have thought that.

It may be best to act as if the probability was one, for lack of ability to represent near-one probabilities in your mind. But I recommend acting and committing to act as if it were impossible to blackmail you, even though you probably (rightly) think the chances someone could successfully do it against you are high enough to be representable.

[Ronny Fernandez]

What evidence could I observe that would lower my probability that I remember a stimulus?

[lessdazed]

If an angry empty crack pipe demands you stop daydreaming, then it should be thought more likely than before that the memory isn’t of something that really happened.

If you are thinking that you are thinking of something, you could be wrong. Thinking the thought “Mouse” is what it feels like to think of a mouse, thinking “I’m thinking of a mouse” is what it feels like to think of thinking of a mouse.

[Ronny Fernandez]

If you can demonstrate that I can be wrong about thinking I’m thinking something I’ll retract my OP.

Of course that I remember the appearance of the crack pipe, does not need for it to be that there was a crack pipe.

[DSimon]

Ideally, whenever you discover that you have mis-remembered something, you should lower your overall confidence in the accuracy of your memories.

Or perhaps you mean it in the sense of “What is the probably that I (the person hearing this sentence in an internal monologue) exist?”. That one is tricky. But, however you resolve the anthropic dilemma there, you still shouldn’t assign it a probability of 1, if only because, very occasionally, other things that seemed just as ludicrously obviously true in the past have later been shown to be false.

[Ronny Fernandez]

You misunderstood, not that the stimulus correlated to anything, but that I myself experienced the memory of the stimulus. This does not mean i had the stimulus, nor that I hallucinated it, simply that I now have the experience of having remembered the stimulus. How could I be wrong about that? What sense does it make to question if you recall something? Again, not if you recall it accurately, but if you recall it all. How might I think I remember one stimulus, and find that i actually remembered another?

[DSimon]

Okay, so you mean something like from my second paragraph, right?

The reason you shouldn’t assign a probability of exactly 1 to any belief, even to things that you can’t imagine being wrong about, is that in the past, intelligent and well-educated people have also believed things that they also couldn’t imagine being wrong about. And then later, been wrong. Usually because they were making some assumption that they weren’t even aware of.

Setting a probability of 1, for a Bayesian, is saying “I will never ever ever adjust my subjective probability for this belief downwards, no matter what.” Putting yourself in that position is not a good idea.

Furthermore, the belief in question has a lot to do with consciousness and internal experience. This is a poorly understood field, so beliefs in this area deserve special scrutiny.

[Stuart_Armstrong]

belief as scientific attire, like wearing a lab coat. Science (unlike religion) has proved its myths—by putting men on the moon, mobile phones in people’s pockets, and curing diseases. It’s payed its dues to reliability. So unless I am willing to look into it myself, I should, as a default, believe most things scientists claim. And, unless I’m willing to study the press extensively, I should defer to uncontradicted press stories about scientific claims, especially if they’re repeated. This makes science into a litterary genre, but it’s the only real option for non-jounalist non-scientists.

The press has reports with things like “scientists attack creationist teachings”, but I’ve never seen “scientists attack common misconceptions about evolutionary theory”. Ergo, bashing creationists is sensible, but worrying about my own poor understanding is not.

As for the issue you identified—attitude towards AI’s—there are some other aspects reinforcing people’s attitudes (this may be why you found this response so prevalent). There is no theoretical barrier to constructing human level AI’s, since humans’s exist. We can improve human intelligence in various ways (slightly), so we can improve these AI’s too (so slightly better than human is possible too). For super-AIs, on the other hand, nothing exists in the world to show that they are possible. And if someone had created them, or were confident of creating them, this would have been reported in the press. So the man on the street, even without apocalyptic litterature, should conclude that super-AI’s are far from today’s technology (of course, the apocalyptic litterature doesn’t help).

Some people I know then make the same mistake you mentioned—super-AI’s aren’t science (no repeatable experiment). They are, however, talked about in a scientific language. Hence they must be pseudo-science.

Jonvon, there is only one human superpower. It makes us what we are. It is our ability to think. I’d add speach, empathy, opposable thumb, quite a long lifespan, and superior social skills to the list. All of them have just as much claim to “making us who we are” (though thinking and social skills have the best claim to “making us who we will be”).

[JonMcGuire]

“There is no theoretical barrier to constructing human level AI’s, since humans’s exist. We can improve human intelligence in various ways (slightly), so we can improve these AI’s too (so slightly better than human is possible too). For super-AIs, on the other hand, nothing exists in the world to show that they are possible.”

If the “man on the street” has sufficient familiarity with The Wonders of Science to accept human-level AIs as genuinely possible, it seems to me that the sheer boost in simple processing power available to an artificial construct is probably enough to support at least a theoretical acceptance of the possibility of super-AI. If somebody understands technology well enough to allow for near-human AI, I would expect to find that they assume super-AI is not just possible, but trivial … witness people attributing all sorts of mysterious and nefarious intelligent behavior to Google’s on-the-fly search predictions, for example...

[Eliezer Yudkowsky]

Science (unlike religion) has proved its myths

See, this is exactly the sort of thing I have a problem with. Science is not just magic that works. People are learning science as though it were merely a true religion: passwords, attire, professions, and all.

The press has reports with things like “scientists attack creationist teachings”, but I’ve never seen “scientists attack common misconceptions about evolutionary theory”.

Press selectivity. Trust me, they do.

I’d add… speech, empathy… superior social skills to the list

I wasn’t aware that speech, empathy, and social skills were functions of the kidneys rather than the brain.

[Larks]

Science (unlike religion) has proved its myths

Isn’t this applying Bayesianism to the scientific method? If the scientific method is a good method of truth seeking, it should lead to useful applications. It has lead to useful applications, so our p: science is a good method of truth-seeking rises.

[Hopefully_Anonymous]

“Science (unlike religion) has proved its myths—by putting men on the moon, mobile phones in people’s pockets, and curing diseases. It’s payed its dues to reliability. So unless I am willing to look into it myself, I should, as a default, believe most things scientists claim.” Stuart, this in my opinion is an awful sentence, and I’m surprised to read it by an overcomingbias contributor.

[Tom_McCabe]

“I wasn’t aware that speech, empathy, and social skills were functions of the kidneys rather than the brain. ”

We know empathy and social skills don’t require general intelligence; plenty of mammals show empathy and social skills. If the definition of “intelligence” is “whatever occurs in the brain”, then a 4004 CPU shows “intelligence” every time it adds two hex digits.

[Anders_Sandberg]

I have noticed that since using the word “progress” has become unseemly, many use “evolution” in its stead. Quite often in the sense of “incremental change”, sometimes in the slightly biology-analogous sense of “the effect of broad trial and error learning”—but hiding the teleological assumption progress was at least open about.

It has been scientifically proven that people use science attire to make their views sound more plausible :-) Throw in some neuroscience, statistics or a claim by a Ph.D. in anything and you show that you are credible. And the worst thing is that it seems to work fairly often. At the price, of course, that increasingly manipulation-savy media consumer start to suspect a Sinister Conspiracy behind every scientific claim. “Gravitational slingshots—who benefits?” “Who is really behind the stem cells?”

But this attire-wearning is likely nothing new. Tartuffe wore the attire of a pious person to manipulate. It might be more problematic from our standpoint that it is currently a largely epistemological profession/​activity that is being used as high-status attire to hide bias and bad epistemology. Having people dress up in moral attire might have been bad for morality and people involved in the moral business, but it didn’t hurt truth-seeking and bias-overcoming directly.

[TGGP3]

I second Stuart’s awful sentence. I’m not seconding the opinion that it is awful, just that it resembles my thoughts.

[tofu]

Unfortunately, you picked the only member of the x-men who turned out to be a goddess and not a mutant (can’t remember what story arc). How disturbing.

[bigjeff5]

All the other mutants destroy evolution the same way.

For example, can you imagine the leap required to be able to blast lasers powerful enough to destroy buildings out of your eyes? That’s no incremental step, that’s skipping at least a couple hundred million years of evolution in a single generation.

[shokwave]

skipping at least a couple hundred million years of evolution

I can’t even imagine how long it would take evolution to produce nuclear fission and lasing chambers in a biological organism. Or what functional mutations it would have to build on to get there.

[DSimon]

Supposedly he’s pulling the energy out from some other dimension or something, rather than producing it in his own body. That’s the latest hand-waving explanation, anyways; originally, Cyclops’ powers were supposed to be powered by photosynthesis in his body! Clearly the early writers were rather over-optimistic about solar power...

[JoshuaZ]

Unfortunately, you picked the only member of the x-men who turned out to be a goddess and not a mutant (can’t remember what story arc). How disturbing.

Replying to a very old comment, but doing so to note that although Storm was worshipped as a goddess by an African tribe I don’t think she was ever a goddess herself (although some people do sometimes refer to as such). However, Storm’s mother in the main Marvel Comics setting is a powerful sorceress, and I think that there’s some canon evidence that Storm inherited some of that ability.

[Douglas_Knight2]

I think people should be more careful about the word “science.” Here are some meanings I see attached to it:

1. knowledge of nature

2. naturalism

3. “the scientific method”

4. institutions practicing the scientific method

5. rules of a particular institution

6. the output of institutions

I feel compelled to add that what I mean by “the scientific method” is that observation should drive belief and that we can put effort into obtaining useful observations (experiments, stamp collecting). Also, it may be useful to distinguish between institutional rules intended to protect the institution from cheaters and rules intended to protect people from their own biases.

[Eliezer Yudkowsky]

Unfortunately, you picked the only member of the x-men who turned out to be a goddess and not a mutant

Ha! I knew someone was going to say that! (Because I looked up the Wikipedia entry, thank you very much.) That’s why I invoked the movie version of Storm, who is a mutant! So there!

[NotAnEditor]

Storm wasn’t a goddess. She was a human mutant with the ability to manipulate and duplicate various weather patterns via telekinetic manipulation of air flows, particularly those involved in electrical storms. An African tribe mistook her for their rain goddess as a child and she grew up, understandably, with something of a god complex.

(Unless that got retconned, I guess.)

That’s the comic Storm, incidentally; I never got around to watching the films.

However, in the comics, so-called “mutant” abilities are genetic “mutations” somehow hidden in human “junk DNA”, placed there by prehistoric aliens (for some reason.) These “mutations” (probably in-universe slang BTW) are often only minor alterations to the affected individual’s biology, since the MU has a lot of strange physics, like psychic powers , alternate worlds, functional magic etc. that make them easier. These powers are usually “latent” and inactive, only manifesting when “triggered” by radiation, electrical shocks, physical injuries or whatever, and this is often mistaken for said “trigger” actually mutating them into super-humans. Whether this is possible or not is debatable, but it always irritates me when people go “those idiots at Marvel think radiation will mutate you into Superman! Ha ha, how superior we are.” and such. (Not saying that was your intention.)

Again, this applies to the comic continuity, and I know nothing whatsoever about the films. But you have to admit “The genetic code … for mutation” makes much more sense if they were aiming to reference this.

(sorry for replying to a 2007 comment, but I hate to think you’d go off thinking this was an example of the uneducated masses thinking evolution consists of animals somehow mutating fully functional organs etc. not present in the parents.)

[Bound_up]

Eh, you sound just a tad smug. But...mmmmmm

maybe you earned it ;)

[CarlShulman]

(http://​​en.wiktionary.org/​​wiki/​​evolution general: a gradual process of development biology: change in the genetic composition of a population over time)

Eliezer,

It seems that when we talk about “evolution by natural selection” as opposed to “Lamarckian evolution,” evolution is the explanandum, and evolutionary theory attempts to provide the explanans.

“I encounter people who very definitely believe in evolution, who sneer at the folly of creationists. And yet they have no idea of what the theory of evolutionary biology permits and prohibits.” One can legitimately sneer at the folly of creationists based on evidence for the explanandum of evolutionary biology (dinosaur bones, dog breeding, Neanderthals, and comparative anatomy) and the creationist response. This level of understanding is inferior to one that incorporates evolutionary theory, but is not simply attire.

“Or even worse, they’ll talk about something completely outside the domain of evolutionary biology, like an improved design for computer chips, or corporations splitting, or humans uploading themselves into computers, and they’ll call that “evolution”. If evolutionary biology could cover that, it could cover anything.” The general sense of the word ‘evolution’ outside of biology is established in English, and some of the people who use it to describe phenomena like ‘the evolution of technology’ do understand evolutionary theory.

[Eliezer Yudkowsky]

The general sense of the word ‘evolution’ outside of biology is established in English, and some of the people who use it to describe phenomena like ‘the evolution of technology’ do understand evolutionary theory.

Okay, but there’s also people who say, “Corporations split—therefore they reproduce—therefore they evolve.” These are the people I’m talking about.

[Ronny Fernandez]

Ok, point taken.

[anonymous7]

Eliezer, are you unaware of the fact that biological evolution is only a subset from general evolution?

[Stuart_Armstrong]

Science (unlike religion) has proved its myths See, this is exactly the sort of thing I have a problem with. Science is not just magic that works. People are learning science as though it were merely a true religion: passwords, attire, professions, and all.

And I agree, that’s terrible. Maybe I wasn’t clear in my post: it’s a disaster for people to learn science as magic. But for people who won’t learn it, for whatever reason, then “science as magic that works” is a sensible view that gives them a cheap tool to assess scientific claims.

But even for people learning science, the fact that “science works” is still critical. That needs to come before anything else. Building models and making them pay for inaccuracies is the correct thing to do… because it works. This is a feature of our world. If our world were run as a huge morality tale by a new testament god with low-brow litterary tastes, then advanced model-building wouldn’t work and doing so would be the wrong thing to do. Scientists in that world would be just a Pythagorean sect, proposing a way of doing stuff that didn’t lead to anything. We need to have some indication that we are not in that sort of world, before we decide to learn the scientific method. The fact that science “put men on the moon” is such an indication.

Then, once we accept that science and rationality work, and decide to study them… that is the moment to put all thoughts of magic away.

The press has reports with things like “scientists attack creationist teachings”, but I’ve never seen “scientists attack common misconceptions about evolutionary theory”. Press selectivity. Trust me, they do.

I know they do, I don’t lack biologists among my friends. I’ve even been in that position myself, trying to inject a proper understanding of evolution into the heads of people whose reasonings are “Telepathy would be useful (and Star Trek has it), so we will soon evolve it”.

But if people have no true understanding of the scientific process (and no desire or need to learn more) then the most accurate picture they can get, for the least effort, is defering to popular press articles on scientific discoveries.

I wasn’t aware that speech, empathy, and social skills were functions of the kidneys rather than the brain. Never implied they were. But they’re definetly not rational thinking, according to the definition you’re been using. But this is far away from the main discussion, anyway.

[Stuart_Armstrong]

Damn the double post! Sorry for that; my incompetence is to blame.

“Science (unlike religion) has proved its myths—by putting men on the moon, mobile phones in people’s pockets, and curing diseases. It’s payed its dues to reliability. So unless I am willing to look into it myself, I should, as a default, believe most things scientists claim.” Stuart, this in my opinion is an awful sentence, and I’m surprised to read it by an overcomingbias contributor.

It lacks something of the pithiness of “Veni, vedi, Vinci”, I’ll admit. But I stand by what I was trying to say; hopefully my more recent post articulates it better.

[Kyle]

“Corporations split—therefore they reproduce—therefore they evolve.”

Okay, now those guys have issues.

[tlhonmey]

Yes… The splitting has absolutely nothing to do with the process in the case of a corporation. And there’s no strict inheritance…

But there is some similarity. Corporations which have a good internal structure and culture are more likely to survive. And they longer they survive the more likely they will be used as role models by other corporations. So it’s the same kind of feedback loop that drives natural selection, only with a very different set of constraints and able to access the processing power of more advanced computing machines than just DNA.

[Hopefully_Anonymous]

anonymous, I think we have good empirical evidence that Eliezer is not “unaware of the fact that biological evolution is only a subset from general evolution”.

Eliezer and Kyle, name names of serious or influential people who posit in a mockworthy “Corporations split—therefore they reproduce—therefore they evolve.”

If you’re just throwing up a foil so we have a smug sense of in-groupedness, are you wasting our time on an overcomingbias blog?

[Eliezer Yudkowsky]

There is no “general evolution”. There is biological evolution. Period. Saying “small incremental changes over time” is not a causal model, it is a surface effect to be explained. If you are talking about the realm of causal forces, of underlying processes, then biological evolution is all there is in science. Natural selection IS NOT a special case of some deeper principle that also explains change in toaster ovens any more than gravity is a special case of a deeper principle that also makes the stock market “fall”.

HA: Not that I have anything against the guy, but, Kevin Kelly.

[Hopefully_Anonymous]

Eliezer, you seem to be making a science vs. engineering distinction. You’re obviously aware of how evolution is used in engineering (as described in the wikipedia entry on evolution).

Took a look at Kevin Kelly’s site. Instead of occasional foilicious potshots, how about a serious critique of the error of these ideas. Let’s not manufacture a dialectic, I think that’s going to get in the way of building the best models of reality.

[Eliezer Yudkowsky]

HA, I don’t remember where all the fallacies I encounter come from. I have a difficult enough time remembering someone’s name after speaking to them for four hours. But before you accuse me of manufacturing strawmen, spend eleven years in my shoes putting up with the likes of this:

http://​​science.slashdot.org/​​comments.pl?&sid=191964

[Hopefully_Anonymous]

Eliezer, I understand the need to contest the wrong-but-influential, but not the wrong-but-insignificant. Kevin Kelly is definitely influential. I just think you have real, worthy opponents (the ultimate one, our apparently pending mortality), I don’t like to see limited energy get sucked up on hack, performed disagreements. I’d rather you get your representational privilege the most useful way to us—solving the hard problems we face, as quickly as possible, not performing ¹⁄₂ of various dialectics.

[Eliezer Yudkowsky]

HA, most influential folks are far beyond persuading. It is the hearts and minds of the unconvinced, who are often novices, for which I fight.

[michael_vassar3]

Influential usually implies higher status. Status makes people effectively stupid, as it makes it harder for them to update their public positions without feeling that they are loosing face.

[Justcurious]

Why this? “because they want to be part of the scientific in-crowd”

Why would everybody want to be part of the “scientific” in-crowd? That wouldn’t leave much for the rest of the world.

[Ronny Fernandez]

Or even worse, they’ll talk about something completely outside the domain of evolutionary biology, like an improved design for computer chips, or corporations splitting, or humans uploading themselves into computers, and they’ll call that “evolution”. If evolutionary biology could cover that, it could cover anything.

That seems unfair. Do they mean evolutionary biology when they say evolution? What if they just mean heredity, variation, and selection? I use “evolution” to describe how technologies spread throughout culture, because it spreads by those three mechanisms, and those seem to be the mechanisms of biology as well. I think of evolution as a class of algorithms which optimize for something (or “select”) without doing prediction, but restraining the search space by keeping only those that pass certain tests, and trying only those nodes out which are closed to ones that passed. and we see the failures in optimization we would expect from such an algorithm in many non-biological mechanisms.

[jooyous]

I hate how movies promote fake ‘science’ like that and then completely disregard actual science that can be used in an exciting movie. For example: we had the awful, fancy-looking cryptographic thinger in Skyfall that leaked its own key! What?! It leaked its own key!! Meanwhile, an action/​heist movie based around Shamir’s secret sharing basically writes itself! There are k shares and we must travel around the world to collect them! That masked man is running away with one of them, chase him! Oh wait, it fell into the river and then exploded! Now we must journey into the mountains to find the other one! But then we shoot this dying man as he is saying there were really k+1 shares! Dramatic ending!

Actually, now that I think about it, that’s exactly what Voldemort did.

[DreamFlasher]

key take-away:

• Many people can’t judge the difference between plausible and implausible scientific explanations, for them it’s just ‘science’

• Science as a new authority to explain things (similar to religion)

[amba12]

Much deeper than attire, it’s folk religion. “Science” is to our time what the One Church Catholic and Universal was to the fourteenth century — the source of cosmology, explainer of existence, consoler for mortality, generator of culture. The people you cite are analogous to those buying saints’ amulets, genuflecting in church on Sundays, hanging a cross over their bed, reflexively repeating the prayers. They do not have the profound understanding of the “theologians”—the scientists. (How many peasants do you think could explain transubstantiation?) In fact, they have all kinds of wild and superstitious misunderstandings.

[EniScien]

In general, when I first read the question about the abilities of the storm, I was so distracted by the fact that I had no idea how it was physically possible that I did not pay attention to the actual mutations and the impossibility from the point of view of the theory of evolution, for me it sounded like Access to Magic and The Atlantean Genetic Marker, which no longer looked so implausible to me.

[deleted]