“Before anyone posts any angry comments: yes, the registration costs actual money this year.”
For comparison: The Singularity Summit at Stanford cost $110K, all of which was provided by SIAI and sponsors. Singularity Summit 2007 undoubtedly cost more, and only $50K of that was raised through ticket sales. All ticket purchases for SS08 will be matched 2:1 by Peter Thiel and Brian Cartmell.
My apologies, but my browser screwed up my comment’s formatting; could an admin please fix it, and then delete this? Thanks.
“Ask anyone, and they’ll say the same thing: they’re pretty open-minded, though they draw the line at things that are really wrong.”
I generally find myself arguing against open-mindedness; because “open-mindedness” is a social virtue, a lot of people apply it indiscriminately, and so they wind up wasting time on long-debunked ideas.
“In the same way that we need statesmen to spare us the abjection of exercising power, we need scholars to spare us the abjection of learning.”
How many people want to exercise government-type power over large numbers of people? A lot of people are, apparently, happy to let someone else tell them what to do. Most of the rest aren’t very ambitious.
“Because giftedness is not to be talked about, no one tells high-IQ children explicitly, forcefully and repeatedly that their intellectual talent is a gift. That they are not superior human beings, but lucky ones. That the gift brings with it obligations to be worthy of it.”
(remembers childhood)
When adults did tell me this, I didn’t believe them- after all, wasn’t it blatantly obvious that there was a strong negative correlation between intelligence and quality of life?
“The best part about math is that, if you have the right answer and someone disagrees with you, it really is because they’re stupid.”
This is true, but only for arbitrarily low values of “stupid”. There are plenty of theorems which are obvious for a superintelligence, but counterintuitive to humans.
“Long-Term Capital Management had faith in diversification. Its history serves as ample notification that eggs in different baskets can and do all break at the same time.”
If I recall correctly, LTCM was so highly leveraged that most of their eggs didn’t have to break- if just 10% or so did, they were hosed anyway.
“In fact, if you’re interested in the field, you should probably try counting the ways yourself, before I continue. And score yourself on how deeply you stated a problem, not just the number of specific cases.”
I got #1, but I mushed #2 and #3 together into “The AI will rewire our brains into computationally cheap super-happy programs with humanesque neurology”, as I was thinking of failure modes and not reasons for why failure modes would be bad.
“The real question is when “Because Eliezer said so!” became a valid moral argument.”
You’re confusing the algorithm Eliezer is trying to approximate with the real, physical Eliezer. If Eliezer was struck by a cosmic ray tomorrow and became a serial killer, me, you, and Eliezer would all agree that this doesn’t make being a serial killer right.
“Tom McCabe: speaking as someone who morally disapproves of murder, I’d like to see the AI reprogram everyone back, or cryosuspend them all indefinitely, or upload them into a sub-matrix where they can think they’re happily murdering each other without all the actual murder. Of course your hypothetical murder-lovers would call this immoral, but I’m not about to start taking the moral arguments of murder-lovers seriously.”
Beware shutting yourself into a self-justifying memetic loop. If you had been born in 1800, and just recently moved here via time travel, would you have refused to listen to all of our modern anti-slavery arguments, on the grounds that no moral argument by negro-lovers could be taken seriously?
“The AI would use the previous morality to select its actions: depending on the content of that morality it might or might not reverse the reprogramming.”
Do you mean would, or should? My question was what the AI should do, not what a human-constructed AI is likely to do.
It should be possible for an AI, upon perceiving any huge changes in renormalized human morality, to scrap its existing moral system and recalibrate from scratch, even if nobody actually codes an AI that way. Obviously, the previous morality will determine the AI’s very next action, but the interesting question is whether the important actions (the ones that directly affect people) map on to a new morality or the previous morality.
“You perceive, of course, that this destroys the world.”
If the AI modifies humans so that humans want whatever happens to already exist (say, diffuse clouds of hydrogen), then this is clearly a failure scenario.
But what if the Dark Lords of the Matrix reprogrammed everyone to like murder, from the perspective of both the murderer and the murderee? Should the AI use everyone’s prior preferences as morality, and reprogram us again to hate murder? Should the AI use prior preferences, and forcibly stop everyone from murdering each other, even if this causes us a great deal of emotional trauma? Or should the AI recalibrate morality to everyone’s current preferences, and start creating lots of new humans to enable more murders?
“However, those objective values probably differ quite a lot from most of what most human beings find important in their lives; for example our obsessions with sex, romance and child-rearing probably aren’t in there.”
Several years ago, I was attracted to pure libertarianism as a possible objective morality for precisely this reason. The idea that, eg., chocolate tastes good can’t possibly be represented directly in an objective morality, as chocolate is unique to Earth and objective moralities need to apply everywhere. However, the idea of immorality stemming from violation of another person’s liberty seemed simple enough to arise spontaneously from the mathematics of utility functions.
It turns out that you do get a morality out of the mathematics of utility functions (sort of), in the sense that utility functions will tend towards certain actions and away from others unless some special conditions are met. Unfortunately, these actions aren’t very Friendly; they involve things like turning the universe into computronium to solve the Riemann Hypothesis (see http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf for some examples). If libertarianism really was a universal morality, Friendly AI would be much simpler, as we could fail on the first try without the UFAI killing us all.
“is true except where general intelligence is at work. It probably takes more complexity to encode an organism that can multiply 7 by 8 and can multiply 432 by 8902 but cannot multiply 6 by 13 than to encode an organism that can do all three,”
This is just a property of algorithms in general, not of general intelligence specifically. Writing a Python/C/assembler program to multiply A and B is simpler than writing a program to multiply A and B unless A % B = 340. It depends on whether you’re thinking of multiplication as an algorithm or a giant lookup table (http://lesswrong.com/lw/l9/artificial_addition/).
“Eventually, the good guys capture an evil alien ship, and go exploring inside it. The captain of the good guys finds the alien bridge, and on the bridge is a lever. “Ah,” says the captain, “this must be the lever that makes the ship dematerialize!” So he pries up the control lever and carries it back to his ship, after which his ship can also dematerialize.”
This type of thing is known to happen in real life, when technology gaps are so large that people have no idea what generates the magic. See http://en.wikipedia.org/wiki/Cargo_cult.
“You will find yourself saying, “If I wanted to kill someone—even if I thought it was right to kill someone—that wouldn’t make it right.” Why? Because what is right is a huge computational property- an abstract computation—not tied to the state of anyone’s brain, including your own brain.”
Coherent Extrapolated Volition (or any roughly similar system) protects against this failure for any specific human, but not in general. Eg., suppose that you use various lawmaking processes to approximate Right(x), and then one person tries to decide independently that Right(Murder) > 0. You can detect the mismatch between the person’s actions and Right(x) by checking against the approximation (the legal code) and finding that murder is wrong. In the limit of the approximation, you can detect even mismatches that people at the time wouldn’t notice (eg., slavery). CEV also protects against specific kinds of group failures, eg., convince everybody that the Christian God exists and that the Bible is literally accurate, and CEV will correct for it by replacing the false belief of “God is real” with the true belief of “God is imaginary”, and then extrapolating the consequences.
However, CEV can’t protect against features of human cognitive architecture that are consistent under reflection, factual accuracy, etc. Suppose that, tomorrow, you used magical powers to rewrite large portions of everyone’s brain. You would expect that people now take actions with lower values of Right(x) than they previously did. But, now, there’s no way to determine the value of anything under Right(x) as we currently understand it. You can’t use previous records (these have all been changed, by act of magic), and you can’t use human intuition (as it too has been changed). So while the external Right(x) still exists somewhere out in thingspace, it’s a moot point, as nobody can access it. This wouldn’t work for, say, arithmetic, as people would rapidly discover that assuming 2 + 2 = 5 in engineering calculations makes bridges fall down.
Wow, there’s a lot of ground to cover. For everyone who hasn’t read Eliezer’s previous writings, he talks about something very similar in Creating Friendly Artificial Intelligence, all the way back in 2001 (link = http://www.singinst.org/upload/CFAI/design/structure/external.html). With reference to Andy Wood’s comment:
“What claim could any person or group have to landing closer to the one-place function?”
Next obvious question: For purposes of Friendly AI, and for correcting mistaken intuitions, how do we approximate the rightness function? How do we determine whether A(x) or B(x) is a closer approximation to Right(x)?
Next obvious answer: The rightness function can be computed by computing humanity’s Coherent Extrapolated Volition, written about by Eliezer in 2004 (http://www.singinst.org/upload/CEV.html). The closer a given algorithm comes to humanity’s CEV, the closer it should come to Right(x).
Note: I did not think of CFAI when I read Eliezer’s previous post, although I did think of CEV as a candidate for morality’s content. CFAI refers to the supergoals of agents in general, while all the previous posts referred to a tangle of stuff surrounding classic philosophical ideas of morality, so I didn’t connect the dots.
“I don’t think I’ve ever touched anything that has endured in the world for longer than that church tower.”
Nitpick: This probably holds true for things of human construction, but there are obviously rocks, bits of dirt, etc. that have endured for far longer than a thousand years.
“What concrete state of the world—which quarks in which positions—corresponds to “There are three apples on the table, and there could be four apples on the table”? Having trouble answering that? Next, say how that world-state is different from “There are three apples on the table, and there couldn’t be four apples on the table.”″
For the former: An ordinary kitchen table with three apples on it. For the latter: An ordinary kitchen table with three apples on it, wired to a pressure-sensitive detonator that will set off 10 kg of C4 if any more weight is added onto the table.
“But “I could have a heart attack at any time” and “I could have a heart attack any time I wanted to” are nonetheless not exactly the same usage of could, though they are confusingly similar.”
They both refer to possible consequences if the initial states were changed, while still obeying a set of constraints. The first refers to a change in initial external states (“there’s a clot in the artery”/”there’s not a clot in the artery”), while the second refers to a change in initial internal states (“my mind activates the induce-heart-attack nerve signal”/”my mind doesn’t activate the induce-heart-attack nerve signal”). Note that “could” only makes sense if the initial conditions are limited to a pre-defined subset. For the above apple-table example, in the second case, you would say that the statement “there could be four apples on the table” is false, but you have to assume that the range of initial states the “could” refers to don’t refer to states in which the detonator is disabled. For the heart-attack example, you have to exclude initial states in which the Mad Scientist Doctor (tm) snuck in in the middle of the night and wired up a deliberation-based heart-attack-inducer.
“But if we assume that Lenin made his decisions after the fashion of an ordinary human brain, and not by virtue of some alien mechanism seizing and overriding his decisions, then Lenin would still be exactly as much of a jerk as before.”
I must admit that I still don’t really understand this. It seems to violate what we usually mean by moral responsibility.
“When, in a highly sophisticated form of helpfulness, I project that you would-want lemonade if you knew everything I knew about the contents of the refrigerator, I do not thereby create a copy of Michael Vassar who screams that it is trapped inside my head.”
This is, I think, because humans are a tiny subset of all possible computers, and not because there’s a qualitative difference between predicting and creating. It is, for instance, possible to look at a variety of factorial algorithms, and rearrange them to predictably compute triangular numbers. This, of course, doesn’t mean that you can look at an arbitrary algorithm and determine whether it computes triangular numbers. I conjecture that, in the general case, it’s impossible to predict the output of an arbitrary Turing machine at any point along its computation without doing a calculation at least as long as the calculations the original Turing machine does. Hence, predicting the output of a mind-in-general would require at least as much computing power as running the mind-in-general.
Incidentally, I think that there’s a selection bias at work here due to our limited technology. Since we don’t yet know how to copy or create a human, all of the predictions about humans that we come up with are, by necessity, easier than creating a human. However, for most predictions on most minds, the reverse should be true. Taking Michael Vassar and creating an electronic copy (uploading), or creating a human from scratch with a set of prespecified characteristics, are both technologically feasible with tools we know how to build. Creating a quantum simulation of Michael Vassar or a generic human to predict their behavior would be utterly beyond the processing power of any classical computer.
“One of the things that always comes up in my mind regarding this is the concept of space relative to these other worlds. Does it make sense to say that they’re “ontop of us” and out of phase so we can’t see them, or do they propagate “sideways”, or is it nonsensical to even talk about it?”
It’s nonsensical. The space that we see is just an artifact of a lower level of reality. See http://www.acceleratingfuture.com/tom/?p=124.
“And you should always take joy in discovery, as long as you personally don’t know a thing.”
I generally give independent, replicated discoveries the same “joy status” (if that makes sense) as first-time-in-this-branch discoveries. However, you should take a hit when you’re just rereading someone else’s work, which isn’t as challenging, or as fun.
I really, really hope that you aren’t going to try and publish a theory of quantum gravity, for practical reasons; even if it’s more elegant than every other theory yet proposed, the lack of experimental evidence and your lack of credentials will make you seem like a crackpot.
First of all, to Eliezer: Great post, but I think you’ll need a few more examples of how stupid chimps are compared to VIs and how stupid Einsteins are compared to Jupiter Brains to convince most of the audience.
“Maybe he felt that the difference between Einstein and a village idiot was larger than between a village idiot and a chimp. Chimps can be pretty clever.”
We see chimps as clever because we have very low expectations of animal intelligence. If a chimp were clever in human terms, it would be able to compete with humans in at least some areas, which is clearly silly. How well would an adult chimp do, if he was teleported into a five-year-old human’s body and thrown into kindergarten?
“But I don’t buy the idea of intelligence as a scalar value.”
Intelligence is obviously not a scalar, but there does seem to be a scalar component of intelligence, at least when dealing with humans. It has long been established that intelligence tests strongly correlate with each other, forming a single scalar known as Spearman’s g (http://en.wikipedia.org/wiki/General_intelligence_factor), which correlates with income, education, etc.
“2) you’re handwaving away deep problems of knowledge and data processing by attributing magical thought powers to your AI.”
Yes. If you have a way to solve those problems, and it’s formal and comprehensive enough to be published in a reputable journal, I will pay you $1,000. Other people on OB will probably pay you much more. Until then, we do the best we can.
“as opposed to simply stating that it could obviously do those things because it’s a superintelligence.”
See the previous post at http://lesswrong.com/lw/qk/that_alien_message/ for what simple overclocking can do.
“We haven’t even established how to measure most aspects of cognitive function—one of the few things we know about how our brains work is that we don’t possess tools to measure most of the things it does.”
Er, yes, we do, actually. See http://lesswrong.com/lw/kj/no_one_knows_what_science_doesnt_know/.
“Some people can do it without much effort at all, and not all of them are autistic, so you can’t just say that they’ve repurposed part of their brain for arithmetic.”
Since when is autism necessary for brain repurposing? Autism specifically refers to difficulty in social interaction and communication. Savantism is actually an excellent example of what we could do with the brain if it worked efficiently.
“By the way, when the best introduction to a supposedly academic field is works of science fiction, it sets off alarm bells in my head. I know that some of the best ideas come from sci-fi and yada, yada, but just throwing that out there.”
Sci-fi is useful for introducing the reader to the idea that there are possibilities for civilization other than 20th-century Earth. It’s not meant to be technical material.
“But I’m skeptical that this uniformity extends to system II. The system II abilities of the best rationalists of today may depend significantly on their having learned a set of reasoning skills developed by their culture over a long period of time.”
That’s precisely the point; the biological difference between humans is not that great, so the huge differences we see in human accomplishment must be due in large part to other factors.
“The simplest best theory we have for precisely predicting an arbitrary 12 grams of carbons behaviour over time requires avogadros of data for the different degrees of freedom of the start state, the electron energy states etc.”
No, it doesn’t; the Standard Model only has eighteen adjustable parameters (physical constants) that must be found through experiment.
“The minor tweaks in brain design allowed enormous improvements in cognitive performance, and I think that the intelligence scale should reflect the performance differences rather than the anatomical ones.”
The difference between humans and chimps is fairly small anatomically; we share 95-98% of our DNA and most of our brain architecture. The huge difference between a civilization inhabited entirely by village idiots and a civilization of chimps is obvious.
“Eliezer, I think this whole frame of analysis has an element of ego-stroking/sour grapes (stroking your ego and perhaps the ego of your reading audience that defines brainy as being Einstein-like, and that defines social success as being inversely correlated, because y’all are more Einstein-like than you’re socially successful).”
Social success will gradually become more irrelevant as society develops further, because social success is a zero-sum game; it doesn’t produce anything of value. Dogs, orangutans, and chimps all have complex social structures. Dogs, orangutans, and chimps would all currently be extinct if we didn’t have domesticated animals and environmentalists.
“The empiricism based seduction community indicates a braininess advantage in being able “to play well with the other kids”.”
If you define braininess as social success, social success is obviously going to correlate with braininess. The ability to find an optimal mate is not why people are successful. Monks, who were the closest thing to scholars during the medieval period, explicitly renounced the quest for a mate, and they didn’t do too badly by the standards of their time period.
“I’ve resisted this thread, but I’m more interested in James Simon and the google founders as an example as the high end of braininess than the Albert Einsteins of today.”
If you’re referring to this James Simon (http://en.wikipedia.org/wiki/James_Simon), he is obviously less accomplished than Newton, Einstein, etc., by any reasonable metric. Larry Page and Sergey Brin are rich primarily because they were more interested in being rich than in publishing papers. They sure as heck didn’t become rich because they knew how to win a high school popularity contest; Bill Gates, the most famous of the dot-com billionaires, is widely reputed to be autistic.
“Celeriac, the distinction is that Tom McCabe seemed to me to be suggesting that the search space was small to begin with—rather than realizing the work it took to cut the search space itself down.”
The search space, within differential geometry, was fairly small by Einstein’s day. It was a great deal of work to narrow the search space, but most of it was done by others (Conservation of Energy, various mathematical theorems, etc., were all known in 1910). The primary difficulties were in realizing that space could be described by differential geometry, and then in deriving GR from known postulates. Neither of these involve large search spaces; the former follows quickly once you realize that your assumptions are inconsistent with Minkowski space, and there’s only one possible derivation of GR if you do the math correctly. I don’t know why the first one is hard, but Einstein showed twice that physicists are very reluctant to question background assumptions (linear time for SR, Euclidean space for GR), so we know it must be. The second one is hard because the human brain does not come equipped with a differential geometry lobe- it took me several hours to fully understand the derivation of the Schwarzschild solution from its postulates, even though the math is simple by GR standards and there is only one possible answer (see http://en.wikipedia.org/wiki/Deriving_the_Schwarzschild_solution).
“And I wonder if that advice will turn out not to help most people, until they’ve personally blown off their own foot, saying to themselves all the while, correctly, “Clearly I’m winning this argument.”″
I fell into this pattern for quite a while. My basic conception was that, if everyone presented their ideas and argued about them, the best ideas would win. Hence, arguing was beneficial for both me and the people on transhumanist forums- we both threw out mistaken ideas and accepted correct ones. Eliezer_2006 even seemed to support my position, with Virtue #5. It never really occurred to me that the best of everyone’s ideas might not be good enough.
“It is Nature that I am facing off against, who does not match Her problems to your skill, who is not obliged to offer you a fair chance to win in return for a diligent effort, who does not care if you are the best who ever lived, if you are not good enough.”
Perhaps we should create an online database of open problems, if one doesn’t exist already. There are several precedents (http://en.wikipedia.org/wiki/Hilbert%27s_problems). So far as I know, if one wishes to attack open problems in physics/chemistry/biology/comp. sci./FAI, the main courses of action are to attack famous problems (where you’re expected to fail and don’t feel bad if you do), or to read the educational literature (where the level of problems is pre-matched to the level of the material).