We’ve had various kinds of Luddism before, but this one is particularly lethal in being a form that appeals to people who had been technophiles. If it spreads enough, best case scenario is the pool of people willing to work on real technological progress shrinks, worst case scenario is regulation that snuffs out progress entirely, and we get to sit around bickering about primate politics until whatever window of time we had runs out.
rwallace
Karma: 1,807
Well, any sequence of events can be placed in a narrative frame with enough of a stretch, but the fact remains that different sequence of events differ in their amenability to this; fiction is not a random sampling from the space of possible things we could imagine happening, and the Singularity is narratively far stronger than most imaginable futures, to a degree that indicates bias we should correct for. I’ve seen a fair bit of strong Singularity fiction at this stage, though being, well, singular, it tends not to be amenable to repeated stories by the same author the way Heinlein’s vision of nuclear powered space colonization was.
We should update away from beliefs that the future will resemble a story, particularly a story whose primary danger will be fought by superheroes (most particularly for those of us who would personally be among the superheroes!) and towards beliefs that the future will resemble the past and the primary dangers will be drearily mundane.
Okay, to look at some of the specifics:
Superior processing power. Evidence against would be the human brain already being close to the physical limits of what is possible.
The linked article is amusing but misleading; the described ‘ultimate laptop’ would essentially be a nuclear explosion. The relevant physical limit is ln(2)kT energy dissipated per bit erased; in SI units at room temperature this is about 4e-21. We don’t know exactly how much computation the human brain performs; middle-of-the-road estimates put it in the ballpark of 1e18 several-bit operations per second for 20 watts, which is not very many orders of magnitude short of even the theoretical limit imposed by thermodynamics, let alone whatever practical limits may arise once we take into account issues like error correction, communication latency and bandwidth, and the need for reprogrammability.
Superior serial power: Evidence against would be an inability to increase the serial power of computers anymore.
Indeed we hit this some years ago. Of course as you observe, it is impossible to prove serial speed won’t start increasing again in the future; that’s inherent in the problem of proving a negative. If such proof is required, then no sequence of observations whatsoever could possibly count as evidence against the Singularity.
Superior parallel power:
Of course uses can always be found for more parallel power. That’s why we humans make use of it all the time, both by assigning multiple humans to a task, and increasingly by placing multiple CPU cores at the disposal of individual humans.
Improved algorithms:
Finding these is (assuming P!=NP) intrinsically difficult; humans and computers can both do it, but neither will ever be able to do it easily.
Designing new mental modules:
As for improved algorithms.
Modifiable motivation systems:
An advantage when they reduce akrasia, a disadvantage when they make you more vulnerable to wireheading.
Copyability: Evidence against would be evidence that minds cannot be effectively copied, maybe because there won’t be enough computing power to run many copies.
Indeed there won’t, at least initially; supercomputers don’t grow on trees. Of course, computing power tends to become cheaper over time, but that does take time, so no support for hard takeoff here.
Alternatively, that copying minds would result in rapidly declining marginal returns and that the various copying advantages discussed by e.g. Hanson and Shulman aren’t as big as they seem.
Matt Mahoney argues that this will indeed happen because an irreducible fraction of the knowledge of how to do a job is specific to that job.
Perfect co-operation:
Some of the more interesting AI work has been on using a virtual market economy to allocate resources between different modules within an AI program, which suggests computers and humans will be on the same playing field.
Superior communication:
Empirically, progress in communication technology between humans outpaces progress in AI, and has done so for as long as digital computers have existed.
Transfer of skills:
Addressed under copyability.
Various biases:
Hard to say, both because it’s very hard to see our own biases, and because a bias that’s adaptive in one situation may be maladaptive in another. But if we believe maladaptive biases run deep, such that we cannot shake them off with any confidence, then we should be all the more skeptical of our far beliefs, which are the most susceptible to bias.
Of course, there is also the fact that humans can and do tap the advantages of digital computers, both by running software on them, and in the long run potentially by uploading to digital substrate.
I discuss some of it at length here: http://lesswrong.com/lw/312/the_curve_of_capability/
I’ll also ask the converse question: given that you can’t typically prove a negative (I can’t prove the nonexistence of psychic powers or flying saucers either), if what we are observing doesn’t constitute evidence against the Singularity in your opinion, then what would?
I understand perfectly well how a hypothetical perfectly logical system would work (leaving aside issues of computational tractability etc.). But then, such a hypothetical perfectly logical system wouldn’t entertain such far mode beliefs in the first place. What I’m discussing is the human mind, and the failure modes it actually exhibits.
So your suggestion is that we should de-compartmentalize, but in the reverse direction to that suggested by the OP, i.e. instead of propagating forward from ridiculous far beliefs, become better at back-propagating and deleting same? There is certainly merit in that suggestion if it can be accomplished. Any thoughts on how?
That’s actually a good question. Let me rephrase it to something hopefully clearer:
Compartmentalization is an essential safety mechanism in the human mind; it prevents erroneous far mode beliefs (which we all adopt from time to time) from having disastrous consequences. A man believes he’ll go to heaven when he dies. Suicide is prohibited in a patch for the obvious problem, but there’s no requirement to make an all-out proactive effort to stay alive. Yet when he gets pneumonia, he gets a prescription for penicillin. Compartmentalization literally saves his life. In some cases many other lives, as we saw when it failed on 9/11.
Here we have a case study where a man of intelligence and goodwill redirected his entire life down a path of negative utility on the basis of reading a single paragraph of sloppy wishful thinking backed up by no evidence whatsoever. (The most straightforward refutation of that paragraph is that creating a machine with even a noteworthy fraction of human intelligence is far beyond the capacity of any human mind; the relevant comparison of such a machine if built would be with that which created it, which would have to be a symbiosis of humanity and its technology as a whole—with that symbiosis necessarily being much more advanced than anything we have today.) What went wrong?
The most obvious part of the answer is that this is an error to which we geeks are particularly prone. (Supporting data: terrorists are disproportionately likely to be trained in some branch of engineering.) Why? Well, we are used to dealing in domains where we can actually apply long chains of logic with success; particularly in the age range when we are old enough to have forgotten how fallible were our first attempts at such logic, yet young enough to be still optimists, it’s an obvious trap to fall into.
Yet most geeks do actually manage to stay out of the trap. What else goes wrong?
It seems to me that there must be a parameter in the human mind for grasping the inertia of the world, for understanding at a gut level how much easier is concept than reality, that we can think in five minutes of ideas that the labor of a million people for a thousand years cannot realize. I suppose in some individuals this parameter must be turned up too high, and they fall too easily into the trap of learned helplessness. And in some it must be turned too low, and those of us for whom this is the case undertake wild projects with little chance of success; and if ninety-nine fail for every one who succeeds, that can yet drive the ratchet of progress.
But we easily forget that progress is not really a ratchet, and the more advanced our communications, the more lethal bad ideas become, for just as our transport networks spread disease like the 1918 flu epidemic which killed more people in a single year than the First World War killed in four years, so our communication networks spread parasite memes deadlier still. And we can’t shut down the networks. We need them too badly.
I’ve seen the Singularity mutate from a harmless, even inspiring fantasy, to a parasite meme that I suspect could well snuff out the entire future of intelligent life. It’s proving itself in many cases immune to any weight of evidence against it; perhaps worst of all, it bypasses ethical defenses, for it can be spread by people of honest goodwill.
Compartmentalization seems to be the primary remaining defense. When that fails, what have we left? This is not a rhetorical question; it may be one of the most important in the world right now.
You are worried that, given your assumptions, civilizations might not be willing to pay an extremely high price to do things that aliens would like if they knew about them, which they don’t.
But one of your assumptions is that every civilization has a moral system that advocates attacking and enslaving everyone they meet who thinks differently from them.
It would be worrying if a slightly bad assumption led to a very bad conclusion, but a very bad assumption leading to a slightly bad conclusion doesn’t strike me as particularly problematic.
Well yes. You give this list of things you claim are universal instrumental values, and it sounds like a plausible idea in our heads, but when we look at the real world, we find humans and other agents tend not in fact possess these, even as instrumental values.
In the case of chess programs, the argument is simply false. Chess programs do not in fact exhibit anything remotely resembling the described behavior, nor would they do so even if given infinite computing power. This despite the fact that they exhibit extremely high performance (playing chess better than any human) and do indeed have a simple goal.
It is more that a wide range of simple goals gives rise to a closely-related class of behaviours
But that is only true by a definition of ‘simple goals’ under which humans and other entities that actually exist do not have simple goals. You can have a theory that explains the behavior that occurs in the real world, or you can have a theory that admits Omohundro’s argument, but they are different theories and you can’t use both in the same argument.
A mind very like humans’ that had the ability to test out new brain components and organizations seems like it would fit it.
Not really, because as you say, there are many constraints preventing it from being applicable, of which difficulty changing our brains’ design is just one, so with that constraint removed, the argument would still not be applicable.
Suppose we grant all this. Very well, then consider what conclusions we can draw from it about the behavior of the hypothetical AI originally under discussion. Clearly no matter what sequence of actions the AI were to carry out, we would be able to explain it with this theory. But a theory that can explain any observations whatsoever, makes no predictions. Therefore, contrary to Omohundro, the theory of optimization does not make any predictions about the behavior of an AI in the absence of specific knowledge of the goals thereof.
Any agent can be expressed as an. O-maximizer
Not just any agent, but any entity. A leaf blown on the wind can be thought of as optimizing the function of following the trajectory dictated by the laws of physics. Which is my point: if you broaden a theory to the point where it can explain anything whatsoever, then it makes no useful predictions.
Hmm—so: publicly soliciting personally identifiable expressions of murderous intent is probably not the best way of going about this
It was a rhetorical question. I’m confident the answer is no—the law only works when most people are basically honest. We think we have a goal, and so we do by the ordinary English meaning of the word, but then there are things we are not prepared to do to achieve it, so it turns out what we have is not a goal by the ultimate criterion of decision theory on which Omohundro draws, and if we try to rescue the overuse of decision theory by appealing to a broader goal, it still doesn’t work; regardless of what level you look at, there is no function such that humans will say “yes, this is my utility function, and I care about nothing but maximizing it.”
The idea of goals in the sense of decision theory is like the idea of particles in the sense of Newtonian physics—a useful approximation for many purposes, provided we remember that it is only an approximation and that if we get a division by zero error the fault is in our overzealous application of the theory, not in reality.
OK—but even plants are optimising. There are multiple optimisation processes
Precisely. There are many optimization processes—and none of them work the way they would need to work for Omohundro’s argument to be relevant.
quite a few definitions of intellligence are all to do with being goal-directed
Of the seventy-some definitions of intelligence that had been gathered last count, most have something to do with achieving goals. That is a very different thing from being goal-directed (which has several additional requirements, the most obvious being an explicit representation of one’s goals).
The main reason most humans don’t murder people to get what they want is because prison sentences confllict with their goals
Would you murder your next-door neighbor if you thought you could get away with it?
Most creatures are as goal-directed as evolution can make them
“As … as evolution can make them” is trivially true in that our assessment of what evolution can do is driven by what it empirically has done. It remains the case that most creatures are not particularly goal-directed. We know that bees stockpile honey to survive the winter, but the bees do not know this. Even the most intelligent animals have planning horizons of minutes compared to lifespans of years to decades.
Memetic parasites are quite significant for humans—but they will probably be quite significant for intelligent machines as well
Indeed, memetic parasites are quite significant for machines today.
Sure, and I’m saying his conclusion is only true for an at best very idiosyncratic definition of ‘sufficiently powerful’ - that the most powerful systems in real life are and will be those that are part of historical processes, not those that try to reinvent themselves by their bootstraps.
Humans and existing programs are approximately goal directed within limited contexts. You might have the goal of making dinner, but you aren’t willing to murder your next-door neighbor so you can fry up his liver with onions, even if your cupboard is empty. Omohundro postulates a system which, unlike any real system, throws unlimited effort and resources into a single goal without upper bound. Trying to draw conclusions about the real world from this thought experiment is like measuring the exponential increase in air velocity from someone sneezing, and concluding that in thirty seconds he’ll have blown the Earth out of orbit.
Sure, and my point is that when you look more closely, ‘sufficiently powerful’ translates to ‘actually pretty much nothing people have built or tried to build within any of these architectures would have this property, no matter how much power you put behind it; instead you would have to build a completely different system with very particular properties, that wouldn’t really use the aforementioned architectures as anything except unusually inefficient virtual machines, and wouldn’t perform well in realistic conditions.’
Or human communications may stop improving because they are good enough to no longer be a major bottleneck, in which case it may not greatly matter whether other possible minds could do better. Amdahl’s law: if something was already only ten percent of total cost, improving it by a factor of infinity would reduce total cost by only that ten percent.