Could someone who voted for unfriendly AI explain how nanotech or biotech isn’t much more of a risk than unfriendly AI (I’ll assume MIRI’s definition here)?
I ask this question because it seems to me that even given a technological singularity there should be enough time for “unfriendly humans” to use precursors to fully fledged artificial general intelligence (e.g. advanced tool AI) in order to solve nanotechnology or advanced biotech. Technologies which themselves will enable unfriendly humans to cause a number of catastrophic risks (e.g. pandemics, nanotech wars, perfect global surveillance (an eternal tyranny) etc.).
Unfriendly AI, as imagined by MIRI, seems to be the end product of a developmental process that provides humans ample opportunity to wreck havoc.
I just don’t see any good reason to believe that the tools and precursors to artificial general intelligence are not themselves disruptive technologies.
And in case you believe advanced nanotechnology to be infeasible, but unfriendly AI to be an existential risk, what concrete scenarios do you imagine on how such an AI could cause human extinction without nanotech?
Presumably many people fear a very rapid “hard takeoff” where the time from “interesting slightly-smarter-than-human AI experiment” to “full-blown technological singularity underway” is measured in at days (or less) rather than months or years.
The AI risk scenario that Eliezer Yudkowsky relatively often uses is that of the AI solving the protein folding problem.
If you believe a “hard takeoff” to be probable, what reason is there to believe that the distance between a.) an AI capable of cracking that specific problem and b.) an AI triggering an intelligence explosion is too short for humans to do something similarly catastrophic as what the AI would have done with the resulting technological breakthrough?
In other words, does the protein folding problem require AI to reach a level of sophistication that would allow humans, or the AI itself, within days or months, to reach the stages where it undergoes an intelligence explosion? How so?
My assumption is that the protein-folding problem is unimaginably easier than an AI doing recursive self-improvement without breaking itself.
Admittedly, Eliezer is describing something harder than the usual interpretation of the protein-folding problem, but it still seems a lot less general than a program making itself more intelligent.
Is this question equivalent to “Is the protein-folding problem equivalently hard to the build-a-smarter-intelligence-than-I-am problem?” ? It seems like it ought to be, but I’m genuinely unsure, as the wording of your question kind of confuses me.
If so, my answer would be that it depends on how intelligent I am, since I expect the second problem to get more difficult as I get more intelligent. If we’re talking about the actual me… yeah, I don’t have higher confidence either way.
Is this question equivalent to “Is the protein-folding problem equivalently hard to the build-a-smarter-intelligence-than-I-am problem?” ?
It is mostly equivalent. Is it easier to design an AI that can solve one specific hard problem than an AI that can solve all hard problems?
Expecting that only a fully-fledged artificial general intelligence is able to solve the protein-folding problem seems to be equivalent to believing the conjunction “an universal problem solver can solve the protein-folding problem” AND “an universal problem solver is easier to solve than the protein-folding problem”. Are there good reasons to believe this?
ETA: My perception is that people who believe unfriendly AI to come sooner than nanotechnology believe that it is easier to devise a computer algorithm to devise a computer algorithm to predict protein structures from their sequences rather than to directly devise a computer algorithm to predict protein structures from their sequences. This seems counter-intuitive.
it is easier to devise a computer algorithm to devise a computer algorithm to predict protein structures from their sequences rather than to directly devise a computer algorithm to predict protein structures from their sequences. This seems counter-intuitive.
Ah, this helps, thanks.
For my own part, the idea that we might build tools better at algorithm-development than our own brains are doesn’t seem counterintuitive at all… we build a lot of tools that are better than our own brains at a lot of things. Neither does it seem implausible that there exist problems that are solvable by algorithm-development, but whose solution requires algorithms that our brains aren’t good enough algorithm-developers to develop algorithms to solve.
So it seems reasonable enough that there are problems which we’ll solve faster by developing algorithm-developers to solve them for us, than by trying to solve the problem itself.
Whether protein-folding is one of those problems, I have absolutely no idea. But it sounds like your position isn’t unique to protein-folding.
For my own part, the idea that we might build tools better at algorithm-development than our own brains are doesn’t seem counterintuitive at all...
So you believe that many mathematical problems are too hard for humans to solve but that humans can solve all of mathematics?
I already asked Timothy Gowers a similar question and I really don’t understand how people can believe this.
In order to create an artificial mathematician it is first necssary to discover, prove and encode the mathematics of discovering and proving non-arbitrary mathematics (i.e. to encode a formalization of the natural language goal “be as good as humans at mathematics”). This seems much more difficult than solving any single problem. And that’s just mathematics...
Neither does it seem implausible that there exist problems that are solvable by algorithm-development, but whose solution requires algorithms that our brains aren’t good enough algorithm-developers to develop algorithms to solve.
I do not disagree with this in theory. After all, evolution is an example of this. But it was not computationally simple for evolution to do so and it did do so by a bottom-up approach, piece by piece.
So it seems reasonable enough that there are problems which we’ll solve faster by developing algorithm-developers to solve them for us, than by trying to solve the problem itself.
To paraphrase your sentence: It seems reasonable that we can design an algorithm that can design algorithms that we are unable to design.
This can only be true in the sense that this algorithm-design-algorithm would run faster on other computational substrates than human brains. I agree that this is possible. But are relevant algorithms in a class for which a speed advantage would be substantial?
Again, in theory, all of this is fine. But how do you know that general algorithm design can be captured by an algorithm that a.) is simpler than most specific algorithms b.) whose execution is faster than that of evolution c.) which can locate useful algorithms within the infinite space of programs and d.) that humans will discover this algorithm?
Some people here seem to be highly confident about this. How?
ETA: Maybe this post better highlights the problems I see.
So you believe that many mathematical problems are too hard for humans to solve but that humans can solve all of mathematics?
All of mathematics? Dunno. I’m not even sure what that phrase refers to. But sure, there exist mathematical problems that humans can’t solve unaided, but which can be solved by tools we create.
I really don’t understand how people can believe this. In order to create an artificial mathematician it is first necssary to discover, prove and encode the mathematics of discovering and proving non-arbitrary mathematics (i.e. to encode a formalization of the natural language goal “be as good as humans at mathematics”). This seems much more difficult than solving any single problem.
In other words: you believe that if we take all possible mathematical problems and sort them by difficulty-to-humans, that one will turn out to be the most difficult?
I don’t mean to put words in your mouth here, I just want to make sure I understood you.
If so… why do you believe that?
To paraphrase your sentence: It seems reasonable that we can design an algorithm that can design algorithms that we are unable to design.
Yes, that’s a fair paraphrase.
This can only be true in the sense that this algorithm-design-algorithm would run faster on other computational substrates than human brains. I agree that this is possible. But are relevant algorithms in a class for which a speed advantage would be substantial?
Nah, I’m not talking about speed.
But how do you know that general algorithm design can be captured by an algorithm that a.) is simpler than most specific algorithms
Can you clarify what you mean by “simpler” here? If you mean in some objective sense, like how many bits would be required to specify it in a maximally compressed form or some such thing, I don’t claim that. If you mean easier for humans to develop… well, of course I don’t know that, but it seems more plausible to me than the idea that human brains happen to be the optimal machine for developing algorithms.
b.) whose execution is faster than that of evolution
We have thus far done pretty good at this; evolution is slow. I don’t expect that to change.
c.) which can locate useful algorithms within the infinite space of programs
Well, this is part of the problem specification. A tool for generating useless algorithms would be much easier to build.
d.) that humans will discover this algorithm?
(shrug) Perhaps we won’t. Perhaps we won’t solve protein-folding, either.
Some people here seem to be highly confident about this. How?
Can you quantify “highly confident” here?
For example, what confidence do you consider appropriate for the idea that there exists at least one useful algorithm A, and at least one artificial algorithm-developer AD, such that it’s easier for humans to develop AD than to develop A, and it’s easier for AD to develop A than it is for humans to develop A?
In other words: you believe that if we take all possible mathematical problems and sort them by difficulty-to-humans, that one will turn out to be the most difficult?
If you want an artificial agent to solve problems for you then you need to somehow constrain it, since there are an infinite number of problems. In this sense it is easier to specify an AI to solve a single problem, such as the protein-folding problem, rather than all problems (whatever that means, supposedly “general intelligence”).
The problem here is that goals and capabilities are not orthogonal. It is more difficult to design an AI that can play all possible games, and then tell it to play a certain game, than designing an AI to play a certain game in the first place.
Can you clarify what you mean by “simpler” here?
The information theoretic complexity of the code of a general problem solver constrained to solve a specific problem should be larger than the constrain itself. I assume here that the constrain is most of the work in getting an algorithm to do useful work. Which I like to exemplify by the difference between playing chess and doing mathematics. Both are rigorously defined activities, one of which has a clear and simple terminal goal, the other being infinite and thus hard to constrain.
For example, what confidence do you consider appropriate for the idea that there exists at least one useful algorithm A, and at least one artificial algorithm-developer AD, such that it’s easier for humans to develop AD than to develop A, and it’s easier for AD to develop A than it is for humans to develop A?
The more general the artificial algorithm-developer is, the less confident I am that it is easier to create than the specific algorithm itself.
I agree that specialized tools to perform particular tasks are easier to design than general-purpose tools. It follows that if I understand a problem well enough to know what tasks must be performed in order to solve that problem, it should be easier to solve that problem by designing specialized tools to perform those tasks, than by designing a general-purpose problem solver.
I agree that the complexity of a general problem solver should be larger than that of whatever constrains it to work on a specific task.
I agree that for a randomly selected algorithm A2, and a randomly selected artificial algorithm-developer AD2, the more general AD2 is the more likely it is that A2 is easier to develop than AD2.
I agree that the complexity of a general problem solver should be larger than that of whatever constrains it to work on a specific task.
What I meant is that if you have a very general and information theoretically simple problem solver, like evolution or AIXI, then in order to make it solve a specific problem you need a complex fitness function, respectively, in the case of AIXI, a substantial head start (the large multiplicative constant mentioned in Hutter’s paper).
When producing e.g. a chair, an AI will have to either know the specifications of the chair (such as its size or the material it is supposed to be made of) or else know how to choose a specification from an otherwise infinite set of possible specifications. Given a poorly designed fitness function, or the inability to refine its fitness function, an AI will either (a) not know what to do or (b) will not be able to converge on a qualitative solution, if at all, given limited computationally resources.
In a sense it is therefore true that an universal problem solver is easier to design than any specialized expert system. But only if you ignore the constrain it takes to “focus” the universal problem solver sufficiently in order to make it solve the right problem efficiently. Which means that the time to develop the universal problem solver plus the time it takes to constrain it might be longer than to develop the specialized solver. Since constraining it means to already know a lot about the problem in question. ETA: Or take science as another example. Once you generated a hypothesis, and an experiment to test it, you have already done most of the work. What reason do I have to believe that this is not true for the protein folding problem?
if you have a very general and information theoretically simple problem solver, like evolution or AIXI, then in order to make it solve a specific problem you need a complex fitness function
I agree with this as well. That said, sometimes that fitness function is implicit in the real world, and need not be explicitly formalized by me.
Once you generated a hypothesis, and an experiment to test it, you have already done most of the work. What reason do I have to believe that this is not true for the protein folding problem?
As I’ve said a couple of times now, I don’t have a dog in the race wrt the protein folding problem, but your argument seems to apply equally well to all conceivable problems. That’s why I asked a while back whether you think algorithm design is the single hardest problem for humans to solve. As I suggested then, I have no particular reason to think the protein-folding problem is harder (or easier) than the algorithm-design problem, but it seems really unlikely that no problem has this property.
That’s why I asked a while back whether you think algorithm design is the single hardest problem for humans to solve.
The problem is that I don’t know what you mean by “algorithm design”. Once you solved “algorithm design”, what do you expect to be able to do with it, and how?
Once you compute this “algorithm design”-algorithm, how will its behavior look like? Will it output all possible algorithms, or just the algorithms that you care about? If the latter, how does it know what algorithms you care about?
There is no brain area for “algorithm design”. There is just this computational substrate that can learn, recognize patterns etc. and whose behavior is defined and constrained by its environmental circumstances.
Say you cloned Donald E. Knuth and made him grow up under completely different circumstances, e.g. as a member of some Amazonian tribe. Now this clone has the same algorithm-design-potential, but he lacks the right input and constrains to output “The Art of Computer Programming”.
What I want to highlight is that “algorithm design”, or even “general intelligence”, is not a sufficient feature in order to get “algorithm that predicts protein structures from their sequences”.
Solving “algorithm design” or “general intelligence” does not give you some sort of oracle. In the same sense as an universal Turing machine does not give you “algorithm design” or “general intelligence”. You have to program the Turing machine in order to compute “algorithm design” or “general intelligence”. In the same sense you have to define what algorithm you want, respectively what problem you want to be solved, in order for your “algorithm design” or “general intelligence” to do what you want.
Just imagine having a human baby, the clone of a 250 IQ eugenics experiment, and ask it to solve protein folding for you. Well, it doesn’t even speak English yet. Even though you have this superior general intelligence, it won’t do what you want it to do without a lot of additional work. And even then it is not clear that it will have the motivation to do so.
You have a good point, in the case of trying to make a narrow intelligence to solve the protein folding problem. Yes, to make it spit out solutions to protein folding (even if given a “general” intelligence), you first must give it a detailed specification of the problem, which may take much work to derive in the first place.
But a solution to the protein solving problem is a means to an end. Generally, through the subgoal of being able to manipulate matter. To put it simply, the information complexity of the “practical facet” of the protein folding problem is actually not that high, because other much more general problems (“the manipulating matter problem”) point to it. An unfriendly AGI with general intelligence above a human’s doesn’t need us to do any work specifying the protein folding problem for them; they’ll find it themselves in their search for solutions to “take over the world”.
Conversely, while an AGI with a goal like the rearranging all the matter in the world a particular way might happen to solve the protein folding problem in the process of its planning, such a machine does not qualify as a useful protein-folding-solver-bot for us humans. Firstly because there’s no guarantee it will actually end up solving protein folding (maybe some other method of rearranging matter turns out to be more useful). Secondly because it doesn’t necessarily care to solve the entire protein solving problem, just the special cases relevant to its goals. Thirdly because it has no interest in giving us the solutions.
That’s why writing an AGI doesn’t violate information theory by giving us a detailed specification of the protein folding problem for free.
An unfriendly AGI with general intelligence above a human’s doesn’t need us to do any work specifying the protein folding problem for them; they’ll find it themselves in their search for solutions to “take over the world”.
First of all, we have narrow AI’s that do not exhibit Omohundro’s “Basic AI Drives”. Secondly, everyone seems to agree that it should be possible to create general AI that does (a) not exhibit those drives or (b) only exhibit AI drives to a limited extent or (c) which focuses AI drives in a manner that agrees with human volition.
The question then—regarding whether a protein-folding solver will be invented before a general AI that solves the same problem for instrumental reasons—is about the algorithmic complexity of an AI whose terminal goal is protein-folding versus an AI that does exhibit the necessary drives in order to solve an equivalent problem for instrumental reasons.
The first sub-question here is whether the aforementioned drives are a feature or a side-effect of general AI. Whether those drives have to be an explicit feature of a general AI or if they are an implicit consequence. The belief around here seems the be the latter.
Given that the necessary drives are implicit, the second sub-question is then about the point at which mostly well-behaved (bounded) AI systems become motivated to act in unbounded and catastrophic ways.
My objections to Omohundro’s “Basic AI Drives” are basically twofold: (a) I do not believe that AIs designed by humans will ever exhibit Omohundro’s “Basic AI Drives” in an unbounded fashion and (b) I believe that AIs that do exhibit Omohundro’s “Basic AI Drives” are either infeasible or require a huge number of constrains to work at all.
(a) The point of transition (step 4 below) between systems that do not exhibit Omohundro’s “Basic AI Drives” and those that do is too vague to count as a non-negligible hypothesis:
(1) Present-day software is better than previous software generations at understanding and doing what humans mean.
(2) There will be future generations of software which will be better than the current generation at understanding and doing what humans mean.
(3) If there is better software, there will be even better software afterwards.
(4) Magic happens.
(5) Software will be superhuman good at understanding what humans mean but catastrophically worse than all previous generations at doing what humans mean.
(b) An AI that does exhibit Omohundro’s “Basic AI Drives” would be paralyzed by infinite choice and low-probability hypotheses that imply vast amounts of expected utility.
There is an infinite choice of paperclip designs to choose from and a choosing a wrong design could have negative consequences that are in the range of −3^^^^3 utils.
Such an AI will not even be able to decide if trying to acquire unlimited computationally resources was instrumentally rational because without more resources it will be unable to decide if the actions that are required to acquire those resources might be instrumentally irrational from the perspective of what it is meant to do (that any terminal goal can be realized in an infinite number of ways, implies an infinite number of instrumental goals to choose from).
Another example is self-protection, which requires a definition of “self”, or otherwise the AI risks destroying itself.
Well, I’ve argued with you about (a) in the past, and it didn’t seem to go anywhere, so I won’t repeat that.
With regards to (b), that sounds like a good list of problems we need to solve in order to obtain AGI. I’m sure someone somewhere is already working on them.
I have no strong opinion on whether a “hard takeoff” is probable. (Because I haven’t thought about it a lot, not because I think the evidence is exquisitely balanced.) I don’t see any particular reason to think that protein folding is the only possible route to a “hard takeoff”.
What is alleged to make for an intelligence explosion is having a somewhat-superhuman AI that’s able to modify itself or make new AIs reasonably quickly. A solution to the protein folding problem might offer one way to make new AIs much more capable than oneself, I suppose, but it’s hardly the only way one can envisage.
If I understand Eliezer’s view, it’s that we can’t be extremely confident of whether artificial superintelligence or perilously advanced nanotechnology will come first, but (a) there aren’t many obvious research projects likely to improve our chances against grey goo, whereas (b) there are numerous obvious research projects likely to improve our changes against unFriendly AI, and (c) inventing Friendly AI would solve both the grey goo problem and the uFAI problem.
Cheer up, the main threat from nanotech may be from brute-forced AI going FOOM and killing everyone long before nanotech is sophisticated enough to reproduce in open-air environments.
The question is what to do about nanotech disaster. As near as I can figure out, the main path into [safety] would be a sufficiently fast upload of humans followed by running them at a high enough speed to solve FAI before everything goes blooey.
But that’s already assuming pretty sophisticated nanotech. I’m not sure what to do about moderately strong nanotech. I’ve never really heard of anything good to do about nanotech. It’s one reason I’m not sending attention there.
Considering … please wait … tttrrrrrr … prima facie, Grey Goo scenarios may seem more likely simply because they make better “Great Filter” candidates; whereas a near-arbitrary Foomy would spread out in all directions at relativistic speeds, with self-replicators no overarching agenty will would accelerate them out across space (the insulation layer with the sparse materials).
So if we approached x-risks through the prism of their consequences (extinction, hence no discernible aliens) and then reasoned our way back to our present predicament, we would note that within AI-power-hierachies (AGI and up) there are few distinct long-term dan-ranks (most such ranks would only be intermediary steps while the AI falls “upwards”), whereas it is much more conceivable that there are self-replicators which can e.g. transform enough carbon into carbon copies (of themselves) to render a planet uninhabitable, but which lack the oomph (and the agency) to do the same to their light cone.
Then I thought that Grey Goo may yet be more of a setback, a restart, not the ultimate planetary tombstone. Once everything got transformed into resident von Neumann machines, evolution amongst those copies would probably occur at some point, until eventually there may be new macroorganisms organized from self-replicating building blocks, which may again show significant agency and turn their gaze towards the stars.
Then again (round and round it goes), Grey Goo would still remain the better transient Great Filter candidate (and thus more likely than uFAI when viewed through the Great Filter spectroscope), simply because of the time scales involved. Assuming the Great Filter is in fact an actual absence of highly evolved civilizations in our neighborhood (as opposed to just hiding or other shenanigans), Grey Goo biosphere-resets may stall the Kardashev climb sufficiently to explain us not having witnessed other civs yet. Also, Grey Goo transformations may burn up all the local negentropy (nanobots don’t work for free), precluding future evolution.
Anyways, I agree that FAI would be the most realistic long-term guardian against accidental nanogoo (ironically, also uFAI).
My own suspicion is that the bulk of the Great Filter is behind us. We’ve awoken into a fairly old universe. (Young in terms of total lifespan, but old in terms of maximally life-sustaining years.) If intelligent agents evolve easily but die out fast, we should expect to see a young universe.
We can also consider the possibility of stronger anthropic effects. Suppose intelligent species always succeed in building AGIs that propagate outward at approximately the speed of light, converting all life-sustaining energy into objects or agents outside our anthropic reference class. Then any particular intelligent species Z will observe a Fermi paradox no matter how common or rare intelligent species are, because if any other high-technology species had arisen first in Z’s past light cone it would have prevented the existence of anything Z-like. (However, species in this scenario will observe much younger universes the smaller a Past Filter there is.)
So grey goo creates an actual Future Filter by killing their creators, but hyper-efficient hungry AGI creates an anthropic illusion of a Future Filter by devouring everything in their observable universe except the creator species. (And possibly devouring the creator species too; that’s unclear. Evolved alien values are less likely to eat the universe than artificial unFriendly-relative-to-alien-values values are, but perhaps not dramatically less likely; and unFriendly-relative-to-creator AI is almost certainly more common than Friendly-relative-to-creator AI.)
Once everything got transformed into resident von Neumann machines, evolution amongst those copies would probably occur at some point, until eventually there may be new macroorganisms organized from self-replicating building blocks, which may again show significant agency and turn their gaze towards the stars.
Probably won’t happen before the heat death of the universe. The scariest thing about nanodevices is that they don’t evolve. A universe ruled by nanodevices is plausibly even worse (relative to human values) than one ruled by uFAI like Clippy, because it’s vastly less interesting.
(Not because paperclips are better than nanites, but because there’s at least one sophisticated mind to be found.)
Two reasons: uFAI is deadlier than nano/biotech and easier to cause by accident.
If you build an AGI and botch friendliness, the world is in big trouble. If you build a nanite and botch friendliness, you have a worthless nanite. If you botch growth-control, it’s still probably not going to eat more than your lab before it runs into micronutrient deficiencies. And if you somehow do build grey goo, people have a chance to call ahead of it and somehow block its spread. What makes uFAI so dangerous is that it can outthink any responders. Grey goo doesn’t do that.
This seems like a consistent answer to my original question. Thank you.
If you botch growth-control, it’s still probably not going to eat more than your lab before it runs into micronutrient deficiencies.
You on the one hand believe that grey goo is not going to eat more than your lab before running out of steam and on the other hand believe that AI in conjunction with nanotechnology will not run out of steam, or only after humanity’s demise.
And if you somehow do build grey goo, people have a chance to call ahead of it and somehow block its spread.
You further believe that AI can’t be stopped but grey goo can.
Accidental grey goo is unlikely to get out of the lab. If I design a nanite to self-replicate and spread through a living brain to report useful data to me, and I have an integer overflow bug in the “stop reproducing” code so that it never stops, I will probably kill the patient but that’s it. Because the nanites are probably using glucose+O2 as their energy source. I never bothered to design them for anything else. Similarly if I sent solar-powered nanites to clean up Chernobyl I probably never gave them copper-refining capability—plenty of copper wiring to eat there—but if I botch the growth code they’ll still stop when there’s no more pre-refined copper to eat. Designing truely dangerous grey goo is hard and would have to be a deliberate effort.
As for stopping grey goo, why not? There’ll be something that destroys it. Extreme heat, maybe. And however fast it spreads, radio goes faster. So someone about to get eaten radios a far-off military base saying “help! grey goo!” and the bomber planes full of incindiaries come forth to meet it.
Contrast uFAI, which has thought of this before it surfaces, and has already radioed forged orders to take all the bomber planes apart for maintenance or something.
Also, the larger the difference between the metabolisms of the nanites and the biosphere, the easier it is to find something toxic to one but not the other.
I think a large part of that may simply be LW’ers being more familiar with UFAI and therefore knowing more details that make it seem like a credible threat / availability heuristic. So for example I would expect e.g. Eliezer’s estimate of the gap between the two to be less than the LW average. (Edit: Actually, I don’t mean that his estimate of the gap would be lower, but something more like it would seem like less of a non-question to him and he would take nanotech a lot more seriously, even if he did still come down firmly on the side of UFAI being a bigger concern.)
How is grey goo realistically a threat, especially without a uFAI guiding it? Remember: grey goo has to out-compete the existing biosphere. This seems hard.
Gray goo designs don’t need to be built up with miniscule steps, each of which makes evolutionary sense, like the evolved biosphere was. This might open up designs that are feasible to invent, very difficult to evolve naturally, and sufficiently different from anything in the natural biosphere to do serious damage even without a billion years of evolutionary optimization.
So far in the history of technology, deliberate design over a period of years has proven consistently less clever (in the sense of “efficiently capturing available mass-energy as living bodies”) than evolution operating over aeons.
And so far the more clever biosphere design is getting its thermodynamical shit handed to it everywhere the hairless apes go and decide to start building and burning stuff.
If a wish to a genie went really wrong and switched the terminal goals of every human on earth into destroying the earth’s biosphere in the most thorough and efficient way possible, the biosphere would be toast, much cleverer than the humans or not. If the wish gave you a billion AGI robots with the that terminal goal, any humans getting in their way would be dead and the biosphere would be toast again. But if the robots were really small and maybe not that smart, then we’d be entirely okay, right?
Think about it: it’s the intelligence that makes things dangerous. Try and engineer a nanoscale robot that’s going to be able to unintelligently disassemble all living matter without getting eaten by a bacterium. Unintelligently, mind you: no invoking superintelligence as your fallback explanation.
That’s not a nanoscale robot, is it? It’s antimatter: it annihilates matter, because that’s what physics says it does. You’re walking around the problem I handed you and just solving the “destroy lots of stuff” problem. Yes, it’s easy to destroy lots of stuff: we knew that already. And yet if I ask you to invent grey goo in specific, you don’t seem able to come up with a feasible design.
How is it not a nanoscale robot? It is a nanoscale device that performs the assigned task. What does a robot have that the nanoscale anticarbon lump doesn’t?
I admit that it’s not the sort of thing one thinks of when one thinks of the word ‘robot’ (to be fair, though, what I think of when I think of the word ‘robot’ is not nanoscale either). But I have found that, often, a simple solution to a problem can be found by, as you put it, ‘walking around’ it to get to the desired outcome.
Humans aren’t superintelligent, and are still able to design macroscale technology that can wipe out biospheres and that can be deployed and propagated with less intelligence than it took to design. I’m not taking the bet that you can’t shrink down the scale of the technology and the amount of intelligence needed to deploy it while keeping around the at least human level designer. That sounds too much like the “I can’t think of a way to do this right now, so it’s obviously impossible” play.
It seems that very few people considered the bad nanotech scenario obviously impossible, merely less likely to cause a near extinction event than uFAI.
In addition, to my best knowledge, trained scientists believe it impossible to turn the sky green and have all humans sprout spider legs. Mostly, they believe these things are impossible because they’re impossible, not because scientists merely lack the leap of superintelligence or superdetermination necessary to kick logic out and do the impossible.
If I wanted to turn the sky green for some reason (and had an infinite budget to work with), then one way to do it would be to release a fine, translucent green powder in the upper atmosphere in large quantities. (This might cause problems when it began to drift down far enough that it can be breathed in, of course). Alternatively, I could encase the planet Earth in a solid shell of green glass.
Please explain, then, without using the word ‘sky’, what exactly you mean by “turning the sky green”.
I had parsed that as “ensuring that a person, looking upwards during the daytime and not seeing an intervening obstacle (such as a ceiling, an aeroplane, or a cloud) would honestly identify the colour that he sees as ‘green’.” It is now evident that this is not what you had meant by the phrase.
The only reason I see blue when I look up during the daytime at something higher than a ceiling, an airplane, or a cloud, is because the atmosphere is composed of reflective blue material (air) intervening between me and the darkness of space. I would still like an explanation from the great-great-grandparent as to what constitutes ‘turning the sky green’.
Category error: neither jet engines nor nuclear weapons capture available/free mass-energy as living (ie: self-reproducing) bodies. Evolution never got to those because it simply doesn’t care about them: nuclear bombs can’t have grandchildren.
There are organisms that use gamma radiation as an energy source. If we lived in an environment richer in naturally occurring radioisotopes, I think I’d expect to see more of this sort of thing—maybe not up to the point of criticality, but maybe so.
Not much point in speculating, really; living on a planet that’s better than four billion years old and of middling metallicity puts something of a damper on the basic biological potential of that pathway.
Not much point in speculating, really; living on a planet that’s better than four billion years old and of middling metallicity puts something of a damper on the basic biological potential of that pathway.
And yet humanity did it, on a much smaller time scale. This is what I’m saying, we are better than evolution at some stuff.
Evolution has got as far as basic jet engines; see the octopus for an example.
Interestingly, this page provides some interesting data; it seems that a squid’s jet is significantly less energy-efficient than a fish’s tail for propulsion. This implies that that’s perhaps why we see so little jet propulsion in the oceans...
So far in the history of technology, deliberate design over a period of years has proven consistently less clever (in the sense of “efficiently capturing available mass-energy as living bodies”)
… because we don’t know how to build “living bodies”. That’s a rather unfair comparison, regardless of whether your point is valid.
Although, of course, we built factory farms for that exact purpose, which are indeed more efficient at that task.
And there’s genetic engineering, which can leapfrog over millions of years of evolution by nicking (simple, at our current tech level) adaptations from other organisms—whereas evolution would have to recreate them from scratch. I reflexively avoid anti-GM stuff due to overexposure when I was younger, but I wouldn’t be surprised if a GM organism could outcompete a wild one, were a mad scientist to choose that as a goal rather than a disaster to be elaborately defended against. (Herbicide-resistant plants, for a start.)
So I suppose it isn’t even very good at biasing the results, since it can still fail—depending, of course, on how true of a scotsman you are, because those do take advantage of prexisting adaptations—and artificially induced ones, in the case of farm animals.
grey goo has to out-compete the existing biosphere. This seems hard.
Really? Von Neumann machines (the universal assembler self-replicating variety, not the computer architecture) versus regular ol’ mitosis, and you think mitosis would win out?
I’ve only ever heard “building self-replicating machinery on a nano-scale is really hard” as the main argument against the immediacy of that particular x-risk, never “even if there were self-replicators on a nano-scale, they would have a hard time out-competing the existing biosphere”. Can you elaborate?
As one of my physics professors put it, “We already have grey goo. They’re called bacteria.”
The intuition behind the grey goo risk appears to be “as soon as someone makes a machine that can make itself, the world is a huge lump of matter and energy just waiting to be converted into copies of that machine.” That is, of course, not true- matter and energy and prized and fought over, and any new contender is going to have to join the fight.
That’s not to say it’s impossible for an artificial self-replicating nanobot to beat the self-replicating nanobots which have evolved naturally, just that it’s hard. For example, it’s not clear to me what part of “regular ol’ mitosis” you think is regular, and easy to improve upon. Is it that the second copy is built internally, preventing it from attack and corruption?
Bacteria et al. are only the locally optimal solution after a long series of selection steps, each of which generally needed to be an improvement upon the previous step, i.e. the result of a greedy algorithm. There are few problems in which you’d expect a greedy algorithm to end up anywhere but in a very local optimum:
DNA is a hilariously inefficient way of storing partly superfluous data (all of which must undergo each mitosis), informational density could be an order/orders of magnitude higher with minor modifications, and the safety redundancies are precarious at best, compared to e.g. Hamming code. A few researchers in a poorly funded government lab can come up with deadlier viruses in a few years (remember the recent controversy) than what nature engineered in millenia. That’s not to say that compared to our current macroscopic technology the informational feats of biological data transmission, duplication etc. aren’t impressive, but that’s only because we’ve not yet achieved molecular manufacturing (a necessity for a Grey Goo scenario). (We could go into more details on gross biological inefficiencies if you’d like.)
Would you expect some antibodies and phagocytosis to defeat an intelligently engineered self-replicating nanobot the size of a virus (but which doesn’t depend on live cells and without the telltale flaws and tradeoffs of Pandemic-reminiscent”can’t kill the host cell too quickly” etc.)?
To me it seems like saying “if you drowned the world in acid, the biosphere could well win the fight in a semi-recognizable form and claim the negentropy for themselves” (yes, cells can survive in extremely adverse environments and survive in some sort of niche, but I wouldn’t exactly call such a pseudo-equilibrium winning, and self-replicators wouldn’t exactly wait for its carbon food source to evolutionary adapt).
A few researchers in a poorly funded government lab can come up with deadlier viruses in a few years (remember the recent controversy) than what nature engineered in millenia.
Killing one human is easier than converting the entire biosphere.
Would you expect some antibodies and phagocytosis to defeat an intelligently engineered self-replicating nanobot the size of a virus (but which doesn’t depend on live cells and without the telltale flaws and tradeoffs of Pandemic-reminiscent”can’t kill the host cell too quickly” etc.)?
Well, that depends on what I think the engineering constraints are. It could be that in order to be the size of a virus, self-assembly has to be outsourced. It could be that in order to be resistant to phagocytosis, it needs exotic materials which limit its growth rate and maximal growth.
To me it seems like saying “if you drowned the world in acid, the biosphere could well win the fight in a semi-recognizable form and claim the negentropy for themselves”
It’s more “in order to drown the world in acid, you need to generate a lot of acid, and that’s actually pretty hard.”
A few researchers in a poorly funded government lab can come up with deadlier viruses in a few years (remember the recent controversy) than what nature engineered in millenia.
Yes, and you may have noticed that bioengineered pandemic was voted top threat.
How is grey goo realistically a threat, especially without a uFAI guiding it?
Is grey goo the only extinction type scenario possible if humans solve advanced nanotechnology? And do you really need an AI whose distance from an intelligence explosion is under 5 years in order to guide something like grey goo?
But yes, this is an answer to my original question. Thanks.
Could someone who voted for unfriendly AI explain how nanotech or biotech isn’t much more of a risk than unfriendly AI (I’ll assume MIRI’s definition here)?
I ask this question because it seems to me that even given a technological singularity there should be enough time for “unfriendly humans” to use precursors to fully fledged artificial general intelligence (e.g. advanced tool AI) in order to solve nanotechnology or advanced biotech. Technologies which themselves will enable unfriendly humans to cause a number of catastrophic risks (e.g. pandemics, nanotech wars, perfect global surveillance (an eternal tyranny) etc.).
Unfriendly AI, as imagined by MIRI, seems to be the end product of a developmental process that provides humans ample opportunity to wreck havoc.
I just don’t see any good reason to believe that the tools and precursors to artificial general intelligence are not themselves disruptive technologies.
And in case you believe advanced nanotechnology to be infeasible, but unfriendly AI to be an existential risk, what concrete scenarios do you imagine on how such an AI could cause human extinction without nanotech?
Presumably many people fear a very rapid “hard takeoff” where the time from “interesting slightly-smarter-than-human AI experiment” to “full-blown technological singularity underway” is measured in at days (or less) rather than months or years.
The AI risk scenario that Eliezer Yudkowsky relatively often uses is that of the AI solving the protein folding problem.
If you believe a “hard takeoff” to be probable, what reason is there to believe that the distance between a.) an AI capable of cracking that specific problem and b.) an AI triggering an intelligence explosion is too short for humans to do something similarly catastrophic as what the AI would have done with the resulting technological breakthrough?
In other words, does the protein folding problem require AI to reach a level of sophistication that would allow humans, or the AI itself, within days or months, to reach the stages where it undergoes an intelligence explosion? How so?
My assumption is that the protein-folding problem is unimaginably easier than an AI doing recursive self-improvement without breaking itself.
Admittedly, Eliezer is describing something harder than the usual interpretation of the protein-folding problem, but it still seems a lot less general than a program making itself more intelligent.
Is this question equivalent to “Is the protein-folding problem equivalently hard to the build-a-smarter-intelligence-than-I-am problem?” ? It seems like it ought to be, but I’m genuinely unsure, as the wording of your question kind of confuses me.
If so, my answer would be that it depends on how intelligent I am, since I expect the second problem to get more difficult as I get more intelligent. If we’re talking about the actual me… yeah, I don’t have higher confidence either way.
It is mostly equivalent. Is it easier to design an AI that can solve one specific hard problem than an AI that can solve all hard problems?
Expecting that only a fully-fledged artificial general intelligence is able to solve the protein-folding problem seems to be equivalent to believing the conjunction “an universal problem solver can solve the protein-folding problem” AND “an universal problem solver is easier to solve than the protein-folding problem”. Are there good reasons to believe this?
ETA: My perception is that people who believe unfriendly AI to come sooner than nanotechnology believe that it is easier to devise a computer algorithm to devise a computer algorithm to predict protein structures from their sequences rather than to directly devise a computer algorithm to predict protein structures from their sequences. This seems counter-intuitive.
Ah, this helps, thanks.
For my own part, the idea that we might build tools better at algorithm-development than our own brains are doesn’t seem counterintuitive at all… we build a lot of tools that are better than our own brains at a lot of things. Neither does it seem implausible that there exist problems that are solvable by algorithm-development, but whose solution requires algorithms that our brains aren’t good enough algorithm-developers to develop algorithms to solve.
So it seems reasonable enough that there are problems which we’ll solve faster by developing algorithm-developers to solve them for us, than by trying to solve the problem itself.
Whether protein-folding is one of those problems, I have absolutely no idea. But it sounds like your position isn’t unique to protein-folding.
So you believe that many mathematical problems are too hard for humans to solve but that humans can solve all of mathematics?
I already asked Timothy Gowers a similar question and I really don’t understand how people can believe this.
In order to create an artificial mathematician it is first necssary to discover, prove and encode the mathematics of discovering and proving non-arbitrary mathematics (i.e. to encode a formalization of the natural language goal “be as good as humans at mathematics”). This seems much more difficult than solving any single problem. And that’s just mathematics...
I do not disagree with this in theory. After all, evolution is an example of this. But it was not computationally simple for evolution to do so and it did do so by a bottom-up approach, piece by piece.
To paraphrase your sentence: It seems reasonable that we can design an algorithm that can design algorithms that we are unable to design.
This can only be true in the sense that this algorithm-design-algorithm would run faster on other computational substrates than human brains. I agree that this is possible. But are relevant algorithms in a class for which a speed advantage would be substantial?
Again, in theory, all of this is fine. But how do you know that general algorithm design can be captured by an algorithm that a.) is simpler than most specific algorithms b.) whose execution is faster than that of evolution c.) which can locate useful algorithms within the infinite space of programs and d.) that humans will discover this algorithm?
Some people here seem to be highly confident about this. How?
ETA: Maybe this post better highlights the problems I see.
Why did you interview Gowers anyway? It’s not like he has any domain knowledge in artificial intelligence.
He works on automatic theorem proving. In addition I was simply curious what a topnotch mathematician thinks about the whole subject.
All of mathematics? Dunno. I’m not even sure what that phrase refers to. But sure, there exist mathematical problems that humans can’t solve unaided, but which can be solved by tools we create.
In other words: you believe that if we take all possible mathematical problems and sort them by difficulty-to-humans, that one will turn out to be the most difficult?
I don’t mean to put words in your mouth here, I just want to make sure I understood you.
If so… why do you believe that?
Yes, that’s a fair paraphrase.
Nah, I’m not talking about speed.
Can you clarify what you mean by “simpler” here? If you mean in some objective sense, like how many bits would be required to specify it in a maximally compressed form or some such thing, I don’t claim that. If you mean easier for humans to develop… well, of course I don’t know that, but it seems more plausible to me than the idea that human brains happen to be the optimal machine for developing algorithms.
We have thus far done pretty good at this; evolution is slow. I don’t expect that to change.
Well, this is part of the problem specification. A tool for generating useless algorithms would be much easier to build.
(shrug) Perhaps we won’t. Perhaps we won’t solve protein-folding, either.
Can you quantify “highly confident” here?
For example, what confidence do you consider appropriate for the idea that there exists at least one useful algorithm A, and at least one artificial algorithm-developer AD, such that it’s easier for humans to develop AD than to develop A, and it’s easier for AD to develop A than it is for humans to develop A?
If you want an artificial agent to solve problems for you then you need to somehow constrain it, since there are an infinite number of problems. In this sense it is easier to specify an AI to solve a single problem, such as the protein-folding problem, rather than all problems (whatever that means, supposedly “general intelligence”).
The problem here is that goals and capabilities are not orthogonal. It is more difficult to design an AI that can play all possible games, and then tell it to play a certain game, than designing an AI to play a certain game in the first place.
The information theoretic complexity of the code of a general problem solver constrained to solve a specific problem should be larger than the constrain itself. I assume here that the constrain is most of the work in getting an algorithm to do useful work. Which I like to exemplify by the difference between playing chess and doing mathematics. Both are rigorously defined activities, one of which has a clear and simple terminal goal, the other being infinite and thus hard to constrain.
The more general the artificial algorithm-developer is, the less confident I am that it is easier to create than the specific algorithm itself.
I agree that specialized tools to perform particular tasks are easier to design than general-purpose tools. It follows that if I understand a problem well enough to know what tasks must be performed in order to solve that problem, it should be easier to solve that problem by designing specialized tools to perform those tasks, than by designing a general-purpose problem solver.
I agree that the complexity of a general problem solver should be larger than that of whatever constrains it to work on a specific task.
I agree that for a randomly selected algorithm A2, and a randomly selected artificial algorithm-developer AD2, the more general AD2 is the more likely it is that A2 is easier to develop than AD2.
What I meant is that if you have a very general and information theoretically simple problem solver, like evolution or AIXI, then in order to make it solve a specific problem you need a complex fitness function, respectively, in the case of AIXI, a substantial head start (the large multiplicative constant mentioned in Hutter’s paper).
When producing e.g. a chair, an AI will have to either know the specifications of the chair (such as its size or the material it is supposed to be made of) or else know how to choose a specification from an otherwise infinite set of possible specifications. Given a poorly designed fitness function, or the inability to refine its fitness function, an AI will either (a) not know what to do or (b) will not be able to converge on a qualitative solution, if at all, given limited computationally resources.
In a sense it is therefore true that an universal problem solver is easier to design than any specialized expert system. But only if you ignore the constrain it takes to “focus” the universal problem solver sufficiently in order to make it solve the right problem efficiently. Which means that the time to develop the universal problem solver plus the time it takes to constrain it might be longer than to develop the specialized solver. Since constraining it means to already know a lot about the problem in question. ETA: Or take science as another example. Once you generated a hypothesis, and an experiment to test it, you have already done most of the work. What reason do I have to believe that this is not true for the protein folding problem?
I agree with this as well. That said, sometimes that fitness function is implicit in the real world, and need not be explicitly formalized by me.
As I’ve said a couple of times now, I don’t have a dog in the race wrt the protein folding problem, but your argument seems to apply equally well to all conceivable problems. That’s why I asked a while back whether you think algorithm design is the single hardest problem for humans to solve. As I suggested then, I have no particular reason to think the protein-folding problem is harder (or easier) than the algorithm-design problem, but it seems really unlikely that no problem has this property.
The problem is that I don’t know what you mean by “algorithm design”. Once you solved “algorithm design”, what do you expect to be able to do with it, and how?
Once you compute this “algorithm design”-algorithm, how will its behavior look like? Will it output all possible algorithms, or just the algorithms that you care about? If the latter, how does it know what algorithms you care about?
There is no brain area for “algorithm design”. There is just this computational substrate that can learn, recognize patterns etc. and whose behavior is defined and constrained by its environmental circumstances.
Say you cloned Donald E. Knuth and made him grow up under completely different circumstances, e.g. as a member of some Amazonian tribe. Now this clone has the same algorithm-design-potential, but he lacks the right input and constrains to output “The Art of Computer Programming”.
What I want to highlight is that “algorithm design”, or even “general intelligence”, is not a sufficient feature in order to get “algorithm that predicts protein structures from their sequences”.
Solving “algorithm design” or “general intelligence” does not give you some sort of oracle. In the same sense as an universal Turing machine does not give you “algorithm design” or “general intelligence”. You have to program the Turing machine in order to compute “algorithm design” or “general intelligence”. In the same sense you have to define what algorithm you want, respectively what problem you want to be solved, in order for your “algorithm design” or “general intelligence” to do what you want.
Just imagine having a human baby, the clone of a 250 IQ eugenics experiment, and ask it to solve protein folding for you. Well, it doesn’t even speak English yet. Even though you have this superior general intelligence, it won’t do what you want it to do without a lot of additional work. And even then it is not clear that it will have the motivation to do so.
Tapping out now.
You have a good point, in the case of trying to make a narrow intelligence to solve the protein folding problem. Yes, to make it spit out solutions to protein folding (even if given a “general” intelligence), you first must give it a detailed specification of the problem, which may take much work to derive in the first place.
But a solution to the protein solving problem is a means to an end. Generally, through the subgoal of being able to manipulate matter. To put it simply, the information complexity of the “practical facet” of the protein folding problem is actually not that high, because other much more general problems (“the manipulating matter problem”) point to it. An unfriendly AGI with general intelligence above a human’s doesn’t need us to do any work specifying the protein folding problem for them; they’ll find it themselves in their search for solutions to “take over the world”.
Conversely, while an AGI with a goal like the rearranging all the matter in the world a particular way might happen to solve the protein folding problem in the process of its planning, such a machine does not qualify as a useful protein-folding-solver-bot for us humans. Firstly because there’s no guarantee it will actually end up solving protein folding (maybe some other method of rearranging matter turns out to be more useful). Secondly because it doesn’t necessarily care to solve the entire protein solving problem, just the special cases relevant to its goals. Thirdly because it has no interest in giving us the solutions.
That’s why writing an AGI doesn’t violate information theory by giving us a detailed specification of the protein folding problem for free.
First of all, we have narrow AI’s that do not exhibit Omohundro’s “Basic AI Drives”. Secondly, everyone seems to agree that it should be possible to create general AI that does (a) not exhibit those drives or (b) only exhibit AI drives to a limited extent or (c) which focuses AI drives in a manner that agrees with human volition.
The question then—regarding whether a protein-folding solver will be invented before a general AI that solves the same problem for instrumental reasons—is about the algorithmic complexity of an AI whose terminal goal is protein-folding versus an AI that does exhibit the necessary drives in order to solve an equivalent problem for instrumental reasons.
The first sub-question here is whether the aforementioned drives are a feature or a side-effect of general AI. Whether those drives have to be an explicit feature of a general AI or if they are an implicit consequence. The belief around here seems the be the latter.
Given that the necessary drives are implicit, the second sub-question is then about the point at which mostly well-behaved (bounded) AI systems become motivated to act in unbounded and catastrophic ways.
My objections to Omohundro’s “Basic AI Drives” are basically twofold: (a) I do not believe that AIs designed by humans will ever exhibit Omohundro’s “Basic AI Drives” in an unbounded fashion and (b) I believe that AIs that do exhibit Omohundro’s “Basic AI Drives” are either infeasible or require a huge number of constrains to work at all.
(a) The point of transition (step 4 below) between systems that do not exhibit Omohundro’s “Basic AI Drives” and those that do is too vague to count as a non-negligible hypothesis:
(1) Present-day software is better than previous software generations at understanding and doing what humans mean.
(2) There will be future generations of software which will be better than the current generation at understanding and doing what humans mean.
(3) If there is better software, there will be even better software afterwards.
(4) Magic happens.
(5) Software will be superhuman good at understanding what humans mean but catastrophically worse than all previous generations at doing what humans mean.
(b) An AI that does exhibit Omohundro’s “Basic AI Drives” would be paralyzed by infinite choice and low-probability hypotheses that imply vast amounts of expected utility.
There is an infinite choice of paperclip designs to choose from and a choosing a wrong design could have negative consequences that are in the range of −3^^^^3 utils.
Such an AI will not even be able to decide if trying to acquire unlimited computationally resources was instrumentally rational because without more resources it will be unable to decide if the actions that are required to acquire those resources might be instrumentally irrational from the perspective of what it is meant to do (that any terminal goal can be realized in an infinite number of ways, implies an infinite number of instrumental goals to choose from).
Another example is self-protection, which requires a definition of “self”, or otherwise the AI risks destroying itself.
Well, I’ve argued with you about (a) in the past, and it didn’t seem to go anywhere, so I won’t repeat that.
With regards to (b), that sounds like a good list of problems we need to solve in order to obtain AGI. I’m sure someone somewhere is already working on them.
I have no strong opinion on whether a “hard takeoff” is probable. (Because I haven’t thought about it a lot, not because I think the evidence is exquisitely balanced.) I don’t see any particular reason to think that protein folding is the only possible route to a “hard takeoff”.
What is alleged to make for an intelligence explosion is having a somewhat-superhuman AI that’s able to modify itself or make new AIs reasonably quickly. A solution to the protein folding problem might offer one way to make new AIs much more capable than oneself, I suppose, but it’s hardly the only way one can envisage.
If I understand Eliezer’s view, it’s that we can’t be extremely confident of whether artificial superintelligence or perilously advanced nanotechnology will come first, but (a) there aren’t many obvious research projects likely to improve our chances against grey goo, whereas (b) there are numerous obvious research projects likely to improve our changes against unFriendly AI, and (c) inventing Friendly AI would solve both the grey goo problem and the uFAI problem.
Considering … please wait … tttrrrrrr … prima facie, Grey Goo scenarios may seem more likely simply because they make better “Great Filter” candidates; whereas a near-arbitrary Foomy would spread out in all directions at relativistic speeds, with self-replicators no overarching agenty will would accelerate them out across space (the insulation layer with the sparse materials).
So if we approached x-risks through the prism of their consequences (extinction, hence no discernible aliens) and then reasoned our way back to our present predicament, we would note that within AI-power-hierachies (AGI and up) there are few distinct long-term dan-ranks (most such ranks would only be intermediary steps while the AI falls “upwards”), whereas it is much more conceivable that there are self-replicators which can e.g. transform enough carbon into carbon copies (of themselves) to render a planet uninhabitable, but which lack the oomph (and the agency) to do the same to their light cone.
Then I thought that Grey Goo may yet be more of a setback, a restart, not the ultimate planetary tombstone. Once everything got transformed into resident von Neumann machines, evolution amongst those copies would probably occur at some point, until eventually there may be new macroorganisms organized from self-replicating building blocks, which may again show significant agency and turn their gaze towards the stars.
Then again (round and round it goes), Grey Goo would still remain the better transient Great Filter candidate (and thus more likely than uFAI when viewed through the Great Filter spectroscope), simply because of the time scales involved. Assuming the Great Filter is in fact an actual absence of highly evolved civilizations in our neighborhood (as opposed to just hiding or other shenanigans), Grey Goo biosphere-resets may stall the Kardashev climb sufficiently to explain us not having witnessed other civs yet. Also, Grey Goo transformations may burn up all the local negentropy (nanobots don’t work for free), precluding future evolution.
Anyways, I agree that FAI would be the most realistic long-term guardian against accidental nanogoo (ironically, also uFAI).
My own suspicion is that the bulk of the Great Filter is behind us. We’ve awoken into a fairly old universe. (Young in terms of total lifespan, but old in terms of maximally life-sustaining years.) If intelligent agents evolve easily but die out fast, we should expect to see a young universe.
We can also consider the possibility of stronger anthropic effects. Suppose intelligent species always succeed in building AGIs that propagate outward at approximately the speed of light, converting all life-sustaining energy into objects or agents outside our anthropic reference class. Then any particular intelligent species Z will observe a Fermi paradox no matter how common or rare intelligent species are, because if any other high-technology species had arisen first in Z’s past light cone it would have prevented the existence of anything Z-like. (However, species in this scenario will observe much younger universes the smaller a Past Filter there is.)
So grey goo creates an actual Future Filter by killing their creators, but hyper-efficient hungry AGI creates an anthropic illusion of a Future Filter by devouring everything in their observable universe except the creator species. (And possibly devouring the creator species too; that’s unclear. Evolved alien values are less likely to eat the universe than artificial unFriendly-relative-to-alien-values values are, but perhaps not dramatically less likely; and unFriendly-relative-to-creator AI is almost certainly more common than Friendly-relative-to-creator AI.)
Probably won’t happen before the heat death of the universe. The scariest thing about nanodevices is that they don’t evolve. A universe ruled by nanodevices is plausibly even worse (relative to human values) than one ruled by uFAI like Clippy, because it’s vastly less interesting.
(Not because paperclips are better than nanites, but because there’s at least one sophisticated mind to be found.)
Two reasons: uFAI is deadlier than nano/biotech and easier to cause by accident.
If you build an AGI and botch friendliness, the world is in big trouble. If you build a nanite and botch friendliness, you have a worthless nanite. If you botch growth-control, it’s still probably not going to eat more than your lab before it runs into micronutrient deficiencies. And if you somehow do build grey goo, people have a chance to call ahead of it and somehow block its spread. What makes uFAI so dangerous is that it can outthink any responders. Grey goo doesn’t do that.
This seems like a consistent answer to my original question. Thank you.
You on the one hand believe that grey goo is not going to eat more than your lab before running out of steam and on the other hand believe that AI in conjunction with nanotechnology will not run out of steam, or only after humanity’s demise.
You further believe that AI can’t be stopped but grey goo can.
Accidental grey goo is unlikely to get out of the lab. If I design a nanite to self-replicate and spread through a living brain to report useful data to me, and I have an integer overflow bug in the “stop reproducing” code so that it never stops, I will probably kill the patient but that’s it. Because the nanites are probably using glucose+O2 as their energy source. I never bothered to design them for anything else. Similarly if I sent solar-powered nanites to clean up Chernobyl I probably never gave them copper-refining capability—plenty of copper wiring to eat there—but if I botch the growth code they’ll still stop when there’s no more pre-refined copper to eat. Designing truely dangerous grey goo is hard and would have to be a deliberate effort.
As for stopping grey goo, why not? There’ll be something that destroys it. Extreme heat, maybe. And however fast it spreads, radio goes faster. So someone about to get eaten radios a far-off military base saying “help! grey goo!” and the bomber planes full of incindiaries come forth to meet it.
Contrast uFAI, which has thought of this before it surfaces, and has already radioed forged orders to take all the bomber planes apart for maintenance or something.
Also, the larger the difference between the metabolisms of the nanites and the biosphere, the easier it is to find something toxic to one but not the other.
I think a large part of that may simply be LW’ers being more familiar with UFAI and therefore knowing more details that make it seem like a credible threat / availability heuristic. So for example I would expect e.g. Eliezer’s estimate of the gap between the two to be less than the LW average. (Edit: Actually, I don’t mean that his estimate of the gap would be lower, but something more like it would seem like less of a non-question to him and he would take nanotech a lot more seriously, even if he did still come down firmly on the side of UFAI being a bigger concern.)
Oooh, that would nicely solve the problem of the other impending apocalypses, wouldn’t it?
How is grey goo realistically a threat, especially without a uFAI guiding it? Remember: grey goo has to out-compete the existing biosphere. This seems hard.
Gray goo designs don’t need to be built up with miniscule steps, each of which makes evolutionary sense, like the evolved biosphere was. This might open up designs that are feasible to invent, very difficult to evolve naturally, and sufficiently different from anything in the natural biosphere to do serious damage even without a billion years of evolutionary optimization.
So far in the history of technology, deliberate design over a period of years has proven consistently less clever (in the sense of “efficiently capturing available mass-energy as living bodies”) than evolution operating over aeons.
And so far the more clever biosphere design is getting its thermodynamical shit handed to it everywhere the hairless apes go and decide to start building and burning stuff.
If a wish to a genie went really wrong and switched the terminal goals of every human on earth into destroying the earth’s biosphere in the most thorough and efficient way possible, the biosphere would be toast, much cleverer than the humans or not. If the wish gave you a billion AGI robots with the that terminal goal, any humans getting in their way would be dead and the biosphere would be toast again. But if the robots were really small and maybe not that smart, then we’d be entirely okay, right?
Think about it: it’s the intelligence that makes things dangerous. Try and engineer a nanoscale robot that’s going to be able to unintelligently disassemble all living matter without getting eaten by a bacterium. Unintelligently, mind you: no invoking superintelligence as your fallback explanation.
Make it out of antimatter? Say, a nanoscale amount of anticarbon—just an unintelligent lump?
Dump enough of those on any (matter) biosphere and all the living matter will be very thoroughly disassembled.
That’s not a nanoscale robot, is it? It’s antimatter: it annihilates matter, because that’s what physics says it does. You’re walking around the problem I handed you and just solving the “destroy lots of stuff” problem. Yes, it’s easy to destroy lots of stuff: we knew that already. And yet if I ask you to invent grey goo in specific, you don’t seem able to come up with a feasible design.
How is it not a nanoscale robot? It is a nanoscale device that performs the assigned task. What does a robot have that the nanoscale anticarbon lump doesn’t?
I admit that it’s not the sort of thing one thinks of when one thinks of the word ‘robot’ (to be fair, though, what I think of when I think of the word ‘robot’ is not nanoscale either). But I have found that, often, a simple solution to a problem can be found by, as you put it, ‘walking around’ it to get to the desired outcome.
Humans aren’t superintelligent, and are still able to design macroscale technology that can wipe out biospheres and that can be deployed and propagated with less intelligence than it took to design. I’m not taking the bet that you can’t shrink down the scale of the technology and the amount of intelligence needed to deploy it while keeping around the at least human level designer. That sounds too much like the “I can’t think of a way to do this right now, so it’s obviously impossible” play.
It seems that very few people considered the bad nanotech scenario obviously impossible, merely less likely to cause a near extinction event than uFAI.
In addition, to my best knowledge, trained scientists believe it impossible to turn the sky green and have all humans sprout spider legs. Mostly, they believe these things are impossible because they’re impossible, not because scientists merely lack the leap of superintelligence or superdetermination necessary to kick logic out and do the impossible.
If I wanted to turn the sky green for some reason (and had an infinite budget to work with), then one way to do it would be to release a fine, translucent green powder in the upper atmosphere in large quantities. (This might cause problems when it began to drift down far enough that it can be breathed in, of course). Alternatively, I could encase the planet Earth in a solid shell of green glass.
In which case you have merely placed reflective green material in the atmosphere. You have not actually turned the sky green.
Please explain, then, without using the word ‘sky’, what exactly you mean by “turning the sky green”.
I had parsed that as “ensuring that a person, looking upwards during the daytime and not seeing an intervening obstacle (such as a ceiling, an aeroplane, or a cloud) would honestly identify the colour that he sees as ‘green’.” It is now evident that this is not what you had meant by the phrase.
That would depend on whether a green shell of glass or a green particle counts as an intervening obstacle.
Do you know, I hadn’t even thought of that?
You are perfectly correct, and I thank you for raising the question.
The only reason I see blue when I look up during the daytime at something higher than a ceiling, an airplane, or a cloud, is because the atmosphere is composed of reflective blue material (air) intervening between me and the darkness of space. I would still like an explanation from the great-great-grandparent as to what constitutes ‘turning the sky green’.
I’ll have to disagree here. Evolution operating over aeons never got to jet engines and nuclear weapons. Maybe it needs more time?
Category error: neither jet engines nor nuclear weapons capture available/free mass-energy as living (ie: self-reproducing) bodies. Evolution never got to those because it simply doesn’t care about them: nuclear bombs can’t have grandchildren.
You can use both jet engines and nuclear weapons to increase your relative fitness.
There are no living nuclear reactors, either, despite the vast potential of energy.
There are organisms that use gamma radiation as an energy source. If we lived in an environment richer in naturally occurring radioisotopes, I think I’d expect to see more of this sort of thing—maybe not up to the point of criticality, but maybe so.
Not much point in speculating, really; living on a planet that’s better than four billion years old and of middling metallicity puts something of a damper on the basic biological potential of that pathway.
And yet humanity did it, on a much smaller time scale. This is what I’m saying, we are better than evolution at some stuff.
Which living beings created by evolution have done—also known as us!
This would be stretching the definition of evolution beyond its breaking point.
Evolution has got as far as basic jet engines; see the octopus for an example.
Interestingly, this page provides some interesting data; it seems that a squid’s jet is significantly less energy-efficient than a fish’s tail for propulsion. This implies that that’s perhaps why we see so little jet propulsion in the oceans...
… because we don’t know how to build “living bodies”. That’s a rather unfair comparison, regardless of whether your point is valid.
Although, of course, we built factory farms for that exact purpose, which are indeed more efficient at that task.
And there’s genetic engineering, which can leapfrog over millions of years of evolution by nicking (simple, at our current tech level) adaptations from other organisms—whereas evolution would have to recreate them from scratch. I reflexively avoid anti-GM stuff due to overexposure when I was younger, but I wouldn’t be surprised if a GM organism could outcompete a wild one, were a mad scientist to choose that as a goal rather than a disaster to be elaborately defended against. (Herbicide-resistant plants, for a start.)
So I suppose it isn’t even very good at biasing the results, since it can still fail—depending, of course, on how true of a scotsman you are, because those do take advantage of prexisting adaptations—and artificially induced ones, in the case of farm animals.
(Should this matter? Discuss.)
Really? Von Neumann machines (the universal assembler self-replicating variety, not the computer architecture) versus regular ol’ mitosis, and you think mitosis would win out?
I’ve only ever heard “building self-replicating machinery on a nano-scale is really hard” as the main argument against the immediacy of that particular x-risk, never “even if there were self-replicators on a nano-scale, they would have a hard time out-competing the existing biosphere”. Can you elaborate?
As one of my physics professors put it, “We already have grey goo. They’re called bacteria.”
The intuition behind the grey goo risk appears to be “as soon as someone makes a machine that can make itself, the world is a huge lump of matter and energy just waiting to be converted into copies of that machine.” That is, of course, not true- matter and energy and prized and fought over, and any new contender is going to have to join the fight.
That’s not to say it’s impossible for an artificial self-replicating nanobot to beat the self-replicating nanobots which have evolved naturally, just that it’s hard. For example, it’s not clear to me what part of “regular ol’ mitosis” you think is regular, and easy to improve upon. Is it that the second copy is built internally, preventing it from attack and corruption?
Bacteria et al. are only the locally optimal solution after a long series of selection steps, each of which generally needed to be an improvement upon the previous step, i.e. the result of a greedy algorithm. There are few problems in which you’d expect a greedy algorithm to end up anywhere but in a very local optimum:
DNA is a hilariously inefficient way of storing partly superfluous data (all of which must undergo each mitosis), informational density could be an order/orders of magnitude higher with minor modifications, and the safety redundancies are precarious at best, compared to e.g. Hamming code. A few researchers in a poorly funded government lab can come up with deadlier viruses in a few years (remember the recent controversy) than what nature engineered in millenia. That’s not to say that compared to our current macroscopic technology the informational feats of biological data transmission, duplication etc. aren’t impressive, but that’s only because we’ve not yet achieved molecular manufacturing (a necessity for a Grey Goo scenario). (We could go into more details on gross biological inefficiencies if you’d like.)
Would you expect some antibodies and phagocytosis to defeat an intelligently engineered self-replicating nanobot the size of a virus (but which doesn’t depend on live cells and without the telltale flaws and tradeoffs of Pandemic-reminiscent”can’t kill the host cell too quickly” etc.)?
To me it seems like saying “if you drowned the world in acid, the biosphere could well win the fight in a semi-recognizable form and claim the negentropy for themselves” (yes, cells can survive in extremely adverse environments and survive in some sort of niche, but I wouldn’t exactly call such a pseudo-equilibrium winning, and self-replicators wouldn’t exactly wait for its carbon food source to evolutionary adapt).
Killing one human is easier than converting the entire biosphere.
Well, that depends on what I think the engineering constraints are. It could be that in order to be the size of a virus, self-assembly has to be outsourced. It could be that in order to be resistant to phagocytosis, it needs exotic materials which limit its growth rate and maximal growth.
It’s more “in order to drown the world in acid, you need to generate a lot of acid, and that’s actually pretty hard.”
Yes, and you may have noticed that bioengineered pandemic was voted top threat.
Is grey goo the only extinction type scenario possible if humans solve advanced nanotechnology? And do you really need an AI whose distance from an intelligence explosion is under 5 years in order to guide something like grey goo?
But yes, this is an answer to my original question. Thanks.