Okay, let’s go on the brain-simulation path. Let’s start with something simple, like a lobster or a dog… oh wait, what if it transcends and isn’t human-friendly. All right, we’ll stick to human brains… oh wait, what if our model of neural function is wrong and we create a sociopathic copy that isn’t human-friendly. All right, we’ll work on human brain regions separately, and absolutely make sure that we have them all right before we do a whole brain… oh wait, what if one of our partial brain models transcends and isn’t human-friendly.
And while you, whose reason for taking this path is to create a human-friendly future, struggle to avoid these pitfalls, there will be others who aren’t so cautious, and who want to conduct experiments like hotwiring together cognitive modules that are merely brain-inspired, just to see what happens, or in the expectation of something cool, or because they want a smarter vacuum cleaner.
We don’t have to try and upgrade any virtual brains to get most of the benefits.
If we could today create an uploaded dog brain that’s just a normal virtual dog running at 1/1000th realtime, that would be a huge win with no meaningful risk. That would lead us down a relatively stable path of obscenely expensive and slow uploads becoming cheaper every year. In this case cheaper means fast and also more numerous, At the start human society can handle a few slightly superior uploads, by the time uploads get way past us, they will be a society of themselves and on roughly equal footing. (this may be bad for people still running at realtime, but human values will persist)
The dangers of someone making a transcendent AI first are there no matter what. This is not a good argument against a FASTER way to get to safe superintelligence.
So, in this scenario we have obtained a big neural network that imprints on a master and can learn complex physical tasks… and we’re just going to ignore the implications of that while we concentrate on trying to duplicate ourselves?
What’s going to stop me from duplicating just the canine prefrontal cortex and experimenting with it? It’s a nice little classifier / decision maker, I’m sure it has other uses…
Just the capacity to reliably emulate major functional regions of vertebrate brain already puts you on the threshold of creating big powerful nonhuman AI. If puploads come first, they’ll be doing more than catching frisbees in Second Life.
I realize that this risk is kinda insignficant compared to the risk of all life on Earth being wiped out… But I’m more than a little scared of the thought of animal uploads, and the possibility of people creating lifeforms that can be cheaply copied and replicated, without them needing to have any of the physical features that usually elict sympathy from people. We already have plenty of people abusing their animals today, and being able to do it perfectly undetected on your home upload isn’t going to help things.
To say nothing about when it becomes easier to run human uploads. I just yesterday re-read a rather disturbing short story about the stuff you could theoretically do with body-repairing nanomachines, a person who enjoys abusing others, and a “pet substitute”. Err, a two-year old human child, that is.
The supercomputers will be there whether we like it or not. Some of what they run will be attempts at AI. This is so far the only approach that someone unaware of Friendliness issues has a high probability of trying and succeeding with (and not immediately killing us all)
Numerous Un-augmented accelerated uplaods is a narrow safe path, and one we probably won’t follow, but it is a safe path. (so far one of 2, so it’s important) I think the likely win is less than FAI, but the dropoff isn’t so steep either as you walk off the path. Any safe AI approach will suggest profitable nonsafe alternatives.
An FAI failure is almost certainly alien, or not there yet. An augmentation failure is probably less-capable, probably not hostile, probably not strictly executing a utility function, and above all: can be surrounded by other, faster, uploads.
If the first pupload costs half a billion dollars and runs very slow, then even tweaking it will be safer than say, letting neural nets evolve in a rich environment on the same hardware.
What’s going to stop me from duplicating just the canine prefrontal cortex and experimenting with it? It’s a nice little classifier / decision maker, I’m sure it has other uses...
What’s going to stop you is that the prefrontal cortex is just one part of a larger whole. It may be possible to isolate that part, but doing so may be very difficult. Now, if your simulation were running in real time, you could just spawn off a bunch of different experiments pursuing different ideas for how to isolate and use the prefrontal cortex, and just keep doing this until you find something that works. But, if your simulation is running at 1/10000th realtime, as JamesAndrix suggests in his hypothetical, the prospects of this type of method seem dim.
Of course, maybe the existence of the dog brain simulation is sufficient to spur advances in neuroscience to the point where you could just isolate the functioning of the cortex, without the need for millions of experimental runs. Even so, your resulting module is still going to be too slow to be an existential threat.
Just the capacity to reliably emulate major functional regions of vertebrate brain already puts you on the threshold of creating big powerful nonhuman AI.
The threshold, yes. But that threshold is still nontrivial to cross. The question is, given that we can reliably emulate major functional regions of the brain, is it easier to cross the threshold to nonhuman AI, or to full emulations of humans? There is virtually no barrier to the second threshold, while the first one still has nontrivial problems to be solved.
It may be possible to isolate that part, but doing so may be very difficult.
Why would it be difficult? How would it be difficult?
There is a rather utopian notion of mind uploading, according to which you blindly scan a brain without understanding it, and then turn that data directly into a simulation. I’m aware of two such scanning paradigms. In one, you freeze the brain, microtome it, and then image the sections. In the other you do high-resolution imaging of the living brain (e.g. fMRI) and then you construct a state-machine model for each small 3D volume.
To turn the images of those microtomed sections into an accurate dynamical model requires a lot of interpretive knowledge. The MRI-plus-inference pathway sounds much more plausible as a blind path to brain simulation. But either way, you are going to know what the physical 3D location of every element in your simulation was, and functional neuroanatomy is already quite sophisticated. It won’t be hard to single out the sim-neurons specific to a particular anatomical macroregion.
There is virtually no barrier to the second threshold, while the first one still has nontrivial problems to be solved.
If you can simulate a human, you can immediately start experimenting with nonhuman cognitive architectures by lobotomizing or lesioning the simulation. But this would already be true for simulated animal brains as well.
It won’t be hard to single out the sim-neurons specific to a particular anatomical macroregion.
That’s true, but ultimately the regions of the brain are not completely islands. The circuitry connecting them is itself intricate. You may, for instance, be able to extract the visual cortex and get it to do some computer vision for you, but I doubt extracting a prefontal cortex will be useful without all the subsystems it depends on. More importantly, how to wire up new configurations (maybe you want to have a double prefrontal cortex: twice the cognitive power!) strikes me as a fundamentally difficulty problem. At that point you probably need to have some legitimate high level understanding of the components and their connective behaviors to succeed. To contrast, a vanilla emulation where you aren’t modifying the architecture or performing virtual surgery requires no such high level understanding.
For those not getting this, the book Accelerandostarts with the main character being called by something with a russian accent that claims to be a neuromorphic AI based off of lobsters grafted into some knowledge management. This AI (roughly “the lobsters”) seeks a human who can help them “defect”.
I recommend the book! The ideas aren’t super deep in retrospect but its “near future” parts have one hilariously juxtaposed geeky allusion after another and the later parts are an interesting take on post-human politics and economics.
I assume the lobsters were chosen because of existing research in this area. For example, there are techniques for keeping bits alive in vitro, there is modeling work from the 1990′s trying to reproduce known neural mechanisms in silico, and I remember (but couldn’t find the link) that a team had some success around 2001(?) doing a moravec transfer to one or more cells in a lobster ganglia (minus the nanotech of course). There are lots of papers in this area. The ones I linked to were easy to find.
Huh? Lobsters have been exploring their own fitness landscape for quite some time and haven’t transcended yet. Evolution doesn’t inevitably lead towards intelligence.
I was way too obscure. I meant: turn it into a Godel machine by modifying the lobster program to explore and evaluate the space of altered lobster programs.
Why do you need a lobster for that? You could start today with any old piece of open source code and any measure of “fitness” you like. People have tried to do this for awhile without much success.
Let’s start with something simple, like a lobster or a dog… oh wait, what if it transcends and isn’t human-friendly.
Lobsters and dogs aren’t general intelligences. A million years of dog-thoughts can’t do the job of a few minutes of human-thoughts. Although a self-improving dog could be pretty friendly. Cats on the other hand… well that would be bad news. :)
what if our model of neural function is wrong and we create a sociopathic copy that isn’t human-friendly.
I find that very unlikely. If you look at diseases or compounds that affect every neuron in the brain, they usually affect all cognitive abilities. Keeping intelligence while eliminating empathy would be pretty hard to do by accident, and if it did happen it would be easy to detect. Humans have experience detecting sociopathic tendencies in other humans. Unlike an AI, an upload can’t easily understand its own code, so self-improving is going to be that much more difficult. It’s not going to be some super-amazing thing that can immediately hack a human mind over a text terminal.
oh wait, what if one of our partial brain models transcends and isn’t human-friendly.
That still seems unlikely. If you look at brains with certain parts missing or injured, you see that they are disabled in very specific ways. Take away just a tiny part of a brain and you’ll end up with things like face blindness, Capgras delusion, or Anton-Babinski syndrome. By only simulating individual parts of the brain, it becomes less likely that the upload will transcend.
So they won’t transcend if we do nothing but run them in copies of their ancestral environments. But how likely is that? They will instead become tools in our software toolbox (see below).
Unlike an AI, an upload can’t easily understand its own code, so self-improving is going to be that much more difficult.
The argument for uploads first is not that by uploading humans, we have solved the problem of Friendliness. The uploads still have to solve that problem. The argument is that the odds are better if the first human-level faster-than-human intelligences are copies of humans rather than nonhuman AIs.
But guaranteeing fidelity in your copy is itself a problem comparable to the problem of Friendliness. It would be incredibly easy for us to miss that (e.g.) a particular neuronal chemical response is of cognitive and not just physiological significance, leave it out of the uploading protocol, and thereby create “copies” which systematically deviate from human cognition in some way, whether subtle or blatant.
By only simulating individual parts of the brain, it becomes less likely that the upload will transcend.
The classic recipe for unsafe self-enhancing AI is that you assemble a collection of software tools, and use them to build better tools, and eventually you delegate even that tool-improving function. The significance of partial uploads is that they can give a big boost to this process.
there will be others who aren’t so cautious, and who want to conduct experiments like hotwiring together cognitive modules that are merely brain-inspired
This is why it’s important that we have high fidelity simulations sooner rather than later, while the necessary hardware rests in the hands of the handful of institutions that can afford top tier supercomputers, rather than an idiot in a garage trying to build a better Roomba. There would be fewer players in the field, making the research easier to monitor, and, more importantly, it would be much more difficult to jerry rig a bunch of modules together. The more cumbersome the hardware the harder experimentation will be, making high fidelity copies more likely to provide computer intelligence before hotwired modules or neuromorphically inspired architectures.
Okay, let’s go on the brain-simulation path. Let’s start with something simple, like a lobster or a dog… oh wait, what if it transcends and isn’t human-friendly. All right, we’ll stick to human brains… oh wait, what if our model of neural function is wrong and we create a sociopathic copy that isn’t human-friendly. All right, we’ll work on human brain regions separately, and absolutely make sure that we have them all right before we do a whole brain… oh wait, what if one of our partial brain models transcends and isn’t human-friendly.
And while you, whose reason for taking this path is to create a human-friendly future, struggle to avoid these pitfalls, there will be others who aren’t so cautious, and who want to conduct experiments like hotwiring together cognitive modules that are merely brain-inspired, just to see what happens, or in the expectation of something cool, or because they want a smarter vacuum cleaner.
We don’t have to try and upgrade any virtual brains to get most of the benefits.
If we could today create an uploaded dog brain that’s just a normal virtual dog running at 1/1000th realtime, that would be a huge win with no meaningful risk. That would lead us down a relatively stable path of obscenely expensive and slow uploads becoming cheaper every year. In this case cheaper means fast and also more numerous, At the start human society can handle a few slightly superior uploads, by the time uploads get way past us, they will be a society of themselves and on roughly equal footing. (this may be bad for people still running at realtime, but human values will persist)
The dangers of someone making a transcendent AI first are there no matter what. This is not a good argument against a FASTER way to get to safe superintelligence.
So, in this scenario we have obtained a big neural network that imprints on a master and can learn complex physical tasks… and we’re just going to ignore the implications of that while we concentrate on trying to duplicate ourselves?
What’s going to stop me from duplicating just the canine prefrontal cortex and experimenting with it? It’s a nice little classifier / decision maker, I’m sure it has other uses…
Just the capacity to reliably emulate major functional regions of vertebrate brain already puts you on the threshold of creating big powerful nonhuman AI. If puploads come first, they’ll be doing more than catching frisbees in Second Life.
I realize that this risk is kinda insignficant compared to the risk of all life on Earth being wiped out… But I’m more than a little scared of the thought of animal uploads, and the possibility of people creating lifeforms that can be cheaply copied and replicated, without them needing to have any of the physical features that usually elict sympathy from people. We already have plenty of people abusing their animals today, and being able to do it perfectly undetected on your home upload isn’t going to help things.
To say nothing about when it becomes easier to run human uploads. I just yesterday re-read a rather disturbing short story about the stuff you could theoretically do with body-repairing nanomachines, a person who enjoys abusing others, and a “pet substitute”. Err, a two-year old human child, that is.
The supercomputers will be there whether we like it or not. Some of what they run will be attempts at AI. This is so far the only approach that someone unaware of Friendliness issues has a high probability of trying and succeeding with (and not immediately killing us all)
Numerous Un-augmented accelerated uplaods is a narrow safe path, and one we probably won’t follow, but it is a safe path. (so far one of 2, so it’s important) I think the likely win is less than FAI, but the dropoff isn’t so steep either as you walk off the path. Any safe AI approach will suggest profitable nonsafe alternatives.
An FAI failure is almost certainly alien, or not there yet. An augmentation failure is probably less-capable, probably not hostile, probably not strictly executing a utility function, and above all: can be surrounded by other, faster, uploads.
If the first pupload costs half a billion dollars and runs very slow, then even tweaking it will be safer than say, letting neural nets evolve in a rich environment on the same hardware.
What’s going to stop you is that the prefrontal cortex is just one part of a larger whole. It may be possible to isolate that part, but doing so may be very difficult. Now, if your simulation were running in real time, you could just spawn off a bunch of different experiments pursuing different ideas for how to isolate and use the prefrontal cortex, and just keep doing this until you find something that works. But, if your simulation is running at 1/10000th realtime, as JamesAndrix suggests in his hypothetical, the prospects of this type of method seem dim.
Of course, maybe the existence of the dog brain simulation is sufficient to spur advances in neuroscience to the point where you could just isolate the functioning of the cortex, without the need for millions of experimental runs. Even so, your resulting module is still going to be too slow to be an existential threat.
The threshold, yes. But that threshold is still nontrivial to cross. The question is, given that we can reliably emulate major functional regions of the brain, is it easier to cross the threshold to nonhuman AI, or to full emulations of humans? There is virtually no barrier to the second threshold, while the first one still has nontrivial problems to be solved.
Why would it be difficult? How would it be difficult?
There is a rather utopian notion of mind uploading, according to which you blindly scan a brain without understanding it, and then turn that data directly into a simulation. I’m aware of two such scanning paradigms. In one, you freeze the brain, microtome it, and then image the sections. In the other you do high-resolution imaging of the living brain (e.g. fMRI) and then you construct a state-machine model for each small 3D volume.
To turn the images of those microtomed sections into an accurate dynamical model requires a lot of interpretive knowledge. The MRI-plus-inference pathway sounds much more plausible as a blind path to brain simulation. But either way, you are going to know what the physical 3D location of every element in your simulation was, and functional neuroanatomy is already quite sophisticated. It won’t be hard to single out the sim-neurons specific to a particular anatomical macroregion.
If you can simulate a human, you can immediately start experimenting with nonhuman cognitive architectures by lobotomizing or lesioning the simulation. But this would already be true for simulated animal brains as well.
That’s true, but ultimately the regions of the brain are not completely islands. The circuitry connecting them is itself intricate. You may, for instance, be able to extract the visual cortex and get it to do some computer vision for you, but I doubt extracting a prefontal cortex will be useful without all the subsystems it depends on. More importantly, how to wire up new configurations (maybe you want to have a double prefrontal cortex: twice the cognitive power!) strikes me as a fundamentally difficulty problem. At that point you probably need to have some legitimate high level understanding of the components and their connective behaviors to succeed. To contrast, a vanilla emulation where you aren’t modifying the architecture or performing virtual surgery requires no such high level understanding.
How does a lobster simulation transcend?
That sounds like a koan.
Clearly people in this thread are not Charles Stross fans.
For those not getting this, the book Accelerando starts with the main character being called by something with a russian accent that claims to be a neuromorphic AI based off of lobsters grafted into some knowledge management. This AI (roughly “the lobsters”) seeks a human who can help them “defect”.
I recommend the book! The ideas aren’t super deep in retrospect but its “near future” parts have one hilariously juxtaposed geeky allusion after another and the later parts are an interesting take on post-human politics and economics.
I assume the lobsters were chosen because of existing research in this area. For example, there are techniques for keeping bits alive in vitro, there is modeling work from the 1990′s trying to reproduce known neural mechanisms in silico, and I remember (but couldn’t find the link) that a team had some success around 2001(?) doing a moravec transfer to one or more cells in a lobster ganglia (minus the nanotech of course). There are lots of papers in this area. The ones I linked to were easy to find.
Melted butter.
Someone uses it to explore its own fitness landscape.
Huh? Lobsters have been exploring their own fitness landscape for quite some time and haven’t transcended yet. Evolution doesn’t inevitably lead towards intelligence.
I was way too obscure. I meant: turn it into a Godel machine by modifying the lobster program to explore and evaluate the space of altered lobster programs.
Why do you need a lobster for that? You could start today with any old piece of open source code and any measure of “fitness” you like. People have tried to do this for awhile without much success.
Lobsters and dogs aren’t general intelligences. A million years of dog-thoughts can’t do the job of a few minutes of human-thoughts. Although a self-improving dog could be pretty friendly. Cats on the other hand… well that would be bad news. :)
I find that very unlikely. If you look at diseases or compounds that affect every neuron in the brain, they usually affect all cognitive abilities. Keeping intelligence while eliminating empathy would be pretty hard to do by accident, and if it did happen it would be easy to detect. Humans have experience detecting sociopathic tendencies in other humans. Unlike an AI, an upload can’t easily understand its own code, so self-improving is going to be that much more difficult. It’s not going to be some super-amazing thing that can immediately hack a human mind over a text terminal.
That still seems unlikely. If you look at brains with certain parts missing or injured, you see that they are disabled in very specific ways. Take away just a tiny part of a brain and you’ll end up with things like face blindness, Capgras delusion, or Anton-Babinski syndrome. By only simulating individual parts of the brain, it becomes less likely that the upload will transcend.
So they won’t transcend if we do nothing but run them in copies of their ancestral environments. But how likely is that? They will instead become tools in our software toolbox (see below).
The argument for uploads first is not that by uploading humans, we have solved the problem of Friendliness. The uploads still have to solve that problem. The argument is that the odds are better if the first human-level faster-than-human intelligences are copies of humans rather than nonhuman AIs.
But guaranteeing fidelity in your copy is itself a problem comparable to the problem of Friendliness. It would be incredibly easy for us to miss that (e.g.) a particular neuronal chemical response is of cognitive and not just physiological significance, leave it out of the uploading protocol, and thereby create “copies” which systematically deviate from human cognition in some way, whether subtle or blatant.
The classic recipe for unsafe self-enhancing AI is that you assemble a collection of software tools, and use them to build better tools, and eventually you delegate even that tool-improving function. The significance of partial uploads is that they can give a big boost to this process.
This is why it’s important that we have high fidelity simulations sooner rather than later, while the necessary hardware rests in the hands of the handful of institutions that can afford top tier supercomputers, rather than an idiot in a garage trying to build a better Roomba. There would be fewer players in the field, making the research easier to monitor, and, more importantly, it would be much more difficult to jerry rig a bunch of modules together. The more cumbersome the hardware the harder experimentation will be, making high fidelity copies more likely to provide computer intelligence before hotwired modules or neuromorphically inspired architectures.