I agree with a lot of your points about the advantages of WBE vs friendly AI. That said, look at the margins. Quite a few people are already working on WBE. Not very many people are working on friendly AI. Taking this into consideration, I think an extra dollar is better spent on FAI research than WBE research.
Also, a world of uploads without FAI would probably not preserve human values for long. The uploads that changed themselves in such a way to grow faster (convert the most resources or make the most copies of themselves) would replace uploads that preserved human values. For example, an upload could probably make more copies of itself it if deleted its capacities for humor and empathy.
We already have a great many relatively stable and sane intelligences.
I don’t think any human being is stable or sane in the way FAI would be stable and sane.
Quite a few people are already working on WBE. Not very many people are working on friendly AI. Taking this into consideration, I think an extra dollar is better spent on FAI research than WBE research.
This is true for the general categories “FAI research” and “WBE research”, but very few of those WBE research dollars are going to studies of safety and policy, such as SIAI does, or (I assume) to projects that take safety and policy at all seriously.
Really, it’s an empty field with the exceptions of the FHI-SIAI axis and a few closely connected people. Well, there are folk who have discussed questions like “are emulations persons?” But policy/x-risk/economic impact has been very limited.
I don’t think it is at all obvious that “an upload could probably make more copies of itself it if deleted its capacities for humor and empathy.” You seem to assume that those features do not serve important functions in current human minds.
Yeah, my example was rather weak. I think humor and empathy are important in current human minds, but uploads could modify their minds much more powerfully and accurately than we can today. Also, uploads would exist in a very different environment from ours. I don’t think current human minds or values would be well-adapted to that environment.
More successful uploads would be those who modified themselves to make more copies or consume/takeover more resources. As they evolved, their values would drift and they would care less about the things we care about. Eventually, they’d be come unfriendly.
Basically, yes. If values are different enough between two species/minds/groups/whatever, then both see the other as resources that could be reorganized into more valuable structures.
To borrow an UFAI example: An upload might not hate you, but your atoms could be reorganized into computronium running thousands of upload copies/children.
“Friendly” values simply means our values (or very close to them—closer than the value spread among us). Preservation of preference means that the agency of far future will prefer (and do) the kinds of things that we would currently prefer to be done in the far future (on reflection, if we knew more, given the specific situation in the future, etc.). In other words, value drift is absence of reflective consistency, and Friendliness is reflective consistency in following our preference. Value drift results in the far future agency having preference very different from ours, and so not doing the things we’d prefer to be done. This turns the far future into the moral wasteland, from the point of view of our preference, little different from what would remain after unleashing a paperclip maximizer or exterminating all life and mind.
(Standard disclaimer: values/preference have little to do with apparent wants or likes.)
Not very many people are working on friendly AI. Taking this into consideration, I think an extra dollar is better spent on FAI research than WBE research.
This doesn’t follow. It’s not clear at all that there is sufficient investment in WBE that substantial diminishing returns have kicked in at the margins.
I didn’t say money spent on WBE research suffered from diminishing returns. I said that $X spent on FAI research probably has more benefit than $X spent on WBE research.
This is because the amount of money spent on WBE is much much greater than that spent on FAI. The Blue Brain Project has funding from Switzerland, Spain, and IBM among others. Just that one project probably has an order of magnitude more money than the whole FAI field. Unless you think WBE offers an order of magnitude greater benefit than FAI, you should favor spending more on FAI.
Unless you think WBE offers an order of magnitude greater benefit than FAI, you should favor spending more on FAI.
No, all that matters is that the increase in utility by increasing WBE funding is greater than an increase in utility by increasing FAI funding. If neither has hit diminishing returns then the amount of current funding is irrelevant to this calculation.
For example, an upload could probably make more copies of itself it if deleted its capacities for humor and empathy.
If you were an upload, would you make copies of yourself? Where’s the fun in that? The only reason I could see doing it is if I wanted to amass knowledge or do a lot of tasks… and if I did that, I’d want the copies to get merged back into a single “me” so I would have the knowledge and experiences. (Okay, and maybe some backups would be good to have around). But why worry about how many copies you could make? That sounds suspiciously Clippy-like to me.
In any case, I think we’d be more likely to be screwed over by uploads’ human qualities and biases, than by a hypothetical desire to become less human.
In a world of uploads which contains some that do want to copy themselves, selection obviously favors the replicators, with tragicresults absent a singleton.
If you were an upload, would you make copies of yourself?
Yes. I’d make as many copies as was optimal for maximising my own power. I would then endeavor to gain dominance over civilisation, probably by joining a coalition of some sort. This may include creating an FAI that could self improve more effectively than I and serve to further my ends. When a stable equilibrium was reached and it was safe to do so I would go back to following this:
Where’s the fun in that? The only reason I could see doing it is if I wanted to amass knowledge or do a lot of tasks… and if I did that, I’d want the copies to get merged back into a single “me” so I would have the knowledge and experiences.
If right now is the final minutes of the game then early in a WBE era is the penalty shootouts. You don’t mess around having fun till you and those that you care about are going to live to see tomorrow.
If you were an upload, would you make copies of yourself? Where’s the fun in that?
You have a moral obligation to do it
Working in concert, thousands of you could save all the orphans from all the fires, and then go on to right a great many wrongs. You have many many good reasons to gain power.
So unless you’re very aware that you will gain power and then abuse power, you will take steps to gain power.
Even from a purely selfish perspective: If 10,000 of you could take over the world and become an elite of 10,000, that’s probably better than your current rank.
We’ve evolved something called “morality” that helps protect us from abuses of power like that. I believe Eliezer expressed it as something that tells you that even if you think it would be right (because of your superior ability) to murder the chief and take over the tribe, it still is not right to murder the chief and take over the tribe.
We do still have problems with abuses of power, but I think we have well-developed ways of spotting this and stopping it.
I believe Eliezer expressed it as something that tells you that even if you think it would be right (because of your superior ability) to murder the chief and take over the tribe, it still is not right to murder the chief and take over the tribe.
That’s exactly the high awareness I was talking about, and most people don’t have it. I wouldn’t be surprised if most people here failed at it, if it presented itself in their real lives.
I mean, are you saying you wouldn’t save the burning orphans?
We do still have problems with abuses of power, but I think we have well-developed ways of spotting this and stopping it.
We have checks and balances of political power, but that works between entities on roughly equal political footing, and doesn’t do much for those outside of that process. We can collectively use physical power to control some criminals who abuse their own limited powers. But we don’t have anything to deal with supervillains.
There is fundamentally no check on violence except more violence, and 10,000 accelerated uploads could quickly become able to win a war against the rest of the world.
It is the fashion in some circles to promote funding for Friendly AI research
as a guard against the existential threat of Unfriendly AI. While this is an
admirable goal, the path to Whole Brain Emulation is in many respects more
straightforward and presents fewer risks.
I believe Eliezer expressed it as something that tells you that even if you
think it would be right (because of your superior ability) to murder the
chief and take over the tribe, it still is not right to murder the chief
and take over the tribe.
That’s exactly the high awareness I was talking about, and most people don’t
have it. I wouldn’t be surprised if most people here failed at it, if it
presented itself in their real lives.
Most people would not act like a Friendly AI therefore “Whole Brain Emulation” only leads to “fewer risks” if you know exactly which brains to emulate and have the ability to choose which brain(s).
If whole brain emulation (for your specific brain) its expensive, it might result in the brain being from a person who starts wars and steals from other countries, so he can get rich.
Most people prefer that 999 people from their country should live at the cost of 1000 people of another country would die, given no other known differences between those 1999 people. Also unlike a “Friendly AI”, their choices are not consistent. Most people will leave the choice at whatever was going to happen if they did not choose, even if they know there are no other effects (like jail) from choosing. If the 1000 people were going to die, unknown to any of them, to save 999, then most people would think “Its none of my business, maybe god wants it to be that way” and let the extra 1 person die. A “Friendly AI” would maximize lives saved if nothing else is known about all those people.
There are many examples why most people are not close to acting like a “Friendly AI” even if we removed all the bad influences on them. We should build a software to be a “Friendly AI” instead of emulating brains and only emulate brains for different reasons, except maybe the few brains that think like a “Friendly AI”. Its probably safer to do it completely in software.
Most people would not act like a Friendly AI therefore “Whole Brain Emulation” only leads to “fewer risks” if you know exactly which brains to emulate and have the ability to choose which brain(s).
I agree entirely that humans are not friendly. Whole brain emulation is humanity-safe if there’s never a point at which one person or small group and run much faster than the rest of humanity (including other uploads)
The uploads may outpace us, but if they can keep each other in check, then uploading is not the same kind of human-values threat.
Even an upload singleton is not a total loss if the uploads have somewhat benign values. It is a crippling of the future, not an erasure.
It’s probably easier to cooperate with copies of yourself than with other people, but you also stand to gain less as all of you start out with the same skill set and the same talents.
But why worry about how many copies you could make? That sounds suspiciously Clippy-like to me.
This is, I think, an echo of Robin Hanson’s ‘crack of a future dawn’, where hyper-Darwinian pressures to multiply cause the discarding of unuseful mental modules like humor or empathy which take up space.
I agree with a lot of your points about the advantages of WBE vs friendly AI. That said, look at the margins. Quite a few people are already working on WBE. Not very many people are working on friendly AI. Taking this into consideration, I think an extra dollar is better spent on FAI research than WBE research.
Also, a world of uploads without FAI would probably not preserve human values for long. The uploads that changed themselves in such a way to grow faster (convert the most resources or make the most copies of themselves) would replace uploads that preserved human values. For example, an upload could probably make more copies of itself it if deleted its capacities for humor and empathy.
I don’t think any human being is stable or sane in the way FAI would be stable and sane.
This is true for the general categories “FAI research” and “WBE research”, but very few of those WBE research dollars are going to studies of safety and policy, such as SIAI does, or (I assume) to projects that take safety and policy at all seriously.
Really, it’s an empty field with the exceptions of the FHI-SIAI axis and a few closely connected people. Well, there are folk who have discussed questions like “are emulations persons?” But policy/x-risk/economic impact has been very limited.
I don’t think it is at all obvious that “an upload could probably make more copies of itself it if deleted its capacities for humor and empathy.” You seem to assume that those features do not serve important functions in current human minds.
Yeah, my example was rather weak. I think humor and empathy are important in current human minds, but uploads could modify their minds much more powerfully and accurately than we can today. Also, uploads would exist in a very different environment from ours. I don’t think current human minds or values would be well-adapted to that environment.
More successful uploads would be those who modified themselves to make more copies or consume/takeover more resources. As they evolved, their values would drift and they would care less about the things we care about. Eventually, they’d be come unfriendly.
Why must value drift eventually make unfriendly values? Do you just define “friendly” values as close values?
Basically, yes. If values are different enough between two species/minds/groups/whatever, then both see the other as resources that could be reorganized into more valuable structures.
To borrow an UFAI example: An upload might not hate you, but your atoms could be reorganized into computronium running thousands of upload copies/children.
“Friendly” values simply means our values (or very close to them—closer than the value spread among us). Preservation of preference means that the agency of far future will prefer (and do) the kinds of things that we would currently prefer to be done in the far future (on reflection, if we knew more, given the specific situation in the future, etc.). In other words, value drift is absence of reflective consistency, and Friendliness is reflective consistency in following our preference. Value drift results in the far future agency having preference very different from ours, and so not doing the things we’d prefer to be done. This turns the far future into the moral wasteland, from the point of view of our preference, little different from what would remain after unleashing a paperclip maximizer or exterminating all life and mind.
(Standard disclaimer: values/preference have little to do with apparent wants or likes.)
This doesn’t follow. It’s not clear at all that there is sufficient investment in WBE that substantial diminishing returns have kicked in at the margins.
I didn’t say money spent on WBE research suffered from diminishing returns. I said that $X spent on FAI research probably has more benefit than $X spent on WBE research.
This is because the amount of money spent on WBE is much much greater than that spent on FAI. The Blue Brain Project has funding from Switzerland, Spain, and IBM among others. Just that one project probably has an order of magnitude more money than the whole FAI field. Unless you think WBE offers an order of magnitude greater benefit than FAI, you should favor spending more on FAI.
No, all that matters is that the increase in utility by increasing WBE funding is greater than an increase in utility by increasing FAI funding. If neither has hit diminishing returns then the amount of current funding is irrelevant to this calculation.
If you were an upload, would you make copies of yourself? Where’s the fun in that? The only reason I could see doing it is if I wanted to amass knowledge or do a lot of tasks… and if I did that, I’d want the copies to get merged back into a single “me” so I would have the knowledge and experiences. (Okay, and maybe some backups would be good to have around). But why worry about how many copies you could make? That sounds suspiciously Clippy-like to me.
In any case, I think we’d be more likely to be screwed over by uploads’ human qualities and biases, than by a hypothetical desire to become less human.
In a world of uploads which contains some that do want to copy themselves, selection obviously favors the replicators, with tragic results absent a singleton.
Note that emulations can enable the creation of a singleton, it doesn’t necessarily have to exist in advance.
Yes, but that’s only likely if the first uploads are FAI researchers.
Yes. I’d make as many copies as was optimal for maximising my own power. I would then endeavor to gain dominance over civilisation, probably by joining a coalition of some sort. This may include creating an FAI that could self improve more effectively than I and serve to further my ends. When a stable equilibrium was reached and it was safe to do so I would go back to following this:
If right now is the final minutes of the game then early in a WBE era is the penalty shootouts. You don’t mess around having fun till you and those that you care about are going to live to see tomorrow.
You have a moral obligation to do it
Working in concert, thousands of you could save all the orphans from all the fires, and then go on to right a great many wrongs. You have many many good reasons to gain power.
So unless you’re very aware that you will gain power and then abuse power, you will take steps to gain power.
Even from a purely selfish perspective: If 10,000 of you could take over the world and become an elite of 10,000, that’s probably better than your current rank.
We’ve evolved something called “morality” that helps protect us from abuses of power like that. I believe Eliezer expressed it as something that tells you that even if you think it would be right (because of your superior ability) to murder the chief and take over the tribe, it still is not right to murder the chief and take over the tribe.
We do still have problems with abuses of power, but I think we have well-developed ways of spotting this and stopping it.
That’s exactly the high awareness I was talking about, and most people don’t have it. I wouldn’t be surprised if most people here failed at it, if it presented itself in their real lives.
I mean, are you saying you wouldn’t save the burning orphans?
We have checks and balances of political power, but that works between entities on roughly equal political footing, and doesn’t do much for those outside of that process. We can collectively use physical power to control some criminals who abuse their own limited powers. But we don’t have anything to deal with supervillains.
There is fundamentally no check on violence except more violence, and 10,000 accelerated uploads could quickly become able to win a war against the rest of the world.
Most people would not act like a Friendly AI therefore “Whole Brain Emulation” only leads to “fewer risks” if you know exactly which brains to emulate and have the ability to choose which brain(s).
If whole brain emulation (for your specific brain) its expensive, it might result in the brain being from a person who starts wars and steals from other countries, so he can get rich.
Most people prefer that 999 people from their country should live at the cost of 1000 people of another country would die, given no other known differences between those 1999 people. Also unlike a “Friendly AI”, their choices are not consistent. Most people will leave the choice at whatever was going to happen if they did not choose, even if they know there are no other effects (like jail) from choosing. If the 1000 people were going to die, unknown to any of them, to save 999, then most people would think “Its none of my business, maybe god wants it to be that way” and let the extra 1 person die. A “Friendly AI” would maximize lives saved if nothing else is known about all those people.
There are many examples why most people are not close to acting like a “Friendly AI” even if we removed all the bad influences on them. We should build a software to be a “Friendly AI” instead of emulating brains and only emulate brains for different reasons, except maybe the few brains that think like a “Friendly AI”. Its probably safer to do it completely in software.
I agree entirely that humans are not friendly. Whole brain emulation is humanity-safe if there’s never a point at which one person or small group and run much faster than the rest of humanity (including other uploads) The uploads may outpace us, but if they can keep each other in check, then uploading is not the same kind of human-values threat.
Even an upload singleton is not a total loss if the uploads have somewhat benign values. It is a crippling of the future, not an erasure.
It’s probably easier to cooperate with copies of yourself than with other people, but you also stand to gain less as all of you start out with the same skill set and the same talents.
This is, I think, an echo of Robin Hanson’s ‘crack of a future dawn’, where hyper-Darwinian pressures to multiply cause the discarding of unuseful mental modules like humor or empathy which take up space.
Where do you get the idea that humor or empathy are not useful mental abiliites?!
From AngryParsley...