if I reflected sufficiently hard, I would place zero terminal value on my own life.
Not even close to zero, but less terminal value than you would assign to other things that an FAI could optimize for. I’m not sure how much extrapolated unity of mankind there would be on this regard. I suspect Eliezer or Anna would counter my 5% with a 95%, and I would Aumann to some extent, but I was giving my impression and not belief. (I think that this is better practice at the start of a ‘debate’: otherwise you might update on the wrong expected evidence. EDIT: To be more clear, I wouldn’t want to update on Eliezer’s evidence if it was some sort of generalization from fictional evidence from Brennan’s world or something, but I would want to update if he had a strong argument that identity has proven to be extremely important to all of human affairs since the dawn of civilization, which is entirely plausible.)
It seems odd to me that out of the 10^40 atoms in the solar system, there would not be any left to revive cryo patients. My impression is that FAI would revive cryo patients, with probability 80%, the remaining 20% being for very odd scenarios that I just can’t think of.
I guess I’m saying the amount of atoms it takes to revive a cryo patient is vastly more wasteful than its weight in computronium. You’re trading off one life for a huge amount of potential lives. A few people, like Alicorn if I understand her correctly, think that people who are already alive are worth a huge number of potential lives, but I don’t quite understand that intuition. Is this a point of disagreement for us?
Yeah, but the cryo patient could be run in software rather than in hardware, which would mean that it would be a rather insignificant amount of extra effort.
Gah, sorry, I keep leaving things out. I’m thinking about the actual physical finding out where cryo patients are, scanning their brains, repairing the damage, and then running them. Mike Blume had a good argument against this point: proportionally, the startup cost of scanning a brain is not much at all compared to the infinity of years of actually running the computation. This is where I should be doing the math… so I’m going to think about it more and try and figure things out. Another point is that an AGI could gain access to infinite computing power in finite time during which it could do everything, but I think I’m just confused about the nature of computations in a Tegmark multiverses here.
actual physical finding out where cryo patients are, scanning their brains, repairing the damage, and then running them.
I hadn’t thought of that; certainly if the AI’s mission was to run as many experience-moments as possible in the amount of space-time-energy it had, then it wouldn’t revive cryo patients.
Note that the same argument says that it would kill all existing persons rather than upload them, and re-use their mass and energy to run ems of generic happy people (maximizing experience moments without regard to any deontological constraints has some weird implications...)
Yes, but this makes people flustered so I prefer not to bring it up as a possibility. I’m not sure if it was Bostrom or just generic SIAI thinking where I heard that an FAI might deconstruct us in order to go out into the universe, solve the problem of astronomical waste, and then run computations of us (or in this case generic transhumans) far in the future.
Of course at this point, the terminology “Friendly” becomes misleading, and we should talk about a Goal-X-controlled-AGI, where Goal X is a variable for the goal that that AGI would optimize for.
There is no unique value for X. Some have suggested the output of CEV as the goal system, but if you look at CEV in detail, you see that it is jam-packed with parameters, all of which make a difference to the actual output.
I would personally lobby against the idea of an AGI that did crazy shit like killing existing people to save a few nanoseconds.
Hm, I’ve noticed before that the term ‘Friendly’ is sort of vague. What would I call an AI that optimizes strictly for my goals (and if I care about others’ goals, so be it)? A Will-AI? I’ve said a few times ‘your Friendly is not my Friendly’ but I think I was just redefining Friendliness in an incorrect way that Eliezer wouldn’t endorse.
What would I call an AI that optimizes strictly for my goals...A Will-AI?
One could say “Friendly towards Will.”
But the problem of nailing down your goals seems to me much harder than the problem of negotiating goals between different people. Thus I don’t see a problem of being vague about the target of Friendliness.
But the problem of nailing down your goals seems to me much harder than the problem of negotiating goals between different people. Thus I don’t see a problem of being vague about the target of Friendliness.
Agreed. And asking the question of what is preference of a specific person, represented in some formal language, seems to be a natural simplification of the problem statement, something that needs to be understood before the problem of preference aggregation can be approached.
but I think I was just redefining Friendliness in an incorrect way that Eliezer wouldn’t endorse.
Beware of the urge to censor thoughts that disagree with authority. I personally agree that there is a serious issue here—the issue of moral antirealism, which implies that there is no “canonical human notion of goodness”, so the terminology “Friendly AI” is actually somewhat misleading, and it might be better to say “average human extrapolated morality AGI” when that’s what we want to talk about, e.g.
“an average human extrapolated morality AGI would oppose a paperclip maximizer”.
Then it sounds less onerous to say that you disagree with what an average human extrapolated morality AGI would do than that you disagree with what a “Friendly AI” would do, because most people on this forum disagree with averaged-out human morality (for example, the average human is a theist). Contrast:
“What, you disagree with the FAI? Are you a bad guy then?”
“Friendly AI” is about as specific/ambiguous as “morality”—something humans mostly have in common, allowing for normal variation, not referring to details about specific people. As with preference (morality) of specific people, we can speak of FAI optimizing the world to preference of specific people. Naturally, for each given person it’s preferable to launch a personal-FAI to a consensus-FAI.
Not even close to zero, but less terminal value than you would assign to other things that an FAI could optimize for. I’m not sure how much extrapolated unity of mankind there would be on this regard. I suspect Eliezer or Anna would counter my 5% with a 95%, and I would Aumann to some extent, but I was giving my impression and not belief. (I think that this is better practice at the start of a ‘debate’: otherwise you might update on the wrong expected evidence. EDIT: To be more clear, I wouldn’t want to update on Eliezer’s evidence if it was some sort of generalization from fictional evidence from Brennan’s world or something, but I would want to update if he had a strong argument that identity has proven to be extremely important to all of human affairs since the dawn of civilization, which is entirely plausible.)
It seems odd to me that out of the 10^40 atoms in the solar system, there would not be any left to revive cryo patients. My impression is that FAI would revive cryo patients, with probability 80%, the remaining 20% being for very odd scenarios that I just can’t think of.
I guess I’m saying the amount of atoms it takes to revive a cryo patient is vastly more wasteful than its weight in computronium. You’re trading off one life for a huge amount of potential lives. A few people, like Alicorn if I understand her correctly, think that people who are already alive are worth a huge number of potential lives, but I don’t quite understand that intuition. Is this a point of disagreement for us?
Yeah, but the cryo patient could be run in software rather than in hardware, which would mean that it would be a rather insignificant amount of extra effort.
Gah, sorry, I keep leaving things out. I’m thinking about the actual physical finding out where cryo patients are, scanning their brains, repairing the damage, and then running them. Mike Blume had a good argument against this point: proportionally, the startup cost of scanning a brain is not much at all compared to the infinity of years of actually running the computation. This is where I should be doing the math… so I’m going to think about it more and try and figure things out. Another point is that an AGI could gain access to infinite computing power in finite time during which it could do everything, but I think I’m just confused about the nature of computations in a Tegmark multiverses here.
I hadn’t thought of that; certainly if the AI’s mission was to run as many experience-moments as possible in the amount of space-time-energy it had, then it wouldn’t revive cryo patients.
Note that the same argument says that it would kill all existing persons rather than upload them, and re-use their mass and energy to run ems of generic happy people (maximizing experience moments without regard to any deontological constraints has some weird implications...)
Yes, but this makes people flustered so I prefer not to bring it up as a possibility. I’m not sure if it was Bostrom or just generic SIAI thinking where I heard that an FAI might deconstruct us in order to go out into the universe, solve the problem of astronomical waste, and then run computations of us (or in this case generic transhumans) far in the future.
Of course at this point, the terminology “Friendly” becomes misleading, and we should talk about a Goal-X-controlled-AGI, where Goal X is a variable for the goal that that AGI would optimize for.
There is no unique value for X. Some have suggested the output of CEV as the goal system, but if you look at CEV in detail, you see that it is jam-packed with parameters, all of which make a difference to the actual output.
I would personally lobby against the idea of an AGI that did crazy shit like killing existing people to save a few nanoseconds.
Hm, I’ve noticed before that the term ‘Friendly’ is sort of vague. What would I call an AI that optimizes strictly for my goals (and if I care about others’ goals, so be it)? A Will-AI? I’ve said a few times ‘your Friendly is not my Friendly’ but I think I was just redefining Friendliness in an incorrect way that Eliezer wouldn’t endorse.
One could say “Friendly towards Will.”
But the problem of nailing down your goals seems to me much harder than the problem of negotiating goals between different people. Thus I don’t see a problem of being vague about the target of Friendliness.
Agreed. And asking the question of what is preference of a specific person, represented in some formal language, seems to be a natural simplification of the problem statement, something that needs to be understood before the problem of preference aggregation can be approached.
Beware of the urge to censor thoughts that disagree with authority. I personally agree that there is a serious issue here—the issue of moral antirealism, which implies that there is no “canonical human notion of goodness”, so the terminology “Friendly AI” is actually somewhat misleading, and it might be better to say “average human extrapolated morality AGI” when that’s what we want to talk about, e.g.
Then it sounds less onerous to say that you disagree with what an average human extrapolated morality AGI would do than that you disagree with what a “Friendly AI” would do, because most people on this forum disagree with averaged-out human morality (for example, the average human is a theist). Contrast:
“Friendly AI” is about as specific/ambiguous as “morality”—something humans mostly have in common, allowing for normal variation, not referring to details about specific people. As with preference (morality) of specific people, we can speak of FAI optimizing the world to preference of specific people. Naturally, for each given person it’s preferable to launch a personal-FAI to a consensus-FAI.