If you meant CEV_(Kawoomba), then it wouldn’t change the outcome of that particular decision. Maybe refer to the definition here?
Ah, but would it really not?
I strongly expect a well-formed fully reflective CEV_(DaFranker) to make different decisions from current_DaFranker. For starters,CEV_(DaFranker) would not have silly scope insensitivity biases, availability bias, and other things that would factor strongly into current_DaFranker’s decision, since we assume CEV_(DaFranker) has immense computational power and can brute-force-optimize their decision process if needed, and current_DaFranker would strongly prefer to have those mental flaws fixed and go for the pure, optimal rationality software and hardware as long as their consciousness and identity is preserved continuously.
Since we’re talking about CEV_(individual), the “poetic” definition would be “[my] wish if [I] knew more, thought faster, were more the [man] [I] wished [I] were, (...), where [my] wishes cohere rather than interfere; extrapolated as [I] wish that extrapolated, interpreted as [I] wish that interpreted.”
Nothing that would change my top priorities, though I’d do a better job convincing Galactus.
Quite sure. I assume you value the life of a sparrow (the bird), all else being equal. Is there a number of sparrows to spare which you would consign yourself and your loved ones to the flames? Is there a hypothetical number of sparrows for which you would choose them living over all currently living humans?
If not, then you are saying that not all goals reduce to a number on a single metric, that there are tiers of values, similar in principle to Maslow’s.
Suppose you had the chance to save the life of one sparrow, but doing so kills you with probability p. For what values of p would you do so?
If the answer is only when p=0, then your value of sparrows should never affect your choices, because it will always be dominated by the greater probability of your own welfare.
This indeed puts me in a conundrum: If I answer anything but p=0, I’m giving a kind of weighting factor that destroys the supposedly strict separation between tiers.
However, if I answer p=0, then indeed as long as there is anything even remotely or possibly affecting my top tier terminal values, I should rationally disregard pursuing any other unrelated goal whatsoever.
Obviously, as evident by my writing here, I do not solely focus all my life’s efforts on my top tier values, even though I claim they outweigh any combination of other values.
So I am dealing with my value system in an irrational way. However, there are two possible conclusions concerning my confusion:
Are my supposed top tier terminal values in fact outweigh-able by others, with “just” a very large conversion coefficient?
or
Do I in fact rank my terminal values as claimed and am just making bad choices effectively matching my behavior to those values, wasting time on things not strictly related to my top values? (Is it just an instrumental rationality failure?) Anything with a terminal value that’s valued infinitely higher than all other values should behave strictly isomorphically to a paperclip maximizer with just that one terminal value, at least in our universe.
This could be resolved by Omega offering me a straight out choice, pressing buttons or something. I know what my consciously reflected decision would be, even if my daily routine does not reflect that.
Another case of “do as I say (I’d do in hypothetical scenarios), not as I do (in daily life)” …
This indeed puts me in a conundrum: If I answer anything but p=0, I’m giving a kind of weighting factor that destroys the supposedly strict separation between tiers.
Even that would be equivalent to an expected utility maximizer using just real numbers, except that there’s a well-defined tie-breaker to be used when two different possible decisions would have the exact same expected utility.
I would like to point out that there is a known bias interfering with said hypothetical scenarios. It’s called “taboo tradeoffs” or “sacred values”, and it’s touched upon here; I don’t think there’s any post that focuses on explaining what it is and how to avoid it, though. One of the more interesting biases, I think.
Of course, your actual preferences could mirror the bias, in this case; lets not fall prey to the fallacy fallacy ;)
If the answer is only when p=0, then your value of sparrows should never affect your choices, because it will always be dominated by the greater probability of your own welfare.
Not sure that holds. Surely there could be situations where you can’t meaningfully calculate whether acting to preserve the life of a sparrow increases or decreases the probability of your death, therefore you act to preserve its life because though you consider it a fundamentally lesser terminal value, it’s still a terminal value.
Surely there could be situations where you can’t meaningfully calculate whether acting to preserve the life of a sparrow increases or decreases the probability of your death
In this case you try harder to figure out a way to calculate the impact on your chance of death. The value of information of such an effort is worth infinite sparrow lives. Lower tier utility functions just don’t matter.
In this case you try harder to figure out a way to calculate the impact on your chance of death. The value of information of such an effort is worth infinite sparrow lives.
What if you’ve already estimated that calculating excessively (e.g. beyond a minute) on this matter will have near-definite negative impact on your well-being?
Then you go do something else that’s relevant to your top-tier utility function.
You can contrive a situation where the lower tier matters, but it looks like someone holding a gun to your head, and threatening to kill you if you don’t choose in the next 5 seconds whether or not they shoot the sparrow. That sort of thing generally doesn’t happen.
And even then, if you have the ability to self-modify, the costs of maintaining a physical representation of the lower tier utility functions is greater than the marginal benefit of choosing to save the sparrow automatically because you lower tier utility function says so over choosing alphabetically.
I think this is more due to diminishing marginal returns for the amount of sparrows in existence, to be honest...
Jokes aside, you are offering a very persuasive argument; I’d be curious to know how you figure out what tier certain values are and whether you ever have reason to (a) change your mind about said tier or (b) create new tiers altogether?
If not, then you are saying that not all goals reduce to a number on a single metric, (...)
Simply false AFAIK. There is a mathematical way to express as a single-number metric every tiered system I’ve ever been capable of conceiving, and I suspect those I haven’t also have such expressions with more mathematics I might not know.
So, I don’t know if the grandparent was saying that, but I assume it wasn’t, and if it was implied somewhere and I missed it then it certainly is false.
But then again, I may simply be interpreting your words uncharitably. I assume you’re already aware that Maslow’s can also be reduced to a single number formula.
A more interesting question than maximizing numbers of sparrows, however, is maximizing other value-factors. Suppose instead of imagining a number of sparrows sufficiently large for which you would trade a human you care about, you imagine a number of different minimum-sufficiency-level values being traded off for that “higher-tiered” life.
One human against a flock of sparrows large enough to guarantee the survival of the species is easy enough. Even the sparrow species doesn’t measure up against a human, or at least that’s what I expect you’d answer.
Now measure it up against a flock of each species of birds, on pain of extinction of birds (but let’s magically handwave away the ecosystem impacts of this—suppose we have advanced technology to compensate for all possible effects).
Now against a flock of each species of birds, a group of each species of nonhuman mammals, a school of each species of fish, a colony of each species of insects, and so forth throughout the entire fauna and flora of the planet—or of all known species of life in the universe. Again we handwave—we can make food using the power of Science and so on.
Now against the same, but without the handwaving. The humans you care about are all fine and healthy, but the world they live in is less interesting, and quite devastated, but Science keeps y’all healthy and stuff.
Now against that, plus having actual human bodies. You’re all jarbrains.
Feel free to keep going, value by value, until you reach a sufficient tradeoff or accept that no amount of destructive alteration to the universe will ever compare to the permanent loss of consciousness of your loved ones, and therefore you’d literally do everything and anything, up to and including warping space and time and sacrificing knowledge or memory or thinking-power or various qualia or experience or capacity for happiness or whatever else you can imagine, all traded for this one absolute value.
“Life/consciousness of loved ones” can also be substituted for whichever is your highest-tiered value if different.
Do you mean that as in “you can describe/encode arbitrary systems as a single number” or something related to that?
Yes.
For my part, I also consider it perfectly plausible (though perhaps less likely than some alternatives) that some humans might actually have tiered systems where certain values really truly never can be traded off in the slightest fraction of opportunity costs against arbitrarily high values of all lower-tiered values at the same time.
For instance, I could imagine an agent that values everything I value but has a hard tier cutoff below the single value that its consciousness must remain continuously aware until the end of the universe if such a time ever arrives (forever otherwise, assuming the simplest alternative). This agent would have no trouble sacrificing the entire solar system if it was proven to raise the expected odds of this survival. Or the agent could also only have to satisfy a soft threshold or some balancing formula where a certain probability of eternal life is desired, but more certainty than that becomes utility-comparable to lower-tier values. Or many other kinds of possible constructs.
So yes, arbitrary systems, for all systems I’ve ever thought of. I like to think of myself as imaginative and as having thought of a lot of possible arbitrary systems, too, though obviously my search space is limited by my intelligence and by the complexity I can formulate.
There are actual tiered systems all around us, even if most examples that come to mind are constructed/thought of by humans.
That aside, I am claiming that I would not trade my highest tier values against arbitrary combinations of all lower tiered values. So … hi!
Re: Just a number; I can encode your previous comments (all of them) in the form of a bitstring, which is a number. Doesn’t mean that doing “+1” on that yields any sensible result. Maybe we’re talking past each other on the “describe/encode” point, but I don’t see how describing a system containing strict tiers as a number somehow makes those tiers go away, unless you were nitpicking about “everything’s just a number that’s interpreted in a certain way” or somesuch.
Ah, on the numbers thing, what I meant was only that AFAIK there always exists some formula for which higher output numbers will correspond to things any abitrary agent (at least, all the logically valid and sound ones that I’ve thought of) would prefer.
So even for a hard tier system, there’s a way to compute a number linearly representative of how happy the agent is with worldstates, where at the extreme all lower-tier values flatline into arbitrarily large negatives (or other, more creative / leakproof weighing) whenever they incur infinitesimal risk of opportunity cost towards the higher-tier values.
The reason I’m said this is because it’s often disputed and/or my audience isn’t aware of it, and I often have to prove even the most basic versions of this claim (such as “you can represent a tiered system where as soon as the higher tier is empty, the lower tier is worthless using a relatively simple mathematical formula”) by showing them the actual equations and explaining how it works.
It might be simpler to compare less sacred values (how many sparrows is a dog worth? How many dogs is a chimp worth?) building up to something greater. Unfortunately, Kawoomba seems to be under the impression that nothing could possibly be worth the life of his family. Not sparrows, not humans, not genocides-prevented.
That is so. Why unfortunately? Also, why “under the impression”? If you were to tell me some of your terminal values, I’d give you the courtesy of assuming you are telling the truth as you subjectively perceive it (you have privileged access to your values, and at least concerning your conscious values, subjective is objective).
I get it that you hold nothing on Earth more sacred than a hypothetical sufficiently high number of sparrows, we differ on that. It is not a question of epistemic beliefs about the world state, of creating a better match between map and territory. It is a difference about values. If Omega gave me a button choice decision, I’m very sure what I would do. That’s where it counts.
For consolidation purposes, this is also meant to answer “How sure? Based on what? What would persuade you otherwise?”—As sure as I can be, based on “what I value above all else”, persuade you otherwise: nothing short of a brain reprogram.
Why unfortunately? Also, why “under the impression”? If you were to tell me some of your terminal values, I’d give you the courtesy of assuming you are telling the truth as you subjectively perceive it (you have privileged access to your values, and at least concerning your conscious values, subjective is objective).
If your values conflict with those of greater humanity (in aggregate,) then you are roughly equivalent to Clippy—not dangerous unless you actually end up being decisive regarding existential risk, but nevertheless only co-operating based on self-interest and bargaining, not because we have a common cause.
Humans are usually operating based on cached thoughts, heuristics whih may conflict with their actual terminal values. Picture a Nazi measuring utility in Jews eliminated. He doesn’t actually, terminally value killing people—but he was persuaded that Jews are undermining civilization, and his brain cached the thought that Jews=Bad. But he isn’t a Paperclipper—if he reexamines this cached thought in light of the truth that Jews are, generally speaking, neurotypical human beings then he will stop killing them.
I get it that you hold nothing on Earth more sacred than a hypothetical sufficiently high number of sparrows, we differ on that.
Well, sacred value is a technical term.
If you genuinely attached infinite utility to your family’s lives, then we could remove the finite terms in your utility function without affecting it’s output. You are not valuing their lives above all else, you are refusing to trade them to gain anything else. There is a difference. Rejecting certain deals because the cost is emotionally charged is suboptimal. Human, but stupid. I (probably) wouldn’t kill to save the sparrows, or for that matter to steal money for children dying in Africa, but that’s not the right choice. That’s just bias/akrasia/the sort of this this site is supposed to fight. If I could press a button and turn into an FAI, then I would. Without question. The fact that I’m not perfectly Friendly is a bad thing.
Anyway.
Considering you’re not typing from a bunker, and indeed probably drive a car, I’m guessing you’re willing to accept small risks to your family. So my question for you is this: how small?
Incidentally, considering the quote this particular branch of this discussion sprouted from, you do realize that killing your son might be the only way to save the rest of your family? Now, if He was claiming that you terminally value killing your son, that would be another thing …
If you genuinely attached infinite utility to your family’s lives, then we could remove the finite terms in your utility function without affecting it’s output. You are not valuing their lives above all else (...)
You do have a point, but there is another explanation to resolve that, see this comment.
We still have a fundamental disagreement on whether rationality is in any way involved when reflecting on your terminal values. I claim that rationality will help the closet murderer who is firm in valuing pain and suffering the same as the altruist, the paperclipper or the FAI. It helps us in pursuing our goals, not in setting the axioms of our value systems (the terminal values).
There is no aspect of Bayes or any reasoning mechanism that tells you whether to value happy humans or dead humans. Reasoning helps you in better achieving your goals, nefarious or angelic as they may be.
My point is that, while an agent that is not confused about its values will not change them in response to rationality (obviously,) one that is confused will. For example, a Nazi realizing Jews are people after all.
Hairyfigment’s answer would also work. The point is that they are as worthy of moral consideration as everyone else, and, to a lesser extent, that they aren’t congenitally predisposed to undermine civilization and so on and so forth.
Interesting analogy. If we accept that utilities are additive, then there is presumably a number of sparrows worth killing for. (Of course, there may be a limit on all possible sparrows or sparrow utilities may be largely due to species preservation or something. As an ethics-based vegetarian, however, I can simply change it to “sparrows tortured.) I would be uncomfortable trying to put a number on it, what with the various sacred value conflicts involved, but I accept that a Friendly AI (even one Friendly only to me) would know and act on it.
Maslow’s Pyramid is not intended as some sort of alternative to utilitarianism, it’s a description of how we should prioritize the needs of humans. An imperfect one, of course, but better than nothing.
How sure? Based on what? What would persuade you otherwise?
They almost certainly are on the margin (think taylor series approx of utility function). Get to the point where you are talking about killing a significant fraction of the sparrow population, then there’s no reason to think so.
To be clear: are you claiming that utilities are not additive? That there is some level of Bad Things that two (a thousand, a billion …) times as much is not worse? I’ve seen the position advocated, but only by appealing to scope insensitivity.
Ah, but would it really not?
I strongly expect a well-formed fully reflective CEV_(DaFranker) to make different decisions from current_DaFranker. For starters,CEV_(DaFranker) would not have silly scope insensitivity biases, availability bias, and other things that would factor strongly into current_DaFranker’s decision, since we assume CEV_(DaFranker) has immense computational power and can brute-force-optimize their decision process if needed, and current_DaFranker would strongly prefer to have those mental flaws fixed and go for the pure, optimal rationality software and hardware as long as their consciousness and identity is preserved continuously.
Since we’re talking about CEV_(individual), the “poetic” definition would be “[my] wish if [I] knew more, thought faster, were more the [man] [I] wished [I] were, (...), where [my] wishes cohere rather than interfere; extrapolated as [I] wish that extrapolated, interpreted as [I] wish that interpreted.”
Nothing that would change my top priorities, though I’d do a better job convincing Galactus.
How sure are you that you haven’t made a mistake somewhere?
Quite sure. I assume you value the life of a sparrow (the bird), all else being equal. Is there a number of sparrows to spare which you would consign yourself and your loved ones to the flames? Is there a hypothetical number of sparrows for which you would choose them living over all currently living humans?
If not, then you are saying that not all goals reduce to a number on a single metric, that there are tiers of values, similar in principle to Maslow’s.
You’re my sparrows.
Suppose you had the chance to save the life of one sparrow, but doing so kills you with probability p. For what values of p would you do so?
If the answer is only when p=0, then your value of sparrows should never affect your choices, because it will always be dominated by the greater probability of your own welfare.
A strong argument, well done.
This indeed puts me in a conundrum: If I answer anything but p=0, I’m giving a kind of weighting factor that destroys the supposedly strict separation between tiers.
However, if I answer p=0, then indeed as long as there is anything even remotely or possibly affecting my top tier terminal values, I should rationally disregard pursuing any other unrelated goal whatsoever.
Obviously, as evident by my writing here, I do not solely focus all my life’s efforts on my top tier values, even though I claim they outweigh any combination of other values.
So I am dealing with my value system in an irrational way. However, there are two possible conclusions concerning my confusion:
Are my supposed top tier terminal values in fact outweigh-able by others, with “just” a very large conversion coefficient?
or
Do I in fact rank my terminal values as claimed and am just making bad choices effectively matching my behavior to those values, wasting time on things not strictly related to my top values? (Is it just an instrumental rationality failure?) Anything with a terminal value that’s valued infinitely higher than all other values should behave strictly isomorphically to a paperclip maximizer with just that one terminal value, at least in our universe.
This could be resolved by Omega offering me a straight out choice, pressing buttons or something. I know what my consciously reflected decision would be, even if my daily routine does not reflect that.
Another case of “do as I say (I’d do in hypothetical scenarios), not as I do (in daily life)” …
Well, you could always play with some fun math…
Even that would be equivalent to an expected utility maximizer using just real numbers, except that there’s a well-defined tie-breaker to be used when two different possible decisions would have the exact same expected utility.
How often do two options have precisely the same expected utility? Not often, I’m guessing. Especially in the real world.
I guess almost never (in the mathematical sense). OTOH, in the real world the difference is often so tiny that it’s hard to tell its sign—but then, the thing to do is gather more information or flip a coin.
I would like to point out that there is a known bias interfering with said hypothetical scenarios. It’s called “taboo tradeoffs” or “sacred values”, and it’s touched upon here; I don’t think there’s any post that focuses on explaining what it is and how to avoid it, though. One of the more interesting biases, I think.
Of course, your actual preferences could mirror the bias, in this case; lets not fall prey to the fallacy fallacy ;)
Not sure that holds. Surely there could be situations where you can’t meaningfully calculate whether acting to preserve the life of a sparrow increases or decreases the probability of your death, therefore you act to preserve its life because though you consider it a fundamentally lesser terminal value, it’s still a terminal value.
In this case you try harder to figure out a way to calculate the impact on your chance of death. The value of information of such an effort is worth infinite sparrow lives. Lower tier utility functions just don’t matter.
What if you’ve already estimated that calculating excessively (e.g. beyond a minute) on this matter will have near-definite negative impact on your well-being?
Then you go do something else that’s relevant to your top-tier utility function.
You can contrive a situation where the lower tier matters, but it looks like someone holding a gun to your head, and threatening to kill you if you don’t choose in the next 5 seconds whether or not they shoot the sparrow. That sort of thing generally doesn’t happen.
And even then, if you have the ability to self-modify, the costs of maintaining a physical representation of the lower tier utility functions is greater than the marginal benefit of choosing to save the sparrow automatically because you lower tier utility function says so over choosing alphabetically.
I think this is more due to diminishing marginal returns for the amount of sparrows in existence, to be honest...
Jokes aside, you are offering a very persuasive argument; I’d be curious to know how you figure out what tier certain values are and whether you ever have reason to (a) change your mind about said tier or (b) create new tiers altogether?
Simply false AFAIK. There is a mathematical way to express as a single-number metric every tiered system I’ve ever been capable of conceiving, and I suspect those I haven’t also have such expressions with more mathematics I might not know.
So, I don’t know if the grandparent was saying that, but I assume it wasn’t, and if it was implied somewhere and I missed it then it certainly is false.
But then again, I may simply be interpreting your words uncharitably. I assume you’re already aware that Maslow’s can also be reduced to a single number formula.
A more interesting question than maximizing numbers of sparrows, however, is maximizing other value-factors. Suppose instead of imagining a number of sparrows sufficiently large for which you would trade a human you care about, you imagine a number of different minimum-sufficiency-level values being traded off for that “higher-tiered” life.
One human against a flock of sparrows large enough to guarantee the survival of the species is easy enough. Even the sparrow species doesn’t measure up against a human, or at least that’s what I expect you’d answer.
Now measure it up against a flock of each species of birds, on pain of extinction of birds (but let’s magically handwave away the ecosystem impacts of this—suppose we have advanced technology to compensate for all possible effects).
Now against a flock of each species of birds, a group of each species of nonhuman mammals, a school of each species of fish, a colony of each species of insects, and so forth throughout the entire fauna and flora of the planet—or of all known species of life in the universe. Again we handwave—we can make food using the power of Science and so on.
Now against the same, but without the handwaving. The humans you care about are all fine and healthy, but the world they live in is less interesting, and quite devastated, but Science keeps y’all healthy and stuff.
Now against that, plus having actual human bodies. You’re all jarbrains.
Feel free to keep going, value by value, until you reach a sufficient tradeoff or accept that no amount of destructive alteration to the universe will ever compare to the permanent loss of consciousness of your loved ones, and therefore you’d literally do everything and anything, up to and including warping space and time and sacrificing knowledge or memory or thinking-power or various qualia or experience or capacity for happiness or whatever else you can imagine, all traded for this one absolute value.
“Life/consciousness of loved ones” can also be substituted for whichever is your highest-tiered value if different.
I don’t understand. Do you mean that as in “you can describe/encode arbitrary systems as a single number” or something related to that?
If not, do you mean that there must be some number of sparrows outweighing everything else as it gets sufficiently large?
Please explain.
Yes.
For my part, I also consider it perfectly plausible (though perhaps less likely than some alternatives) that some humans might actually have tiered systems where certain values really truly never can be traded off in the slightest fraction of opportunity costs against arbitrarily high values of all lower-tiered values at the same time.
For instance, I could imagine an agent that values everything I value but has a hard tier cutoff below the single value that its consciousness must remain continuously aware until the end of the universe if such a time ever arrives (forever otherwise, assuming the simplest alternative). This agent would have no trouble sacrificing the entire solar system if it was proven to raise the expected odds of this survival. Or the agent could also only have to satisfy a soft threshold or some balancing formula where a certain probability of eternal life is desired, but more certainty than that becomes utility-comparable to lower-tier values. Or many other kinds of possible constructs.
So yes, arbitrary systems, for all systems I’ve ever thought of. I like to think of myself as imaginative and as having thought of a lot of possible arbitrary systems, too, though obviously my search space is limited by my intelligence and by the complexity I can formulate.
There are actual tiered systems all around us, even if most examples that come to mind are constructed/thought of by humans.
That aside, I am claiming that I would not trade my highest tier values against arbitrary combinations of all lower tiered values. So … hi!
Re: Just a number; I can encode your previous comments (all of them) in the form of a bitstring, which is a number. Doesn’t mean that doing “+1” on that yields any sensible result. Maybe we’re talking past each other on the “describe/encode” point, but I don’t see how describing a system containing strict tiers as a number somehow makes those tiers go away, unless you were nitpicking about “everything’s just a number that’s interpreted in a certain way” or somesuch.
Ah, on the numbers thing, what I meant was only that AFAIK there always exists some formula for which higher output numbers will correspond to things any abitrary agent (at least, all the logically valid and sound ones that I’ve thought of) would prefer.
So even for a hard tier system, there’s a way to compute a number linearly representative of how happy the agent is with worldstates, where at the extreme all lower-tier values flatline into arbitrarily large negatives (or other, more creative / leakproof weighing) whenever they incur infinitesimal risk of opportunity cost towards the higher-tier values.
The reason I’m said this is because it’s often disputed and/or my audience isn’t aware of it, and I often have to prove even the most basic versions of this claim (such as “you can represent a tiered system where as soon as the higher tier is empty, the lower tier is worthless using a relatively simple mathematical formula”) by showing them the actual equations and explaining how it works.
It might be simpler to compare less sacred values (how many sparrows is a dog worth? How many dogs is a chimp worth?) building up to something greater. Unfortunately, Kawoomba seems to be under the impression that nothing could possibly be worth the life of his family. Not sparrows, not humans, not genocides-prevented.
That is so. Why unfortunately? Also, why “under the impression”? If you were to tell me some of your terminal values, I’d give you the courtesy of assuming you are telling the truth as you subjectively perceive it (you have privileged access to your values, and at least concerning your conscious values, subjective is objective).
I get it that you hold nothing on Earth more sacred than a hypothetical sufficiently high number of sparrows, we differ on that. It is not a question of epistemic beliefs about the world state, of creating a better match between map and territory. It is a difference about values. If Omega gave me a button choice decision, I’m very sure what I would do. That’s where it counts.
For consolidation purposes, this is also meant to answer “How sure? Based on what? What would persuade you otherwise?”—As sure as I can be, based on “what I value above all else”, persuade you otherwise: nothing short of a brain reprogram.
Diction impaired by C2H6O.
If your values conflict with those of greater humanity (in aggregate,) then you are roughly equivalent to Clippy—not dangerous unless you actually end up being decisive regarding existential risk, but nevertheless only co-operating based on self-interest and bargaining, not because we have a common cause.
Humans are usually operating based on cached thoughts, heuristics whih may conflict with their actual terminal values. Picture a Nazi measuring utility in Jews eliminated. He doesn’t actually, terminally value killing people—but he was persuaded that Jews are undermining civilization, and his brain cached the thought that Jews=Bad. But he isn’t a Paperclipper—if he reexamines this cached thought in light of the truth that Jews are, generally speaking, neurotypical human beings then he will stop killing them.
Well, sacred value is a technical term.
If you genuinely attached infinite utility to your family’s lives, then we could remove the finite terms in your utility function without affecting it’s output. You are not valuing their lives above all else, you are refusing to trade them to gain anything else. There is a difference. Rejecting certain deals because the cost is emotionally charged is suboptimal. Human, but stupid. I (probably) wouldn’t kill to save the sparrows, or for that matter to steal money for children dying in Africa, but that’s not the right choice. That’s just bias/akrasia/the sort of this this site is supposed to fight. If I could press a button and turn into an FAI, then I would. Without question. The fact that I’m not perfectly Friendly is a bad thing.
Anyway.
Considering you’re not typing from a bunker, and indeed probably drive a car, I’m guessing you’re willing to accept small risks to your family. So my question for you is this: how small?
Incidentally, considering the quote this particular branch of this discussion sprouted from, you do realize that killing your son might be the only way to save the rest of your family? Now, if He was claiming that you terminally value killing your son, that would be another thing …
You do have a point, but there is another explanation to resolve that, see this comment.
We still have a fundamental disagreement on whether rationality is in any way involved when reflecting on your terminal values. I claim that rationality will help the closet murderer who is firm in valuing pain and suffering the same as the altruist, the paperclipper or the FAI. It helps us in pursuing our goals, not in setting the axioms of our value systems (the terminal values).
There is no aspect of Bayes or any reasoning mechanism that tells you whether to value happy humans or dead humans. Reasoning helps you in better achieving your goals, nefarious or angelic as they may be.
I see your psychopath and raise you one Nazi.
I’m sorry, does that label impact our debate whether rationality implies terminal values?
My point is that, while an agent that is not confused about its values will not change them in response to rationality (obviously,) one that is confused will. For example, a Nazi realizing Jews are people after all.
Sorry if that wasn’t clear.
Taboo “people”.
‘Share many human characteristics with the Nazi, and in particular suffered in similar ways from the economic conditions that helped produce Nazism.’
“not Evil Mutants”
Hairyfigment’s answer would also work. The point is that they are as worthy of moral consideration as everyone else, and, to a lesser extent, that they aren’t congenitally predisposed to undermine civilization and so on and so forth.
Interesting analogy. If we accept that utilities are additive, then there is presumably a number of sparrows worth killing for. (Of course, there may be a limit on all possible sparrows or sparrow utilities may be largely due to species preservation or something. As an ethics-based vegetarian, however, I can simply change it to “sparrows tortured.) I would be uncomfortable trying to put a number on it, what with the various sacred value conflicts involved, but I accept that a Friendly AI (even one Friendly only to me) would know and act on it.
Maslow’s Pyramid is not intended as some sort of alternative to utilitarianism, it’s a description of how we should prioritize the needs of humans. An imperfect one, of course, but better than nothing.
How sure? Based on what? What would persuade you otherwise?
Why?
They almost certainly are on the margin (think taylor series approx of utility function). Get to the point where you are talking about killing a significant fraction of the sparrow population, then there’s no reason to think so.
True, but this doesn’t apply to MugaSofer’s claim that
To be clear: are you claiming that utilities are not additive? That there is some level of Bad Things that two (a thousand, a billion …) times as much is not worse? I’ve seen the position advocated, but only by appealing to scope insensitivity.