I think what you mean to tell me is: “say ‘my preferences’ instead of ‘my utility function’”. I acknowledge that I was incorrectly using these interchangeably.
I do think it was clear what I meant when I called it “my” function and talked about it not conforming to VNM rules, so this response felt tautological to me.
I notice we’re not understanding each other, but I don’t know why. Let’s step back a bit. What problem is “radiation poisoning for looking at magnitude of utility” supposed to be solving?
We’re not talking about adding N to both sides of a comparison. We’re talking about taking a relation where we are only allowed to know that A < B, multiplying B by some probability factor, and then trying to make some judgment about the new relationship between A and xB. The rule against looking at magnitudes prevents that. So we can’t give an answer to the question: “Is the sandwich day better than the expected value of 1⁄400 chance of a whale day?”
If we’re allowed to compare A to xB, then we have to do that before the magnitude rule goes into effect. I don’t see how this model is supposed to account for that.
It’s too late for me. It might work to tell the average person to use “awesomeness” as their black box for moral reasoning as long as they never ever look inside it. Unfortunately, all of us have now looked, and so whatever value it had as a black box has disappeared.
You can’t tell me now to go back and revert to my original version of awesome unless you have a supply of blue pills whenever I need them.
If the power of this tool evaporates as soon as you start investigating it, that strikes me as a rather strong point of evidence against it. It was fun while it lasted, though.
Ooops, you tried to feel a utility. Go directly to type theory hell; do not pass go, do not collect 200 utils.
I don’t think this example is evidence against trying to ‘feel’ a utility. You didn’t account for scope insensitivity and the qualitative difference between the two things you think you’re comparing.
You need to compare the feeling of the turtle thrown against the wall to the cumulative feeling when you think about EACH individual beheading, shooting, orphaned child, open grave, and every other atrocity of the genocide. Thinking about the vague concept “genocide” doesn’t use the same part of your brain as thinking about the turtle incident.
What I mean by “normalized” is that you’re compressing the utility values into the range between 0 and 1. I am not aware of another definition that would apply here.
Your rule says you’re allowed to compare, but your other rule says you’re not allowed to compare by magnitude. You were serious enough about this second rule to equate it with radiation death.
You can’t apply probabilities to utilities and be left with anything meaningful unless you’re allowed to compare by magnitude. This is a fatal contradiction in your thesis. Using your own example, you assign a value of 1 to whaling and 1⁄500 to the sandwich. If you’re not allowed to compare the two using their magnitude, then you can’t compare the utility of 1⁄400 chance of the whale day with the sandwich, because you’re not allowed to think about how much better it is to be a whale.
No, I mean if my utility function violates transitivity or other axioms of VNM, I more want to fix it than to throw out VNM as being invalid.
I think I have updated slightly in the direction of requiring my utility function to conform to VNM and away from being inclined to throw it out if my preferences aren’t consistent. This is probably mostly due to smart people being asked to give an example of a circular preference and my not finding any answer compelling.
Expectation. VNM isn’t really useful without uncertainty. Without uncertainty, transitive preferences are enough.
I think I see the point you’re trying to make, which is that we want to have a normalized scale of utility to apply probability to. This directly contradicts the prohibition against “looking at the sign or magnitude”. You are comparing 1⁄400 EU and 1⁄500 EU using their magnitudes, and jumping headfirst into the radiation. Am I missing something?
That was one of the major points. Do not play with naked utilities. For any decision, find the 0 anchor and the 1 anchor, and rank other stuff relative to them.
I understood your major point about the radioactivity of the single real number for each utility, but I got confused by what you intended the process to look like with your hell example. I think you need to be a little more explicit about your algorithm when you say “find the 0 anchor and the 1 anchor”. I defaulted to a generic idea of moral intuition about best and worst, then only made it as far as thinking it required naked utilities to find the anchors in the first place. Is your process something like: “compare each option against the next until you find the worst and best?”
It is becoming clear from this and other comments that you consider at least the transitivity property of VNM to be axiomatic. Without it, you couldn’t find what is your best option if the only operation you’re allowed to do is compare one option against another. If VNM is required, it seems sort of hard to throw it out after the fact if it causes too much trouble.
What is the point of ranking other stuff relative to the 0 and 1 anchor if you already know the 1 anchor is your optimal choice? Am I misunderstanding the meaning of the 0 and 1 anchor, and it’s possible to go less than 0 or greater than 1?
“Awesomeness” is IMO the simplest effective pointer to morality that we currently have, but that morality is still inconsistent and dynamic.
The more I think about “awesomeness” as a proxy for moral reasoning, the less awesome it becomes and the more like the original painful exercise of rationality it looks.
I’ve been very entertained by this framing of the problem—very fun to read!
I find it strange that you claim the date with Satan is clearly the best option, but almost in the same breath say that the utility of whaling in the lake of fire is only 0.1% worse. It sounds like your definition of clarity is a little bit different from mine.
On the Satan date, souls are tortured, steered toward destruction, and tossed in a lake of fire. You are indifferent to those outcomes because they would have happened anyway (we can grant this a premise of the scenario). But I very much doubt you are indifferent to your role in those outcomes. I assume that you negatively value having participated in torture, damnation, and watching others suffer, but it’s not immediately clear if you had already done those things on the previous 78044 days.
Are you taking into account duration neglect? If so, is the pain of rape only slightly worse than burning in fire?
This probably sounds nitpicky; the point I’m trying to make is that computing utilities using the human brain has all kinds of strange artifacts that you probably can’t gloss over by saying “first calculate the utility of all outcomes as a number then compare all your numbers on relative scale”. We’re just not built to compute naked utilities without reference anchors, and there does not appear to be a single reference anchor to which all outcomes can be compared.
Your system seems straightforward when only 2 or 3 options are in play, but how do you compare even 10 options? 100? 1000? In the process you probably do uncover examples of your preferences that will cause you to realize you are not VNM-compliant, but what rule system do you replace it with? Or is VNM correct and the procedure is to resolve the conflict with your own broken utility function somehow?
TL;DR: I think axiom #1 (utility can be represented as a single real number) is false for human hardware, especially when paired with #5.
Edited, thanks for the style correction.
I suspect you’re probably right that more examples makes this more interesting, given the lack of upvotes. In fact, I probably found the quote relevant mostly because it more or less summed up the experience of my OWN life at the time I read it years ago.
I spent much of my youth being contrarian for contradiction’s sake, and thinking myself to be revolutionary or somehow different from those who just joined the cliques and conformed, or blindly followed their parents, or any other authority.
When I realized that defining myself against social norms, or my parents, or society was really fundamentally no different from blind conformity, only then was I free to figure out who I really was and wanted to be. Probably related: this quote.
However, the parent says this is NOT an epistemological principle, that one should prefer to get the most benefit when choosing between equally-punished crimes.
So is it saying that epistemology should not allow for equal punishments for unequal crimes? That seems less like epistemology and more like ethics.
Should our epistemology simply not waste time judging which untrue things are more false than others because we shouldn’t be believing false things anyway?
It would be great if Jason would give us more context about this one, since the meaning doesn’t seem clear without it.
BART: It’s weird, Lis: I miss him as a friend, but I miss him even more as an enemy.
LISA: I think you need Skinner, Bart. Everybody needs a nemesis. Sherlock Holmes had his Dr. Moriarty, Mountain Dew has its Mellow Yellow, even Maggie has that baby with the one eyebrow.Everyone may need a nemesis, but while Holmes had a distinct character all his own and thus used Dr. Moriarty simply to test formidable skills, Bart actually seems to create or define himself precisely in opposition to authority, as the other to authority, and not as some identifiable character in his own right.
- Mark T. Conrad, “Thus Spake Bart: On Nietzche and the Virtues of Being Bad”, The Simpsons and Philosophy: The D’Oh of Homer
I think you set a false dichotomy here—we can generate relatively safe nuclear power (thorium reactors) without existential risk, and without creating the byproducts necessary to create nuclear weapons. This is not an argument against the root comment, however.
FWIW, those that are ‘hostile’ don’t generally believe they’re going to convince the people they’re being hostile to. They’re after the peanut gallery; the undecided.
Can anyone please explain the reference to the horror seen firsthand at http://www.mail-archive.com/agi@v2.listbox.com/? I tried going back in the archives to see if something happened in August 2008 or earlier (the date of Eliezer’s post), but the list archive site doesn’t have anything older than October 2008 currently. My curiosity is piqued and I need closure on the anecdote. If nothing else, others might benefit from knowing what horrors might be avoided during AGI research.
There is no sense in which what we want may be correct or incorrect for the universe at large, because the universe does not care. Caring is a thing that minds do, and the universe is not a mind.
Yes, I agree, and I realize that that isn’t what I was actually trying to say. What I meant was, there is a set of possible, superlatively rational intelligences that may make better use of the universe than humanity (or humanity + a constrained FAI). If Omega reveals to you that such an intelligence would come about if you implement AGI with no Friendly constraint, at the cost of the extinction of humanity, would you build it? This to me drives directly to the heart of whether you value rationality over existence. You don’t personally ‘win’, humanity doesn’t ‘win’, but rationality is maximized.
My goals require that humans stick around, so our purpose with respect to my goal system does not involve disappearing into the past. I think most peoples’ goal systems are similar.
I think we need to unpack that a little, because I don’t think you mean “humans stick around more or less unchanged from their current state”. This is what I was trying to drive at about the Neanderthals. In some sense we ARE Neanderthals, slightly farther along an evolutionary timescale, but you wouldn’t likely feel any moral qualms about their extinction.
So if you do expect that humanity will continue to evolve, probably into something unrecognizable to 21st century humans, in what sense does humanity actually “stick around”? Do you mean you, personally, want to maintain your own conscious self indefinitely, so that no matter what the future, “you” will in some sense be part of it? Or do you mean “whatever intelligent life exists in the future, its ancestry is strictly human”?
When I read this parable, I was already looking for a reason to understand why Friendly AI necessarily meant “friendly to human interests or with respect to human moral systems”. Hence, my conclusion from this parable was that Eliezer was trying to show how, from the perspective of AGI, human goals and ambitions are little more than trying to find a good way to pile up our pebbles. It probably doesn’t matter that the pattern we’re currently on to is “bigger and bigger piles of primes”, since pebble-sorting isn’t certain at all to be the right mountain to be climbing. An FAI might be able to convince us that 108301 is a good pile from within our own paradigm, but how can it ever convince us that we have the wrong paradigm altogether, especially if that appears counter to our own interests?
What if Eliezer were to suddenly find himself alone among neanderthals? Knowing, with his advanced knowledge and intelligence, that neanderthals were doomed to extinction, would he be immoral or unfriendly to continue to devote his efforts to developing greater and greater intelligences, instead of trying to find a way to sustain the neanderthal paradigm for its own sake? Similarly, why should we try to restrain future AGI so that it maintains the human paradigm?
The obvious answer is that we want to stay alive, and we don’t want our atoms used for other things. But why does it matter what we want, if we aren’t ever able to know if what we want is correct for the universe at large? What if our only purpose is to simply enable the next stage of intelligence, then to disappear into the past? It seems more rational to me to abandon focus specifically on FAI, and just build AGI as quickly as possible before humanity destroys itself.
Isn’t the true mark of rationality the ability to reach a correct conclusion even if you don’t like the answer?
[edited out emotional commentary/snark]
If you can’t multiply B by a probability factor, then it’s meaningless in the context of xB + (1-x)C, also. xB by itself isn’t meaningless; it roughly means “the expected utility on a normalized scale between the utility of the outcome I least prefer and the outcome I most prefer”. nyan_sandwich even agrees that 0 and 1 aren’t magic numbers, they’re just rescaled utility values.
I’m 99% confident that that’s not what nyan_sandwich means by radiation poisoning in the original post, considering the fact that comparing utilities to 0 and 1 is exactly what he does in the hell example. If you’re not allowed to compare utilities by magnitude, then you can’t obtain an expected utility by multiplying by a probability distribution. Show the math if you think you can prove otherwise.
It’s getting hard to reference back to the original post because it keeps changing with no annotations to highlight the edits, but I think the only useful argument in the radiation poisoning section is: “don’t use units of sandwiches, whales, or orgasms because you’ll get confused by trying to experience them”. However, I don’t see any good argument for not even using Utils as a unit for a single person’s preferences. In fact, using units of Awesomes seems to me even worse than Utils, because it’s easier to accidentally experience an Awesome than a Util. Converting from Utils to unitless measurement may avoid some infinitesimal amount of radiation poisoning, but it’s no magic bullet for anything.