(examples chosen for being at different points in the spectrum between the two options, not for being likely)
Moral Universalism could be true in some sense, but not automatically compelling, and the AI would need to be programmed to find and/or follow it.
There could be a uniquely specified human morality that fulfills much of the same purpose Moral Universalism does for humans.
It might be possible to specify what we want in a more dynamic way than freezing in current customs.
The only similarity between those cases is that they involve utility calculations you disagree with. Otherwise every single detail is completely different. (e. g. the sort of utility considered, two negative utilities being traded against each other vs. trading utility elsewhere (positive and negative) for positive utility, which side of the trade the single person with the large individual utility difference is on, the presence of perverse incentives, etc, etc).
If anything it would be more logical to equate Felix with the tortured person and treat this as a reductio ad absurdum of your position on the dust speck problem. (But that would be wrong too, since the numbers aren’t actually the problem with Felix, the fact that there’s an incentive to manipulate your own utility function that way is (among other things).)
Yes, and the FFN update alert already went out. My guess is that Eliezer posted the chapter and deleted it immediately afterwards, perhaps due to some formatting problem.
If he apologizes he’ll probably either do it in person or in a similar way to last time, when he apologized for being unfair after Fawkes started shouting via Flitwick.
One major problem with such a list is that he currently doesn’t know how difficult it would be to earn more money.
And there is no problem with that if it’s restricted to non-blackmail interactions (except perhaps to the degree it’s mistaken by others to also apply to blackmail). Not responding to blackmail as a principled position and not valuing the life of the hostage highly enough for the amount asked for are completely different things.
Otherwise it would have made sense for Voldemort (who wouldn’t care about Death Eater families) to keep taking family members hostage and ask for lower and lower amounts until hitting the sum they are valued at. Either that sum would have been low enough to devastate the morale of the Order members (e. g, 100 galleons and Voldemort asks for 101 the next time) or it would be high enough to drain their funds.
A refusal to respond to blackmail needs to be unconditional.
Dumbledore seems a bit off in equating the two situations. Lucius isn’t threatening to send Hermione to Azkaban in the hope of getting something from Harry/Dumbledore; in fact he made clear that he would rather send her to Azkaban than receive the money. Therefore paying of the blood debt does not equal giving in to blackmail and Harry can save her while still maintaining a consistent position of not giving in to blackmail. Engineering similar situations without making apparent that they are engineered (and therefore blackmail) is probably too impractical to be worth the effort.
Hence the word “largely”.
Saying that moral weight is based on sentience is IMO largely a tautology. Sentience is mostly the word we use for “whatever poorly defined features of a mind give it moral weight”.
Harry could still get a false negative. Remember, Harry will feel the impulse to offer a drink to Alice if and only if if Alice is about to be surprised.
Again, we don’t know that. The soda working in two steps as you seem to suggest (detecting future surprise, then determining whether that surprise is sufficient to cause soda spitting when drunk at the right time) is consistent with what we know about the soda. But that’s not the only possibility consistent with what we know. The soda could also work in a single step and detect whether soda drunk at various points would be spit, without directly detecting surprise at all.
And more significantly:
“Severus,” Albus Dumbledore said, and his voice almost cracked, “do you realize what you are saying? If Harry Potter and Voldemort fight their war with Muggle weapons there will be nothing left of the world but fire!”
“What?” said Minerva. She had heard of guns, of course, but they weren’t that dangerous to an experienced witch -
Severus spoke as though she weren’t in the room. “Then perhaps, Headmaster, he is sending a deliberate warning to Harry Potter of exactly that; saying that any attack with Muggle weapons will be met with retaliation in kind. Command Mr. Potter to cease his use of Muggle technology in his battles; that will show him the message is received… and not give him any more ideas.” Severus frowned. “Though, come to think of it, Mr. Malfoy—and of course Miss Granger—well, on second thought a blanket prohibition on technology seems wiser—”
The old wizard pressed both his hands to his forehead, and from his lips came an unsteady voice, “I begin to hope that it is Harry behind this escape… oh, Merlin defend us all, what have I done, what have I done, what will become of the world?”
There aren’t really any other good candidates for what he might have done to cause this particular problem (even if he felt responsibility on account of e g. not having been able to beat Voldemort permanently himself it seems unlikely to phrase it like that).
We don’t know that, committing to saying Vldemort is alive conditional on actually giving them a can might suffice.
Unlike most of the room she knows Harry well enough that even him scaring a Dementor, no matter how surprising, wouldn’t make her personally afraid of Harry; she might be worried about what trouble he could cause but she knows perfectly well that he wouldn’t do anything to her. Besides it was less of a surprise for her since Dumbledore already told her Harry had developed a new charm.
The Interdict of Merlin blocking transmission of non-magical knowledge between living minds?
Could you summarize to spare us from gazing into that abyss?
We know this not to be true since Quirrel changed back into his normal appearance when the polyjuice wore off in Azkaban while he was unconscious.
Why does Dumbledore not give a quick Summary of the worst consequences of being in debt to Lucius Malfoy? It’s hard to see how that could necessitate telling secrets that cannot be revealed in public, the laws involved should already be known. Naming a few of the “certain rights” Lucius would have shouldn’t take more time than Dumbledore actually spends trying to convince Harry.
Different, but neither noticeably better nor worse. In any case those who would rather not read the speculation can just stay out of the thread, or discussion and speculation could be separated from each other.
Assigning a high probability to a complicated hypothesis and assigning a high probability to the negation of a complicated hypothesis are two very different things. I’m more than 99% sure that my neighbor is not a bank robber, even though I don’t know all that much about him and would consider a 90% certainty that he was a bank robber with comparable information massively overconfident (the example isn’t supposed to be an exact equivalent, just to illustrate the general principle, and I don’t necessarily share ArisKatsaris’ specific positions).
If you keep refraining from editing it the comment won’t display the asterisk after the positing time signifying that the comment has been edited, so people will know you made the prediction at the posting time.
This is either wrong (the utility functions of the people involved aren’t queried in the dust speck problem) or so generic as to be encompassed in the concept of “utility calculation”.
Aggregating utility functions across different people is an unsolved problem, but not necessarily an unsolvable one. One way of avoiding utility monsters would be to normalize utility functions. The obvious way to do that leads to problems such as arachnophobes getting less cake even if they like cake equally much, but IMO that’s better than utility monsters.