I do believe you get in an interesting confusion when talking about the blue tentacle example. When you’re asked to answer why, you’re not prompted for scenarios which are likely beforehand, you’re prompted for argmax(P(Ai|T)) (T is the situation of waking up that day with a blue tentacle, argmax taken to choose from i’s), which equals—by theoreme of Bayes—argmax(P(T|Ai)*P(Ai)/P(T))=argmax(P(T|Ai)*P(Ai))=argmax(P(Ai&T)). And this—despite the fact that ALL P(Ai&T) are very small and the P(T), equaling their sum, is also too small to worry about it in usual life—is an allowed mathematical task in itself. The task of asking “who is taller, John or Mary” does not entail either John or Mary is tall—in fact, they can be two dwarves. Same logic applies here.
I have noticed this too! But I think if we unwind the “explain how you got the blue tentacle for your hand” in the sense it is used in this post, it would be “give a reason why waking up with a blue tentacle is not unimaginable”. In other words, it’s used as “human explain” not “formal explain”. I would guess that a lot of people when given this question will concoct some explanation on the edge of their understanding where it becomes nontechnical, some may even say “mutation”. Ultimately, it would be an answer that feels “fit” a whole lot more than they fear waking up with a tentacle going to bed. Eliezer makes up a story of being teleported to a comic, which, I suppose, also feels fit a lot more than we actually fear being teleported when going to bed, because the mere fact of assessing an explanation for fitness allocates way more brain resources than needed to consider this possibility.
Now thinking about it, I can see that questions like “blue tentacle” really show where one’s understanding stops being technical and starts assigning more fitness to impossible things. For some people it’s “mutation”, but when you actually know how mutations work this answer simply won’t occur to you as an actual explanation you are requested to give, it only occurs as a bad example.
I do believe you get in an interesting confusion when talking about the blue tentacle example. When you’re asked to answer why, you’re not prompted for scenarios which are likely beforehand, you’re prompted for argmax(P(Ai|T)) (T is the situation of waking up that day with a blue tentacle, argmax taken to choose from i’s), which equals—by theoreme of Bayes—argmax(P(T|Ai)*P(Ai)/P(T))=argmax(P(T|Ai)*P(Ai))=argmax(P(Ai&T)). And this—despite the fact that ALL P(Ai&T) are very small and the P(T), equaling their sum, is also too small to worry about it in usual life—is an allowed mathematical task in itself. The task of asking “who is taller, John or Mary” does not entail either John or Mary is tall—in fact, they can be two dwarves. Same logic applies here.
I have noticed this too! But I think if we unwind the “explain how you got the blue tentacle for your hand” in the sense it is used in this post, it would be “give a reason why waking up with a blue tentacle is not unimaginable”. In other words, it’s used as “human explain” not “formal explain”. I would guess that a lot of people when given this question will concoct some explanation on the edge of their understanding where it becomes nontechnical, some may even say “mutation”. Ultimately, it would be an answer that feels “fit” a whole lot more than they fear waking up with a tentacle going to bed. Eliezer makes up a story of being teleported to a comic, which, I suppose, also feels fit a lot more than we actually fear being teleported when going to bed, because the mere fact of assessing an explanation for fitness allocates way more brain resources than needed to consider this possibility.
Now thinking about it, I can see that questions like “blue tentacle” really show where one’s understanding stops being technical and starts assigning more fitness to impossible things. For some people it’s “mutation”, but when you actually know how mutations work this answer simply won’t occur to you as an actual explanation you are requested to give, it only occurs as a bad example.