I have noticed this too! But I think if we unwind the “explain how you got the blue tentacle for your hand” in the sense it is used in this post, it would be “give a reason why waking up with a blue tentacle is not unimaginable”. In other words, it’s used as “human explain” not “formal explain”. I would guess that a lot of people when given this question will concoct some explanation on the edge of their understanding where it becomes nontechnical, some may even say “mutation”. Ultimately, it would be an answer that feels “fit” a whole lot more than they fear waking up with a tentacle going to bed. Eliezer makes up a story of being teleported to a comic, which, I suppose, also feels fit a lot more than we actually fear being teleported when going to bed, because the mere fact of assessing an explanation for fitness allocates way more brain resources than needed to consider this possibility.
Now thinking about it, I can see that questions like “blue tentacle” really show where one’s understanding stops being technical and starts assigning more fitness to impossible things. For some people it’s “mutation”, but when you actually know how mutations work this answer simply won’t occur to you as an actual explanation you are requested to give, it only occurs as a bad example.
I have noticed this too! But I think if we unwind the “explain how you got the blue tentacle for your hand” in the sense it is used in this post, it would be “give a reason why waking up with a blue tentacle is not unimaginable”. In other words, it’s used as “human explain” not “formal explain”. I would guess that a lot of people when given this question will concoct some explanation on the edge of their understanding where it becomes nontechnical, some may even say “mutation”. Ultimately, it would be an answer that feels “fit” a whole lot more than they fear waking up with a tentacle going to bed. Eliezer makes up a story of being teleported to a comic, which, I suppose, also feels fit a lot more than we actually fear being teleported when going to bed, because the mere fact of assessing an explanation for fitness allocates way more brain resources than needed to consider this possibility.
Now thinking about it, I can see that questions like “blue tentacle” really show where one’s understanding stops being technical and starts assigning more fitness to impossible things. For some people it’s “mutation”, but when you actually know how mutations work this answer simply won’t occur to you as an actual explanation you are requested to give, it only occurs as a bad example.