In giving humans reason at all, evolution took a huge risk. Surely it must have wished there was some other way, some path that made us big-brained enough to understand tradition, but not big-brained enough to question it. Maybe it searched for a mind design like that and couldn’t find one. So it was left with this ticking time-bomb, this ape that was constantly going to be able to convince itself of hare-brained and probably-fatal ideas.
This quote reminds me of much of the discussion around mesa-optimizers — how much optimization do you want the selection pressure doing, and how much optimization do you want the system you build doing? It seems that evolution ended up giving a lot of power to the system, and that was when humans really took off.
In the spoilers below I write some comments that assume you have 100% completed HPMOR (do not read otherwise):
This feels remarkably similar to the fates plotting everything to… give Harry the opportunity to actually try to figure out what to do for himself. He had enough explicit reasoning to destroy everything, but they couldn’t do the rest themselves, so they tried to put in the right safeguards, and then leave it up to him.
This quote reminds me of much of the discussion around mesa-optimizers — how much optimization do you want the selection pressure doing, and how much optimization do you want the system you build doing? It seems that evolution ended up giving a lot of power to the system, and that was when humans really took off.
In the spoilers below I write some comments that assume you have 100% completed HPMOR (do not read otherwise):
This feels remarkably similar to the fates plotting everything to… give Harry the opportunity to actually try to figure out what to do for himself. He had enough explicit reasoning to destroy everything, but they couldn’t do the rest themselves, so they tried to put in the right safeguards, and then leave it up to him.