Things are actually a bit worse than this, because there is also no theorem that says there is only one valley, so there’s no guarantee that even after you climb out of this valley, your next step won’t cause you to go off a precipice.
BTW, there’s a very similar issue in economics, which goes under the name of the Theory of the Second Best. Markets will allocate resources efficiently if they are perfectly competitive and complete, but there is no guarantee that any incremental progress towards that state, such creating some markets that were previously missing, or making some markets more competitive, will improve social welfare.
I can see at least one second valley instance in my own experience—someone born religious initially thinks a creator probably exists, then learns evidence against, and assigns fairly high probability of no creator. later on, he becomes more rational and considers simulation arguments, and needs to re-adjust up his estimate. (Bostrom I think was at ~1/3 probability that we are simulated, based on his Simulation Argument paper). Am I interpreting second valley the same way you are, Wei?
Things are actually a bit worse than this, because there is also no theorem that says there is only one valley, so there’s no guarantee that even after you climb out of this valley, your next step won’t cause you to go off a precipice.
BTW, there’s a very similar issue in economics, which goes under the name of the Theory of the Second Best. Markets will allocate resources efficiently if they are perfectly competitive and complete, but there is no guarantee that any incremental progress towards that state, such creating some markets that were previously missing, or making some markets more competitive, will improve social welfare.
I agree there’s no guarantee in principle, but I can’t recall ever running into a second valley in practice.
I can see at least one second valley instance in my own experience—someone born religious initially thinks a creator probably exists, then learns evidence against, and assigns fairly high probability of no creator. later on, he becomes more rational and considers simulation arguments, and needs to re-adjust up his estimate. (Bostrom I think was at ~1/3 probability that we are simulated, based on his Simulation Argument paper). Am I interpreting second valley the same way you are, Wei?