One question around the “Long Reflection” or around “What will AGI do?” is something like, “How bottlenecked will be by scientific advances that we’ll need to then spend significant resources on?”
I think some assumptions that this model typically holds are:
There will be decision-relevant unknowns.
Many decision-relevant unkowns will be EV-positive to work on.
Of the decision-relevant unknowns that are EV-positive to work on, these will take between 1% to 99% of our time.
(3) seems quite uncertain to me in the steady state. I believe it makes an intuitive estimate between 2 orders of magnitude, while the actual uncertainty is much higher than that. If this were the case, it would mean:
Almost all possible experiments are either trivial (<0.01% of resources, in total), or not cost-effective.
If some things are cost-effective and still expensive (they will take over 1% of the AGI lifespan), it’s likely that they will take 100%+ of the time. Even if they would take 10^10% of the time, in expectation, they could still be EV-positive to pursue. I wouldn’t be surprised if there were one single optimal thing like this in the steady-state. So this strategy would look something like, “Do all the easy things, then spend a huge amount of resources on one gigantic-sized, but EV-high challenge.”
(This was inspired by a talk that Anders Sandberg gave)
One question around the “Long Reflection” or around “What will AGI do?” is something like, “How bottlenecked will be by scientific advances that we’ll need to then spend significant resources on?”
I think some assumptions that this model typically holds are:
There will be decision-relevant unknowns.
Many decision-relevant unkowns will be EV-positive to work on.
Of the decision-relevant unknowns that are EV-positive to work on, these will take between 1% to 99% of our time.
(3) seems quite uncertain to me in the steady state. I believe it makes an intuitive estimate between 2 orders of magnitude, while the actual uncertainty is much higher than that. If this were the case, it would mean:
Almost all possible experiments are either trivial (<0.01% of resources, in total), or not cost-effective.
If some things are cost-effective and still expensive (they will take over 1% of the AGI lifespan), it’s likely that they will take 100%+ of the time. Even if they would take 10^10% of the time, in expectation, they could still be EV-positive to pursue. I wouldn’t be surprised if there were one single optimal thing like this in the steady-state. So this strategy would look something like, “Do all the easy things, then spend a huge amount of resources on one gigantic-sized, but EV-high challenge.”
(This was inspired by a talk that Anders Sandberg gave)