Just to add to your thinking: consider also your hypothetical “experiment A vs experiment B”. Suppose the AI tasked with the decision is both more capable than the best humans, but by a plausible margin (it’s only 50 percent better) and can make the decision in 1 hour. (At 10 tokens a second it deliberates for a while, using tools and so on).
But the experiment is an AI training run and results won’t be available for 3 weeks.
So the actual performance comparison is the human took one week and had a 50 percent pSuccess, and the AI took 1 hour and had a 75 percent pSuccess.
So your success per day is 75/(21 days) and for the human it’s 50/(28 days). Or in real world terms, the AI is 2 times as effective.
In this example it is an enormous amount smarter, completing 40-80 hours of work in 1 hour and better than the best human experts by a 50 percent margin. Probably the amount of compute required to accomplish this (and the amount of electricity and patterned silicon) is also large.
Yet in real world terms it is “only” twice as good. I suspect this generalizes a lot of places, where AGI is a large advance but it won’t be enough to foom due to the real world gain being much smaller.
Just to add to your thinking: consider also your hypothetical “experiment A vs experiment B”. Suppose the AI tasked with the decision is both more capable than the best humans, but by a plausible margin (it’s only 50 percent better) and can make the decision in 1 hour. (At 10 tokens a second it deliberates for a while, using tools and so on).
But the experiment is an AI training run and results won’t be available for 3 weeks.
So the actual performance comparison is the human took one week and had a 50 percent pSuccess, and the AI took 1 hour and had a 75 percent pSuccess.
So your success per day is 75/(21 days) and for the human it’s 50/(28 days). Or in real world terms, the AI is 2 times as effective.
In this example it is an enormous amount smarter, completing 40-80 hours of work in 1 hour and better than the best human experts by a 50 percent margin. Probably the amount of compute required to accomplish this (and the amount of electricity and patterned silicon) is also large.
Yet in real world terms it is “only” twice as good. I suspect this generalizes a lot of places, where AGI is a large advance but it won’t be enough to foom due to the real world gain being much smaller.