Probably whoever controls AGI will be able to use it to get to ASI shortly thereafter—maybe in another year, give or take a year.
2. Wait a second. How fast are humans building ICs for AI compute? Let’s suppose humans double the total AI compute available on the planet over 2 years (Moore’s law + effort has gone to wartime levels of investment since AI IC’s are money printers). An AGI means there is now a large economic incentive to ‘greedy’ maximize the gains from the AGI, why take a risk on further R&D?
But say all the new compute goes into AI R&D.
a. How much of a compute multiplier do you need for AGI->ASI training?
b. How much more compute does an ASI instance take up? You have noticed that there is diminishing throughput for high serial speed, are humans going to want to run an ASI instance that takes OOMs more compute for marginally more performance?
c. How much better is the new ASI? If you can ‘only’ spare 10x more compute than for the AGI, why do you believe it will be able to:
Probably whoever controls ASI will have access to a spread of powerful skills/abilities and will be able to build and wield technologies that seem like magic to us, just as modern tech would seem like magic to medievals.
This will probably give them godlike powers over whoever doesn’t control ASI.
Looks like ~4x better pass rate for ~3k times as much compute?
And then if we predict forward for the ASI, we’re dividing the error rate by another factor of 4 in exchange for 3k times as much compute?
Is that going to be enough for magic? Might it also require large industrial facilities to construct prototypes and learn from experiments? Perhaps some colliders larger than CERN? Those take time to build...
For another data source:
Assuming the tokens processed is linearly proportional to compute required, Deepmind burned 2.3 times the compute and used algorithmic advances for Gemini 1 for barely more performance than GPT-4.
I think your other argument will be that algorithmic advances are possible that are enormous? Could you get to an empirical bounds on that, such as looking at the diminishing series of performance:(architectural improvement) and projecting forward?
5. Agree
6. Conditional on having an ASI strong enough that you can’t control it the easy way
7. sure
8. conditional on needing to do this
9. conditional on having a choice, no point in being skeptical if you must build ASI or lose
10. Agree
I think could be an issue with your model, @Daniel Kokotajlo . It’s correct for the short term, but you have essentially the full singularity happening all at once over a few years. If it took 50 years for the steps you think will take 2-5 it would still be insanely quick by the prior history for human innovation...
Truthseeking note : I just want to know what will happen. We have some evidence now. You personally have access to more evidence as an insider, as you can get the direct data for OAI’s models, and you probably can ask the latest new joiner from deepmind for what they remember. With that evidence you could more tightly bound your model and see if the math checks out.
reasonable
2. Wait a second. How fast are humans building ICs for AI compute? Let’s suppose humans double the total AI compute available on the planet over 2 years (Moore’s law + effort has gone to wartime levels of investment since AI IC’s are money printers). An AGI means there is now a large economic incentive to ‘greedy’ maximize the gains from the AGI, why take a risk on further R&D?
But say all the new compute goes into AI R&D.
a. How much of a compute multiplier do you need for AGI->ASI training?
b. How much more compute does an ASI instance take up? You have noticed that there is diminishing throughput for high serial speed, are humans going to want to run an ASI instance that takes OOMs more compute for marginally more performance?
c. How much better is the new ASI? If you can ‘only’ spare 10x more compute than for the AGI, why do you believe it will be able to:
Looks like ~4x better pass rate for ~3k times as much compute?
And then if we predict forward for the ASI, we’re dividing the error rate by another factor of 4 in exchange for 3k times as much compute?
Is that going to be enough for magic? Might it also require large industrial facilities to construct prototypes and learn from experiments? Perhaps some colliders larger than CERN? Those take time to build...
For another data source:
Assuming the tokens processed is linearly proportional to compute required, Deepmind burned 2.3 times the compute and used algorithmic advances for Gemini 1 for barely more performance than GPT-4.
I think your other argument will be that algorithmic advances are possible that are enormous? Could you get to an empirical bounds on that, such as looking at the diminishing series of performance:(architectural improvement) and projecting forward?
5. Agree
6. Conditional on having an ASI strong enough that you can’t control it the easy way
7. sure
8. conditional on needing to do this
9. conditional on having a choice, no point in being skeptical if you must build ASI or lose
10. Agree
I think could be an issue with your model, @Daniel Kokotajlo . It’s correct for the short term, but you have essentially the full singularity happening all at once over a few years. If it took 50 years for the steps you think will take 2-5 it would still be insanely quick by the prior history for human innovation...
Truthseeking note : I just want to know what will happen. We have some evidence now. You personally have access to more evidence as an insider, as you can get the direct data for OAI’s models, and you probably can ask the latest new joiner from deepmind for what they remember. With that evidence you could more tightly bound your model and see if the math checks out.