Humans building AGI ASI likely leads to human extinction.
I disagree: we have many other routes of expansion, including biological improvement, cyborgism, etc. This seems akin to a cultic thinking and akin to Spartan ideas of “only hoplite warfare must be adopted or defeat ensues.”
The “limitations of physics” is quite extensive, and applies even to the pipeline leading up to anything like ASI. I am quite confident that any genuine dedication to the grim game would be more than enough to prevent it, and defiance of it leads to much more likelihood of nuclear winter worlds than ASI dominance.
But I also disagree on your prior of “this world in months”, I suppose we will see in December.
I stated “years if not months”. I agree there is probably not yet enough compute even built to find a true ASI. I assume we will need to explore many cognitive architectures, which means repeating gpt-4 scale training runs thousands of times in order to learn what actually works.
“Months” would be if I am wrong and it’s just a bit of RL away
I find it happy that we probably don’t have enough compute and it is likely this will be restricted even at this fairly early level, long before more extreme measures are needed.
Additionally, I think one should support the Grim Trigger even if you want ASI, because it forces development along more “safe” lines to prevent being Grimmed. It also encourages non-ASI advancement as alternate routes, effectively being a form of regulation.
We will see. There is incredible economic pressure right now to build as much compute as physically possible. Without coordinated government action across all countries capable of building the hardware, this is the default outcome.
Humans building AGI ASI likely leads to human extinction.
I disagree: we have many other routes of expansion, including biological improvement, cyborgism, etc. This seems akin to a cultic thinking and akin to Spartan ideas of “only hoplite warfare must be adopted or defeat ensues.”
The “limitations of physics” is quite extensive, and applies even to the pipeline leading up to anything like ASI. I am quite confident that any genuine dedication to the grim game would be more than enough to prevent it, and defiance of it leads to much more likelihood of nuclear winter worlds than ASI dominance.
But I also disagree on your prior of “this world in months”, I suppose we will see in December.
I stated “years if not months”. I agree there is probably not yet enough compute even built to find a true ASI. I assume we will need to explore many cognitive architectures, which means repeating gpt-4 scale training runs thousands of times in order to learn what actually works.
“Months” would be if I am wrong and it’s just a bit of RL away
I find it happy that we probably don’t have enough compute and it is likely this will be restricted even at this fairly early level, long before more extreme measures are needed.
Additionally, I think one should support the Grim Trigger even if you want ASI, because it forces development along more “safe” lines to prevent being Grimmed. It also encourages non-ASI advancement as alternate routes, effectively being a form of regulation.
We will see. There is incredible economic pressure right now to build as much compute as physically possible. Without coordinated government action across all countries capable of building the hardware, this is the default outcome.