As the planetary death rate is 100 percent, and the increase in capabilities from tool AGI is realistically the only thing that could change that within the lifetimes of all living humans today, why is working on AI capabilities bad?
How confident are you that it is bad? Have you seen the CAIS proposal?
Not Zvi, but flourishing human lives are still good even when they end in death. If you only care about humans alive today then faster AGI development is still very likely negative unless you are a lot more optimistic about alignment than most of us are.
If you care about future humans in addition to current humans then faster AGI development is massively negative, because of the likelihood of extinction.
150,000 people die every day. That’s not a small price for any delays to AGI development. Now, we need to do this right: AGI without alignment just kills everyone; it doesn’t solve anything. But the faster we get aligned AI, the better. And trying to slow down capabilities research without much thought into the endgame seems remarkably callous.
Eliezer has mentioned the idea of trying to invent a new paradigm for AI, outside of the conventional neural net/backpropagation model. The context was more “what would you do with unlimited time and money” than “what do you intend irl”, but this seems to be his ideal play. Now, I wish him the best of luck with the endeavor if he tries it, but do we have any evidence that another paradigm is possible?
Evolved minds use something remarkably close to the backprop model, and the only other model we’ve seen work is highly mechanistic AI like Deep Blue. The Deep Blue model doesn’t generalize well, nor is it capable of much creativity. A priori, it seems somewhat unlikely that any other AI paradigm exists: why would math just happen to permit it? And if we oppose capabilities research until we find something like a new AI model, there’s a good chance that we oppose it all the way to the singularity, rather than ever contributing to a Friendly system. That’s not an outcome anyone wants, and it seems to be the default outcome of incautious pessimism.
Another general way to look at it is think about what a policy IS.
A policy is the set of rules any AI system uses to derive the input from the output. It’s how a human or humanoid robot walks, or talks, or any intelligent act. (Since there are rules that update the rules)
Well any non trivial task you realize the policy HAS to account for thousands of variables, including intermediates generated during the policy calculation. It trivially, for any “competitive” policy that does complex things, can exceed the complexity that a human being can grok.
So no matter the method you use to generate a policy it will exceed your ability to review it, and COULD contain “if condition do bad thing” in it.
Also not Zvi, but reducing the death rate from 100% still requires at least some of humanity to survive long enough to recognize those gains. If the AI paperclips everyone, there’d be no one left to immortalize, unless it decides to make new humans down the road for some reason.
Zvi, you didn’t respond in the last thread.
As the planetary death rate is 100 percent, and the increase in capabilities from tool AGI is realistically the only thing that could change that within the lifetimes of all living humans today, why is working on AI capabilities bad?
How confident are you that it is bad? Have you seen the CAIS proposal?
Not Zvi, but flourishing human lives are still good even when they end in death. If you only care about humans alive today then faster AGI development is still very likely negative unless you are a lot more optimistic about alignment than most of us are.
If you care about future humans in addition to current humans then faster AGI development is massively negative, because of the likelihood of extinction.
150,000 people die every day. That’s not a small price for any delays to AGI development. Now, we need to do this right: AGI without alignment just kills everyone; it doesn’t solve anything. But the faster we get aligned AI, the better. And trying to slow down capabilities research without much thought into the endgame seems remarkably callous.
Eliezer has mentioned the idea of trying to invent a new paradigm for AI, outside of the conventional neural net/backpropagation model. The context was more “what would you do with unlimited time and money” than “what do you intend irl”, but this seems to be his ideal play. Now, I wish him the best of luck with the endeavor if he tries it, but do we have any evidence that another paradigm is possible?
Evolved minds use something remarkably close to the backprop model, and the only other model we’ve seen work is highly mechanistic AI like Deep Blue. The Deep Blue model doesn’t generalize well, nor is it capable of much creativity. A priori, it seems somewhat unlikely that any other AI paradigm exists: why would math just happen to permit it? And if we oppose capabilities research until we find something like a new AI model, there’s a good chance that we oppose it all the way to the singularity, rather than ever contributing to a Friendly system. That’s not an outcome anyone wants, and it seems to be the default outcome of incautious pessimism.
Another general way to look at it is think about what a policy IS.
A policy is the set of rules any AI system uses to derive the input from the output. It’s how a human or humanoid robot walks, or talks, or any intelligent act. (Since there are rules that update the rules)
Well any non trivial task you realize the policy HAS to account for thousands of variables, including intermediates generated during the policy calculation. It trivially, for any “competitive” policy that does complex things, can exceed the complexity that a human being can grok.
So no matter the method you use to generate a policy it will exceed your ability to review it, and COULD contain “if condition do bad thing” in it.
unless you are a lot more optimistic about alignment than most of us are.
CAIS is using current known engineering techniques. It may work.
Also not Zvi, but reducing the death rate from 100% still requires at least some of humanity to survive long enough to recognize those gains. If the AI paperclips everyone, there’d be no one left to immortalize, unless it decides to make new humans down the road for some reason.