This is one of the clearest top-to-bottom accounts of the alignment problem and related world situation that I’ve seen here in a while. Thank you for writing it.
i believe, akin to the yudkowsky-moore law of mad science, that the amount of resources it takes for the world to be destroyed — whether on purpose or by accident — keeps decreasing.
Yes it seems that in this particular way the world is becoming more and more unstable
pretty soon (probly this decade or the next), an artificial intelligence capable of undergoing recursive self-improvement (RSI) until it becomes a singleton, and at that point the fate of at least the entire future lightcone will be determined by the goals of that AI.
I think the risk is that one way or another we lock in some mostly-worthless goal to a powerful optimization process. I don’t actually think RSI is necessary for that. Beyond that, in practical ML work, we keep seeing systems that are implemented with a world model that is very far from being able to make sense of their own implementation, and actually we seem to be moving further in this direction over time. Google’s SayCan, for example, seems to be further from understanding its own implementation than some much more old-fashioned robotics tech from the 1990s (which wasn’t exactly close to being able to reason about its own implementation within its world model, either)
the values we want are a very narrow target and we currently have no solid idea how to do alignment, so when AI does take over everything we’re probly going to die. or worse, if for example we botch alignment.
Don’t assume that the correct solution to the alignment problem consists of alignment to a utility function defined over physical world states. We don’t know that for sure and many schools of moral philosophy have formulated ethics in a way that doesn’t supervene on physical world states. It’s not really clear to me that even hedonistic utilitarianism can be formulated as a utility function over physical world states.
what i call “sponge coordination” is getting everyone who’s working on AI to only build systems that are weak and safe just like a sponge, instead of building powerful AIs that take over or destroy everything
I really like the term “sponge coordination” and the definition you’ve given! And I agree that it’s not viable. The basic problem is that we humans are ourselves rapidly becoming unbounded optimizers and so the current world situation is fundamentally not an equilibrium, and we can’t just make it an equilibrium by trying to. A solution looks like a deeper change than just asking everyone to keep doing mostly what they’re currently doing except not building super-powerful AIs.
[pivotal acts] right now, we’re at least able to work on alignment, and the largest AI capability organizations are at least somewhat interacting with the alignment community; it’s not clear how that might evolve in the future if the alignment community is percieved to be trying to harm the productivity of AI capability organizations
There are two basic responses possible: either don’t perform pivotal acts, or move to a situation where we can perform pivotal acts. It is very difficult to resolve the world situation with AI if one is trying to do so while not harming the productivity of any AI companies. That would be like trying to upgrade a bird to a jumbo jet while keeping it alive throughout.
it’s not clear that we have to aim for first non-singleton aligned AIs, and then FAS; it could be that the best way to maximize our expected utility is to aim straight for FAS.
This is true. Thank you for having the courage to say it.
in retrospect of this now-formalized view of the work at hand, a lot of my criticisms of approaches to AI alignment i’ve seen are that they’re either [hand-wavey or uncompetitive]
Indeed. It’s a difficult problem and it’s okay to formulate and discuss hand-wavey or uncompetitive plans as a stepping stone to formulating precise and competitive plans.
finally, tractable solutions — by virtue of being easier to implement — risk boosting AI capability, and when you’re causing damage you really want to know that it’s helping alignment enough to be an overall expected net good
This is a very good sentence
this is the sort of general robustness i think we’ll be needing, to trust an AI with singletonhood. and without singletonhood
I think you’re right to focus on both formalism and trustworthiness, though please do also investigate whether and in what way the former actually leads to the latter.
nothing guarantees that, just because ML is how we got to highly capable unaligned systems, it’s also the shortest route to highly capable aligned systems
Yeah well said.
a much more reasonable approach is to first figure out what “aligned” means, and then figure out how to build something which from the ground up is designed to have that property
This is my favorite sentence in your essay
in my opinion we should figure out what alignment means, what desiderata would formalize it, and then build something that has those.
Again, just notice that you are assuming a kind of pre-baked blueprint here, just in putting in the middle step “what desiderata would formalize it”. Formal systems are a tool like any other tool: powerful, useful, important to understand the contexts in which they are effective and the contexts in which they are ineffective.
This is one of the clearest top-to-bottom accounts of the alignment problem and related world situation that I’ve seen here in a while. Thank you for writing it.
Yes it seems that in this particular way the world is becoming more and more unstable
I think the risk is that one way or another we lock in some mostly-worthless goal to a powerful optimization process. I don’t actually think RSI is necessary for that. Beyond that, in practical ML work, we keep seeing systems that are implemented with a world model that is very far from being able to make sense of their own implementation, and actually we seem to be moving further in this direction over time. Google’s SayCan, for example, seems to be further from understanding its own implementation than some much more old-fashioned robotics tech from the 1990s (which wasn’t exactly close to being able to reason about its own implementation within its world model, either)
Don’t assume that the correct solution to the alignment problem consists of alignment to a utility function defined over physical world states. We don’t know that for sure and many schools of moral philosophy have formulated ethics in a way that doesn’t supervene on physical world states. It’s not really clear to me that even hedonistic utilitarianism can be formulated as a utility function over physical world states.
I really like the term “sponge coordination” and the definition you’ve given! And I agree that it’s not viable. The basic problem is that we humans are ourselves rapidly becoming unbounded optimizers and so the current world situation is fundamentally not an equilibrium, and we can’t just make it an equilibrium by trying to. A solution looks like a deeper change than just asking everyone to keep doing mostly what they’re currently doing except not building super-powerful AIs.
There are two basic responses possible: either don’t perform pivotal acts, or move to a situation where we can perform pivotal acts. It is very difficult to resolve the world situation with AI if one is trying to do so while not harming the productivity of any AI companies. That would be like trying to upgrade a bird to a jumbo jet while keeping it alive throughout.
This is true. Thank you for having the courage to say it.
Indeed. It’s a difficult problem and it’s okay to formulate and discuss hand-wavey or uncompetitive plans as a stepping stone to formulating precise and competitive plans.
This is a very good sentence
I think you’re right to focus on both formalism and trustworthiness, though please do also investigate whether and in what way the former actually leads to the latter.
Yeah well said.
This is my favorite sentence in your essay
Again, just notice that you are assuming a kind of pre-baked blueprint here, just in putting in the middle step “what desiderata would formalize it”. Formal systems are a tool like any other tool: powerful, useful, important to understand the contexts in which they are effective and the contexts in which they are ineffective.