Note: This is presumably not novel, but I think it ought to be better-known.
This indeed ought to be better-known. The real question is: why is it not better-known?
What I notice in the EA/Rationalist based alignment world is that a lot of people seem to believe in the conventional wisdom that nobody knows how to build myopic agents, nobody knows how to build corrigible agents, etc.
When you then ask people why they believe that, you usually get some answer ‘because MIRI’, and then when you ask further it turns out these people did not actually read MIRI’s more technical papers, they just heard about them.
The conventional wisdom ‘nobody knows how to build myopic agents’ is not true for the class of all agents, as your post illustrates. In the real world, applied AI practitioners use actually existing AI technology to build myopic agents, and corrigible agents, all the time. There are plenty of alignment papers showing how to do these things for certain models of AGI too: in the comment thread here I recently posted a list.
I speculate that the conventional rationalist/EA wisdom of ‘nobody knows how to do this’ persists because of several factors. One of them is just how social media works, Eternal September, and People Do Not Read Math, but two more interesting and technical ones are the following:
It is popular to build analytical models of AGI where your AGI will have an infinite time horizon by definition. Inside those models, making the AGI myopic without turning it into a non-AGI is then of course logically impossible. Analytical models built out of hard math can suffer from this built-in problem, and so can analytical models built out of common-sense verbal reasoning, In the hard math model case, people often discover an easy fix. In verbal models, this usually does not happen.
You can always break an agent alignment scheme by inventing an environment for the agent that breaks the agent or the scheme. See johnswentworth’s comment elsewhere in the comment section for an example of this. So it is always possible to walk away from a discussion believing that the ‘real’ alignment problem has not been solved.
This indeed ought to be better-known. The real question is: why is it not better-known?
What I notice in the EA/Rationalist based alignment world is that a lot of people seem to believe in the conventional wisdom that nobody knows how to build myopic agents, nobody knows how to build corrigible agents, etc.
When you then ask people why they believe that, you usually get some answer ‘because MIRI’, and then when you ask further it turns out these people did not actually read MIRI’s more technical papers, they just heard about them.
The conventional wisdom ‘nobody knows how to build myopic agents’ is not true for the class of all agents, as your post illustrates. In the real world, applied AI practitioners use actually existing AI technology to build myopic agents, and corrigible agents, all the time. There are plenty of alignment papers showing how to do these things for certain models of AGI too: in the comment thread here I recently posted a list.
I speculate that the conventional rationalist/EA wisdom of ‘nobody knows how to do this’ persists because of several factors. One of them is just how social media works, Eternal September, and People Do Not Read Math, but two more interesting and technical ones are the following:
It is popular to build analytical models of AGI where your AGI will have an infinite time horizon by definition. Inside those models, making the AGI myopic without turning it into a non-AGI is then of course logically impossible. Analytical models built out of hard math can suffer from this built-in problem, and so can analytical models built out of common-sense verbal reasoning, In the hard math model case, people often discover an easy fix. In verbal models, this usually does not happen.
You can always break an agent alignment scheme by inventing an environment for the agent that breaks the agent or the scheme. See johnswentworth’s comment elsewhere in the comment section for an example of this. So it is always possible to walk away from a discussion believing that the ‘real’ alignment problem has not been solved.