Assuming that users can figure out intended goals for the AGI that are valuable and pivotal, the identification problem for describing what constitutes a safe performance of that Task, might be simpler than giving the AGI a complete description of normativity in general. [...] Relative to the problem of building a Sovereign, trying to build a Task AGI instead might step down the problem from “impossibly difficult” to “insanely difficult”, while still maintaining enough power in the AI to perform pivotal acts.
That is fascinating. I hadn’t seen his “task AGI” plan, and I agree it’s highly overlapping with this proposal—more so than any other work I was aware of. What’s most fascinating is that YK doesn’t currently endorse that plan, even though it looks to me as though on main reason he calls it “insanely difficult” has been mitigated greatly by the success of LLMs in understanding human semantics and therefore preferences. We are already well up his Do-What-I-Mean hierarchy, arguably at an adequate level for safety/success even before inevitable improvements on the way to AGI. In addition, the slow takeoff path we’re on seems to also make the project easier (although less likely to allow a pivotal act before we have many AGIs causing coordination problems).
So, why does YK think we should Shut It Down instead of build DWIM AGI? Ii’ve been trying to figure this out. I think his principal reasons are two: reinforcement learning sounds like a good way to get any central goal somewhat wrong, and being somewhat wrong could well be too much for survival. As I mentioned in the article, I think we have good alternatives to RL alignment, particularly for the AGI we’re most likely to build first, and I don’t think YK has ever considered proposals of that type. Second, he thinks that humans are stunningly foolish, and that competitive race dynamiccs will make them even more prone to critical errors, even for a project that’s in-principal quite accomplishable. On this, I’m afraid I agree. So if I were in charge, I would indeed Shut It Down instead of shooting for DWIM alignment. But I’m not, and neither is YK. He thinks it’s worth trying, to at least slow down AGI progress; I think it’s more critical to use the time we’ve got to refine the alignment approaches that are most likely to actually be deployed.
I’m not so sure it’s the same—my interpretation was something like:
Yudkowsky plan: Make an AI that designs a certain kind of nanobot
Seth plan: Make an AI that does what I tell it to do, and then I will tell it to design a certain kind of nanobot
For example, in this comment, @Rob Bensinger was brainstorming nanobot-specific things that one might put into the source code. (Warning that Rob is not Eliezer.) (Related.)
I’m sure it’s not the same, particularly since neither one has really been fully fleshed out and thought through. In particular, Yudkowsky doesn’t focus on the advantages of instructing the AGI to tell you the truth, and interacting with it as it gets smarter. I’d guess that’s because he was still anticipating a faster takeoff than network-based AGI affords.
But to give credit where it’s due, I think that literal instruction-following was probably part of (but not the whole of) his conception of task-based AGI. From the discussion thread w Paul Christiano following the task directed AGI article on Greater Wrong:
The AI is getting short-term objectives from humans and carrying them out under some general imperative to do things conservatively or with ‘low unnecessary impact’ in some sense of that, and describes plans and probable consequences that are subject to further human checking, and then does them, and then the humans observe the results and file more requests.
And the first line of that article:
A task-based AGI is an AGI intended to follow a series of human-originated orders, with these orders each being of limited scope [...]
These sections, in connection with the lack of reference to instructions and checking for most of the presentation, suggest to me that he probably was thinking of things like hard-coding it to design nanotech, melt down GPUs (or whatever) and then delete itself, but also of more online, continuous instruction-following AGI more similar to my conception of likely AGI projects. Bensinger may have been pursuing one part of that broader conception.
This plan seems to be roughly the same as Yudkowsky’s plan.
That is fascinating. I hadn’t seen his “task AGI” plan, and I agree it’s highly overlapping with this proposal—more so than any other work I was aware of. What’s most fascinating is that YK doesn’t currently endorse that plan, even though it looks to me as though on main reason he calls it “insanely difficult” has been mitigated greatly by the success of LLMs in understanding human semantics and therefore preferences. We are already well up his Do-What-I-Mean hierarchy, arguably at an adequate level for safety/success even before inevitable improvements on the way to AGI. In addition, the slow takeoff path we’re on seems to also make the project easier (although less likely to allow a pivotal act before we have many AGIs causing coordination problems).
So, why does YK think we should Shut It Down instead of build DWIM AGI? Ii’ve been trying to figure this out. I think his principal reasons are two: reinforcement learning sounds like a good way to get any central goal somewhat wrong, and being somewhat wrong could well be too much for survival. As I mentioned in the article, I think we have good alternatives to RL alignment, particularly for the AGI we’re most likely to build first, and I don’t think YK has ever considered proposals of that type. Second, he thinks that humans are stunningly foolish, and that competitive race dynamiccs will make them even more prone to critical errors, even for a project that’s in-principal quite accomplishable. On this, I’m afraid I agree. So if I were in charge, I would indeed Shut It Down instead of shooting for DWIM alignment. But I’m not, and neither is YK. He thinks it’s worth trying, to at least slow down AGI progress; I think it’s more critical to use the time we’ve got to refine the alignment approaches that are most likely to actually be deployed.
I’m not so sure it’s the same—my interpretation was something like:
Yudkowsky plan: Make an AI that designs a certain kind of nanobot
Seth plan: Make an AI that does what I tell it to do, and then I will tell it to design a certain kind of nanobot
For example, in this comment, @Rob Bensinger was brainstorming nanobot-specific things that one might put into the source code. (Warning that Rob is not Eliezer.) (Related.)
I’m sure it’s not the same, particularly since neither one has really been fully fleshed out and thought through. In particular, Yudkowsky doesn’t focus on the advantages of instructing the AGI to tell you the truth, and interacting with it as it gets smarter. I’d guess that’s because he was still anticipating a faster takeoff than network-based AGI affords.
But to give credit where it’s due, I think that literal instruction-following was probably part of (but not the whole of) his conception of task-based AGI. From the discussion thread w Paul Christiano following the task directed AGI article on Greater Wrong:
And the first line of that article:
These sections, in connection with the lack of reference to instructions and checking for most of the presentation, suggest to me that he probably was thinking of things like hard-coding it to design nanotech, melt down GPUs (or whatever) and then delete itself, but also of more online, continuous instruction-following AGI more similar to my conception of likely AGI projects. Bensinger may have been pursuing one part of that broader conception.