Thanks. Still not convinced, but it will take me a full post to explain why exactly. :)
Though possibly some of this is due to a difference in definitions. When you say this:
what I consider AGI—which importantly is fully general in that it can learn new things, but will not meet the bar of doing 95% of remote jobs because it’s not likely to be human-level in all areas right away
Do you have a sense of how long you expect it will take for it to go from “can learn new things” to “doing 95% of remote jobs”? If you e.g. expect that it might still take several years for AGI to master most jobs once it has been created, then that might be more compatible with my model.
I do think our models may be pretty similar once we get past slightly different definitions of AGI.
It’s pretty hard to say how fast the types of agents I’m envisioning would take off. It could be a while between what I’m calling real AGI that can learn anything, and having it learn well and quickly enough, and be smart enough, to do 95% of remote jobs. If there aren’t breakthroughs in learning and memory systems, it could take as much as three years to really start doing substantial work, and be a slow progression toward 95% of jobs as it’s taught and teaches itself new skills. The incremental improvements on existing memory systems—RAG, vector databases, and fine-tuning for skills and new knowledge—would remain clumsier than human learning for a while.
This would be potentially very good for safety. Semi-competent agents that aren’t yet takeover-capable might wake people up to the alignment and safety issues. And I’m optimistic about the agent route for technical alignment; of course that’s a more complex issue. Intent alignment as a stepping-stone to value alignment gives the broad outline and links to more work on how instruction-following language model agents might bypass some of the worst concerns about goal mis-specification and mis-generalization and risks from optimization.
You made a good point in the linked comment that these systems will be clumsier to train and improve if they have more moving parts. My impression from the little information I have on agent projects is that this is true. But I haven’t heard of a large and skilled team taking on this task yet; it will be interesting to see what one can do. And at some point, an agent directing its own learning and performance gains an advantage that can offset the disadvantage of being harder for humans to improve and optimize the underlying system.
I look forward to that post if you get around to writing it. I’ve been toying with the idea of writing a more complete post on my short timelines and slow takeoff scenario. Thanks for posing the question and getting me to dash off a short version at least.
Thanks. Still not convinced, but it will take me a full post to explain why exactly. :)
Though possibly some of this is due to a difference in definitions. When you say this:
Do you have a sense of how long you expect it will take for it to go from “can learn new things” to “doing 95% of remote jobs”? If you e.g. expect that it might still take several years for AGI to master most jobs once it has been created, then that might be more compatible with my model.
I do think our models may be pretty similar once we get past slightly different definitions of AGI.
It’s pretty hard to say how fast the types of agents I’m envisioning would take off. It could be a while between what I’m calling real AGI that can learn anything, and having it learn well and quickly enough, and be smart enough, to do 95% of remote jobs. If there aren’t breakthroughs in learning and memory systems, it could take as much as three years to really start doing substantial work, and be a slow progression toward 95% of jobs as it’s taught and teaches itself new skills. The incremental improvements on existing memory systems—RAG, vector databases, and fine-tuning for skills and new knowledge—would remain clumsier than human learning for a while.
This would be potentially very good for safety. Semi-competent agents that aren’t yet takeover-capable might wake people up to the alignment and safety issues. And I’m optimistic about the agent route for technical alignment; of course that’s a more complex issue. Intent alignment as a stepping-stone to value alignment gives the broad outline and links to more work on how instruction-following language model agents might bypass some of the worst concerns about goal mis-specification and mis-generalization and risks from optimization.
You made a good point in the linked comment that these systems will be clumsier to train and improve if they have more moving parts. My impression from the little information I have on agent projects is that this is true. But I haven’t heard of a large and skilled team taking on this task yet; it will be interesting to see what one can do. And at some point, an agent directing its own learning and performance gains an advantage that can offset the disadvantage of being harder for humans to improve and optimize the underlying system.
I look forward to that post if you get around to writing it. I’ve been toying with the idea of writing a more complete post on my short timelines and slow takeoff scenario. Thanks for posing the question and getting me to dash off a short version at least.