Thank you! Outside perspectives from someone who’s bothered to spend their time looking at the arguments are really useful.
I’m disturbed that the majority of community responses seem defensive in tone. Responding to attempts at constructive criticism with defensiveness is a really bad sign for becoming Less Wrong.
I think the major argument missing from what you’ve read is that giving an AGI a goal that works for humanity is surprisingly really hard. Accurately expressing human goals, let alone as an RL training set, in a way that stays stable long-term one an AGI has (almost inevitably) escapes your control, is really difficult.
But that’s on the object level, which isn’t the point of your post. I include it as my suggestion for the biggest thing we’re leaving out in brief summaries of the arguments.
I think the community at large tends to be really good at alignment logic, and pretty bad at communicating succinctly with the world at large, and we had better correct this or it might get us all killed. Thanks so much for trying to push us in that direction!
Thank you! Outside perspectives from someone who’s bothered to spend their time looking at the arguments are really useful.
I’m disturbed that the majority of community responses seem defensive in tone. Responding to attempts at constructive criticism with defensiveness is a really bad sign for becoming Less Wrong.
I think the major argument missing from what you’ve read is that giving an AGI a goal that works for humanity is surprisingly really hard. Accurately expressing human goals, let alone as an RL training set, in a way that stays stable long-term one an AGI has (almost inevitably) escapes your control, is really difficult.
But that’s on the object level, which isn’t the point of your post. I include it as my suggestion for the biggest thing we’re leaving out in brief summaries of the arguments.
I think the community at large tends to be really good at alignment logic, and pretty bad at communicating succinctly with the world at large, and we had better correct this or it might get us all killed. Thanks so much for trying to push us in that direction!