I think I communicated unclearly and it’s my fault, sorry for that: I shouldn’t have used the phrase “any easily specifiable task” for what I meant, because I didn’t mean it to include “optimize the entire human lightcone w.r.t. human values”. In fact, I was being vague and probably there isn’t really a sensible notion that I was trying to point to. However, to clarify what I really was trying to say: What I mean by “hard problem of alignment” is : “develop an AI system that keeps humanity permanently safe from misaligned AI (and maybe other x risks), and otherwise leaves humanity to figure out what it wants and do what it wants without restricting it in much of any way except some relatively small volume of behaviour around ‘things that cause existential catastrophe’ ” (maybe this ends up being to develop a second version AI that then gets free reign to optimize the universe w.r.t. human values, but I’m a bit skeptical). I agree that “solve all of human psychology and moral …” is significantly harder than that (as a technical problem). (maybe I’d call this the “even harder problem”).
Ehh, maybe I am changing my mind and also agree that even what I’m calling the hard problem is significantly more difficult than the pivotal act you’re describing, if you can really do it without modelling humans, by going to mars and doing WBE. But then still the whole thing would have to rely on the WBE, and I find it implausible to do it without it (currently, but you’ve been updating me about lack of need of human modelling so maybe I’ll update here too). Basically the pivotal act is very badly described as merely “melt the gpus”, and is much more crazy than what I thought it was meant to refer to.
Regarding “rogue”: I just looked up the meaning and I thought it meant “independent from established authority”, but it seems to mean “cheating/dishonest/mischievous”, so I take back that statement about rogueness.
I think I communicated unclearly and it’s my fault, sorry for that: I shouldn’t have used the phrase “any easily specifiable task” for what I meant, because I didn’t mean it to include “optimize the entire human lightcone w.r.t. human values”. In fact, I was being vague and probably there isn’t really a sensible notion that I was trying to point to. However, to clarify what I really was trying to say: What I mean by “hard problem of alignment” is : “develop an AI system that keeps humanity permanently safe from misaligned AI (and maybe other x risks), and otherwise leaves humanity to figure out what it wants and do what it wants without restricting it in much of any way except some relatively small volume of behaviour around ‘things that cause existential catastrophe’ ” (maybe this ends up being to develop a second version AI that then gets free reign to optimize the universe w.r.t. human values, but I’m a bit skeptical). I agree that “solve all of human psychology and moral …” is significantly harder than that (as a technical problem). (maybe I’d call this the “even harder problem”).
Ehh, maybe I am changing my mind and also agree that even what I’m calling the hard problem is significantly more difficult than the pivotal act you’re describing, if you can really do it without modelling humans, by going to mars and doing WBE. But then still the whole thing would have to rely on the WBE, and I find it implausible to do it without it (currently, but you’ve been updating me about lack of need of human modelling so maybe I’ll update here too). Basically the pivotal act is very badly described as merely “melt the gpus”, and is much more crazy than what I thought it was meant to refer to.
Regarding “rogue”: I just looked up the meaning and I thought it meant “independent from established authority”, but it seems to mean “cheating/dishonest/mischievous”, so I take back that statement about rogueness.
I’ll respond to the “public opinion” thing later.