Thank you for this post. “should” being a label for results of the human planning algorithm in backward-chaining mode the same way that “could” is a label for results of the forward-chaining mode explains a lot. It’s obvious in retrospect (and unfortunately, only in retrospect) to me that the human brain would do both kinds of search in parallel; in big search spaces, the computational advantages are too big not to do it.
I found two minor syntax errors in the post:
“Could make sense to …”—did you mean “Could it make sense to …”?
“(something that has a charge of should-ness”—that parenthesis is never closed.
Unknown wrote:
As I’ve stated before, we are all morally obliged to prevent Eliezer from programming an AI.
Speak for yourself. I don’t think EliezerYudkowsky::Right is quite the same function as SebastianHagen::Right, but I don’t see a real chance of getting an AI that optimizes only for SebastianHagen::Right accepted as sysop. I’d rather settle for an acceptable compromise in what values our successor-civilization will be built on than see our civilization being stomped into dust by an entirely alien RPOP, or destroyed by another kind of existential catastrophe.
Thank you for this post. “should” being a label for results of the human planning algorithm in backward-chaining mode the same way that “could” is a label for results of the forward-chaining mode explains a lot. It’s obvious in retrospect (and unfortunately, only in retrospect) to me that the human brain would do both kinds of search in parallel; in big search spaces, the computational advantages are too big not to do it.
I found two minor syntax errors in the post: “Could make sense to …”—did you mean “Could it make sense to …”? “(something that has a charge of should-ness”—that parenthesis is never closed.
Unknown wrote:
Speak for yourself. I don’t think EliezerYudkowsky::Right is quite the same function as SebastianHagen::Right, but I don’t see a real chance of getting an AI that optimizes only for SebastianHagen::Right accepted as sysop. I’d rather settle for an acceptable compromise in what values our successor-civilization will be built on than see our civilization being stomped into dust by an entirely alien RPOP, or destroyed by another kind of existential catastrophe.