Gut feeling: I’d probably sacrifice myself to create a superhuman artilect, but not my kids…. I do have huge ambitions and interests going way beyond the human race – but I’m still a human.
[...]
And the better an AGI theory we have, the more intelligently we’ll be able to bias the odds. But I doubt we’ll be able to get a good AGI theory via pure armchair theorizing. I think we’ll get there via an evolving combination of theory and experiment – experiment meaning, building and interacting with early-stage proto-AGI systems of various sorts.
experiment meaning, building and interacting with early-stage proto-AGI systems of various sorts.
I’m not very familiar with Goertzel’s ideas. Does he recognize the importance of not letting the proto-AGI systems self-improve while their values are uncertain?
From what I’ve gathered Ben thinks that these experiments will reveal that friendliness is impossible, that ‘be nice to humans’ is not a stable value. I’m not sure why he thinks this.
Here is an interesting interview between Hugo de Garis and Ben Goertzel:
[...]
I’m not very familiar with Goertzel’s ideas. Does he recognize the importance of not letting the proto-AGI systems self-improve while their values are uncertain?
From what I’ve gathered Ben thinks that these experiments will reveal that friendliness is impossible, that ‘be nice to humans’ is not a stable value. I’m not sure why he thinks this.