I’d think it’d be great if people stopped thinking in terms of some fuzzy abstraction “AI” which is basically a basket for all sorts of biases. If we consider the software that can self improve ‘intelligently’ in our opinion, in general, the minimal such software is something like an optimizing compiler that when compiling it’s source will even optimize its ability to optimize. This sort of thing is truly alien (beyond any actual “aliens”), you get to it by employing your engineering thought ability, unlike paperclip maximizer at which you get by dressing up a phenomenon of human pleasure maximizer such as a serial murderer and killer, and making it look like something more general than that by making it be about paperclips rather than sex.
I’d think it’d be great if people stopped thinking in terms of some fuzzy abstraction “AI” which is basically a basket for all sorts of biases. If we consider the software that can self improve ‘intelligently’ in our opinion, in general, the minimal such software is something like an optimizing compiler that when compiling it’s source will even optimize its ability to optimize. This sort of thing is truly alien (beyond any actual “aliens”), you get to it by employing your engineering thought ability, unlike paperclip maximizer at which you get by dressing up a phenomenon of human pleasure maximizer such as a serial murderer and killer, and making it look like something more general than that by making it be about paperclips rather than sex.