It counts as failure to preserve humanity. An AI that does that is probably unfriendly (barring the coercion by external powerful agents. Eliezer actually wrote a story about such scenario, without AIs though.)
Interesting. I think I may even agree with you. In that story each race would need to conclude that the other races are “unfriendly”. So Eliezer has written a story in which all the NATURAL intelligences (except us of course) are “unfriendly,” and in which a human would need to agree that from the point of view of the other intelligent races, human intelligence was “unfriendly.”
Perhaps all intelligences are necessarily “unfriendly” to all other intelligences. This could even apply at the micro level, perhaps each human intelligence is “unfriendly” to all other human intelligences. This actually looks pretty real and pretty much like what happens in a world where survival is the only enforced value. Humans have the fascinating conundrum that even though we are unfriendly to the other humans, we have a much better chance of surviving and thriving by working with the other humans. The alliances and technical abilities and so on are, if not balance across all humans and all groups, at least balanced enough across many of them so that the result is a plethora of competing / cooperating intelligences where the jury is still out on who is the ultimate winner. Breeding in to us the ability (the value?) that “others” are our allies against “the enemies” clearly has resulted in collective efforts of cooperation that have produced quickly cascading production ability in our species. “We” worried about the Nazis FOOMing and winning, we worried the Soviets might FOOM and win. Our ancestors fought against every tribe that lived 5 miles away from them, before cultural evolution allowed them (us) to cooperate in groups of hundreds of millions.
So in Eliezer’s story, 3 NI’s have FOOMed and then finally run into each other. And they CANNOT resist getting up in each other’s grills. And why not? what are the chances that the final intelligence IF only one is left will have been one which was shy about destroying potential competitors before they destroyed it?
Interesting. I think I may even agree with you. In that story each race would need to conclude that the other races are “unfriendly”. So Eliezer has written a story in which all the NATURAL intelligences (except us of course) are “unfriendly,” and in which a human would need to agree that from the point of view of the other intelligent races, human intelligence was “unfriendly.”
Perhaps all intelligences are necessarily “unfriendly” to all other intelligences. This could even apply at the micro level, perhaps each human intelligence is “unfriendly” to all other human intelligences. This actually looks pretty real and pretty much like what happens in a world where survival is the only enforced value. Humans have the fascinating conundrum that even though we are unfriendly to the other humans, we have a much better chance of surviving and thriving by working with the other humans. The alliances and technical abilities and so on are, if not balance across all humans and all groups, at least balanced enough across many of them so that the result is a plethora of competing / cooperating intelligences where the jury is still out on who is the ultimate winner. Breeding in to us the ability (the value?) that “others” are our allies against “the enemies” clearly has resulted in collective efforts of cooperation that have produced quickly cascading production ability in our species. “We” worried about the Nazis FOOMing and winning, we worried the Soviets might FOOM and win. Our ancestors fought against every tribe that lived 5 miles away from them, before cultural evolution allowed them (us) to cooperate in groups of hundreds of millions.
So in Eliezer’s story, 3 NI’s have FOOMed and then finally run into each other. And they CANNOT resist getting up in each other’s grills. And why not? what are the chances that the final intelligence IF only one is left will have been one which was shy about destroying potential competitors before they destroyed it?