Person A: Why are we going to create horribly misaligned artificial intelligence?
Person B: Because the AI will end up optimizing hard on subtly askew values that don’t approximate well out of distribution, like humans did relative to IGF.
Person A (who is attacking the weak analogy instead of something more substantive): But humans seem to be doing pretty well at that inadvertently? Humans got a lot better at optimizing things and now there are eight billion humans. Yes, we, don’t maximize IGF as much as we would if we explicitly optimized for that, but, if we become a nice spacefaring civilization, I expect there will be many orders of magnitude more bioconservatives still living in basically human bodies running on something like DNA.
Person B: Well, that’s not going to hold, because we’re going to misalign artificial intelligence, and that will kill everybody.
FWIW the difference between “we get 10^-18x the utility we would’ve gotten if our AI had been a human satisfaction maximizer” and “we get zero utility” is basically nothing. But it’s not literally nothing.
I think A and B are both wrong in the quoted text because they’re forgetting that IGF doesn’t apply at the species level. A species can evolve to extinction to optimize IGF.
The general lack of concern about existential risk is very compatible with the IGF-optimization hypothesis. If the godshatter hypothesis or shard theory hypothesis is true then we have to also conclude that people are short-sighted idiots. Which isn’t a big leap.
FWIW the difference between “we get 10^-18x the utility we would’ve gotten if our AI had been a human satisfaction maximizer” and “we get zero utility” is basically nothing. But it’s not literally nothing.
I think A and B are both wrong in the quoted text because they’re forgetting that IGF doesn’t apply at the species level. A species can evolve to extinction to optimize IGF.
The general lack of concern about existential risk is very compatible with the IGF-optimization hypothesis. If the godshatter hypothesis or shard theory hypothesis is true then we have to also conclude that people are short-sighted idiots. Which isn’t a big leap.