I wasn’t arguing for “99+% chance that an AI, even if trained specifically to care about humans, would not end up caring about humans at all” just addressing the questions about humans in the limit of intelligence and power in the comment I replied to.
Tru
It does seem to me that there is substantial chance that humans eventually do stop having human children in the limit of intelligence and power.
A uniform fertility below 2.1 means extinction, yes, but in no country is the fertility rate uniformly below 2.1. Instead, some humans decide they want lots of children despite the existence of contraception and educational opportunity, and others do not. It seems to me that a substantial proportion of humans would stop having children in the limit of intelligence and power. It also seems to me like a substantial number of humans continue (and would continue) to have such children as if they value it for its own sake.
This suggests that the problems Nate is highlighting, while real, are not sufficient to guarantee complete failure—even when the training process is not being designed with those problems in mind, and there are no attempts at iterated amplification whatsoever. This nuance is important because it affects how far we should think a naive SGD RL approach is from limited “1% success”, and whether or not simple modifications are likely to greatly increase survival odds.
Tru
A uniform fertility below 2.1 means extinction, yes, but in no country is the fertility rate uniformly below 2.1. Instead, some humans decide they want lots of children despite the existence of contraception and educational opportunity, and others do not. It seems to me that a substantial proportion of humans would stop having children in the limit of intelligence and power. It also seems to me like a substantial number of humans continue (and would continue) to have such children as if they value it for its own sake.
This suggests that the problems Nate is highlighting, while real, are not sufficient to guarantee complete failure—even when the training process is not being designed with those problems in mind, and there are no attempts at iterated amplification whatsoever. This nuance is important because it affects how far we should think a naive SGD RL approach is from limited “1% success”, and whether or not simple modifications are likely to greatly increase survival odds.