5. Empirical evidence: human intelligence generalised far without staying aligned with its optimisation target.
I think this one is debatable. It seems to me that human intelligence has remained reasonably well aligned with its optimization target, if its optimization target is defined as “being well-fed, getting social status, remaining healthy, having sex, raising children, etc.”, i.e. the things that evolution actually could optimize humans for rather than something like inclusive fitness that it couldn’t directly optimize for. Yes there are individual humans who are less interested in pursuing particular pieces on that list (e.g. many prefer not to have children), but that’s because the actual thing being optimized is a combination of those variables that’s sensitive to local conditions. Any goal drift is then from a changed environment that acts as an input to the optimization target, rather than from an increase in capabilities as such.
The point isn’t about goal misalignment but capability generalisation. It is surprising to some degree that just selecting on reproductive fitness through its proxies of being well-fed, social status etc humans have obtained the capability to go to the moon. It points toward a coherent notion & existence of ‘general intelligence’ as opposed to specific capabilities.
I think what you say makes sense, but to be clear the argument does not consider those things as the optimisation target but rather considers fitness or reproductive capacity as the optimisation target. (A reasonable counterargument is that the analogy doesn’t hold up because fitness-as-optimisation-target isn’t a good way to characterise evolution as an optimiser.)
A reasonable counterargument is that the analogy doesn’t hold up because fitness-as-optimisation-target isn’t a good way to characterise evolution as an optimiser.
Yeah, that’s the main counterargument. Evolution is purposeless and doesn’t care about anything for specific species or nature itself, and evolution isn’t telelogical, so Argument 5 fails.
evolution isn’t exactly purposeless; it has very little purpose, but to the degree a purpose could be described, the purpose for which things evolve is to survive in competition. that’s more than nothing. the search process is mutation, and the selection process is <anything that survives>. inferring additional constraints that this purpose implies seems potentially fruitful; non-local optimizers like ourselves can look at that objective and design constraints that unilaterally increase durability. our ability to reason over game theory means we’re not constrained to only evolutionary game theory; for example, we can make tit-for-tat-with-forgiveness more durable by noticing that it has a tendency to be replaced with cooperation when society is cooperative, and we can reintroduce tit-for-tat-with-forgiveness into contexts where cooperatebot-style reasoning has taken over.
I think this one is debatable. It seems to me that human intelligence has remained reasonably well aligned with its optimization target, if its optimization target is defined as “being well-fed, getting social status, remaining healthy, having sex, raising children, etc.”, i.e. the things that evolution actually could optimize humans for rather than something like inclusive fitness that it couldn’t directly optimize for. Yes there are individual humans who are less interested in pursuing particular pieces on that list (e.g. many prefer not to have children), but that’s because the actual thing being optimized is a combination of those variables that’s sensitive to local conditions. Any goal drift is then from a changed environment that acts as an input to the optimization target, rather than from an increase in capabilities as such.
The point isn’t about goal misalignment but capability generalisation. It is surprising to some degree that just selecting on reproductive fitness through its proxies of being well-fed, social status etc humans have obtained the capability to go to the moon. It points toward a coherent notion & existence of ‘general intelligence’ as opposed to specific capabilities.
I think what you say makes sense, but to be clear the argument does not consider those things as the optimisation target but rather considers fitness or reproductive capacity as the optimisation target. (A reasonable counterargument is that the analogy doesn’t hold up because fitness-as-optimisation-target isn’t a good way to characterise evolution as an optimiser.)
Yes, that was my argument in the comment that I linked. :)
Yeah, that’s the main counterargument. Evolution is purposeless and doesn’t care about anything for specific species or nature itself, and evolution isn’t telelogical, so Argument 5 fails.
evolution isn’t exactly purposeless; it has very little purpose, but to the degree a purpose could be described, the purpose for which things evolve is to survive in competition. that’s more than nothing. the search process is mutation, and the selection process is <anything that survives>. inferring additional constraints that this purpose implies seems potentially fruitful; non-local optimizers like ourselves can look at that objective and design constraints that unilaterally increase durability. our ability to reason over game theory means we’re not constrained to only evolutionary game theory; for example, we can make tit-for-tat-with-forgiveness more durable by noticing that it has a tendency to be replaced with cooperation when society is cooperative, and we can reintroduce tit-for-tat-with-forgiveness into contexts where cooperatebot-style reasoning has taken over.