I think the whole set of concepts here requires as a complement concept of zone of applicability.
Like, heart is “supposed to” pump blood under actually very narrow range of circumstances like oxygen/ electrolyte/sugar/osmotic protein concentration, body temperature, etc. The trick is that we expect this narrow range of circumstances to be present and actively support them.
When something is “evolved to do X” it actually means that in some ecological niche mechanisms enabling doing X are spreading via reproduction.
The function of database is to give information to authorized users and refuse to everyone else. The task of hacker is to change circumstances in a way that stops database from performing its function.
The function of hammer is to hammer nails into surface while being wielded by human, but if you forget hammer inside gears of complex mechanism and mechanism gets broken by hammer, it’s not its function, because “being inside gears” is not zone of applicability.
The problem: it’s hard to derive zone of applicability from the first principles. You know that you need to hold hammer with whole palm and what nail is, but derive this can be very hard for superintelligence trained to predict molecular structures, even if said superintelligence has all general-intelligence-related cognitive engines like general-purpose search, because it has never encountered anything like hammer before. And, conversively, it is hard for human to infer functions of output of alien optimization processes, like evolution.
I think the whole set of concepts here requires as a complement concept of zone of applicability.
Like, heart is “supposed to” pump blood under actually very narrow range of circumstances like oxygen/ electrolyte/sugar/osmotic protein concentration, body temperature, etc. The trick is that we expect this narrow range of circumstances to be present and actively support them.
When something is “evolved to do X” it actually means that in some ecological niche mechanisms enabling doing X are spreading via reproduction.
The function of database is to give information to authorized users and refuse to everyone else. The task of hacker is to change circumstances in a way that stops database from performing its function.
The function of hammer is to hammer nails into surface while being wielded by human, but if you forget hammer inside gears of complex mechanism and mechanism gets broken by hammer, it’s not its function, because “being inside gears” is not zone of applicability.
The problem: it’s hard to derive zone of applicability from the first principles. You know that you need to hold hammer with whole palm and what nail is, but derive this can be very hard for superintelligence trained to predict molecular structures, even if said superintelligence has all general-intelligence-related cognitive engines like general-purpose search, because it has never encountered anything like hammer before. And, conversively, it is hard for human to infer functions of output of alien optimization processes, like evolution.