What on Earth could someone possibly be thinking, when they propose creating a superintelligence whose behaviors are reinforced by human smiles?
Wiring almost anything about today’s humans into a proposed long-term utility function seems rather obviously dangerous. Keeping unmodified humans around would cripple our civilisation’s spaceworthyness—and would probably eventually lead to our distant descendants being smothered by aliens who didn’t have such a limited and backwards notion of what’s actually important.
Humans should probably not be allowed to persist for too long past their sell-by date—or they may fill the whole planet with the foul stench of their excrement. Fortunately, it doesn’t seem like a likely outcome. Our main asset is our brains—and, fairly soon, that asset will be practically worthless.
Wiring almost anything about today’s humans into a proposed long-term utility function seems rather obviously dangerous. Keeping unmodified humans around would cripple our civilisation’s spaceworthyness—and would probably eventually lead to our distant descendants being smothered by aliens who didn’t have such a limited and backwards notion of what’s actually important.
Humans should probably not be allowed to persist for too long past their sell-by date—or they may fill the whole planet with the foul stench of their excrement. Fortunately, it doesn’t seem like a likely outcome. Our main asset is our brains—and, fairly soon, that asset will be practically worthless.