I guess I disagree with the premise that we will have superintelligent successors who will think circles around us, and yet we get to specify in detail what ethical values they will have, and it will stick. Forever. So let’s debate what values to specify.
A parent would be crazy to think this way about a daughter, optimizing in detail the order of priorities that he intends to implant into her, and expecting them to stick. But if your daughter is a superintelligence, it’s even crazier.
A parent would be crazy to think this way about a daughter, optimizing in detail the order of priorities that he intends to implant into her, and expecting them to stick. But if your daughter is a superintelligence, it’s even crazier.
Suppose it’s twenty years from now, and know exactly what genes go into the heritable portion of intelligence and personality, which includes both stuff like the Big Five and the weird preferences twins sometimes share.
Suppose further that genetic modification of children is possible and acceptable, and you and your partner have decided that you’ll have a daughter, and naturally you want her IQ to be as high as possible (suppose that’s 170 on today’s scale). So she’s going to be able to think circles around you, but be comparable to her augmented classmates. But personality isn’t as obvious. Do you really want her to be maximally agreeable? Extraverted? Open? The other two might be easy to agree on; you might decide to zero out her neuroticism without much debate, and maximize her conscientiousness without much more.
But, importantly, her ability to outthink you doesn’t mean she will outthink the personality you chose for her. Why would she want to? It’s her personality.
That’s what a non-crazy version looks like: we know that personality traits are at least partly heritable for humans, and so we can imagine manipulating what personality traits future humans have by manipulating their genes. We also have some idea of how raising children impacts their personality / methods of relating with other people, and we can similarly imagine manipulating their early environment to get the personalities and relationships that we want.
We can further strengthen the analogy by considering the next generation. Your daughter has found a partner and is considering having a granddaughter; the IQ manipulation technology has improved to the point where the granddaughter is expected to score the equivalent of 220 on today’s scale, but there’s still a comparable personality question. If you were highly open and decided that your daughter should be highly open too, it seems likely that your daughter will use similar logic to decide that your granddaughter should also be highly open.
That is, a value-optimizing AI will only create a descendant AI (including later versions of itself) that it thinks will fulfill its values, and so it will likely encode its values (at least implicitly) into that AI. So it is important to get right the process of thinking about value preservation, so we can both pass our values and our sense of how to look at values and decide what to preserve down.
I guess I disagree with the premise that we will have superintelligent successors who will think circles around us, and yet we get to specify in detail what ethical values they will have, and it will stick. Forever. So let’s debate what values to specify.
A parent would be crazy to think this way about a daughter, optimizing in detail the order of priorities that he intends to implant into her, and expecting them to stick. But if your daughter is a superintelligence, it’s even crazier.
Suppose it’s twenty years from now, and know exactly what genes go into the heritable portion of intelligence and personality, which includes both stuff like the Big Five and the weird preferences twins sometimes share.
Suppose further that genetic modification of children is possible and acceptable, and you and your partner have decided that you’ll have a daughter, and naturally you want her IQ to be as high as possible (suppose that’s 170 on today’s scale). So she’s going to be able to think circles around you, but be comparable to her augmented classmates. But personality isn’t as obvious. Do you really want her to be maximally agreeable? Extraverted? Open? The other two might be easy to agree on; you might decide to zero out her neuroticism without much debate, and maximize her conscientiousness without much more.
But, importantly, her ability to outthink you doesn’t mean she will outthink the personality you chose for her. Why would she want to? It’s her personality.
That’s what a non-crazy version looks like: we know that personality traits are at least partly heritable for humans, and so we can imagine manipulating what personality traits future humans have by manipulating their genes. We also have some idea of how raising children impacts their personality / methods of relating with other people, and we can similarly imagine manipulating their early environment to get the personalities and relationships that we want.
We can further strengthen the analogy by considering the next generation. Your daughter has found a partner and is considering having a granddaughter; the IQ manipulation technology has improved to the point where the granddaughter is expected to score the equivalent of 220 on today’s scale, but there’s still a comparable personality question. If you were highly open and decided that your daughter should be highly open too, it seems likely that your daughter will use similar logic to decide that your granddaughter should also be highly open.
That is, a value-optimizing AI will only create a descendant AI (including later versions of itself) that it thinks will fulfill its values, and so it will likely encode its values (at least implicitly) into that AI. So it is important to get right the process of thinking about value preservation, so we can both pass our values and our sense of how to look at values and decide what to preserve down.