The first problem is the wish would have to be extremely good at pointing.
This sounds silly but what I mean is that humans are COMPLICATED. “Pointing” at a human and telling an AI to deduce things about it will come up with HUGE swathes of data which you have to have already prepared it to ignore or pay attention to. To give a classic simple example, smiles are a sign of happiness but we do not want to tile the universe in smiley faces or create an artificial virus that constricts your face into a rictus and is highly contagious.
Second: assuming that works, it works primarily for one person, which is giving that person a lot more power than I think most people want to give any one person. But if we could guarantee an AI would fulfill the values of A person rather than of multiple people and someone else was developing AI that wasn’t guarunteed to fulfill any values I’d probably take it.
The first problem is the wish would have to be extremely good at pointing.
This sounds silly but what I mean is that humans are COMPLICATED. “Pointing” at a human and telling an AI to deduce things about it will come up with HUGE swathes of data which you have to have already prepared it to ignore or pay attention to. To give a classic simple example, smiles are a sign of happiness but we do not want to tile the universe in smiley faces or create an artificial virus that constricts your face into a rictus and is highly contagious.
Second: assuming that works, it works primarily for one person, which is giving that person a lot more power than I think most people want to give any one person. But if we could guarantee an AI would fulfill the values of A person rather than of multiple people and someone else was developing AI that wasn’t guarunteed to fulfill any values I’d probably take it.