I agree this is part of the problem, but like others here I think you might be making it out to be harder than it is. We know, in principle, how to translate a utility function into a physical description of an object: by coding it as an AI and then specifying the AI along with its substrate down to the quantum level. So, again in principle, we can go backwards: take a physical description of an object, consider all possible implementations of all possible utility functions, and see if any of them matches the object.
We know, in principle, how to translate a utility function into a physical description of an object: by coding it as an AI and then specifying the AI along with its substrate down to the quantum level. So, again in principle, we can go backwards: take a physical description of an object, consider all possible implementations of all possible utility functions, and see if any of them matches the object.
I think it’s enough to consider computer programs and dispense with details of physics—everything else can be discovered by the program. You are assuming the “bottom” level of physics, “quantum level”, but there is no bottom, not really, there is only the beginning where our own minds are implemented, and the process of discovery that defines the way we see the rest of the world.
If you start with an AI design parameterized by preference, you are not going to enumerate all programs, only a small fraction of programs that have the specific form of your AI with some preference, and so for a given arbitrary program there will be no match. Furthermore, you are not interested in finding a match: if a human was equal to the AI, you are already done! It’s necessary to explicitly go the other way, starting from arbitrary programs and understanding what a program is, deeply enough to see preference in it. This understanding may give an idea of a mapping for translating a crazy ape into an efficient FAI.
If you start with an AI design parameterized by preference, you are not going to enumerate all programs, only a small fraction of programs that have the specific form of your AI with some preference, and so for a given arbitrary program there will be no match.
When I said “all possible implementations of all possible utility functions”, I meant to include flawed implementations. But then two different utility functions might map onto the same physical object, so we’d also need a theory of implementation flaws that tells us, given two implementations of a utility function, which is more flawed.
When I said “all possible implementations of all possible utility functions”, I meant to include flawed implementations. But then two different utility functions might map onto the same physical object, so we’d also need a theory of implementation flaws that tells us, given two implementations of a utility function, which is more flawed.
This is WAY too hand-wavy an explanation for “in principle, we can go backwards” (from a system to its preference). I believe that in principle, we can, but not via injecting fuzziness of “implementation flaws”.
Here’s another statement of the problem: One agent’s bias is another agent’s heuristic. And the “two agents” might be physically the same, but just interpreted differently.
I agree this is part of the problem, but like others here I think you might be making it out to be harder than it is. We know, in principle, how to translate a utility function into a physical description of an object: by coding it as an AI and then specifying the AI along with its substrate down to the quantum level. So, again in principle, we can go backwards: take a physical description of an object, consider all possible implementations of all possible utility functions, and see if any of them matches the object.
I think it’s enough to consider computer programs and dispense with details of physics—everything else can be discovered by the program. You are assuming the “bottom” level of physics, “quantum level”, but there is no bottom, not really, there is only the beginning where our own minds are implemented, and the process of discovery that defines the way we see the rest of the world.
If you start with an AI design parameterized by preference, you are not going to enumerate all programs, only a small fraction of programs that have the specific form of your AI with some preference, and so for a given arbitrary program there will be no match. Furthermore, you are not interested in finding a match: if a human was equal to the AI, you are already done! It’s necessary to explicitly go the other way, starting from arbitrary programs and understanding what a program is, deeply enough to see preference in it. This understanding may give an idea of a mapping for translating a crazy ape into an efficient FAI.
When I said “all possible implementations of all possible utility functions”, I meant to include flawed implementations. But then two different utility functions might map onto the same physical object, so we’d also need a theory of implementation flaws that tells us, given two implementations of a utility function, which is more flawed.
This is WAY too hand-wavy an explanation for “in principle, we can go backwards” (from a system to its preference). I believe that in principle, we can, but not via injecting fuzziness of “implementation flaws”.
Here’s another statement of the problem: One agent’s bias is another agent’s heuristic. And the “two agents” might be physically the same, but just interpreted differently.