The simplest known complete representation of my utility function is my brain, combined with some supporting infrastructure and a question-asking procedure. Any utility function that is substantially simpler than my brain has almost certainly left something out.
The first step in extracting a human’s volition is to develop a complete understanding of the brain, including the ability to simulate it. We are currently stuck there. We have some high-level approximations that work around needing this understanding, but their accuracy is questionable.
The simplest known complete representation of my utility function is my brain, combined with some supporting infrastructure and a question-asking procedure. Any utility function that is substantially simpler than my brain has almost certainly left something out.
The first step in extracting a human’s volition is to develop a complete understanding of the brain, including the ability to simulate it. We are currently stuck there. We have some high-level approximations that work around needing this understanding, but their accuracy is questionable.
Assume you have a working simulation. What next?