Simplicity has its value, but one shouldn’t pursue plans any simpler than the simplest plan that might actually work. We don’t know whether it’s even in principle possible for us to knowingly construct an explicit goal system that matches our values and doesn’t offer any room for a super-human AI to cleverly “optimize” in unexpected ways without referencing our values themselves. But I would be extremely skeptical of any attempt inspired by contract law. In your analogy this plan sounds more like “balloon that goes really high” to me. Perhaps the Singinst is pursuing Verneian cannons and no one has thought of rockets yet.
Simplicity has its value, but one shouldn’t pursue plans any simpler than the simplest plan that might actually work. We don’t know whether it’s even in principle possible for us to knowingly construct an explicit goal system that matches our values and doesn’t offer any room for a super-human AI to cleverly “optimize” in unexpected ways without referencing our values themselves. But I would be extremely skeptical of any attempt inspired by contract law. In your analogy this plan sounds more like “balloon that goes really high” to me. Perhaps the Singinst is pursuing Verneian cannons and no one has thought of rockets yet.