My purposefully open-ended question would simply be, “What is good?” My hope is that finding the nature of what good is as its super goal would keep the AI on course to the future we want as it would pass through its recursive self-improvements.
You have to tell the AI how to find out how well it has done. To ask “What is a good definition of ‘good’?”, you already have to define good. At least if we ever find a definition of good, we can ask an AI with it to judge it.
My purposefully open-ended question would simply be, “What is good?” My hope is that finding the nature of what good is as its super goal would keep the AI on course to the future we want as it would pass through its recursive self-improvements.
You have to tell the AI how to find out how well it has done. To ask “What is a good definition of ‘good’?”, you already have to define good. At least if we ever find a definition of good, we can ask an AI with it to judge it.