Open Question: Working with concepts that the human can’t understand
Question: when we need to assemble complex concepts by learning/interacting with the environment, rather than using H’s concepts directly, and when those concepts influence reasoning in subtle/abstract ways, how do we retain corrigibility/alignment?
Paul: I don’t have any general answer to this, seems like we should probably choose some example cases. I’m probably going to be advocating something like “Search over a bunch of possible concepts and find one that does what you want / has the desired properties.”
E.g. for elegant proofs, you want a heuristic that gives successful lines of inquiry higher scores. You can explore a bunch of concepts that do that, evaluate each one according to how well it discriminates good from bad lines of inquiry, and also evaluate other stuff like “What would I infer from learning that a proof is `elegant` other than that it will work” and make sure that you are OK with that.
Andreas: Suppose you don’t have the concepts of “proof” and “inquiry”, but learned them (or some more sophisticated analogs) using the sort of procedure you outlined below. I guess I’m trying to see in more detail that you can do a good job at “making sure you’re OK with reasoning in ways X” in cases where X is far removed from H’s concepts. (Unfortunately, it seems to be difficult to make progress on this by discussing particular examples, since examples are necessarily about concepts we know pretty well.)
This may be related to the more general question of what sorts of instructions you’d give H to ensure that if they follow the instructions, the overall process remains corrigible/aligned.
Open Question: Working with concepts that the human can’t understand
Question: when we need to assemble complex concepts by learning/interacting with the environment, rather than using H’s concepts directly, and when those concepts influence reasoning in subtle/abstract ways, how do we retain corrigibility/alignment?
Paul: I don’t have any general answer to this, seems like we should probably choose some example cases. I’m probably going to be advocating something like “Search over a bunch of possible concepts and find one that does what you want / has the desired properties.”
E.g. for elegant proofs, you want a heuristic that gives successful lines of inquiry higher scores. You can explore a bunch of concepts that do that, evaluate each one according to how well it discriminates good from bad lines of inquiry, and also evaluate other stuff like “What would I infer from learning that a proof is `elegant` other than that it will work” and make sure that you are OK with that.
Andreas: Suppose you don’t have the concepts of “proof” and “inquiry”, but learned them (or some more sophisticated analogs) using the sort of procedure you outlined below. I guess I’m trying to see in more detail that you can do a good job at “making sure you’re OK with reasoning in ways X” in cases where X is far removed from H’s concepts. (Unfortunately, it seems to be difficult to make progress on this by discussing particular examples, since examples are necessarily about concepts we know pretty well.)
This may be related to the more general question of what sorts of instructions you’d give H to ensure that if they follow the instructions, the overall process remains corrigible/aligned.