I strongly agree with you that it’ll eventually be very difficult for humans to tell apart AI-generated alignment proposals that look good and aren’t good from ones that look good and are actually good.
There is a much stronger version of the claim “alignment proposals are easier to evaluate than to generate” that I think we’re discussing in this thread, where you claim that humans will be able to tell all good alignment proposals apart from bad ones or at least not accept any bad ones (precision matters much more than recall here since you can compensate bad recall with compute). If this strong claim is true, then conceptually RLHF/reward modeling should be sufficient as an alignment technique for the minimal viable product. Personally I think that this strong version of the claim is unlikely to be true, but I’m not certain that I will be false for the first systems that can do useful alignment research.
As William points out below, if we get AI-assisted human evaluation to work well, then we can uncover flaws in alignment proposals that are too hard to find for unassisted humans. This is a weaker version of the claim, because you’re just claiming that humans + AI assistance are better at evaluating alignment proposals than human + AI assistance are at generating them. Generally I’m pretty optimistic about that level of supervision actually allowing us to supervise superhuman alignment research; I’ve written more about this here: https://aligned.substack.com/p/ai-assisted-human-feedback
I strongly agree with you that it’ll eventually be very difficult for humans to tell apart AI-generated alignment proposals that look good and aren’t good from ones that look good and are actually good.
There is a much stronger version of the claim “alignment proposals are easier to evaluate than to generate” that I think we’re discussing in this thread, where you claim that humans will be able to tell all good alignment proposals apart from bad ones or at least not accept any bad ones (precision matters much more than recall here since you can compensate bad recall with compute). If this strong claim is true, then conceptually RLHF/reward modeling should be sufficient as an alignment technique for the minimal viable product. Personally I think that this strong version of the claim is unlikely to be true, but I’m not certain that I will be false for the first systems that can do useful alignment research.
As William points out below, if we get AI-assisted human evaluation to work well, then we can uncover flaws in alignment proposals that are too hard to find for unassisted humans. This is a weaker version of the claim, because you’re just claiming that humans + AI assistance are better at evaluating alignment proposals than human + AI assistance are at generating them. Generally I’m pretty optimistic about that level of supervision actually allowing us to supervise superhuman alignment research; I’ve written more about this here: https://aligned.substack.com/p/ai-assisted-human-feedback