Re (14), I guess the ideas are very similar, where the mesaoptimizer scenario is like a sharp example of the more general concept Eliezer points at, that different classes of difficulties may appear at different capability levels.
Re (15), “Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously”, which is about how we may have reasons to expect aligned output that are brittle under rapid capability gain: your quote from Richard is just about “fast capability gain seems possible and likely”, and isn’t about connecting that to increased difficulty in succeeding at the alignment problem?
Re (32), I don’t think your quote isn’t talking about the thing Eliezer is talking about, which is that in order to be human level at modelling human-generated text, your AI must be doing something on par with human thought that figures out what humans would say. Your quote just isn’t discussing this, namely that strong imitation requires cognition that is dangerous.
So I guess I don’t take much issue with (14) or (15), but I think you’re quite off the mark about (32). In any case, I still have a strong sense that Eliezer is successfully being more on the mark here than the rest of us manage. Kudos of course to you and others that are working on writing things up and figuring things out. Though I remain sympathetic to Eliezer’s complaint.
Re (14), I guess the ideas are very similar, where the mesaoptimizer scenario is like a sharp example of the more general concept Eliezer points at, that different classes of difficulties may appear at different capability levels.
Re (15), “Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously”, which is about how we may have reasons to expect aligned output that are brittle under rapid capability gain: your quote from Richard is just about “fast capability gain seems possible and likely”, and isn’t about connecting that to increased difficulty in succeeding at the alignment problem?
Re (32), I don’t think your quote isn’t talking about the thing Eliezer is talking about, which is that in order to be human level at modelling human-generated text, your AI must be doing something on par with human thought that figures out what humans would say. Your quote just isn’t discussing this, namely that strong imitation requires cognition that is dangerous.
So I guess I don’t take much issue with (14) or (15), but I think you’re quite off the mark about (32). In any case, I still have a strong sense that Eliezer is successfully being more on the mark here than the rest of us manage. Kudos of course to you and others that are working on writing things up and figuring things out. Though I remain sympathetic to Eliezer’s complaint.