[...] do people who for example proposed (C)IRL without acknowledging the difficulties you described in this post actually understand these difficulties and just want to write papers that show some sort of forward progress, or are they not aware of them?
As someone who has worked on IRL a little bit, my impression is that such algorithms are not intended to capture human value to its full extent, but rather to learn shorter-term instrumental preferences. Paul gives some arguments for such “narrow value learning” here. This scenario, where human abilities are augmented using AI assistants, falls under your AI/human ecosystem category. I don’t think many people view “moral philosophy” as a separate type of activity that differentially benefits less from augmentation. Rather, AI assistants are seen as helping with essentially all tasks, including analyzing the consequences of decisions that have potentially far-reaching impacts, deciding when to keep our options open, and engineering the next-generation AI/human system in a way that maintains alignment. I don’t think this sort of bootstrapping process is understood very well, though.
As someone who has worked on IRL a little bit, my impression is that such algorithms are not intended to capture human value to its full extent, but rather to learn shorter-term instrumental preferences. Paul gives some arguments for such “narrow value learning” here. This scenario, where human abilities are augmented using AI assistants, falls under your AI/human ecosystem category. I don’t think many people view “moral philosophy” as a separate type of activity that differentially benefits less from augmentation. Rather, AI assistants are seen as helping with essentially all tasks, including analyzing the consequences of decisions that have potentially far-reaching impacts, deciding when to keep our options open, and engineering the next-generation AI/human system in a way that maintains alignment. I don’t think this sort of bootstrapping process is understood very well, though.