I think it’s a common notion that if you were just good enough you wouldn’t respond to incentives. I used to think it more myself. It’s a key element to realize that the system will still create these outcomes even if nobody is consciously choosing to fall prey to them, and that to ‘avoid incentives’ in a system like that you’d need to actually model what the incentives are and what outcome they’re systematically choosing, via selection effects, by correctly optimizing according to feedback in a complex domain, and so on. One cannot have model-free integrity.
So it feels to me like a very fundamental insight, written up well. (It’s also related I think to some of the discussion about lying vs unconscious bias between Scott and Zack and Jessica and so on.)
I think it’s a common notion that if you were just good enough you wouldn’t respond to incentives. I used to think it more myself. It’s a key element to realize that the system will still create these outcomes even if nobody is consciously choosing to fall prey to them, and that to ‘avoid incentives’ in a system like that you’d need to actually model what the incentives are and what outcome they’re systematically choosing, via selection effects, by correctly optimizing according to feedback in a complex domain, and so on. One cannot have model-free integrity.
So it feels to me like a very fundamental insight, written up well. (It’s also related I think to some of the discussion about lying vs unconscious bias between Scott and Zack and Jessica and so on.)