Specifically, it means that you have to deal with generalizing your values to new situations, but without the IID assumption, you can’t just interpolate from existing values anymore, and you will likely overfit to your IID data points, and that’s the better case. In other words, your behavior will be dominated by your inductive biases and priors. And my fear is that given real life examples of intelligence differences that violate IID distributions, things end up misaligned really fast. I’m not saying that we are doomed, but I want to call this out since I think breaking IID will most likely cause Turner to do something really bad to his brother if we allow even one order of magnitude more compute.
Scale this up to human civilization relying on IID distributions in intelligence, and I’m much more careful than Turner is in trying to extrapolate.
Specifically, it means that you have to deal with generalizing your values to new situations, but without the IID assumption, you can’t just interpolate from existing values anymore, and you will likely overfit to your IID data points, and that’s the better case. In other words, your behavior will be dominated by your inductive biases and priors. And my fear is that given real life examples of intelligence differences that violate IID distributions, things end up misaligned really fast. I’m not saying that we are doomed, but I want to call this out since I think breaking IID will most likely cause Turner to do something really bad to his brother if we allow even one order of magnitude more compute.
Scale this up to human civilization relying on IID distributions in intelligence, and I’m much more careful than Turner is in trying to extrapolate.