Any feasible comparison of potential worlds is actually a comparison of predicted summaries of those worlds. Both the prediction and the summary are lossy, thus that asepect of Goodhart’s law.
I did not mean to say that “everything is an optimization process”. I did mean to say that decisions are an optimization process, and I now realize even that’s too strong. I suspect all I can actually assert is that “intentionality is an optimization process”.
I did not mean to say that “everything is an optimization process”. I did mean to say that decisions are an optimization process, and I now realize even that’s too strong. I suspect all I can actually assert is that “intentionality is an optimization process”.
Oh, I didn’t mean to accuse you of that. It’s more that this is a common implicit frame of reference (including/especially on LW).
I rather suspect the correct direction is to break down “optimization” into more careful concepts (starting, but not finishing, with something like selection vs control).
Awesome—I think I agree with most of this. Specifically, https://www.lesswrong.com/posts/A8iGaZ3uHNNGgJeaD/an-orthodox-case-against-utility-functions is very compatible with the possibility that the function is too complex for any agent to actually compute. It’s quite likely that there are more potential worlds than an agent can rank, with more features than an agent can measure.
Any feasible comparison of potential worlds is actually a comparison of predicted summaries of those worlds. Both the prediction and the summary are lossy, thus that asepect of Goodhart’s law.
I did not mean to say that “everything is an optimization process”. I did mean to say that decisions are an optimization process, and I now realize even that’s too strong. I suspect all I can actually assert is that “intentionality is an optimization process”.
Oh, I didn’t mean to accuse you of that. It’s more that this is a common implicit frame of reference (including/especially on LW).
I rather suspect the correct direction is to break down “optimization” into more careful concepts (starting, but not finishing, with something like selection vs control).