To my eye, this seems like it mostly establishes ‘it’s not impossible in principle for an optimizer to have a goal that relates to the physical world’. But we had no reason to doubt this in the first place, and it doesn’t give us a way to reliably pick in advance which physical things the optimizer cares about. “It’s not impossible” is a given for basically everything in AI, in principle, if you have arbitrary amounts of time and arbitrarily deep understanding.
As I said (a few times!) in the discussion about orthogonality, indifference about the measure of “agents” that have particular properties seems crazy to me. Having an example of “agents” that behave in a particular way is a enormously different to having an unproven claim that such agents might be mathematically possible.
To my eye, this seems like it mostly establishes ‘it’s not impossible in principle for an optimizer to have a goal that relates to the physical world’. But we had no reason to doubt this in the first place, and it doesn’t give us a way to reliably pick in advance which physical things the optimizer cares about. “It’s not impossible” is a given for basically everything in AI, in principle, if you have arbitrary amounts of time and arbitrarily deep understanding.
As I said (a few times!) in the discussion about orthogonality, indifference about the measure of “agents” that have particular properties seems crazy to me. Having an example of “agents” that behave in a particular way is a enormously different to having an unproven claim that such agents might be mathematically possible.