The problem with x2+y2=1 is that it’s not clear why x would seem like a good proxy in the first place. With an inequality constraint, x has positive correlation with the objective everywhere except the boundary. You get at this idea with u(x,y)=xy knowing only x, but I think it’s more a property of dimensionality than of objective complexity—even with a complicated objective, it’s usually easy to tell how to change a single variable to improve the objective if everything else is held constant.
It’s the “held constant” part that really matters—changing one variable while holding all else constant only makes sense in the interior of the set, so it runs into Goodhart-type tradeoffs once you hit the boundary. But you still need the interior in order for the proxy to look good in the first place.
The problem with x2+y2=1 is that it’s not clear why x would seem like a good proxy in the first place. With an inequality constraint, x has positive correlation with the objective everywhere except the boundary. You get at this idea with u(x,y)=xy knowing only x, but I think it’s more a property of dimensionality than of objective complexity—even with a complicated objective, it’s usually easy to tell how to change a single variable to improve the objective if everything else is held constant.
It’s the “held constant” part that really matters—changing one variable while holding all else constant only makes sense in the interior of the set, so it runs into Goodhart-type tradeoffs once you hit the boundary. But you still need the interior in order for the proxy to look good in the first place.