“So my brain is sitting there with mirror-twin goals of maximize exposure to low scale numbers and minimize exposure to high scale numbers, “
I think my answer would be don’t give it those goals. Give it rule following goals and model updating goals. Create rules from the models. If following the rules don’t actually get you slimmer, update the model and update the rules.
I think you missed the central claim? I’m not saying those rules are good, nor that they are consciously installed and reflectively endorsed. I’m saying that your subconscious has goals like that that you, whpearson’s conscious verbal loop, aren’t fully aware of and don’t notice and are manipulated (or at least influenced) by. This isn’t a thing that’s fixed by simply deciding on different rules—S1 doesn’t communicate verbally, except indirectly by responding to stories and narratives.
Also, the idea that the solution to Goodhart’s Law is “create rules” makes me feel like I failed to communicate Claim 1.
I was not say that they were consciously endorsed, just that they were a product of taking a particular mindset which was consciously endorsed. E.g. my goal is to lose weight.
What I am suggesting, which is not a panacea but might not suck too much, is taking a different mindset. So “My goal is to understand the relationship between my activities and weight”. So the scientific point of view. Once you have gotten a good understanding that you can actually try and optimise your weight. More details on what I am trying to get across can be found in this post which introduces it and post which gives a hypothetical example of its application to a field and has a more formal description of what I mean in one of the comments.
Yeah, but I guess I’m still not communicating the part where a very large and important section of your brain doesn’t just adopt the goals you consciously give it. The stuff you said above is true, but irrelevant in this context/overwhelmed or undermined by the effect the post is pointing at, which makes me continue to feel like you’re not receiving the point I’m trying to convey.
I hundred percent agree that most of the brain is below conscious control and doesn’t adopt your goals. What I think Goodhearts law should be guiding us towards is how we set the bits of the brain that are.
For example of losing weight by measuring weight and using that as a metric is *literally* setting the metric and measure the same. I was trying to point out a way that a measure could be used in pursuit of a goal, but that it could not be a treated like a metric.
I didn’t get that impression from your post that you thought that there was any sort of conscious thing you could do before hand to try and head off the worst bits of Goodhearts. It was only post-hoc noticing things are going wrong. Like noticing “hey I am about to start on a poorly understood problem, that my subconscious brain will optimise wrong if I just set solving it as my goal. Maybe instead I should try to understand it first, using the measure I was about to use as a metric.”
“So my brain is sitting there with mirror-twin goals of maximize exposure to low scale numbers and minimize exposure to high scale numbers, “
I think my answer would be don’t give it those goals. Give it rule following goals and model updating goals. Create rules from the models. If following the rules don’t actually get you slimmer, update the model and update the rules.
Uh
I think you missed the central claim? I’m not saying those rules are good, nor that they are consciously installed and reflectively endorsed. I’m saying that your subconscious has goals like that that you, whpearson’s conscious verbal loop, aren’t fully aware of and don’t notice and are manipulated (or at least influenced) by. This isn’t a thing that’s fixed by simply deciding on different rules—S1 doesn’t communicate verbally, except indirectly by responding to stories and narratives.
Also, the idea that the solution to Goodhart’s Law is “create rules” makes me feel like I failed to communicate Claim 1.
I was not say that they were consciously endorsed, just that they were a product of taking a particular mindset which was consciously endorsed. E.g. my goal is to lose weight.
What I am suggesting, which is not a panacea but might not suck too much, is taking a different mindset. So “My goal is to understand the relationship between my activities and weight”. So the scientific point of view. Once you have gotten a good understanding that you can actually try and optimise your weight. More details on what I am trying to get across can be found in this post which introduces it and post which gives a hypothetical example of its application to a field and has a more formal description of what I mean in one of the comments.
Yeah, but I guess I’m still not communicating the part where a very large and important section of your brain doesn’t just adopt the goals you consciously give it. The stuff you said above is true, but irrelevant in this context/overwhelmed or undermined by the effect the post is pointing at, which makes me continue to feel like you’re not receiving the point I’m trying to convey.
We may be agreeing, but not being clear!
I hundred percent agree that most of the brain is below conscious control and doesn’t adopt your goals. What I think Goodhearts law should be guiding us towards is how we set the bits of the brain that are.
For example of losing weight by measuring weight and using that as a metric is *literally* setting the metric and measure the same. I was trying to point out a way that a measure could be used in pursuit of a goal, but that it could not be a treated like a metric.
I didn’t get that impression from your post that you thought that there was any sort of conscious thing you could do before hand to try and head off the worst bits of Goodhearts. It was only post-hoc noticing things are going wrong. Like noticing “hey I am about to start on a poorly understood problem, that my subconscious brain will optimise wrong if I just set solving it as my goal. Maybe instead I should try to understand it first, using the measure I was about to use as a metric.”