I find the structure of this post very clear, but I’m confused about which are the ‘reality-masking’ problems that you say you spent a while puzzling. You list three bullets in that section, let me rephrase them as problems.
How to not throw things out just because they seem absurd
How to update on bayesian evidence even if it isn’t ‘legible, socially approved evidence’
How to cause beliefs to propagate through one’s model of the world
I guess this generally connects with my confusion around the ontology of the post. I think it would make sense for the post to be ‘here are some problems where puzzling at them helped me understand reality’ and ‘here are some problems where puzzling at them caused me to hide parts of reality from myself’, but you seem to think it’s an attribute of the puzzle, not the way one approaches it, and I don’t have a compelling sense of why you think that.
You give an example of teaching people math, and finding that you were training particular bad patterns of thought in yourself (and the students). That’s valid, and I expect a widespread experience. I personally have done some math tutoring that I don’t think had that property, due to background factors that affected how I approached it. In particular, I wasn’t getting paid, my mum told me I had to do it (she’s a private english teacher who also offers maths, but knows I grok maths better than her), and so I didn’t have much incentive to achieve results. I mostly just spoke with kids about what they understood, drew diagrams, etc, and had a fun time. I wasn’t too results-driven, mostly just having fun, and this effect didn’t occur.
More generally, many problems will teach you bad things if you locally hill-climb or optimise in a very short-sighted way. I remember as a 14 year old, I read Thinking Physics, spent about 5 mins per question, and learned nothing from repeatedly just reading the answers. Nowadays I do Thinking Physics problems weekly, and I spend like 2-3 hours per question. This seems more like a fact about how I approached it than a fact about the thing itself.
Looking up at the three bullets I pointed to, all three of them are important things to get right, that most people could be doing better on. I can imagine healthy and unhealthy ways of approaching them, but I’m not sure what an ‘unhealthy puzzle’ looks like.
I like your example about your math tutoring, where you “had a fun time” and “[weren’t] too results driven” and reality-masking phenomena seemed not to occur.
It reminds me of Eliezer talking about how the first virtue of rationality is curiosity.
I wonder how general this is. I recently read the book “Zen Mind, Beginner’s Mind,” where the author suggests that difficulty sticking to such principles as “don’t lie,” “don’t cheat,” “don’t steal,” comes from people being afraid that they otherwise won’t get a particular result, and recommends that people instead… well, “leave a line of retreat” wasn’t his suggested ritual, but I could imagine “just repeatedly leave a line of retreat, a lot” working for getting unattached.
Also, I just realized (halfway through typing this) that cousin_it and Said Achmiz say the same thing in another comment.
Thanks; you naming what was confusing was helpful to me. I tried to clarify here; let me know if it worked. The short version is that what I mean by a “puzzle” is indeed person-specific.
A separate clarification: on my view, reality-masking processes are one of several possible causes of disorientation and error; not the only one. (Sort of like how rationalization is one of several possible causes of people getting the wrong answers on math tests; not the only one.) In particular, I think singularity scenarios are sufficiently far from what folks normally expect that the sheer unfamiliarity of the situation can cause disorientation and errors (even without any reality-masking processes; though those can then make things worse).
I read ‘unhealthy puzzle’ as a situation in which (without trying to redesign it) you are likely to fall into a pattern that hides the most useful information about your true progress. Situation where you seek confirmatory evidence of your success, but the measures are only proxy measures can often have this feature (relating to Goodhart’s law).
example: If I want to be a better communicator I might accidentally spend more time with those I can already communicate well. Thus I feel like I’m making progress “the percentage of time that I’m well understood has increased” but not actually have made any change to my communication skills.
example: If I want to teach well it would be easier to seem like I’m making progress if I do things that make it harder for the student to explicitly show their confusion—e.g. I might answer my own questions before the student has time to think about them, I might give lots of prompts to students on areas they should be able to answer, I might talk too much and not listen enough.
example: If I’m trying to research something I might focus on the areas the theory is already known to succeed.
All of this could be done without realising that you are accidentally optimising for fake-progress.
I find the structure of this post very clear, but I’m confused about which are the ‘reality-masking’ problems that you say you spent a while puzzling. You list three bullets in that section, let me rephrase them as problems.
How to not throw things out just because they seem absurd
How to update on bayesian evidence even if it isn’t ‘legible, socially approved evidence’
How to cause beliefs to propagate through one’s model of the world
I guess this generally connects with my confusion around the ontology of the post. I think it would make sense for the post to be ‘here are some problems where puzzling at them helped me understand reality’ and ‘here are some problems where puzzling at them caused me to hide parts of reality from myself’, but you seem to think it’s an attribute of the puzzle, not the way one approaches it, and I don’t have a compelling sense of why you think that.
You give an example of teaching people math, and finding that you were training particular bad patterns of thought in yourself (and the students). That’s valid, and I expect a widespread experience. I personally have done some math tutoring that I don’t think had that property, due to background factors that affected how I approached it. In particular, I wasn’t getting paid, my mum told me I had to do it (she’s a private english teacher who also offers maths, but knows I grok maths better than her), and so I didn’t have much incentive to achieve results. I mostly just spoke with kids about what they understood, drew diagrams, etc, and had a fun time. I wasn’t too results-driven, mostly just having fun, and this effect didn’t occur.
More generally, many problems will teach you bad things if you locally hill-climb or optimise in a very short-sighted way. I remember as a 14 year old, I read Thinking Physics, spent about 5 mins per question, and learned nothing from repeatedly just reading the answers. Nowadays I do Thinking Physics problems weekly, and I spend like 2-3 hours per question. This seems more like a fact about how I approached it than a fact about the thing itself.
Looking up at the three bullets I pointed to, all three of them are important things to get right, that most people could be doing better on. I can imagine healthy and unhealthy ways of approaching them, but I’m not sure what an ‘unhealthy puzzle’ looks like.
I like your example about your math tutoring, where you “had a fun time” and “[weren’t] too results driven” and reality-masking phenomena seemed not to occur.
It reminds me of Eliezer talking about how the first virtue of rationality is curiosity.
I wonder how general this is. I recently read the book “Zen Mind, Beginner’s Mind,” where the author suggests that difficulty sticking to such principles as “don’t lie,” “don’t cheat,” “don’t steal,” comes from people being afraid that they otherwise won’t get a particular result, and recommends that people instead… well, “leave a line of retreat” wasn’t his suggested ritual, but I could imagine “just repeatedly leave a line of retreat, a lot” working for getting unattached.
Also, I just realized (halfway through typing this) that cousin_it and Said Achmiz say the same thing in another comment.
Thanks; you naming what was confusing was helpful to me. I tried to clarify here; let me know if it worked. The short version is that what I mean by a “puzzle” is indeed person-specific.
A separate clarification: on my view, reality-masking processes are one of several possible causes of disorientation and error; not the only one. (Sort of like how rationalization is one of several possible causes of people getting the wrong answers on math tests; not the only one.) In particular, I think singularity scenarios are sufficiently far from what folks normally expect that the sheer unfamiliarity of the situation can cause disorientation and errors (even without any reality-masking processes; though those can then make things worse).
I read ‘unhealthy puzzle’ as a situation in which (without trying to redesign it) you are likely to fall into a pattern that hides the most useful information about your true progress. Situation where you seek confirmatory evidence of your success, but the measures are only proxy measures can often have this feature (relating to Goodhart’s law).
example: If I want to be a better communicator I might accidentally spend more time with those I can already communicate well. Thus I feel like I’m making progress “the percentage of time that I’m well understood has increased” but not actually have made any change to my communication skills.
example: If I want to teach well it would be easier to seem like I’m making progress if I do things that make it harder for the student to explicitly show their confusion—e.g. I might answer my own questions before the student has time to think about them, I might give lots of prompts to students on areas they should be able to answer, I might talk too much and not listen enough.
example: If I’m trying to research something I might focus on the areas the theory is already known to succeed.
All of this could be done without realising that you are accidentally optimising for fake-progress.