One thing that I’m pretty sure is going on and that might be a sufficient explanation is that it takes time to develop fluency in a hard problem. You can solve a simple problem in one go if you can hold enough of it in your mind to see the next steps of a plan that prove useful, and the same happens with the results of those steps, and eventually you reach a solution. For a harder problem, you might fail to see a specific plan, so you develop various observations about the problem and additional representations of its aspects, without having a clear sense of which of them will be useful, and there are too many of these to hold in your mind at the same time, as even though the new observations may be obtained by the methods you know well, they are in themselves new facts that are not yet thoroughly familiar.
To get to the next step, it might be necessary to be able to access a lot of these observations easily, without spending attention on recreating them. It takes time to familiarize yourself with the new observations (and with the way they connect to the original problem and to each other), to commit all these details to long term memory and to train your imagination to easily retrace the connections between them. But once you’ve done so, you obtain new superpowers with respect to that problem (and perhaps others analogous to it). A proof that would’ve taken 15 steps in terms of the ideas you had when you started working on the problem (and so wasn’t apparent), now takes only 4 steps in terms of the new auxiliary ideas you’ve developed in the meantime, and you can see it at a glance. (Perhaps if your mind wanders when you’re stepping onto a bus, and spends a few seconds on the problem, this proves sufficient to take advantage of the prior training and notice the solution.)
This analysis suggests that if you are stuck on a problem you need to solve, and you have enough time on your hands, (1) you should deliberately and systematically study the observations associated with your problem, even the ones that don’t seem immediately useful, and those that are easy to obtain, until these observations and their connections to the rest become obvious without the need to concentrate on reconstructing them, (2) revisit all (relevant) parts of the problem when you expect that new observations have been internalized since the last time you’ve revisited the problem.
This seems very insightful to me. In physics, it’s definitely my experience that over time I gain fluency with more and more powerful concepts that let me derive new things in much faster and simpler ways. And I find myself consciously working ideas over in my mind with, I think, the explicit goal of advancing this process.
The funny thing about this is that before I gain these “superpowers,” I’ll read an explanation in a textbook, which is in terms of high-level ideas that I haven’t completely grasped yet, so the reading doesn’t help as much as it should. The book claims, “this follows immediately from Lorentz invariance,” and I don’t really see what’s going on. Then, later, after I’ve understood those ideas, I find myself explaining things to myself in much the same words as the textbook: “I see! It’s simple! It follows immediately from Lorentz invariance!”—but now this really is an explanation, and the words have a lot more meaning.
One thing that I’m pretty sure is going on and that might be a sufficient explanation is that it takes time to develop fluency in a hard problem. You can solve a simple problem in one go if you can hold enough of it in your mind to see the next steps of a plan that prove useful, and the same happens with the results of those steps, and eventually you reach a solution. For a harder problem, you might fail to see a specific plan, so you develop various observations about the problem and additional representations of its aspects, without having a clear sense of which of them will be useful, and there are too many of these to hold in your mind at the same time, as even though the new observations may be obtained by the methods you know well, they are in themselves new facts that are not yet thoroughly familiar.
To get to the next step, it might be necessary to be able to access a lot of these observations easily, without spending attention on recreating them. It takes time to familiarize yourself with the new observations (and with the way they connect to the original problem and to each other), to commit all these details to long term memory and to train your imagination to easily retrace the connections between them. But once you’ve done so, you obtain new superpowers with respect to that problem (and perhaps others analogous to it). A proof that would’ve taken 15 steps in terms of the ideas you had when you started working on the problem (and so wasn’t apparent), now takes only 4 steps in terms of the new auxiliary ideas you’ve developed in the meantime, and you can see it at a glance. (Perhaps if your mind wanders when you’re stepping onto a bus, and spends a few seconds on the problem, this proves sufficient to take advantage of the prior training and notice the solution.)
This analysis suggests that if you are stuck on a problem you need to solve, and you have enough time on your hands, (1) you should deliberately and systematically study the observations associated with your problem, even the ones that don’t seem immediately useful, and those that are easy to obtain, until these observations and their connections to the rest become obvious without the need to concentrate on reconstructing them, (2) revisit all (relevant) parts of the problem when you expect that new observations have been internalized since the last time you’ve revisited the problem.
This seems very insightful to me. In physics, it’s definitely my experience that over time I gain fluency with more and more powerful concepts that let me derive new things in much faster and simpler ways. And I find myself consciously working ideas over in my mind with, I think, the explicit goal of advancing this process.
The funny thing about this is that before I gain these “superpowers,” I’ll read an explanation in a textbook, which is in terms of high-level ideas that I haven’t completely grasped yet, so the reading doesn’t help as much as it should. The book claims, “this follows immediately from Lorentz invariance,” and I don’t really see what’s going on. Then, later, after I’ve understood those ideas, I find myself explaining things to myself in much the same words as the textbook: “I see! It’s simple! It follows immediately from Lorentz invariance!”—but now this really is an explanation, and the words have a lot more meaning.
I’m reminded of the Interdict of Merlin in HMPOR.