First and foremost, it was quite an interesting post and my goal of the comment is to try to connect my own frame of thinking with the one presented here. My main question is about the relationship between emotions/implicit thoughts and explicit thinking.
My first thought was on the frame of thinking versus feeling and how these flow together. If we think of emotions as probability clouds that tell us whether to go in one direction or another, we can see them as systems for making decisions in highly complex environments, such as when working on impossible problems.
I think something like research taste is exactly this—highly trained implicit thoughts and emotions. Continuing from something like tuning your cognitive systems, I notice that this is mostly done with System 2 and I can’t help but feel that it’s missing some System 1 stuff here.
I will give an analogy similar to a meditation analogy as this is the general direction I’m pointing in:
If we imagine that we’re faced with a wall of rock, it looks like a very big problem. You’re thinking to yourself, “fuck, how in the hell are we ever going to get past that thing?”
So first you just approach it and you start using a pickaxe to hack away at it, you make some local progress yet it is hard to reflect on where to go. You think hard, what are the properties of this rock that allows me to go through it faster?
You continue yet you’re starting to feel discouraged as you’re not making any progress, you think to yourself “Fuck this goddamn rock man, this shit is stupid.”
You’re not getting any feedback since it is an almost impossible problem.
Above is the base analogy, following are two points on the post from this analogy:
1. Let’s start with a continuation to the analogy, imagine that your goal, the thing behind huge piece of rock is a source of gravity and you’re water.
You’re continuously striving towards it yet the way that you do it is that you flow over the surface. You’re probing for holes in the rock, crevices that run deep, structural instability in the rock yet you’re not thinking—you’re feeling it out. You’re flowing in the problem space, allowing implicit thoughts and emotions guide you and from time to time you make a cut. Yet your evaluation loop is a lot longer than your improvement loop. It doesn’t matter if you haven’t found anything yet because gravity is pulling you in that direction and if you succeed is a question of finding the crevice rather than your individual successes with your pickaxe.
You apply all the rules of local gradient search and similar, you’re not a stupid fluid yet you’re fine with failing because you know it gives you information about where the crevice might be, and it isn’t until you find it that you will make major progress.
2. If you have other people with you then you can see what others are doing and check whether your strategies are stupid or not. They give you an appropriate measuring stick for working on an impossible problem. You may not know how well you’re doing in solving the problem but you know your relative rating and so you can get feedback through that (as long as it is causally related to the problem you’re solving).
What are your thoughts on the trade-off between emotional understanding and more hardcore system 2 thinking? If one applies the process above, do you think there’s something that is missed out?
My overall frame is it’s best to have emotional understanding and system 2 deeply integrated. How to handle local tradeoffs unfortunately depends a lot on your current state, and where your bottlenecks are.
Could you provide a specific, real-world example where the tradeoff comes up and you’re either unsure of how to navigate it, or, you think I might suggested navigating it differently?
So, I’ve had this research agenda into agent foundations for a while which essentially mirrors developmental interpretability a bit in that it wants to say things about what a robust development process is rather than something about post-training sampling.
The idea is to be able to predict “optimisation daemons” or inner optimisers as they arise in a system.
The problem that I’ve had is that it is very non-obvious to me what a good mathematical basis for this is. I’ve read through a lot of the existent agent foundations literature but I’m not satisfied with finite factored sets nor the existing boundaries definitions since they don’t tell you about the dynamics.
What I would want is a dynamical systems inspired theory of the formation of inner misalignment. It’s been in my head in the background for almost 2 years now and it feels really difficult to make any progress, from time to time I have a thought that brings me closer but I don’t usually make it closer by just thinking about it.
I guess something I’m questioning in my head is the deliberate practice versus exploration part of this. For me this is probably the hardest problem I’m working on and whilst I could think more deliberately on what I should be doing here I generally follow my curiosity, which I think has worked better than deliberate practice in this area?
I’m currently following a strategy where this theoretical foundation is on the side whilst I build real world skills of running organisations, fundraising, product-building and networking. I then from time to time find some gems such as applied category theory or Michael Levin’s work on Boundaries in cells and Active Inference that I find can really help elucidate some of the deeper foundations of this problem.
I do feel like I’m floating more here, going with the interest and coming back to the problems over time in order to see if I’ve unlocked any new insights. This feels more like flow than it does deliberate practice? Like I’m building up my skills of having loose probability clouds and seeing where they guide me?
I’m not sure if you agree that this is the right strategy but I guess that there’s this frame difference between a focus on the emotional, intuition or research taste side of things versus the deliberate practice side of things?
Nod. So, first of all: I don’t know. My own guesses would depend on a lot of details of the individual person (even those in a similar situation to you).
(This feels somewhat outside the scope of the main thrust of this post, but, definitely related to my broader agenda of ‘figure out a training paradigm conveying the skills and tools necessary to solve very difficult, confusing problems’)
But, riffing so far. And first, summarizing what seemed like the key points:
You want to predict optimization daemons as they arise in a system, and want a good mathematical basis for that, and don’t feel satisfied with the existing tools.
You’re currently exploring this on-the-side while working on some more tractable problems.
You’ve identified two broad-strategies, which are:
somehow “deliberate practice” this,
somehow explore and follow your curiosity intermitently
Three things I’d note:
“Deliberate practice” is very openended. You can deliberate practice noticing and cultivating veins of curiosity, for example.
You can strategize about how to pursue curiosity, or explore (without routing through the practice angle)
There might be action-spaces other than “deliberate practice” or “explore/curiosity” that will turn out to be useful.
My current angle for deliberate practice is to find problem sets that feel somehow-analogous to the one you’re trying to tackle, but simpler/shorter. They should be difficult enough that they feel sort of impossible while you’re working on them, but, also actually solvable. They should be varied enough that you aren’t overfitting to one particular sort of puzzle.
After the exercise, apply the Think It Faster meta-exercise to it.
Part of the point here is notice strategies like “apply explicit systematic thinking” and strategies like “take a break, come back to it when you feel more inspired”, and start to develop your own sense of which strategies work best for you.
First and foremost, it was quite an interesting post and my goal of the comment is to try to connect my own frame of thinking with the one presented here. My main question is about the relationship between emotions/implicit thoughts and explicit thinking.
My first thought was on the frame of thinking versus feeling and how these flow together. If we think of emotions as probability clouds that tell us whether to go in one direction or another, we can see them as systems for making decisions in highly complex environments, such as when working on impossible problems.
I think something like research taste is exactly this—highly trained implicit thoughts and emotions. Continuing from something like tuning your cognitive systems, I notice that this is mostly done with System 2 and I can’t help but feel that it’s missing some System 1 stuff here.
I will give an analogy similar to a meditation analogy as this is the general direction I’m pointing in:
If we imagine that we’re faced with a wall of rock, it looks like a very big problem. You’re thinking to yourself, “fuck, how in the hell are we ever going to get past that thing?”
So first you just approach it and you start using a pickaxe to hack away at it, you make some local progress yet it is hard to reflect on where to go. You think hard, what are the properties of this rock that allows me to go through it faster?
You continue yet you’re starting to feel discouraged as you’re not making any progress, you think to yourself “Fuck this goddamn rock man, this shit is stupid.”
You’re not getting any feedback since it is an almost impossible problem.
Above is the base analogy, following are two points on the post from this analogy:
1.
Let’s start with a continuation to the analogy, imagine that your goal, the thing behind huge piece of rock is a source of gravity and you’re water.
You’re continuously striving towards it yet the way that you do it is that you flow over the surface. You’re probing for holes in the rock, crevices that run deep, structural instability in the rock yet you’re not thinking—you’re feeling it out. You’re flowing in the problem space, allowing implicit thoughts and emotions guide you and from time to time you make a cut. Yet your evaluation loop is a lot longer than your improvement loop. It doesn’t matter if you haven’t found anything yet because gravity is pulling you in that direction and if you succeed is a question of finding the crevice rather than your individual successes with your pickaxe.
You apply all the rules of local gradient search and similar, you’re not a stupid fluid yet you’re fine with failing because you know it gives you information about where the crevice might be, and it isn’t until you find it that you will make major progress.
2.
If you have other people with you then you can see what others are doing and check whether your strategies are stupid or not. They give you an appropriate measuring stick for working on an impossible problem. You may not know how well you’re doing in solving the problem but you know your relative rating and so you can get feedback through that (as long as it is causally related to the problem you’re solving).
What are your thoughts on the trade-off between emotional understanding and more hardcore system 2 thinking? If one applies the process above, do you think there’s something that is missed out?
My overall frame is it’s best to have emotional understanding and system 2 deeply integrated. How to handle local tradeoffs unfortunately depends a lot on your current state, and where your bottlenecks are.
Could you provide a specific, real-world example where the tradeoff comes up and you’re either unsure of how to navigate it, or, you think I might suggested navigating it differently?
Yeah sure!
So, I’ve had this research agenda into agent foundations for a while which essentially mirrors developmental interpretability a bit in that it wants to say things about what a robust development process is rather than something about post-training sampling.
The idea is to be able to predict “optimisation daemons” or inner optimisers as they arise in a system.
The problem that I’ve had is that it is very non-obvious to me what a good mathematical basis for this is. I’ve read through a lot of the existent agent foundations literature but I’m not satisfied with finite factored sets nor the existing boundaries definitions since they don’t tell you about the dynamics.
What I would want is a dynamical systems inspired theory of the formation of inner misalignment. It’s been in my head in the background for almost 2 years now and it feels really difficult to make any progress, from time to time I have a thought that brings me closer but I don’t usually make it closer by just thinking about it.
I guess something I’m questioning in my head is the deliberate practice versus exploration part of this. For me this is probably the hardest problem I’m working on and whilst I could think more deliberately on what I should be doing here I generally follow my curiosity, which I think has worked better than deliberate practice in this area?
I’m currently following a strategy where this theoretical foundation is on the side whilst I build real world skills of running organisations, fundraising, product-building and networking. I then from time to time find some gems such as applied category theory or Michael Levin’s work on Boundaries in cells and Active Inference that I find can really help elucidate some of the deeper foundations of this problem.
I do feel like I’m floating more here, going with the interest and coming back to the problems over time in order to see if I’ve unlocked any new insights. This feels more like flow than it does deliberate practice? Like I’m building up my skills of having loose probability clouds and seeing where they guide me?
I’m not sure if you agree that this is the right strategy but I guess that there’s this frame difference between a focus on the emotional, intuition or research taste side of things versus the deliberate practice side of things?
Nod. So, first of all: I don’t know. My own guesses would depend on a lot of details of the individual person (even those in a similar situation to you).
(This feels somewhat outside the scope of the main thrust of this post, but, definitely related to my broader agenda of ‘figure out a training paradigm conveying the skills and tools necessary to solve very difficult, confusing problems’)
But, riffing so far. And first, summarizing what seemed like the key points:
You want to predict optimization daemons as they arise in a system, and want a good mathematical basis for that, and don’t feel satisfied with the existing tools.
You’re currently exploring this on-the-side while working on some more tractable problems.
You’ve identified two broad-strategies, which are:
somehow “deliberate practice” this,
somehow explore and follow your curiosity intermitently
Three things I’d note:
“Deliberate practice” is very openended. You can deliberate practice noticing and cultivating veins of curiosity, for example.
You can strategize about how to pursue curiosity, or explore (without routing through the practice angle)
There might be action-spaces other than “deliberate practice” or “explore/curiosity” that will turn out to be useful.
My current angle for deliberate practice is to find problem sets that feel somehow-analogous to the one you’re trying to tackle, but simpler/shorter. They should be difficult enough that they feel sort of impossible while you’re working on them, but, also actually solvable. They should be varied enough that you aren’t overfitting to one particular sort of puzzle.
After the exercise, apply the Think It Faster meta-exercise to it.
Part of the point here is notice strategies like “apply explicit systematic thinking” and strategies like “take a break, come back to it when you feel more inspired”, and start to develop your own sense of which strategies work best for you.