I think this post is a red flag about your mental health. “I work so hard that I ignore broken glass and then walk on it” is not healthy.
Seems like a rational prioritization to me if they were in an important moment of thought and didn’t want to disrupt it. (Noting of course that ‘walking on it’ was not intentional and was caused by forgetting it was there.)
Also, I would feel pretty bad if someone wrote a comment like this after I posted something. (Maybe it would have been better as a PM)
Seems like a rational prioritization to me if they were in an important moment of thought and didn’t want to disrupt it. (Noting of course that ‘walking on it’ was not intentional and was caused by forgetting it was there.)
This sounds like you’re saying that they made a rational prioritization and then, separately from that, forgot that it was there. But those two events are not separate: the forgetting-and-then-walking-on-it was a predictable consequence of the earlier decision to ignore it and instead focus on work. I think if you model the first decision as a decision to continue working and to also take on a significant risk of hurting your feet, it doesn’t seem so obviously rational anymore. (Of course it could be that the thought in question was just so important that it was worth the risk. But that seems unlikely to me.)
As the OP says, a “normal person might stop and remove all the glass splinters”. Most people, in thinking whether to continue working or whether to clean up the splinters, wouldn’t need to explicitly consider the possibility that they might forget about the splinters and step on them later. This would be incorporated into the decision-making process implicitly and automatically, by the presence of splinters making them feel uneasy until they were cleaned up. The fact that this didn’t happen suggests that the OP might also ignore other signals relevant to their well-being.
The fact that the OP seems to consider this event a virtue to highlight in the title of their post, is also a sign that they are systematically undervaluing their own well-being in a way that to me seems very worrying.
Also, I would feel pretty bad if someone wrote a comment like this after I posted something. (Maybe it would have been better as a PM)
Probably most people would. But I think it’s also really important for there to be clear, public signals that the community wants people to take their well-being seriously and doesn’t endorse people hurting themselves “for the sake of the cause”.
The EA and rationalist communities are infamous for having lots of people burning themselves out through extreme self-sacrifice. If someone makes a post where they present the act of working until their feet start bleeding as a personal virtue, and there’s no public pushback to that, then that sends the implicit signal that the community endorses that reasoning. That will then contribute to unhealthy social norms that cause people to burn themselves out. The only way to counteract that is by public comments that make it clear that the community wants people to take care of themselves, even if that makes them (temporarily) less effective.
To the OP: please prioritize your well-being first. Self-preservation is one of the instrumental convergent drives; you can only continue to work if you are in good shape.
I am probably bad at valuing my well-being correctly. That said I don’t think the initial comment made me feel bad (but maybe I am bad at noticing if it would). Rather now with this entire comment stream, I realize that I have again failed to communicate.
Yes, I think this was irrational to not clean up the glass. That is the point I want to make. I don’t think it is virtuous to have failed in this way at all. What I want to say is: “Look I am running into failure modes because I want to work so much.”
Not running into these failure modes is important, but these failure modes where you are working too much are much easier to handle than the failure mode of “I can’t get myself to put in at least 50 hours of work per week consistently.”
While I do think that it is true, I am probably very bad in general at optimizing for myself to be happy. But the thing is while I was working so hard during AISC I was most of the time very happy. The same when I made these games. Most of the time I did these things because I deeply wanted to.
There where moments during AISC where I felt like I was close to burning out, but this was the minority. Mostly I was much happier than baseline. I think usually I don’t manage to work as hard and as long as I’d like, and that is a major source of unhappiness for me.
So it seems that the problem that Alex seems to see, in me working very hard (that I am failing to take my happiness into account) is actually solved by me working very hard, which is quite funny.
Yes, I think this was irrational to not clean up the glass. That is the point I want to make. I don’t think it is virtual to have failed in this way at all. What I want to say is: “Look I am running into failure modes because I want to work so much.”
Ah! I completely missed that, that changes my interpretation significantly. Thank you for the clarification, now I’m less worried for you since it no longer sounds like you have a blindspot around it.
Not running into these failure modes is important, but these failure modes where you are working too much are much easier to handle than the failure mode of “I can’t get myself to put in at least 50 hours of work per week consistently.”
While I do think that it is true, I am probably very bad in general at optimizing for myself to be happy. But the thing is while I was working so hard during AISC I was most of the time very happy. The same when I made these games. Most of the time I did these things because I deeply wanted to.
It sounds right that these failure modes are easier to handle than the failure mode of not being able to do much work.
Though working too much can lead to the failure mode of “I can’t get myself put in work consistently”. I’d be cautious in that it’s possible to feel like you really enjoy your work… and then burn out anyway! I’ve heard several people report this happening to them. The way I model that is something like… there are some parts of the person that are obsessed with the work, and become really happy about being able to completely focus on the obsession. But meanwhile, that single-minded focus can lead to the person’s other needs not being met, and eventually those unmet needs add up and cause a collapse.
I don’t know how much you need to be worried about that, but it’s at least good to be aware of.
This sounds like you’re saying that they made a rational prioritization and then, separately from that, forgot that it was there
That implication wasn’t intended. I agree that (for basic reasons) the probability of a small cut was higher given their choice.
Rather, the action itself seems rational to me when considering:
That outcome seems unprobable (at least if they were sitting down), but actual in this particular timeline.
The effects of a cut on the foot are really low (with I’d guess >99.5% probability, for an otherwise healthy person—on reflection, maybe not cumulatively low enough for the also-small payoff?), and if so ~certain to not significantly curtail progress.
That doesn’t necessarily imply the policy which produced the action is rational, though. But when considering the two hypotheses: (1) OP is mentally unwell, and (2) They have some them-specific reason[1] for following a policy which outputs actions like this, I considered (2) to be a lot more probable.
Meta: This comment is (genuinely) very hard/overwhelming-feeling for me to try to reply to, for a few reasons specific to my mind, mainly about {unmarked assumptions} and {parts seeming to be for rhetorical effect}. (For that reason I’ll let others discuss this instead of saying much further)
I think it’s also really important for there to be clear, public signals that the community wants people to take their well-being seriously
I agree with this, but I think any ‘community norm reinforcing messages’ should be clearly about norms rather than framed about an individual, in cases like this where there’s just a weak datapoint about the individual.
A simple example would be “Having introspected and tested different policies before determining that they’re not at risk of burnout from the policy which gives this action.”
A more complex example would be “a particular action can be irrational in isolation but downstream of a (suboptimal but human-attainable) policy which produces irrational behavior less than is typical”, which (now) seems to me to be what OP was trying to show with this example given their comment
Jog, don’t sprint. Skeptics of the “most important century” hypothesis will sometimes say things like “If you really believe this, why are you working normal amounts of hours instead of extreme amounts? Why do you have hobbies (or children, etc.) at all?” And I’ve seen a number of people with an attitude like: “THIS IS THE MOST IMPORTANT TIME IN HISTORY. I NEED TO WORK 24⁄7 AND FORGET ABOUT EVERYTHING ELSE. NO VACATIONS.”
I think that’s a very bad idea.
Trying to reduce risks from advanced AI is, as of today, a frustrating and disorienting thing to be doing. It’s very hard to tell whether you’re being helpful (and as I’ve mentioned, many will inevitably think you’re being harmful).
I think the difference between “not mattering,” “doing some good” and “doing enormous good” comes down to how you choose the job, how good at it you are, and how good your judgment is (including what risks you’re most focused on and how you model them). Going “all in” on a particular objective seems bad on these fronts: it poses risks to open-mindedness, to mental health and to good decision-making (I am speaking from observations here, not just theory).
That is, I think it’s a bad idea to try to be 100% emotionally bought into the full stakes of the most important century—I think the stakes are just too high for that to make sense for any human being.
Instead, I think the best way to handle “the fate of humanity is at stake” is probably to find a nice job and work about as hard as you’d work at another job, rather than trying to make heroic efforts to work extra hard. (I criticized heroic efforts in general here.)
I think this basic formula (working in some job that is a good fit, while having some amount of balance in your life) is what’s behind a lot of the most important positive events in history to date, and presents possibly historically large opportunities today.
It’s more important to have good judgment than to dedicate 100% of your life to an EA project. If output scales linearly with work hours, then you can hit 60% of your maximum possible impact with 60% of your work hours. But if bad judgment causes you to miss one or two multipliers, you could make less than 10% of your maximum impact. (But note that working really hard can sometimes enable multipliers—see this comment by Mathieu Putz.)
I think both “jog, don’t sprint” and “sprint, don’t jog” is too low-dimensional as advice. It’s good to try to spend 100% of one’s resources on doing good—sorta tautologically. What allows Johannes to work as hard as he does, I think, is not (just) that he’s obsessed with the work, it’s rather that he understands his own mind well enough to navigate around its limits. And that self-insight is also what enables him aim his cognition at what matters—which is a trait I care more about than ability to work hard.
People who are good at aiming their cognition at what matters sometimes choose to purposefwly flout[1]various social expectations in order to communicate “I see through this distracting social convention and I’m willing to break it in order to aim myself more purely at what matters”. Readers who haven’t noticed that some of their expectations are actually superfluous or misaligned with altruistic impact, will mistakenly think the flouter has low impact-potential or is just socially incompetent.
By writing the way he does, Johannes signals that he’s distancing himself from status-related putative proxies-for-effectiveness, and I think that’s a hard requirement for aiming more purely at the conjunction of multipliers[2] that matter. But his signals will be invisible to people who aren’t also highly attuned to that conjunction.
I think the post uses an odd definition of “conjunction”, but it points to something important regardless. My term for this bag of nearby considerations is “costs of compromise”:
there are exponential costs to compromising what you are optimizing for in order to appeal to a wider variety of interests
I think what quila is pointing at is their belief in the supposed fragility of thoughts at the edge of research questions. From that perspective I think their rebuttal is understandable, and your response completely misses the point: you can be someone who spends only four hours a day working and the rest of the time relaxing, but also care a lot about not losing the subtle and supposedly fragile threads of your thought when working.
Note: I have a different model of research thought, one that involves a systematic process towards insight, and because of that I also disagree with Johannes’ decisions.
Rather, I’m confident that executing my research process will over time lead to something good.
Yeah, this is a sentiment I agree with and believe. I think that it makes sense to have a cognitive process that self-corrects and systematically moves towards solving whatever problem it is faced with. In terms of computability theory, one could imagine it as an effectively computable function that you expect will return you the answer—and the only ‘obstacle’ is time / compute invested.
I think being confident, i.e. not feeling hopeless in doing anything, is important. The important takeaway here is that you don’t need to be confident in any particular idea that you come up with. Instead, you can be confident in the broader picture of what you are doing, i.e. your processes.
I share your sentiment, although the causal model for it is different in my head. A generalized feeling of hopelessness is an indicator of mistaken assumptions and causal models in my head, and I use that as a cue to investigate why I feel that way. This usually results in me having hopelessness about specific paths, and a general purposefulness (for I have an idea of what I want to do next), and this is downstream of updates to my causal model that attempts to track reality as best as possible.
Note that when I said I disagree with your decisions, I specifically meant the sort of myopia in the glass shard story—and specifically because I believe that if your research process / cognition algorithm is fragile enough that you’d be willing to take physical damage to hold onto an inchoate thought, maybe consider making your cognition algorithm more robust.
On my current models of theoretical[1] insight-making, the beginning of an insight will necessarily—afaict—be “non-robust”/chaotic. I think it looks something like this:
A gradual build-up and propagation of salience wrt some tiny discrepancy between highly confident specific beliefs
This maybe corresponds to simultaneously-salient neural ensembles whose oscillations are inharmonic[2]
Or in the frame of predictive processing: unresolved prediction-error between successive layers
Immediately followed by a resolution of that discrepancy if the insight is successfwl
This maybe corresponds to the brain having found a combination of salient ensembles—including the originally inharmonic ensembles—whose oscillations are adequately harmonic.
Super-speculative but: If the “question phase” in step 1 was salient enough, and the compression in step 2 great enough, this causes an insight-frisson[3] and a wave of pleasant sensations across your scalp, spine, and associated sensory areas.
This maps to a fragile/chaotic high-energy “question phase” during which the violation of expectation is maximized (in order to adequately propagate the implications of the original discrepancy), followed by a compressive low-energy “solution phase” where correctness of expectation is maximized again.
In order to make this work, I think the brain is specifically designed to avoid being “robust”—though here I’m using a more narrow definition of the word than I suspect you intended. Specifically, there are several homeostatic mechanisms which make the brain-state hug the border between phase-transitions as tightly as possible. In other words, the brain maximizes dynamic correlation length between neurons[4], which is when they have the greatest ability to influence each other across long distances (aka “communicate”). This is called the critical brain hypothesis, and it suggests that good thinking is necessarily chaotic in some sense.
Another point is that insight-making is anti-inductive.[5] Theoretical reasoning is a frontier that’s continuously being exploited based on the brain’s native Value-of-Information-estimator, which means that the forests with the highest naively-calculated-VoI are also less likely to have any low-hanging fruit remaining. What this implies is that novel insights are likely to be very narrow targets—which means they could be really hard to hold on to for the brief moment between initial hunch and build-up of salience. (Concise handle: epistemic frontiers are anti-inductive.)
I scope my arguments only to “theoretical processing” (i.e. purely introspective stuff like math), and I don’t think they apply to “empirical processing”.
Harmonic (red) vs inharmonic (blue) waveforms. When a waveform is harmonic, efferent neural ensembles can quickly entrain to it and stay in sync with minimal metabolic cost. Alternatively, in the context of predictive processing, we can say that “top-down predictions” quickly “learn to predict” bottom-up stimuli.
I basically think musical pleasure (and aesthetic pleasure more generally) maps to 1) the build-up of expectations, 2) the violation of those expectations, and 3) the resolution of those violated expectations. Good art has to constantly balance between breaking and affirming automatic expectations. I think the aesthetic chills associates with insights are caused by the same structure as appogiaturas—the one-period delay of an expected tone at the end of a highly predictable sequence.
I think the term originates from Eliezer, but Q Home has more relevant discussion on it—also I’m just a big fan of their chaoticoptimal reasoning style in general. Can recommend! 🍵
I think what quila is pointing at is their belief in the supposed fragility of thoughts at the edge of research questions.
Yes, thanks for noticing and making it explicit. It seems I was modelling Johannes as having a similar cognition type, since it would explain their behavior, which actually had a different cause.
I believe that if your research process / cognition algorithm is fragile enough that you’d be willing to take physical damage[1] to hold onto an inchoate thought, maybe consider making your cognition algorithm more robust.
My main response to ‘try to change your cognition algorithm if it is fragile’ is to remind that human minds tend to work differently on unexpected dimensions. (Of course, you know this abstractly, and have probably read the same post about the ‘typical mind fallacy’. But the suggestion seems like harmful advice to follow for some of the minds it’s directed at.) (Alternatively, since you wrote ‘maybe’, this comment can be seen as describing a kind of case where it would be harmful)
My highest value mental states are fragile: they are hard to re-enter at will once left, and they take some subconscious effort to preserve/cultivate. They can also feel totally immersing and overwhelming, when I manage to enter them. (I don’t feel confident in my ability to qualitatively write more, as much as I would like to (maybe not here or now)).
This is analogous to Johannes’ situation in a way. They believe the problem they have of working too hard is less bad to have than the standard problem of not feeling motive to work. The specific irrational behavior their problem caused also ‘stands out’ more to onlookers, since it’s not typical. (One wouldn’t expect the top comment here if one described succumbing to akrasia; but if akrasia was rare in humans, such that the distribution over most probable causes included some worrying possibilities, we might)
In the same way, I feel like my cognition-algorithm is in a local optima which is better than the standard one, where one lesser-problem I face is that my highest output mental states are ‘fragile’, and because this is not typical it may (when read of in isolation) seem like a sign of ‘a negative deviation from the normal local optima, which this person would be better off if they corrected’.
From my inside perspective, I don’t want to try to avoid fragile mental states, because I think it would only be a possible change as a more general directional change away from ‘how my cognition works (at its best)’ towards ‘how human cognition typically works’.
(And because the fragility-of-thought feels like a small problem, once I learned to work around it, e.g learning to preserve states and augmenting with external notes. At least when compared to the problem most have of not having a chance at generating insights of a high enough quality as our situation necessitates.)
… although, if you knew of a method to reduce fragility while not reducing other things, then I’d love to try it :)
On ‘willing to take physical damage …’, footnoted because it seems like a minor point—This seems like another case of avoiding the typical-mind-fallacy being important, since different minds have different pain tolerances / levels of experienced pain from a cut.
I think you’re right that I missed their point, thanks for pointing it out.
I have had experiences similar to Johannes’ anecdote re: ignoring broken glass to not lose fragile threads of thought; they usually entailed extended deep work periods past healthy thresholds for unclear marginal gain, so the quotes above felt personally relevant as guardrails. But also my experiences don’t necessarily generalize (as your hypothetical shows).
I’d be curious to know your model, and how it compares to some of John Wentworth’s posts on the same IIRC.
I think that conceptual alignment research of the sort that Johannes is doing (and that I also am doing, which I call “deconfusion”) is just really difficult. It involves skills that are not taught to people, that seems very unlikely that you’d learn by being mentored in traditional academia (including when doing theoretical CS or non-applied math PhDs), that I only started wrapping my head around after some mentorship from two MIRI researchers (that I believe I was pretty lucky to get), and even then I’ve spent a ridiculous amount of time by myself trying to tease out patterns to figure out a more systematic process of doing this.
Oh, and the more theoretical CS (and related math such as mathematical logic) you know, the better you probably are at this—see how Johannes tries to create concrete models of the inchoate concepts in his head? Well, if you know relevant theoretical CS and useful math, you don’t have to rebuild the mathematical scaffolding all by yourself.
I don’t have a good enough model of John Wentworth’s model for alignment research to understand the differences, but I don’t think I learned all that much from John’s writings and his training sessions that were a part of his MATS 4.0 training regimen, as compared to the stuff I described above.
What <mathematical scaffolding/theoretical CS> do you think I am recreating? What observations did you use to make this inference? (These questions are not intended to imply any subtext meaning.)
Well, if you know relevant theoretical CS and useful math, you don’t have to rebuild the mathematical scaffolding all by yourself.
I didn’t intend to imply in my message that you have mathematical scaffolding that you are recreating, although I expect it may be likely (Pearlian causality perhaps? I’ve been looking into it recently and clearly knowing Bayes nets is very helpful). I specifically used “you” to imply that in general this is the case. I haven’t looked very deep into the stuff you are doing, unfortunately—it is on my to-do list.
Seems like a rational prioritization to me if they were in an important moment of thought and didn’t want to disrupt it. (Noting of course that ‘walking on it’ was not intentional and was caused by forgetting it was there.)
Also, I would feel pretty bad if someone wrote a comment like this after I posted something. (Maybe it would have been better as a PM)
This sounds like you’re saying that they made a rational prioritization and then, separately from that, forgot that it was there. But those two events are not separate: the forgetting-and-then-walking-on-it was a predictable consequence of the earlier decision to ignore it and instead focus on work. I think if you model the first decision as a decision to continue working and to also take on a significant risk of hurting your feet, it doesn’t seem so obviously rational anymore. (Of course it could be that the thought in question was just so important that it was worth the risk. But that seems unlikely to me.)
As the OP says, a “normal person might stop and remove all the glass splinters”. Most people, in thinking whether to continue working or whether to clean up the splinters, wouldn’t need to explicitly consider the possibility that they might forget about the splinters and step on them later. This would be incorporated into the decision-making process implicitly and automatically, by the presence of splinters making them feel uneasy until they were cleaned up. The fact that this didn’t happen suggests that the OP might also ignore other signals relevant to their well-being.
The fact that the OP seems to consider this event a virtue to highlight in the title of their post, is also a sign that they are systematically undervaluing their own well-being in a way that to me seems very worrying.
Probably most people would. But I think it’s also really important for there to be clear, public signals that the community wants people to take their well-being seriously and doesn’t endorse people hurting themselves “for the sake of the cause”.
The EA and rationalist communities are infamous for having lots of people burning themselves out through extreme self-sacrifice. If someone makes a post where they present the act of working until their feet start bleeding as a personal virtue, and there’s no public pushback to that, then that sends the implicit signal that the community endorses that reasoning. That will then contribute to unhealthy social norms that cause people to burn themselves out. The only way to counteract that is by public comments that make it clear that the community wants people to take care of themselves, even if that makes them (temporarily) less effective.
To the OP: please prioritize your well-being first. Self-preservation is one of the instrumental convergent drives; you can only continue to work if you are in good shape.
I am probably bad at valuing my well-being correctly. That said I don’t think the initial comment made me feel bad (but maybe I am bad at noticing if it would). Rather now with this entire comment stream, I realize that I have again failed to communicate.
Yes, I think this was irrational to not clean up the glass. That is the point I want to make. I don’t think it is virtuous to have failed in this way at all. What I want to say is: “Look I am running into failure modes because I want to work so much.”
Not running into these failure modes is important, but these failure modes where you are working too much are much easier to handle than the failure mode of “I can’t get myself to put in at least 50 hours of work per week consistently.”
While I do think that it is true, I am probably very bad in general at optimizing for myself to be happy. But the thing is while I was working so hard during AISC I was most of the time very happy. The same when I made these games. Most of the time I did these things because I deeply wanted to.
There where moments during AISC where I felt like I was close to burning out, but this was the minority. Mostly I was much happier than baseline. I think usually I don’t manage to work as hard and as long as I’d like, and that is a major source of unhappiness for me.
So it seems that the problem that Alex seems to see, in me working very hard (that I am failing to take my happiness into account) is actually solved by me working very hard, which is quite funny.
Ah! I completely missed that, that changes my interpretation significantly. Thank you for the clarification, now I’m less worried for you since it no longer sounds like you have a blindspot around it.
It sounds right that these failure modes are easier to handle than the failure mode of not being able to do much work.
Though working too much can lead to the failure mode of “I can’t get myself put in work consistently”. I’d be cautious in that it’s possible to feel like you really enjoy your work… and then burn out anyway! I’ve heard several people report this happening to them. The way I model that is something like… there are some parts of the person that are obsessed with the work, and become really happy about being able to completely focus on the obsession. But meanwhile, that single-minded focus can lead to the person’s other needs not being met, and eventually those unmet needs add up and cause a collapse.
I don’t know how much you need to be worried about that, but it’s at least good to be aware of.
That implication wasn’t intended. I agree that (for basic reasons) the probability of a small cut was higher given their choice.
Rather, the action itself seems rational to me when considering:
That outcome seems unprobable (at least if they were sitting down), but actual in this particular timeline.
The effects of a cut on the foot are really low (with I’d guess >99.5% probability, for an otherwise healthy person—on reflection, maybe not cumulatively low enough for the also-small payoff?), and if so ~certain to not significantly curtail progress.
That doesn’t necessarily imply the policy which produced the action is rational, though. But when considering the two hypotheses: (1) OP is mentally unwell, and (2) They have some them-specific reason[1] for following a policy which outputs actions like this, I considered (2) to be a lot more probable.
Meta: This comment is (genuinely) very hard/overwhelming-feeling for me to try to reply to, for a few reasons specific to my mind, mainly about {unmarked assumptions} and {parts seeming to be for rhetorical effect}. (For that reason I’ll let others discuss this instead of saying much further)
I agree with this, but I think any ‘community norm reinforcing messages’ should be clearly about norms rather than framed about an individual, in cases like this where there’s just a weak datapoint about the individual.
A simple example would be “Having introspected and tested different policies before determining that they’re not at risk of burnout from the policy which gives this action.”
A more complex example would be “a particular action can be irrational in isolation but downstream of a (suboptimal but human-attainable) policy which produces irrational behavior less than is typical”, which (now) seems to me to be what OP was trying to show with this example given their comment
Holden advised against this:
Also relevant are the takeaways from Thomas Kwa’s effectiveness as a conjunction of multipliers, in particular:
I think both “jog, don’t sprint” and “sprint, don’t jog” is too low-dimensional as advice. It’s good to try to spend 100% of one’s resources on doing good—sorta tautologically. What allows Johannes to work as hard as he does, I think, is not (just) that he’s obsessed with the work, it’s rather that he understands his own mind well enough to navigate around its limits. And that self-insight is also what enables him aim his cognition at what matters—which is a trait I care more about than ability to work hard.
People who are good at aiming their cognition at what matters sometimes choose to purposefwly flout[1] various social expectations in order to communicate “I see through this distracting social convention and I’m willing to break it in order to aim myself more purely at what matters”. Readers who haven’t noticed that some of their expectations are actually superfluous or misaligned with altruistic impact, will mistakenly think the flouter has low impact-potential or is just socially incompetent.
By writing the way he does, Johannes signals that he’s distancing himself from status-related putative proxies-for-effectiveness, and I think that’s a hard requirement for aiming more purely at the conjunction of multipliers[2] that matter. But his signals will be invisible to people who aren’t also highly attuned to that conjunction.
“flouting a social expectation”: choosing to disregard it while being fully aware of its existence, in a not-mean-spirited way.
I think the post uses an odd definition of “conjunction”, but it points to something important regardless. My term for this bag of nearby considerations is “costs of compromise”:
I think what quila is pointing at is their belief in the supposed fragility of thoughts at the edge of research questions. From that perspective I think their rebuttal is understandable, and your response completely misses the point: you can be someone who spends only four hours a day working and the rest of the time relaxing, but also care a lot about not losing the subtle and supposedly fragile threads of your thought when working.
Note: I have a different model of research thought, one that involves a systematic process towards insight, and because of that I also disagree with Johannes’ decisions.
How much does this line up with your model.
Quoted from the linked comment:
Yeah, this is a sentiment I agree with and believe. I think that it makes sense to have a cognitive process that self-corrects and systematically moves towards solving whatever problem it is faced with. In terms of computability theory, one could imagine it as an effectively computable function that you expect will return you the answer—and the only ‘obstacle’ is time / compute invested.
I share your sentiment, although the causal model for it is different in my head. A generalized feeling of hopelessness is an indicator of mistaken assumptions and causal models in my head, and I use that as a cue to investigate why I feel that way. This usually results in me having hopelessness about specific paths, and a general purposefulness (for I have an idea of what I want to do next), and this is downstream of updates to my causal model that attempts to track reality as best as possible.
Note that when I said I disagree with your decisions, I specifically meant the sort of myopia in the glass shard story—and specifically because I believe that if your research process / cognition algorithm is fragile enough that you’d be willing to take physical damage to hold onto an inchoate thought, maybe consider making your cognition algorithm more robust.
Edit: made it a post.
On my current models of theoretical[1] insight-making, the beginning of an insight will necessarily—afaict—be “non-robust”/chaotic. I think it looks something like this:
A gradual build-up and propagation of salience wrt some tiny discrepancy between highly confident specific beliefs
This maybe corresponds to simultaneously-salient neural ensembles whose oscillations are inharmonic[2]
Or in the frame of predictive processing: unresolved prediction-error between successive layers
Immediately followed by a resolution of that discrepancy if the insight is successfwl
This maybe corresponds to the brain having found a combination of salient ensembles—including the originally inharmonic ensembles—whose oscillations are adequately harmonic.
Super-speculative but: If the “question phase” in step 1 was salient enough, and the compression in step 2 great enough, this causes an insight-frisson[3] and a wave of pleasant sensations across your scalp, spine, and associated sensory areas.
This maps to a fragile/chaotic high-energy “question phase” during which the violation of expectation is maximized (in order to adequately propagate the implications of the original discrepancy), followed by a compressive low-energy “solution phase” where correctness of expectation is maximized again.
In order to make this work, I think the brain is specifically designed to avoid being “robust”—though here I’m using a more narrow definition of the word than I suspect you intended. Specifically, there are several homeostatic mechanisms which make the brain-state hug the border between phase-transitions as tightly as possible. In other words, the brain maximizes dynamic correlation length between neurons[4], which is when they have the greatest ability to influence each other across long distances (aka “communicate”). This is called the critical brain hypothesis, and it suggests that good thinking is necessarily chaotic in some sense.
Another point is that insight-making is anti-inductive.[5] Theoretical reasoning is a frontier that’s continuously being exploited based on the brain’s native Value-of-Information-estimator, which means that the forests with the highest naively-calculated-VoI are also less likely to have any low-hanging fruit remaining. What this implies is that novel insights are likely to be very narrow targets—which means they could be really hard to hold on to for the brief moment between initial hunch and build-up of salience. (Concise handle: epistemic frontiers are anti-inductive.)
I scope my arguments only to “theoretical processing” (i.e. purely introspective stuff like math), and I don’t think they apply to “empirical processing”.
Harmonic (red) vs inharmonic (blue) waveforms. When a waveform is harmonic, efferent neural ensembles can quickly entrain to it and stay in sync with minimal metabolic cost. Alternatively, in the context of predictive processing, we can say that “top-down predictions” quickly “learn to predict” bottom-up stimuli.
I basically think musical pleasure (and aesthetic pleasure more generally) maps to 1) the build-up of expectations, 2) the violation of those expectations, and 3) the resolution of those violated expectations. Good art has to constantly balance between breaking and affirming automatic expectations. I think the aesthetic chills associates with insights are caused by the same structure as appogiaturas—the one-period delay of an expected tone at the end of a highly predictable sequence.
I highly recommend this entire YT series!
I think the term originates from Eliezer, but Q Home has more relevant discussion on it—also I’m just a big fan of their
chaoticoptimal reasoning style in general. Can recommend! 🍵Yes, thanks for noticing and making it explicit. It seems I was modelling Johannes as having a similar cognition type, since it would explain their behavior, which actually had a different cause.
My main response to ‘try to change your cognition algorithm if it is fragile’ is to remind that human minds tend to work differently on unexpected dimensions. (Of course, you know this abstractly, and have probably read the same post about the ‘typical mind fallacy’. But the suggestion seems like harmful advice to follow for some of the minds it’s directed at.) (Alternatively, since you wrote ‘maybe’, this comment can be seen as describing a kind of case where it would be harmful)
My highest value mental states are fragile: they are hard to re-enter at will once left, and they take some subconscious effort to preserve/cultivate. They can also feel totally immersing and overwhelming, when I manage to enter them. (I don’t feel confident in my ability to qualitatively write more, as much as I would like to (maybe not here or now)).
This is analogous to Johannes’ situation in a way. They believe the problem they have of working too hard is less bad to have than the standard problem of not feeling motive to work. The specific irrational behavior their problem caused also ‘stands out’ more to onlookers, since it’s not typical. (One wouldn’t expect the top comment here if one described succumbing to akrasia; but if akrasia was rare in humans, such that the distribution over most probable causes included some worrying possibilities, we might)
In the same way, I feel like my cognition-algorithm is in a local optima which is better than the standard one, where one lesser-problem I face is that my highest output mental states are ‘fragile’, and because this is not typical it may (when read of in isolation) seem like a sign of ‘a negative deviation from the normal local optima, which this person would be better off if they corrected’.
From my inside perspective, I don’t want to try to avoid fragile mental states, because I think it would only be a possible change as a more general directional change away from ‘how my cognition works (at its best)’ towards ‘how human cognition typically works’.
(And because the fragility-of-thought feels like a small problem, once I learned to work around it, e.g learning to preserve states and augmenting with external notes. At least when compared to the problem most have of not having a chance at generating insights of a high enough quality as our situation necessitates.)
… although, if you knew of a method to reduce fragility while not reducing other things, then I’d love to try it :)
On ‘willing to take physical damage …’, footnoted because it seems like a minor point—This seems like another case of avoiding the typical-mind-fallacy being important, since different minds have different pain tolerances / levels of experienced pain from a cut.
I think you’re right that I missed their point, thanks for pointing it out.
I have had experiences similar to Johannes’ anecdote re: ignoring broken glass to not lose fragile threads of thought; they usually entailed extended deep work periods past healthy thresholds for unclear marginal gain, so the quotes above felt personally relevant as guardrails. But also my experiences don’t necessarily generalize (as your hypothetical shows).
I’d be curious to know your model, and how it compares to some of John Wentworth’s posts on the same IIRC.
I wrote a bit about it in this comment.
I think that conceptual alignment research of the sort that Johannes is doing (and that I also am doing, which I call “deconfusion”) is just really difficult. It involves skills that are not taught to people, that seems very unlikely that you’d learn by being mentored in traditional academia (including when doing theoretical CS or non-applied math PhDs), that I only started wrapping my head around after some mentorship from two MIRI researchers (that I believe I was pretty lucky to get), and even then I’ve spent a ridiculous amount of time by myself trying to tease out patterns to figure out a more systematic process of doing this.
Oh, and the more theoretical CS (and related math such as mathematical logic) you know, the better you probably are at this—see how Johannes tries to create concrete models of the inchoate concepts in his head? Well, if you know relevant theoretical CS and useful math, you don’t have to rebuild the mathematical scaffolding all by yourself.
I don’t have a good enough model of John Wentworth’s model for alignment research to understand the differences, but I don’t think I learned all that much from John’s writings and his training sessions that were a part of his MATS 4.0 training regimen, as compared to the stuff I described above.
What <mathematical scaffolding/theoretical CS> do you think I am recreating? What observations did you use to make this inference? (These questions are not intended to imply any subtext meaning.)
I didn’t intend to imply in my message that you have mathematical scaffolding that you are recreating, although I expect it may be likely (Pearlian causality perhaps? I’ve been looking into it recently and clearly knowing Bayes nets is very helpful). I specifically used “you” to imply that in general this is the case. I haven’t looked very deep into the stuff you are doing, unfortunately—it is on my to-do list.