I notice that I find Valentine’s posts somewhat insightful, and believe they point at incredibly neglected research directions, but I notice a huge the distance seems to exist between what Valentine intends to communicate and what most readers seem to get.
Off the top of my head:
Here’s the Exit is written in a style that belies an astounding confidence that what Valentine says is applicable to the reader, no matter what. After a few commenters correctly critique the post, Valentine backs down and claims that it was meant to be an “invitation” for people who recognize themselves in the post to explore the thoughts that Valentine espouses in the post, and not a set of claims to evaluate. This feels slightly like a bait-and-switch, and worse, I feel like Valentine was acting in complete good faith while doing so, with a sort of very out-of-distribution model of communication they endorse for themselves.
In We’re already in AI takeoff, Valentine seems to claim that humans should not try to intervene at the egregore-level, because we are too weak to do so. When someone points out that this may not necessarily be correct, Valentineclarifies that what they meant was more that humans should not use shoulds to try to get themselves to do something that one could be confident that they physically cannot accomplish, and that solving AI alignment or existential risk can be one example of such a thing for many people. Again, I notice how Valentine makes a ton of sense, and points [badly] at very valuable rationalist insights and concepts when requested to clarify, but the way they pointed at it in their OP was, in my opinion, disastrously bad.
The reason I wrote this down is because I think Valentine (and other people reading this) might find this helpful, and I didn’t feel like it made sense to post this as a comment in any specific individual post.
It has been six months since I wrote this, and I want to note an update: I now grok what Valentine is trying to say and what he is pointing at in Here’s the Exit and We’re already in AI takeoff. That is, I have a detailed enough model of Valentine’s model of the things he talks about, such that I understand the things he is saying.
The fundamental issue is that we are communicating in language, the medium of ideas, so it is easy to get stuck in ideas. The only way to get someone to start looking, insofar as that is possible, is to point at things using words, and to get them to do things. This is why I tell you to do things like wave your arms about or attack someone with your personal bubble or try to initiate the action of touching a hot stove element.
Most of that isn’t grounded in reality, but that fact is hard to miss because the thinker isn’t distinguishing between thoughts and reality.
Looking is just the skill of looking at reality prior to thought. It’s really not complicated. It’s just very, very easy to misunderstand if you fixate on mentally understanding it instead of doing it. Which sadly seems to be the default response to the idea of Looking.
I am unsure if this differs from mundane metacognitive skills like “notice the inchoate cognitions that arise in your mind-body, that aren’t necessarily verbal”. I assume that Valentine is pointing at a certain class of cognition, one that is essentially entirely free of interpretation. Or perhaps before ‘value-ness’ is attached to an experience—such as “this experience is good because <elaborate strategic chain>” or “this experience is bad because it hurts!”
I understand how a better metacognitive skillset would lead to the benefits Valentine mentioned, but I don’t think it requires you to only stay at the level of “direct embodied perception prior to thought”.
I may have a better answer for the concrete thing that it allows you to do: it’s fully generalizing the move of un-goodharting. Buddhism seems to be about doing this for happiness/inverse-suffering, though in principle you could pick a different navigational target (maybe).
Concretely, this should show up as being able to decondition induced reward loops and thus not be caught up in any negative compulsive behaviors.
I think that “fully generalized un-goodharting” is a pretty vague phrase and I could probably come up with a better one, but it is an acceptable pointer term for now. So I assume it is something like ‘anti-myopia’? Hard to know at this point. I’d need more experience and experimentation and thought to get a better idea of this.
I believe that Here’s the Exit, We’re already in AI Takeoff, and Slack matters more than any outcome all were pointing at the same cluster of skills and thought—about realizing the existence of psyops, systematic vulnerabilities or issues that leads you (whatever ‘you’ means) to forgetting the ‘bigger picture’, and that the resulting myopia causes significantly bad outcomes from the perspective of the ‘whole’ individual/society/whatever.
In general, Lexicogenesis seems like a really important sub-skill for deconfusion.
And yet, I get the sentiment that Valentine seems to have been trying to communicate—it sure seems like there are epistemic rationality techniques that seem incredibly valuable and neglected, and one could discover them in the course of doing something about as useless as paperwork, and talking about how you became more efficient at paperwork would seem like a waste of time to everyone involved.
Is this a real example or one that you’ve made up? That is, do you actually have cases in mind where someone discovered valuable and neglected epistemic rationality techniques in the course of doing paperwork?
I apologize for not providing a good enough example—yes, it was made up. Here’s a more accurate explanation of what causes me to believe that Valentine’s sentiment has merit:
It seems to me that a lot of epistemic confusion can be traced to almost unrelated upstream misconceptions. Examples: thinking that people must be suspended upside down below the equator, once someone understands the notion of an approximately spherical Earth; the illusion that mirrors create horizontal asymmetry but retain vertical symmetry; the notion that an AGI will automatically be moral. Similarly, it seems plausible to me that while attempting to fix one issue (similar to attempting to fix a confusion of the sort just listed), one could find themselves making almost unrelated upstream epistemic discoveries that might just be significantly more valuable). I do acknowledge that these epistemic discoveries do also seem object-level and communicable, and I do think that the sentiment that Valentine showed could make sense.
It also seems that a lot of rationality skill involves starting out with a bug one notices (“hey, I seem to be really bad at going to the gym”), and then making multiple attempts to fix the problem (ideally focusing on making an intervention as close to the ‘root’ of the issue as possible), and then discovering epistemic rationality techniques that may be applicable in many places. I agree that it seems like really bad strategy to then not try to explain why the technique is useful by giving another example where the technique is useful and results in good object-level outcomes, instead of simply talking about (given my original example) paperwork for a sentence and then spending paragraphs talking about some rationality technique in the abstract.
thinking that people must be suspended upside down below the equator, once someone understands the notion of an approximately spherical Earth
That page seems to be talking about a four-year-old child, who has not yet learned about space, how gravity works, etc. It’s not clear to me that there’s anything to conclude from this about what sorts of epistemic rationality techniques might be useful to adults.
More importantly, it’s not clear to me how any of your examples are supposed to be examples of “epistemic confusion [that] can be traced to almost unrelated upstream misconceptions”. Could you perhaps make the connection more explicitly?
Similarly, it seems plausible to me that while attempting to fix one issue (similar to attempting to fix a confusion of the sort just listed), one could find themselves making almost unrelated upstream epistemic discoveries that might just be significantly more valuable).
And… do you have any examples of this?
It also seems that a lot of rationality skill involves starting out with a bug one notices (“hey, I seem to be really bad at going to the gym”), and then making multiple attempts to fix the problem (ideally focusing on making an intervention as close to the ‘root’ of the issue as possible), and then discovering epistemic rationality techniques that may be applicable in many places.
There’s a lot of “<whatever> seems like it could be true” in your comment. Are you really basing your views on this subject on nothing more than abstract intuition?
I agree that it seems like really bad strategy to then not try to explain why the technique is useful by giving another example where the technique is useful and results in good object-level outcomes, instead of simply talking about (given my original example) paperwork for a sentence and then spending paragraphs talking about some rationality technique in the abstract.
If, hypothetically, you discovered some alleged epistemic rationality technique while doing paperwork, I would certainly want you to either explain how you applied this technique originally (with a worked example involving your paperwork), or explain how the reader might (or how you did) apply the technique to some other domain (with a worked example involving something else, not paperwork), or (even better!) both.
It would be very silly to just talk about the alleged technique, with no demonstration of its purported utility.
If, hypothetically, you discovered some alleged epistemic rationality technique while doing paperwork, I would certainly want you to either explain how you applied this technique originally (with a worked example involving your paperwork), or explain how the reader might (or how you did) apply the technique to some other domain (with a worked example involving something else, not paperwork), or (even better!) both.
This seems sensible, yes.
It would be very silly to just talk about the alleged technique, with no demonstration of its purported utility.
I agree that it seems silly to not demonstrate the utility of a technique when trying to discuss it! I try to give examples to support my reasoning when possible. What I attempted to do with that one passage that you seemed to have taken offense to was show that I could guess at one causal cognitive chain that would have led Valentine to feel the way they did and therefore act and communicate the way they did, not that I endorse the way Kensho was written—because I did not get anything out of the original post.
There’s a lot of “<whatever> seems like it could be true” in your comment.
Here’s a low investment attempt to point at the cause of what seems to you a verbal tic:
I can tell you that when I put “it seems to me” at the front of so many of my sentences, it’s not false humility, or insecurity, or a verbal tic. (It’s a deliberate reflection on the distance between what exists in reality, and the constellations I’ve sketched on my map.)
If you need me to write up a concrete elaboration to help you get a better idea about this, please tell me.
Are you really basing your views on this subject on nothing more than abstract intuition?
My intuitions on my claim related to rationality skill seem to be informed by concrete personal experience, which I haven’t yet described in length, mainly because I expected that using a simple plausible made-up example would serve as well. I apologize for not adding a “(based on experience)” in that original quote, although I guess I assumed that was deducible.
That page seems to be talking about a four-year-old child, who has not yet learned about space, how gravity works, etc. It’s not clear to me that there’s anything to conclude from this about what sorts of epistemic rationality techniques might be useful to adults.
I’m specifically pointing at examples of deconfusion here, which I consider the main (and probably the only?) strand of epistemic rationality techniques. I concede that I haven’t provided you useful information about how to do it—but that isn’t something I’d like to get into right now, when I am still wrapping my mind around deconfusion.
More importantly, it’s not clear to me how any of your examples are supposed to be examples of “epistemic confusion [that] can be traced to almost unrelated upstream misconceptions”. Could you perhaps make the connection more explicitly?
For the gravity example, the ‘upstream misconception’ is that the kid did not realize that ‘up and down’ is relative to the direction in which Earth’s gravity acts on the body, and therefore the kid tries to fit the square peg of “Okay, I see that humans have heads that point up and legs that point down” into the round hole of “Below the equator, humans are pulled upward, and humans heads are up, so humans’ heads point to the ground”.
For the AI example, the ‘upstream misconception’ can be[1] conflating the notion of intelligence with ‘human’s behavior and tendencies that I recognize as intelligence’ (and this in turn can be due to other misconceptions, such as not understanding how alien the selection process that underlies evolution is; not understanding how intelligence is not the same as saying impressive things in a social party but the ability to squeeze the probability distribution of future outcomes into a smaller space; et cetera), and then making a reasoning error that seems like anthromorphizing an AI, and concluding that the more intelligent a system would be, the more it would care about the ‘right things’ that us humans seem to care about.
The second example is a bit expensive to elaborate on, so I will not do so right now. I apologize.
Anyway, I intended to write this stuff up when I felt like I understood deconfusion enough that I could explain it to other people.
Similarly, it seems plausible to me that while attempting to fix one issue (similar to attempting to fix a confusion of the sort just listed), one could find themselves making almost unrelated upstream epistemic discoveries that might just be significantly more valuable).
And… do you have any examples of this?
I find this plausible based on my experience with deconfusion and my current state of understanding of the skill. I do not believe I understand deconfusion well enough to communicate it to people who have an inferential distance as huge as the one between you and I, so I do not intend to try.
[1]: There are a myriad of ways you can be confused, and only one way you can be deconfused.
I notice that I find Valentine’s posts somewhat insightful, and believe they point at incredibly neglected research directions, but I notice a huge the distance seems to exist between what Valentine intends to communicate and what most readers seem to get.
Off the top of my head:
Here’s the Exit is written in a style that belies an astounding confidence that what Valentine says is applicable to the reader, no matter what. After a few commenters correctly critique the post, Valentine backs down and claims that it was meant to be an “invitation” for people who recognize themselves in the post to explore the thoughts that Valentine espouses in the post, and not a set of claims to evaluate. This feels slightly like a bait-and-switch, and worse, I feel like Valentine was acting in complete good faith while doing so, with a sort of very out-of-distribution model of communication they endorse for themselves.
In We’re already in AI takeoff, Valentine seems to claim that humans should not try to intervene at the egregore-level, because we are too weak to do so. When someone points out that this may not necessarily be correct, Valentine clarifies that what they meant was more that humans should not use shoulds to try to get themselves to do something that one could be confident that they physically cannot accomplish, and that solving AI alignment or existential risk can be one example of such a thing for many people. Again, I notice how Valentine makes a ton of sense, and points [badly] at very valuable rationalist insights and concepts when requested to clarify, but the way they pointed at it in their OP was, in my opinion, disastrously bad.
The only thing I recall from Kensho is the Said baking metaphor for people not explaining why one should care about the meta-level thing by showing an object-level achievement done using the meta-level thing. And yet, I get the sentiment that Valentine seems to have been trying to communicate—it sure seems like there are epistemic rationality techniques that seem incredibly valuable and neglected, and one could discover them in the course of doing something about as useless as paperwork, and talking about how you became more efficient at paperwork would seem like a waste of time to everyone involved.
The reason I wrote this down is because I think Valentine (and other people reading this) might find this helpful, and I didn’t feel like it made sense to post this as a comment in any specific individual post.
It has been six months since I wrote this, and I want to note an update: I now grok what Valentine is trying to say and what he is pointing at in Here’s the Exit and We’re already in AI takeoff. That is, I have a detailed enough model of Valentine’s model of the things he talks about, such that I understand the things he is saying.
I still don’t feel like I understand Kensho. I get the pattern of the epistemic puzzle he is demonstrating, but I don’t know if I get the object-level thing he points at. Based on a reread of the comments, maybe what Valentine means by Looking is essentially gnosis, as opposed to doxa. An understanding grounded in your experience rather than an ungrounded one you absorbed from someone else’s claims. See this comment by someone else who is not Valentine in that post:
Alternately, Valentine describes the process of Looking as “Direct embodied perception prior to thought.”:
I am unsure if this differs from mundane metacognitive skills like “notice the inchoate cognitions that arise in your mind-body, that aren’t necessarily verbal”. I assume that Valentine is pointing at a certain class of cognition, one that is essentially entirely free of interpretation. Or perhaps before ‘value-ness’ is attached to an experience—such as “this experience is good because <elaborate strategic chain>” or “this experience is bad because it hurts!”
I understand how a better metacognitive skillset would lead to the benefits Valentine mentioned, but I don’t think it requires you to only stay at the level of “direct embodied perception prior to thought”.
As for kensho, it seems to be a term for some skill that leads you to be able to do what romeostevensit calls ‘fully generalized un-goodharting’:
I think that “fully generalized un-goodharting” is a pretty vague phrase and I could probably come up with a better one, but it is an acceptable pointer term for now. So I assume it is something like ‘anti-myopia’? Hard to know at this point. I’d need more experience and experimentation and thought to get a better idea of this.
I believe that Here’s the Exit, We’re already in AI Takeoff, and Slack matters more than any outcome all were pointing at the same cluster of skills and thought—about realizing the existence of psyops, systematic vulnerabilities or issues that leads you (whatever ‘you’ means) to forgetting the ‘bigger picture’, and that the resulting myopia causes significantly bad outcomes from the perspective of the ‘whole’ individual/society/whatever.
In general, Lexicogenesis seems like a really important sub-skill for deconfusion.
Is this a real example or one that you’ve made up? That is, do you actually have cases in mind where someone discovered valuable and neglected epistemic rationality techniques in the course of doing paperwork?
I apologize for not providing a good enough example—yes, it was made up. Here’s a more accurate explanation of what causes me to believe that Valentine’s sentiment has merit:
It seems to me that a lot of epistemic confusion can be traced to almost unrelated upstream misconceptions. Examples: thinking that people must be suspended upside down below the equator, once someone understands the notion of an approximately spherical Earth; the illusion that mirrors create horizontal asymmetry but retain vertical symmetry; the notion that an AGI will automatically be moral. Similarly, it seems plausible to me that while attempting to fix one issue (similar to attempting to fix a confusion of the sort just listed), one could find themselves making almost unrelated upstream epistemic discoveries that might just be significantly more valuable). I do acknowledge that these epistemic discoveries do also seem object-level and communicable, and I do think that the sentiment that Valentine showed could make sense.
It also seems that a lot of rationality skill involves starting out with a bug one notices (“hey, I seem to be really bad at going to the gym”), and then making multiple attempts to fix the problem (ideally focusing on making an intervention as close to the ‘root’ of the issue as possible), and then discovering epistemic rationality techniques that may be applicable in many places. I agree that it seems like really bad strategy to then not try to explain why the technique is useful by giving another example where the technique is useful and results in good object-level outcomes, instead of simply talking about (given my original example) paperwork for a sentence and then spending paragraphs talking about some rationality technique in the abstract.
That page seems to be talking about a four-year-old child, who has not yet learned about space, how gravity works, etc. It’s not clear to me that there’s anything to conclude from this about what sorts of epistemic rationality techniques might be useful to adults.
More importantly, it’s not clear to me how any of your examples are supposed to be examples of “epistemic confusion [that] can be traced to almost unrelated upstream misconceptions”. Could you perhaps make the connection more explicitly?
And… do you have any examples of this?
There’s a lot of “<whatever> seems like it could be true” in your comment. Are you really basing your views on this subject on nothing more than abstract intuition?
If, hypothetically, you discovered some alleged epistemic rationality technique while doing paperwork, I would certainly want you to either explain how you applied this technique originally (with a worked example involving your paperwork), or explain how the reader might (or how you did) apply the technique to some other domain (with a worked example involving something else, not paperwork), or (even better!) both.
It would be very silly to just talk about the alleged technique, with no demonstration of its purported utility.
This seems sensible, yes.
I agree that it seems silly to not demonstrate the utility of a technique when trying to discuss it! I try to give examples to support my reasoning when possible. What I attempted to do with that one passage that you seemed to have taken offense to was show that I could guess at one causal cognitive chain that would have led Valentine to feel the way they did and therefore act and communicate the way they did, not that I endorse the way Kensho was written—because I did not get anything out of the original post.
Here’s a low investment attempt to point at the cause of what seems to you a verbal tic:
If you need me to write up a concrete elaboration to help you get a better idea about this, please tell me.
My intuitions on my claim related to rationality skill seem to be informed by concrete personal experience, which I haven’t yet described in length, mainly because I expected that using a simple plausible made-up example would serve as well. I apologize for not adding a “(based on experience)” in that original quote, although I guess I assumed that was deducible.
I’m specifically pointing at examples of deconfusion here, which I consider the main (and probably the only?) strand of epistemic rationality techniques. I concede that I haven’t provided you useful information about how to do it—but that isn’t something I’d like to get into right now, when I am still wrapping my mind around deconfusion.
For the gravity example, the ‘upstream misconception’ is that the kid did not realize that ‘up and down’ is relative to the direction in which Earth’s gravity acts on the body, and therefore the kid tries to fit the square peg of “Okay, I see that humans have heads that point up and legs that point down” into the round hole of “Below the equator, humans are pulled upward, and humans heads are up, so humans’ heads point to the ground”.
For the AI example, the ‘upstream misconception’ can be[1] conflating the notion of intelligence with ‘human’s behavior and tendencies that I recognize as intelligence’ (and this in turn can be due to other misconceptions, such as not understanding how alien the selection process that underlies evolution is; not understanding how intelligence is not the same as saying impressive things in a social party but the ability to squeeze the probability distribution of future outcomes into a smaller space; et cetera), and then making a reasoning error that seems like anthromorphizing an AI, and concluding that the more intelligent a system would be, the more it would care about the ‘right things’ that us humans seem to care about.
The second example is a bit expensive to elaborate on, so I will not do so right now. I apologize.
Anyway, I intended to write this stuff up when I felt like I understood deconfusion enough that I could explain it to other people.
I find this plausible based on my experience with deconfusion and my current state of understanding of the skill. I do not believe I understand deconfusion well enough to communicate it to people who have an inferential distance as huge as the one between you and I, so I do not intend to try.
[1]: There are a myriad of ways you can be confused, and only one way you can be deconfused.