Disclaimer: I currently work for MIRI in a non-technical capacity, mostly surrounding low-level ops and communications (e.g. I spent much of the COVID times disinfecting mail for MIRI employees). I did not overlap with Jessica and am not speaking on behalf of MIRI.
I’m having a very hard time with the first few thousand words here, for epistemic reasons. It’s fuzzy and vague in ways that leave me feeling confused and sleight-of-handed and motte-bailey’d and 1984′d. I only have the spoons to work through the top 13-point summary at the moment.
I acknowledge here, and will re-acknowledge at the end of this comment, that there is an obvious problem with addressing only a summary; it is quite possible that much of what I have to say about the summary is resolved within the larger text.
But as Jessica notes, many people will only read the summary and it was written with those people in mind. This makes it something of a standalone document, and in my culture would mean that it’s held to a somewhat higher standard of care; there’s a difference between points that are just loosely meant to gesture at longer sections, and points which are known to be [the whole story] for a substantial chunk of the audience. Additionally, it sets the tone for the ten-thousand-or-so words to follow. I sort of fear the impact of the following ten thousand words on people who were just fine with the first three thousand; people whose epistemic immune systems did not boot up right at the start.
”Claim 0″
As a MIRI employee I was coerced into a frame where I was extremely powerful and likely to by-default cause immense damage with this power, and therefore potentially responsible for astronomical amounts of harm. I was discouraged from engaging with people who had criticisms of this frame, and had reason to fear for my life if I published some criticisms of it. Because of this and other important contributing factors, I took this frame more seriously than I ought to have and eventually developed psychotic delusions of, among other things, starting World War 3 and creating hell. Later, I discovered that others in similar situations killed themselves and that there were distributed attempts to cover up the causes of their deaths.
The passive voice throws me. “I was coerced” reads as a pretty strong statement of fact about the universe; “this simply is.” I would have liked to hear, even in a brief summary paragraph, something more like “I was coerced by A, B, and C, via methods that included X, Y, and Z, into a frame where etc,” because I do not yet know whether I should trust Jessica’s assessment of what constitutes coercion.
Ditto “I had reason to fear for my life.” From whom? To what extent? What constitutes [sufficient] reason? Is this a report of fear of being actually murdered? Is this a report of fear of one’s own risk of suicide? To some extent this might be clarified by the followup in Claim 1, but it’s not clear whether Claim 1 covers the whole of that phrase or whether it’s just part of it. In all, the whole intro feels … clickbaity? Maximum-attention-grabbing while being minimum-substantive?
“Claim 1”
Multiple people in the communities I am describing have died of suicide in the past few years. Many others have worked to conceal the circumstances of their deaths due to infohazard concerns. I am concerned that in my case as well, people will not really investigate the circumstances that made my death more likely, and will discourage others from investigating, but will continue to make strong moral judgments about the situation anyway.
The phrase “many others have worked to conceal the circumstances of their deaths due to infohazard concerns” unambiguously implies something like a conspiracy, or at least a deliberate deceptiveness on the part of five-plus (?) people. This is quite a strong claim. It seems to me that it could in fact be true. In worlds where it is true, I think that (even in the summary) it should be straightforward about who it is accusing, and of what. “Many others” should be replaced with a list of names, or at least a magnitude; is it four? Fourteen? Forty? “Worked to conceal” should tell me whether there were lies told, or whether evidence was destroyed, or whether people were bribed or threatened, or whether people simply played things close to the vest.
(In general, all the motte-and-baileys in this should have been crafted to be much less motte-and-bailey-ish, in my opinion. It seems genuinely irresponsible to leave them as vague and as fill-in-the-gaps-with-your-own-preconceptions; virtually-all-interpretations-are-nonzero-defensible as they are. Claims like these should claim things, and they should claim them clearly so that they can later be unambiguously judged true or false.)
“Claim 2”
My official job responsibilities as a researcher at MIRI included thinking seriously about hypothetical scenarios, including the possibility that someone might cause a future artificial intelligence to torture astronomical numbers of people. While we considered such a scenario unlikely, it was considered bad enough if it happened to be relevant to our decision-making framework. My psychotic break in which I imagined myself creating hell was a natural extension of this line of thought.
I don’t know what it means to be “a natural extension of this line of thought.” I don’t think it is the case that >5% of people who are aware of the concept of an AI hellscape become concerned that their actions might directly result in the creation of hell. Some dots need connecting. This is not a criticism I would ordinarily level, were it not for the things I’ve already said above; at some point in the first ten sentences, I smelled something like an attempt to ensnare my mind and my shields went up and now I’m noticing all the things, whereas if I were in a less defensive mood I might not mention this one at all.
“Claim 3”
Scott asserts that Michael Vassar thinks “regular society is infinitely corrupt and conformist and traumatizing”. This is hyperbolic (infinite corruption would leave nothing to steal) but Michael and I do believe that the problems I experienced at MIRI and CFAR were not unique or unusually severe for people in the professional-managerial class. By the law of excluded middle, the only possible alternative hypothesis is that the problems I experienced at MIRI and CFAR were unique or at least unusually severe, significantly worse than companies like Google for employees’ mental well-being.
This section encourages thinking of MIRI and CFAR as a single unit, which I argued against at length on the original post. It’s somewhat like describing “my experiences in Berkeley and Oakland” in a single breath. I do not believe that there are very many of Jessica’s experiences that can’t be clearly separated into “descended from experiences at MIRI, and the responsibility of MIRI and its culture” and “descended from experiences at CFAR, and the responsibility of CFAR and its culture.” I think that distinction is pretty important.
Separately, I feel encouraged to adopt a false dichotomy, in which either [some unspecified combination of experiences, some of which involved MIRI and others of which involved CFAR] were fairly mundane and within the range of normal workplace experiences or they were unique or severely bad. I feel like this dichotomy leaves out all sorts of nuance that’s pretty important, such as how things might be good on one day and bad on another, or interactions with one staff member might be positive while interactions with another are harmful, or experience X might have bothered person A quite a lot while having no impact on person B whatsoever. Invoking the law of the excluded middle does the same thing as using the passive voice does up above—it makes things feel floaty and absolute and unchallengeable, rather than grounded in the mundane realm of “X did Y and it resulted in Z.”
I don’t think there exists a version of Claim 3 which is not fundamentally misleading (whereas Claims 1 and 2 feel like they could have been written in a non-misleading fashion and still communicated what they were trying to communicate).
“Claim 4”
Scott asserts that Michael Vassar thinks people need to “jailbreak” themselves using psychedelics and tough conversations. Michael does not often use the word “jailbreak” but he believes that psychedelics and tough conversations can promote psychological growth. This view is rapidly becoming mainstream, validated by research performed by MAPS and at Johns Hopkins, and FDA approval for psychedelic psychotherapy is widely anticipated in the field.
“Because some instances of [the thing referred to as jailbreaking] are good and growing in popularity, criticisms of [the thing referred to as jailbreaking] are invalid or should at least be treated with suspicion”?
That’s what it seems to me that Claim 4 wants me to believe. In Claim 4′s defense, it may just be responding-in-kind to a similar kind of rounding-off in Scott’s original comment. But if so, it’s the second wrong that doesn’t make right.
It is entirely possible for Scott to have been correctly critical of a thing, and for psychedelics and tough conversations to finally be coming out from under an unfair cloud of suspicion. The whole challenge is figuring out whether X is [the bad thing it looks like] or [the good thing that bears an unfortunate resemblance to the bad thing]. The text of Claim 4 tries to make me forget this fact. It nudges me toward a bucket error in which I have to treat Michael Arc (née Vassar) and MAPS and Johns Hopkins and the FDA all the same. Either they’re all right, and Scott is wrong, or they’re all wrong, and Scott is right. This does not make it easier for me to see and think clearly around whatever-happened-here.
“Claim 5”
I was taking psychedelics before talking extensively with Michael Vassar. From the evidence available to me, including a report from a friend along the lines of “CFAR can’t legally recommend that you try [a specific psychedelic], but...”, I infer that psychedelic use was common in that social circle whether or not there was an endorsement from CFAR. I don’t regret having tried psychedelics. Devi Borg reports that Michael encouraged her to take fewer, not more, drugs; Zack Davis reports that Michael recommended psychedelics to him but he refused.
I do not see why this claim does not simply say: “I was taking psychedelics before talking extensively with Michael Vassar. I don’t regret having tried psychedelics. Devi Borg reports that Michael encouraged her to take fewer, not more, drugs; Zack Davis reports that Michael recommended psychedelics to him but he refused.”
I think it does not simply say that because it wants me to believe (without having to actually demonstrate) that psychedelic use was common in [not clearly defined “social circle” that I suppose is intended to include CFAR leadership].
If there is a defense of “Michael Vassar wasn’t pushing psychedelics,” it seems to me that it can and should be lodged separately from an accusation that “CFAR and its social circle (?) tacitly encouraged psychedelics, or at least included a lot of psychedelic use.”
“Claim 6”
Scott asserts that Michael made people including me paranoid about MIRI/CFAR and that this contributes to psychosis. Before talking with Michael, I had already had a sense that people around me were acting harmfully towards me and/or the organization’s mission. Michael and others talked with me about these problems, and I found this a relief.
Nothing wrong with this claim. Did not cause me to feel confused or mentally yanked around.
“Claim 7”
If I hadn’t noticed such harmful behavior, I would not have been fit for my nominal job. MIRI leaders were already privately encouraging me to adopt a kind of conflict theory in which many AI organizations were trying to destroy the world on <20-year timescales and could not be reasoned with about the alignment problem, such that aligned AGI projects including MIRI would have to compete with them.
I think this is the same sort of confusion as in Claim 4. All members of a category are not the same. I do not see how “MIRI leadership encouraged me to view other AI orgs through an adversarial lens” (which is a claim I’m much more suspicious of in light of the previous few hundred words, and how they seem to be trying to hypnotize me) necessarily implies “I would have been doing a bad job if I hadn’t been viewing my colleagues with suspicion and scanning their actions for potential threats to me personally and to the org as a whole.”
Obviously these two are compatible; it is absolutely possible for there to be a single mindset which was both useful for thinking about other AI orgs and also appropriate to turn on one’s own experiences within MIRI, and it’s possible for that very mindset to have been a necessary prerequisite for the type of work one was doing at MIRI.
But there are many, many more worlds in which two separate things are going on, and I do not like that Claim 7 tried to handwave me into not noticing this fact, and into thinking that two very-likely-separate things must obviously be linked (so obviously that the link does not need to be described).
“Claim 8”
MIRI’s information security policies and other forms of local information suppression thus contributed to my psychosis. I was given ridiculous statements and assignments including the claim that MIRI already knew about a working AGI design and that it would not be that hard for me to come up with a working AGI design on short notice just by thinking about it, without being given hints. The information required to judge the necessity of the information security practices was itself hidden by these practices. While psychotic, I was extremely distressed about there being a universal cover-up of things-in-general.
I do not see the “thus.”
On the first layer, it is not clear to me that the infosec policies and information suppression are being accurately described, since I am not at all sure that Jessica and I would use the word “ridiculous” to describe similar things. In my culture, I would have wanted this to say “ridiculous-to-me” rather than the more authoritative, statement-of-fact-sounding “ridiculous [period].”
On the second layer, it is not clear to me that, even if accurately described, this can be said to “contribute to psychosis” by any mechanism other than “they happened around someone who was psychotic, and thus became a part of the psychosis.” I grant that, if one is becoming psychotic, and one is in a professionally-paranoid setting, this will not help. But the claim leaves me with a tinge of “this is MIRI’s fault” and I don’t like having to swallow that tinge without knowing where it came from and whether I can trust the process that created it.
(Here I pause to reiterate two things: first, I am self-awaredly only diving into the summary, and the text likely contains much of the detail I’m seeking. But these summaries are really strong, and really want me to draw certain conclusions, and they don’t “own” their persuasive/assertive nature. They keep leaning toward “and that’s the way it was” in a way that is not demanded by the constraint of being-a-summary. They could be much less epistemically weaponized and still be short and digestible.
Second, I’m pointing at everything that bothers me in a way that might seem churlish or unreasonable, but the reason I’m doing that is because I started to feel like I was being manipulated, and as a result I switched into “don’t let yourself get manipulated” mode and that caused a lot of little things to leap out. If I were not feeling like I was being manipulated, many of these things would not reach the level of being concerning, and might not even be consciously noticeable. But once I notice a cluster of ten little manipulations, each of which was just below the radar, I raise the sensitivity of the radar.)
“Claim 9”
Scott asserts that the psychosis cluster was a “Vassar-related phenomenon”. There were many memetic and personal influences on my psychosis, a small minority of which were due to Michael Vassar (my present highly-uncertain guess is that, to the extent that assigning causality to individuals makes sense at all, Nate Soares and Eliezer Yudkowsky each individually contributed more to my psychosis than did Michael Vassar, but that structural factors were important in such a way that attributing causality to specific individuals is to some degree nonsensical). Other people (Zack Davis and Devi Borg) who have been psychotic and were talking with Michael significantly commented to say that Michael Vassar was not the main cause. One person (Eric Bruylant) cited his fixation on Michael Vassar as a precipitating factor, but clarified that he had spoken very little with Michael and most of his exposure to Michael was mediated by others who likely introduced their own ideas and agendas.
This is excellent in my culture. This is what I wish the other claims were like.
“Claim 10”
Scott asserts that Michael Vassar treats borderline psychosis as success. A text message from Michael Vassar to Zack Davis confirms that he did not treat my clinical psychosis as a success. His belief that mental states somewhat in the direction of psychosis, such as those had by family members of schizophrenics, are helpful for some forms of intellectual productivity is also shared by Scott Alexander and many academics.
A text message from Michael Arc (née Vassar) to Zack Davis provides marginal evidence that he does not explicitly claim to treat one instance of clinical psychosis as a success.
I believe that Scott’s accusation is serious, and bears the burden of proof; I don’t think that it’s Jessica’s or Michael’s job to prove the accusation false. But nevertheless, a single text message whose context is not described isn’t proof of anything. I cannot update on this, and I don’t like that this claim takes me-updating-on-this for granted, and simply asserts confirmation from what is at best weak evidence.
The sentence beginning with “His belief” is untrustworthy. What is the level of psychosis of “family members of schizophrenics”? Is that a wide range? What spots within the range are being pointed at? Would Michael and Scott agree that they are pointing at the same states, when they both assert that they are “helpful”? Would they mean similar things by “helpful”? Would they be pointing at the same “forms of intellectual productivity”? What “many academics”?
It’s not that I expect a brief summary to answer all these questions. It’s more that, if you can’t say something more clear and less confusing (/outright misleading) than something like this, then I think you should not include any such sentence at all.
A far better sentence (in my culture) would be something like “To the best of my own ability to understand the positions of both Michael and Scott, they have similar beliefs about which points on the sliding scale between [normal] and [psychotic] are useful, and for what reasons, and furthermore I think that their shared view is reasonably typical of many academics.”
“Claim 11”
Scott asserts that Michael Vassar discourages people from seeking mental health treatment. Some mutual friends tried treating me at home for a week as I was losing sleep and becoming increasingly mentally disorganized before deciding to send me to a psychiatric institution, which was a reasonable decision in retrospect.
These two sentences bear no relation. I think I am intended to be hypnotized into thinking that the second sentence provides circumstantial evidence against the first.
“Claim 12”
Scott asserts that most local psychosis cases were “involved with the Vassarites or Zizians”. At least two former MIRI employees who were not significantly talking with Vassar or Ziz experienced psychosis in the past few years. Also, most or all of the people involved were talking significantly with others such as Anna Salamon (and read and highly regarded Eliezer Yudkowsky’s extensive writing about how to structure one’s mind, and read Scott Alexander’s fiction writing about hell). There are about equally plausible mechanisms by which each of these were likely to contribute to psychosis, so this doesn’t single out Michael Vassar or Ziz.
Again I have the sense that Jessica may be responding to sloppiness with sloppiness, which at least provides the justification of “I didn’t break the peace treaty first, here.” But: it is not the case that all psychoses should be lumped together. It is not the case that two people not-in-contact with Vassar or Ziz falsifies the claim that “most” of the cases were involved with the Vassarites or Zizians. It is not the case that Eliezer and Scott’s writings should be treated as being similar to extensive and intense in-person interactions (though the mention of Anna Salamon does seem like relevant and useful info (though again I would much prefer more of the specifically who and specifically what)).
But nothing after the first sentence actually contradicts the first sentence, and I really really really do not like how it tries to make me think that it did.
“Claim 13”
Scott Alexander asserts that MIRI should have discouraged me from talking about “auras” and “demons” and that such talk should be treated as a “psychiatric emergency”. This increases the chance that someone like me could be psychiatrically incarcerated for talking about things that a substantial percentage of the general public (e.g. New Age people and Christians) talk about, and which could be explained in terms that don’t use magical concepts. This is inappropriately enforcing the norms of a minority ideological community as if they were widely accepted professional standards.
Such talk as a sudden departure from one’s previous norms of speech and thought seems to me to be quite likely to be a psychiatric emergency, in actual fact.
I do not believe that our psychiatric systems are anywhere near perfect; I know of at least two cases of involuntary incarceration that seem to me to have been outrageously unjustified and some fear here seems reasonable. But nevertheless, it does not seem likely to me that someone who can easily explain “What? Oh, sorry—that’s shorthand. The actual non-crazy concept that the shorthand is for is not hard to explain, let me give you the five sentence version—” is at substantial risk for being thrown into a mental institution.
I’m not sure why this section wants me to be really very confident that [the proposed intervention] would not have helped prevent a slide into psychosis which Jessica seems to be largely laying at MIRI’s feet.
I spent something like an hour on this. I don’t know if it is helpful to anyone besides myself, but it is helpful to me. I feel much better equipped to navigate the remainder of this piece without “falling under its spell,” so to speak. I know to be on the lookout for:
Summaries which may or may not be apt descriptions of the more detailed thing they are trying to summarize (e.g. assertions that something was “ridiculous” or “extensive”)
Language which is more authoritative, universal, or emphatic than it should be
Attempts to link things via mere juxtaposition, without spelling out the connection between them
Umbrella statements that admit of a wide range of interpretations and possibly equivocate between them.
etc.
The absence of those things, and similar, makes it easier to see and think clearly, at least for me.
Their presence makes it harder to see and think clearly, at least for me.
Where it’s important to see and think clearly, I think it’s extra important to care about that.
Acknowledging one last time: I only dealt with the top-level summary; this is somewhat uncharitable and incomplete and a better version of me would have been able to make it through the whole thing, first. Responding to that top-level summary was the best I could manage with the resources I had at my disposal.
EDIT: I have as of this edit now read half of the larger text, and no, these issues are largely not resolved and are in many places exacerbated. Reading this without “shields up” would cause a person to become quite seriously misled/confused, such as (for a single representative example) when Jessica begins with:
a specific individual conveying to her a rumor that members of the precursor org to MIRI (prior to its split into MIRI and CFAR) had years earlier (seriously? jokingly?) discussed assassinating AGI researchers
… and a paragraph later casually refers to these discussions as if their existence is an absolute fact, saying “The obvious alternative hypothesis is that MIRI is not for real, and therefore hypothetical discussions about assassinations were just dramatic posturing.” And furthermore seems to want the reader to just … nod along? … with this being the sort of thing that would reasonably cause a person to fear for their own life, or at least contribute substantially to the development of such a fear/meaningfully justify such a fear.
I want to endorse this as a clear and concise elucidation of the concerns I laid out in my comment, which are primarily with the mismatch between what the text seems to want me to believe, vs. what conclusions are actually valid given the available information.
The passive voice throws me. “I was coerced” reads as a pretty strong statement of fact about the universe; “this simply is.” I would have liked to hear, even in a brief summary paragraph, something more like “I was coerced by A, B, and C, via methods that included X, Y, and Z, into a frame where etc.”
It seems to me that you can’t expect a summary to make the claims as detailed as possible. You don’t criticize scientific papers either because their abstract doesn’t fully prove the claims it makes, that’s for what you have the full article.
It seems to me that you can’t expect a summary to make the claims as detailed as possible.
Just noting that this was explicitly addressed in a few places in my comment, and I believe I correctly compensated for it/took this truth into account. “Make the claims in the summary as detailed as possible” is not what I was recommending.
If you didn’t read the post and are complaining that the short summary didn’t contain the details that the full post contained, then… I don’t know how to respond. It’s equivalent to complaining that the intro paragraph of an essay doesn’t prove each sentence it states.
With respect to the criticism of the post body:
a specific individual conveying to her a rumor that members of the precursor org to MIRI (prior to its split into MIRI and CFAR) had (seriously or unseriously) discussed assassinating AGI researchers
… and a paragraph later casually refers to these discussions as if their existence is an absolute fact, saying “The obvious alternative hypothesis is that MIRI is not for real, and therefore hypothetical discussions about assassinations were just dramatic posturing.”
Yes, “this person was wrong/lying so the rumor was wrong” is an alternative, but I assigned low probability to it (in part due to a subsequent conversation with a MIRI person about this rumor), so it wasn’t the most obvious alternative.
If you didn’t read the post and are complaining that the short summary didn’t contain the details that the full post contained, then
That is very explicitly a strawman of what I am objecting to. As in: that interpretation is explicitly ruled out, multiple times within my comment, including right up at the very top, and so you reaching for it lands with me as deliberately disingenuous.
What I am objecting to is lots and lots and lots of statements that are crafted to confuse/mislead (if not straightforwardly deceive).
Okay, I can respond to the specific intro paragraph talking about this.
But as Jessica notes, many people will only read the summary and it was written with those people in mind. This makes it something of a standalone document, and in my culture would mean that it’s held to a somewhat higher standard of care; there’s a difference between points that are just loosely meant to gesture at longer sections, and points which are known to be [the whole story] for a substantial chunk of the audience. Additionally, it sets the tone for the ten-thousand-or-so words to follow. I sort of fear the impact of the following ten thousand words on people who were just fine with the first three thousand; people whose epistemic immune systems did not boot up right at the start.
I don’t expect people who only read the summary to automatically believe what I’m saying with high confidence. I expect them to believe they have an idea of what I am saying. Once they have this idea, they can decide to investigate or not investigate why I believe these things. If they don’t, they can’t know whether these things are true.
Maybe it messes with people’s immune systems by being misleading… but how could you tell the summary is misleading without reading most of the post? Seems like a circular argument.
It’s not a circular argument. The summary is misleading in its very structure/nature, as I have detailed above at great length. It’s misleading independent of the rest of the post.
Upon going further and reading the rest of the post, I confirmed that the problems evinced by the summary, which I stated up-front might have been addressed within the longer piece (so as not to mislead or confuse any readers of my comment), in fact only get worse.
This is not a piece which visibly tries to, or succeeds at, helping people see and think more clearly. It does the exact opposite, in service of ???
I would be tempted to label this a psy-op, if I thought its confusing and manipulative nature was intentional rather than just something you didn’t actively try not to do.
The passive voice throws me. “I was coerced” reads as a pretty strong statement of fact about the universe; “this simply is.” I would have liked to hear, even in a brief summary paragraph, something more like “I was coerced by A, B, and C, via methods that included X, Y, and Z, into a frame where etc.”
The rest of the paragraph says some parts of how I was coerced, e.g. I was discouraged from engaging with critics of the frame and from publishing my own criticisms.
Ditto “I had reason to fear for my life.” From whom? To what extent? What constitutes [sufficient] reason? Is this a report of fear of being actually murdered? Is this a report of fear of one’s own risk of suicide? To some extent this might be clarified by the followup in Claim 1, but it’s not clear whether Claim 1 covers the whole of that phrase or whether it’s just part of it.
If you keep reading you see that I heard about the possibility of assassination. The suicides are also worrying, although the causality on those is unclear.
Maybe this isn’t a particularly strong argument you gave for the summary being misleading. If so I’d want to know which you think are particularly strong so I don’t have to refute a bunch of weak arguments.
“I’d want to know which arguments you think are particularly strong so I don’t have to refute a bunch of weak ones” is my feeling, here, too.
Would’ve been nice if you’d just stated your claims instead of burying them in 13000 words of meandering, often misleading, not-at-all-upfront-about-epistemic-status insinuation. I’m frustrated because your previous post received exactly this kind of criticism, and that criticism was highly upvoted, and you do not seem to have felt it was worth adjusting your style.
EDIT: A relevant term here is “gish gallop.”
What I am able to gather from the OP is that you believe lots of bad rumors when you hear them, use that already-negative lens to adversarially interpret all subsequent information, get real anxious about it, and … think everyone should know this?
This is a double bind. If I state the claims in the summary I’m being misleading by not providing details or evidence for them close to the claims themselves. If I don’t then I’m doing a “gish gallop” by embedding claims in the body of the post. The post as a whole has lots of numbered lists that make most of the claims I’m making pretty clear.
It’s not a double bind, and my foremost hypothesis is now that you are deliberately strawmanning, so as to avoid addressing my real point.
Not only did I highlight two separate entries in your list of thirteen that do the thing properly, I also provided some example partial rewrites of other entries, some of which made them shorter rather than longer.
The point is not that you need to include more and more detail, and it’s disingenuous to pretend that’s what I’m saying. It’s that you need to be less deceptive and misleading. Say what you think you know, clearly and unambiguously, and say why you think you know it, directly and explicitly, instead of flooding the channel with passive voice and confident summaries that obscure the thick layer of interpretation atop the actual observable facts.
[After writing this comment, I realized that maybe I’m just missing what’s happening altogether, since maybe I read the post in a fairly strongly “sandboxed” way, so I’m failing to empathize with the mental yanks. That said, maybe it has some value.]
FWIW, my sense (not particularly well-founded?) isn’t that jessicata is deliberately strawmanning here, but isn’t getting your point or doesn’t agree.
You write above:
It’s more that, if you can’t say something more clear and less confusing (/outright misleading) than something like this, then I think you should not include any such sentence at all.
This is sort of mixing multiple things together: there’s the clarity/confusingness, and then there’s the slant/misleadingness. These are related, in that one can mislead more easily when one is being unclear/ambiguous.
You write:
Say what you think you know, clearly and unambiguously, and say why you think you know it, directly and explicitly, instead of flooding the channel with passive voice and confident summaries that obscure the thick layer of interpretation atop the actual observable facts.
Some of your original criticisms read, to me, more like asking a bunch of questions about details (which is a reasonable thing to do; some questions are answered in the post, some not), and then saying that the summary claims are bad for not including those details and instead summarizing them at a higher level. (Other of your criticisms are more directly about misleadingness IMO, e.g. the false dichotomy thing.)
E.g. you write:
The phrase “many others have worked to conceal the circumstances of their deaths due to infohazard concerns” unambiguously implies something like a conspiracy, or at least a deliberate deceptiveness on the part of five-plus (?) people. This is quite a strong claim. It seems to me that it could in fact be true. In worlds where it is true, I think that (even in the summary) it should be straightforward about who it is accusing, and of what. “Many others” should be replaced with a list of names, or at least a magnitude; is it four? Fourteen? Forty? “Worked to conceal” should tell me whether there were lies told, or whether evidence was destroyed, or whether people were bribed or threatened, or whether people simply played things close to the vest.
The phrase “many others have worked to conceal the circumstances of their deaths due to infohazard concerns” seems to me (and I initially read it as) making a straightforward factual claim: many people took actions that prevented the circumstances of their deaths from being known; and they did so because they were worried about something about infohazards. That sentence definitely doesn’t unambiguously imply a conspiracy, since it doesn’t say anything about conspiracy (though the alleged fact it expresses is of course relevant to conspiracy hypotheses). You’re then saying that there was insufficient detail to support this supposed implication. To me it just looks like a reasonable way of compressing a longer list of facts.
I don’t think I’m getting where you’re coming from here. From this:
But the claim leaves me with a tinge of “this is MIRI’s fault” and I don’t like having to swallow that tinge without knowing where it came from and whether I can trust the process that created it.
it sounds like you’re interpreting jessicata’s statements as being aimed at getting you to make a judgement, and also it was sort of working, or it would/could have been sort of working if you weren’t being vigilant. Like, something about reading that part of the post, led to you having a sense that “this is MIRI’s fault”, and then you noticed that on reflection the sense seemed incorrect. Is that right?
Disclaimer: I currently work for MIRI in a non-technical capacity, mostly surrounding low-level ops and communications (e.g. I spent much of the COVID times disinfecting mail for MIRI employees). I did not overlap with Jessica and am not speaking on behalf of MIRI.
I’m having a very hard time with the first few thousand words here, for epistemic reasons. It’s fuzzy and vague in ways that leave me feeling confused and sleight-of-handed and motte-bailey’d and 1984′d. I only have the spoons to work through the top 13-point summary at the moment.
I acknowledge here, and will re-acknowledge at the end of this comment, that there is an obvious problem with addressing only a summary; it is quite possible that much of what I have to say about the summary is resolved within the larger text.
But as Jessica notes, many people will only read the summary and it was written with those people in mind. This makes it something of a standalone document, and in my culture would mean that it’s held to a somewhat higher standard of care; there’s a difference between points that are just loosely meant to gesture at longer sections, and points which are known to be [the whole story] for a substantial chunk of the audience. Additionally, it sets the tone for the ten-thousand-or-so words to follow. I sort of fear the impact of the following ten thousand words on people who were just fine with the first three thousand; people whose epistemic immune systems did not boot up right at the start.
”Claim 0″
The passive voice throws me. “I was coerced” reads as a pretty strong statement of fact about the universe; “this simply is.” I would have liked to hear, even in a brief summary paragraph, something more like “I was coerced by A, B, and C, via methods that included X, Y, and Z, into a frame where etc,” because I do not yet know whether I should trust Jessica’s assessment of what constitutes coercion.
Ditto “I had reason to fear for my life.” From whom? To what extent? What constitutes [sufficient] reason? Is this a report of fear of being actually murdered? Is this a report of fear of one’s own risk of suicide? To some extent this might be clarified by the followup in Claim 1, but it’s not clear whether Claim 1 covers the whole of that phrase or whether it’s just part of it. In all, the whole intro feels … clickbaity? Maximum-attention-grabbing while being minimum-substantive?
“Claim 1”
The phrase “many others have worked to conceal the circumstances of their deaths due to infohazard concerns” unambiguously implies something like a conspiracy, or at least a deliberate deceptiveness on the part of five-plus (?) people. This is quite a strong claim. It seems to me that it could in fact be true. In worlds where it is true, I think that (even in the summary) it should be straightforward about who it is accusing, and of what. “Many others” should be replaced with a list of names, or at least a magnitude; is it four? Fourteen? Forty? “Worked to conceal” should tell me whether there were lies told, or whether evidence was destroyed, or whether people were bribed or threatened, or whether people simply played things close to the vest.
(In general, all the motte-and-baileys in this should have been crafted to be much less motte-and-bailey-ish, in my opinion. It seems genuinely irresponsible to leave them as vague and as fill-in-the-gaps-with-your-own-preconceptions; virtually-all-interpretations-are-nonzero-defensible as they are. Claims like these should claim things, and they should claim them clearly so that they can later be unambiguously judged true or false.)
“Claim 2”
I don’t know what it means to be “a natural extension of this line of thought.” I don’t think it is the case that >5% of people who are aware of the concept of an AI hellscape become concerned that their actions might directly result in the creation of hell. Some dots need connecting. This is not a criticism I would ordinarily level, were it not for the things I’ve already said above; at some point in the first ten sentences, I smelled something like an attempt to ensnare my mind and my shields went up and now I’m noticing all the things, whereas if I were in a less defensive mood I might not mention this one at all.
“Claim 3”
This section encourages thinking of MIRI and CFAR as a single unit, which I argued against at length on the original post. It’s somewhat like describing “my experiences in Berkeley and Oakland” in a single breath. I do not believe that there are very many of Jessica’s experiences that can’t be clearly separated into “descended from experiences at MIRI, and the responsibility of MIRI and its culture” and “descended from experiences at CFAR, and the responsibility of CFAR and its culture.” I think that distinction is pretty important.
Separately, I feel encouraged to adopt a false dichotomy, in which either [some unspecified combination of experiences, some of which involved MIRI and others of which involved CFAR] were fairly mundane and within the range of normal workplace experiences or they were unique or severely bad. I feel like this dichotomy leaves out all sorts of nuance that’s pretty important, such as how things might be good on one day and bad on another, or interactions with one staff member might be positive while interactions with another are harmful, or experience X might have bothered person A quite a lot while having no impact on person B whatsoever. Invoking the law of the excluded middle does the same thing as using the passive voice does up above—it makes things feel floaty and absolute and unchallengeable, rather than grounded in the mundane realm of “X did Y and it resulted in Z.”
I don’t think there exists a version of Claim 3 which is not fundamentally misleading (whereas Claims 1 and 2 feel like they could have been written in a non-misleading fashion and still communicated what they were trying to communicate).
“Claim 4”
“Because some instances of [the thing referred to as jailbreaking] are good and growing in popularity, criticisms of [the thing referred to as jailbreaking] are invalid or should at least be treated with suspicion”?
That’s what it seems to me that Claim 4 wants me to believe. In Claim 4′s defense, it may just be responding-in-kind to a similar kind of rounding-off in Scott’s original comment. But if so, it’s the second wrong that doesn’t make right.
It is entirely possible for Scott to have been correctly critical of a thing, and for psychedelics and tough conversations to finally be coming out from under an unfair cloud of suspicion. The whole challenge is figuring out whether X is [the bad thing it looks like] or [the good thing that bears an unfortunate resemblance to the bad thing]. The text of Claim 4 tries to make me forget this fact. It nudges me toward a bucket error in which I have to treat Michael Arc (née Vassar) and MAPS and Johns Hopkins and the FDA all the same. Either they’re all right, and Scott is wrong, or they’re all wrong, and Scott is right. This does not make it easier for me to see and think clearly around whatever-happened-here.
“Claim 5”
I do not see why this claim does not simply say: “I was taking psychedelics before talking extensively with Michael Vassar. I don’t regret having tried psychedelics. Devi Borg reports that Michael encouraged her to take fewer, not more, drugs; Zack Davis reports that Michael recommended psychedelics to him but he refused.”
I think it does not simply say that because it wants me to believe (without having to actually demonstrate) that psychedelic use was common in [not clearly defined “social circle” that I suppose is intended to include CFAR leadership].
If there is a defense of “Michael Vassar wasn’t pushing psychedelics,” it seems to me that it can and should be lodged separately from an accusation that “CFAR and its social circle (?) tacitly encouraged psychedelics, or at least included a lot of psychedelic use.”
“Claim 6”
Nothing wrong with this claim. Did not cause me to feel confused or mentally yanked around.
“Claim 7”
I think this is the same sort of confusion as in Claim 4. All members of a category are not the same. I do not see how “MIRI leadership encouraged me to view other AI orgs through an adversarial lens” (which is a claim I’m much more suspicious of in light of the previous few hundred words, and how they seem to be trying to hypnotize me) necessarily implies “I would have been doing a bad job if I hadn’t been viewing my colleagues with suspicion and scanning their actions for potential threats to me personally and to the org as a whole.”
Obviously these two are compatible; it is absolutely possible for there to be a single mindset which was both useful for thinking about other AI orgs and also appropriate to turn on one’s own experiences within MIRI, and it’s possible for that very mindset to have been a necessary prerequisite for the type of work one was doing at MIRI.
But there are many, many more worlds in which two separate things are going on, and I do not like that Claim 7 tried to handwave me into not noticing this fact, and into thinking that two very-likely-separate things must obviously be linked (so obviously that the link does not need to be described).
“Claim 8”
I do not see the “thus.”
On the first layer, it is not clear to me that the infosec policies and information suppression are being accurately described, since I am not at all sure that Jessica and I would use the word “ridiculous” to describe similar things. In my culture, I would have wanted this to say “ridiculous-to-me” rather than the more authoritative, statement-of-fact-sounding “ridiculous [period].”
On the second layer, it is not clear to me that, even if accurately described, this can be said to “contribute to psychosis” by any mechanism other than “they happened around someone who was psychotic, and thus became a part of the psychosis.” I grant that, if one is becoming psychotic, and one is in a professionally-paranoid setting, this will not help. But the claim leaves me with a tinge of “this is MIRI’s fault” and I don’t like having to swallow that tinge without knowing where it came from and whether I can trust the process that created it.
(Here I pause to reiterate two things: first, I am self-awaredly only diving into the summary, and the text likely contains much of the detail I’m seeking. But these summaries are really strong, and really want me to draw certain conclusions, and they don’t “own” their persuasive/assertive nature. They keep leaning toward “and that’s the way it was” in a way that is not demanded by the constraint of being-a-summary. They could be much less epistemically weaponized and still be short and digestible.
Second, I’m pointing at everything that bothers me in a way that might seem churlish or unreasonable, but the reason I’m doing that is because I started to feel like I was being manipulated, and as a result I switched into “don’t let yourself get manipulated” mode and that caused a lot of little things to leap out. If I were not feeling like I was being manipulated, many of these things would not reach the level of being concerning, and might not even be consciously noticeable. But once I notice a cluster of ten little manipulations, each of which was just below the radar, I raise the sensitivity of the radar.)
“Claim 9”
This is excellent in my culture. This is what I wish the other claims were like.
“Claim 10”
A text message from Michael Arc (née Vassar) to Zack Davis provides marginal evidence that he does not explicitly claim to treat one instance of clinical psychosis as a success.
I believe that Scott’s accusation is serious, and bears the burden of proof; I don’t think that it’s Jessica’s or Michael’s job to prove the accusation false. But nevertheless, a single text message whose context is not described isn’t proof of anything. I cannot update on this, and I don’t like that this claim takes me-updating-on-this for granted, and simply asserts confirmation from what is at best weak evidence.
The sentence beginning with “His belief” is untrustworthy. What is the level of psychosis of “family members of schizophrenics”? Is that a wide range? What spots within the range are being pointed at? Would Michael and Scott agree that they are pointing at the same states, when they both assert that they are “helpful”? Would they mean similar things by “helpful”? Would they be pointing at the same “forms of intellectual productivity”? What “many academics”?
It’s not that I expect a brief summary to answer all these questions. It’s more that, if you can’t say something more clear and less confusing (/outright misleading) than something like this, then I think you should not include any such sentence at all.
A far better sentence (in my culture) would be something like “To the best of my own ability to understand the positions of both Michael and Scott, they have similar beliefs about which points on the sliding scale between [normal] and [psychotic] are useful, and for what reasons, and furthermore I think that their shared view is reasonably typical of many academics.”
“Claim 11”
These two sentences bear no relation. I think I am intended to be hypnotized into thinking that the second sentence provides circumstantial evidence against the first.
“Claim 12”
Again I have the sense that Jessica may be responding to sloppiness with sloppiness, which at least provides the justification of “I didn’t break the peace treaty first, here.” But: it is not the case that all psychoses should be lumped together. It is not the case that two people not-in-contact with Vassar or Ziz falsifies the claim that “most” of the cases were involved with the Vassarites or Zizians. It is not the case that Eliezer and Scott’s writings should be treated as being similar to extensive and intense in-person interactions (though the mention of Anna Salamon does seem like relevant and useful info (though again I would much prefer more of the specifically who and specifically what)).
But nothing after the first sentence actually contradicts the first sentence, and I really really really do not like how it tries to make me think that it did.
“Claim 13”
Such talk as a sudden departure from one’s previous norms of speech and thought seems to me to be quite likely to be a psychiatric emergency, in actual fact.
I do not believe that our psychiatric systems are anywhere near perfect; I know of at least two cases of involuntary incarceration that seem to me to have been outrageously unjustified and some fear here seems reasonable. But nevertheless, it does not seem likely to me that someone who can easily explain “What? Oh, sorry—that’s shorthand. The actual non-crazy concept that the shorthand is for is not hard to explain, let me give you the five sentence version—” is at substantial risk for being thrown into a mental institution.
I’m not sure why this section wants me to be really very confident that [the proposed intervention] would not have helped prevent a slide into psychosis which Jessica seems to be largely laying at MIRI’s feet.
I spent something like an hour on this. I don’t know if it is helpful to anyone besides myself, but it is helpful to me. I feel much better equipped to navigate the remainder of this piece without “falling under its spell,” so to speak. I know to be on the lookout for:
Summaries which may or may not be apt descriptions of the more detailed thing they are trying to summarize (e.g. assertions that something was “ridiculous” or “extensive”)
Language which is more authoritative, universal, or emphatic than it should be
Attempts to link things via mere juxtaposition, without spelling out the connection between them
Umbrella statements that admit of a wide range of interpretations and possibly equivocate between them.
etc.
The absence of those things, and similar, makes it easier to see and think clearly, at least for me.
Their presence makes it harder to see and think clearly, at least for me.
Where it’s important to see and think clearly, I think it’s extra important to care about that.
Acknowledging one last time: I only dealt with the top-level summary; this is somewhat uncharitable and incomplete and a better version of me would have been able to make it through the whole thing, first. Responding to that top-level summary was the best I could manage with the resources I had at my disposal.
EDIT: I have as of this edit now read half of the larger text, and no, these issues are largely not resolved and are in many places exacerbated. Reading this without “shields up” would cause a person to become quite seriously misled/confused, such as (for a single representative example) when Jessica begins with:
a specific individual conveying to her a rumor that members of the precursor org to MIRI (prior to its split into MIRI and CFAR) had years earlier (seriously? jokingly?) discussed assassinating AGI researchers
… and a paragraph later casually refers to these discussions as if their existence is an absolute fact, saying “The obvious alternative hypothesis is that MIRI is not for real, and therefore hypothetical discussions about assassinations were just dramatic posturing.” And furthermore seems to want the reader to just … nod along? … with this being the sort of thing that would reasonably cause a person to fear for their own life, or at least contribute substantially to the development of such a fear/meaningfully justify such a fear.
I have strong downvoted this post.
I want to endorse this as a clear and concise elucidation of the concerns I laid out in my comment, which are primarily with the mismatch between what the text seems to want me to believe, vs. what conclusions are actually valid given the available information.
It seems to me that you can’t expect a summary to make the claims as detailed as possible. You don’t criticize scientific papers either because their abstract doesn’t fully prove the claims it makes, that’s for what you have the full article.
Just noting that this was explicitly addressed in a few places in my comment, and I believe I correctly compensated for it/took this truth into account. “Make the claims in the summary as detailed as possible” is not what I was recommending.
If you didn’t read the post and are complaining that the short summary didn’t contain the details that the full post contained, then… I don’t know how to respond. It’s equivalent to complaining that the intro paragraph of an essay doesn’t prove each sentence it states.
With respect to the criticism of the post body:
Yes, “this person was wrong/lying so the rumor was wrong” is an alternative, but I assigned low probability to it (in part due to a subsequent conversation with a MIRI person about this rumor), so it wasn’t the most obvious alternative.
That is very explicitly a strawman of what I am objecting to. As in: that interpretation is explicitly ruled out, multiple times within my comment, including right up at the very top, and so you reaching for it lands with me as deliberately disingenuous.
What I am objecting to is lots and lots and lots of statements that are crafted to confuse/mislead (if not straightforwardly deceive).
Okay, I can respond to the specific intro paragraph talking about this.
I don’t expect people who only read the summary to automatically believe what I’m saying with high confidence. I expect them to believe they have an idea of what I am saying. Once they have this idea, they can decide to investigate or not investigate why I believe these things. If they don’t, they can’t know whether these things are true.
Maybe it messes with people’s immune systems by being misleading… but how could you tell the summary is misleading without reading most of the post? Seems like a circular argument.
It’s not a circular argument. The summary is misleading in its very structure/nature, as I have detailed above at great length. It’s misleading independent of the rest of the post.
Upon going further and reading the rest of the post, I confirmed that the problems evinced by the summary, which I stated up-front might have been addressed within the longer piece (so as not to mislead or confuse any readers of my comment), in fact only get worse.
This is not a piece which visibly tries to, or succeeds at, helping people see and think more clearly. It does the exact opposite, in service of ???
I would be tempted to label this a psy-op, if I thought its confusing and manipulative nature was intentional rather than just something you didn’t actively try not to do.
Here’s an example (Claim 0):
The rest of the paragraph says some parts of how I was coerced, e.g. I was discouraged from engaging with critics of the frame and from publishing my own criticisms.
If you keep reading you see that I heard about the possibility of assassination. The suicides are also worrying, although the causality on those is unclear.
Maybe this isn’t a particularly strong argument you gave for the summary being misleading. If so I’d want to know which you think are particularly strong so I don’t have to refute a bunch of weak arguments.
“I’d want to know which arguments you think are particularly strong so I don’t have to refute a bunch of weak ones” is my feeling, here, too.
Would’ve been nice if you’d just stated your claims instead of burying them in 13000 words of meandering, often misleading, not-at-all-upfront-about-epistemic-status insinuation. I’m frustrated because your previous post received exactly this kind of criticism, and that criticism was highly upvoted, and you do not seem to have felt it was worth adjusting your style.
EDIT: A relevant term here is “gish gallop.”
What I am able to gather from the OP is that you believe lots of bad rumors when you hear them, use that already-negative lens to adversarially interpret all subsequent information, get real anxious about it, and … think everyone should know this?
This is a double bind. If I state the claims in the summary I’m being misleading by not providing details or evidence for them close to the claims themselves. If I don’t then I’m doing a “gish gallop” by embedding claims in the body of the post. The post as a whole has lots of numbered lists that make most of the claims I’m making pretty clear.
It’s not a double bind, and my foremost hypothesis is now that you are deliberately strawmanning, so as to avoid addressing my real point.
Not only did I highlight two separate entries in your list of thirteen that do the thing properly, I also provided some example partial rewrites of other entries, some of which made them shorter rather than longer.
The point is not that you need to include more and more detail, and it’s disingenuous to pretend that’s what I’m saying. It’s that you need to be less deceptive and misleading. Say what you think you know, clearly and unambiguously, and say why you think you know it, directly and explicitly, instead of flooding the channel with passive voice and confident summaries that obscure the thick layer of interpretation atop the actual observable facts.
[After writing this comment, I realized that maybe I’m just missing what’s happening altogether, since maybe I read the post in a fairly strongly “sandboxed” way, so I’m failing to empathize with the mental yanks. That said, maybe it has some value.]
FWIW, my sense (not particularly well-founded?) isn’t that jessicata is deliberately strawmanning here, but isn’t getting your point or doesn’t agree.
You write above:
This is sort of mixing multiple things together: there’s the clarity/confusingness, and then there’s the slant/misleadingness. These are related, in that one can mislead more easily when one is being unclear/ambiguous.
You write:
Some of your original criticisms read, to me, more like asking a bunch of questions about details (which is a reasonable thing to do; some questions are answered in the post, some not), and then saying that the summary claims are bad for not including those details and instead summarizing them at a higher level. (Other of your criticisms are more directly about misleadingness IMO, e.g. the false dichotomy thing.)
E.g. you write:
The phrase “many others have worked to conceal the circumstances of their deaths due to infohazard concerns” seems to me (and I initially read it as) making a straightforward factual claim: many people took actions that prevented the circumstances of their deaths from being known; and they did so because they were worried about something about infohazards. That sentence definitely doesn’t unambiguously imply a conspiracy, since it doesn’t say anything about conspiracy (though the alleged fact it expresses is of course relevant to conspiracy hypotheses). You’re then saying that there was insufficient detail to support this supposed implication. To me it just looks like a reasonable way of compressing a longer list of facts.
I don’t think I’m getting where you’re coming from here. From this:
it sounds like you’re interpreting jessicata’s statements as being aimed at getting you to make a judgement, and also it was sort of working, or it would/could have been sort of working if you weren’t being vigilant. Like, something about reading that part of the post, led to you having a sense that “this is MIRI’s fault”, and then you noticed that on reflection the sense seemed incorrect. Is that right?