Life is not graded on a curve and you can update yourself incrementally? I also hold all of those people in high regard—relative to the rest of the world. (And Anna is a personal friend.) I think all of those people hold me in high regard, relative to the rest of the world. Nevertheless, it seems like there ought to be a time and a place to talk about people having been culpably wrong about some things, even while the same people have also done a lot of things right?
(I think I apply this symmetrically; if someone wants to write a 22,000 word blog post about the ways in which my intellectual conduct failed to live up to standards, that’s fine with me.)
The thing me and my allies were hoping for a “court ruling” on was not about who should or shouldn’t be held in high regard, but about the philosophical claim that one “ought to accept an unexpected [X] or two deep inside the conceptual boundaries of what would normally be considered [Y] if [positive consequence]”. (I think this is false.)That’s what really matters, not who we should or shouldn’t hold in high regard.
People’s reputations only come into it because of considerations like, if you think Scott is right about the philosophical claim, that should be to his credit and to my detriment, and vice versa. The reason I’m telling this Whole Dumb Story about people instead of just making object-level arguments about ideas, is that I tried making object-level arguments about ideas first, for seven years, and it wasn’t very effective. At some point, I think it makes sense to jump up a meta level and try to reason about people to figure out why the ideas aren’t landing.
I understand what you’re trying to do. “Hold in high regard” is maybe the wrong choice of phrase since it connotes something more status-y than I intended; what I’m really saying is more general: this post failed to convince me of anything in particular, at any level of meta. I’m not inclined to wade into the actual arguments or explain why I feel that way, but I commented anyway since this post does impugn on various people’s internal motivations and reputations pretty directly, and because if my sentiment is widely shared, that might inform others’ decision about whether to respond or engage themselves.
I realize this might be frustrating and come across as intellectually rude or lazy to you or anyone who has a different assessment. Blunt / harsh / vague feedback still seemed better than no feedback, and I wanted to gauge sentiment and offer others a low-effort way to give slightly more nuanced feedback than a vote on the top-level post.
(I think I apply this symmetrically; if someone wants to write a 22,000 word blog post about the ways in which my intellectual conduct failed to live up to standards, that’s fine with me.)
Sure, I’m generally fine with you or others doing this (about anyone), but I can’t imagine many people wanting to, and I’m hypothesizing above that the ROI would be low.
The reason I’m telling this Whole Dumb Story about people instead of just making object-level arguments about ideas, is that I tried making object-level arguments about ideas first, for seven years, and it wasn’t very effective. At some point, I think it makes sense to jump up a meta level and try to reason about people to figure out why the ideas aren’t landing.
I think that’s a very good reason to go meta! And I sympathize deeply with the feeling of having your object-level ideas ignored or misunderstood no matter how many times you explain. But I also think reasoning about the internal motivations / sanity failures / rationality of others is very hard to do responsibly, and even harder to do correctly.
So, more blunt feedback: I think you have mostly succeeded in responsibly presenting your case. This post is a bit rambly, but otherwise clearly written and extremely detailed and… I wouldn’t exactly call it “objective”, but I mostly trust that is an honest and accurate depiction of the key facts, and that you are not close to impinging on any rights or expectations of privacy. And I have no problem with going meta and reasoning about others’ internal motivations and thoughts when done responsibly. It’s just that the reasoning also has to be correct, and I claim (again, rudely without explanation) that isn’t the case here.
I’m not inclined to wade into the actual arguments or explain why I feel that way
Would you do it for $40? I can do PayPal or mail you a check.
Sorry, I know that’s not very much money. I’m low-balling this one partially because I’m trying to be more conservative with money since I’m not dayjobbing right now, and partially because I don’t really expect you to have any good arguments (and I feel OK about throwing away $40 to call your bluff, but I would feel bad spending more than that on lousy arguments).
Usually when people have good arguments and are motivated to comment at all, they use the comment to explain the arguments. So far, you’ve spent 475 words telling me nothing of substance except that you disagree, somehow, about something. This is not a promising sign.
Blunt / harsh / vague feedback
I don’t know why you’re grouping these together. Blunt is great. Harsh is great. Vague is useless.
The reason that blunt is great and harsh is great is because detailed, intellectually substantive feedback phrased in a blunt or harsh manner can be evaluated on its intellectual merits. The merits can’t be faked—or at least, I think I’m pretty good at distinguishing between good and bad criticisms.
The reason vague is useless is because vague feedback is trivially faked. Anyone can just say “this post failed to convince me of anything in particular, at any level of meta” or “the reasoning also has to be correct, and I claim [...] that isn’t the case here.” Why should I trust you? How do I know you’re not bluffing?
If I got something wrong, I want to know about it! (I probably got lots of things wrong; humans aren’t good at writing 22,000 word documents without making any mistakes, and the Iceman/Said interaction already makes me think the last five paragraphs need a rewrite.) But I’m not a mind-reader: in order for me to become less wrong, someone does, actually, have to tell me what I got wrong, specifically. Seems like an easy way to earn forty bucks if you ask me!
Accepted, but I want to register that I am responding because you have successfully exerted social pressure, not because of any monetary incentive. I don’t mind the offer / ask (or the emotional appeals / explanations), but in the future I would prefer that you (or anyone) make such offers via PM.
Semi-relatedly, on cheerful prices, you wrote:
But that seemed sufficiently psychologically coercive and socially weird that I wasn’t sure I wanted to go there.
I don’t think there’s anything weird or coercive about offering to pay someone to respond or engage. But there are often preconditions that must be met between the buyer and seller for a particular cheerful price transaction to clear. For the seller to feel cheerful about a transaction at any price, the seller might need to be reasonably confident that (a) the buyer understands what they are buying (b) the buyer will not later regret having transacted (c) the seller will not later regret having transacted.
This requires a fair amount of trust between the buyer and seller, the buyer to have enough self-knowledge and stability, and for the seller to know enough about the buyer (or perform some due diligence) to be reasonably confident that all of these conditions obtain.
In the post, you recognize that someone might decline a cheerful price offer for reasons related to status, politics, or time / energy constraints. I think you’ve failed to consider a bunch of other reasons people might not want to engage, in public or in private.
Anyway, on to some specific objections:
Basically everywhere you speculate about someone’s internal motivations or thoughts, or claim that someone is doing something for political reasons or status reasons or energy reasons, I think you’re failing to consider alternate hypotheses adequately, starting from false or misleading premises, or drawing conclusions unjustifiably. Also, I disagree with your framing about just about everything, everywhere. I’ll go through a few examples to make my point:
But because I was so cowardly as to need social proof (because I believed that an ordinary programmer such as me was as a mere worm in the presence of the great Eliezer Yudkowsky), it must have just looked to him like an illegible social plot originating from Michael.
Or… engaging with you and your allies is likely to be unpleasant. And empirically you don’t seem capable of gracefully handling a refusal to engage, without making potentially incorrect and public updates about the refuser’s internal state of mind based on that refusal. I guess that’s your right, but it increases the transaction costs of engaging with you, and others might choose to glomarize as a blanket policy in response.
Okay, so Yudkowsky had prevaricated about his own philosophy of language
[shaky premise]
for transparently political reasons
[unjustified assertion about someone else’s internal mental state that fails to consider alternate hypotheses]
Yudkowsky had set in motion a marketing machine (the “rationalist community”)
Loaded framing.
that was continuing to raise funds and demand work from people for below-market rates
“demand”?
based on the claim that while nearly everyone else was criminally insane (causing huge amounts of damage due to disconnect from reality, in a way that would be criminal if done knowingly), he, almost uniquely, was not.
I don’t think Eliezer has ever made this claim in the sense that you connote, and is certainly not “making demands” based on it.
“Work for me or the world ends badly,” basically.
Again, I’m pretty sure Eliezer has never claimed anything like this, except in the literal sense that he has both claimed that AGI is likely to lead to human extinction, and presumably at some point asked people to work for or with him to prevent that. Eliezer is somewhat more certain of this claim than others, but it’s not exactly an abnormal or extreme view within this community on its own, and asking people to work for you for pay is a pretty fundamental and generally prosocial economic activity.
As written, this connotes both “work for me else the world ends badly” and “the world continues to exist iff everyone works for me”, both of which are untrue and I don’t think Eliezer has ever claimed.
(Also, you offered to pay me to respond, and I didn’t call it a “demand” or treat it that way, even though you exerted a bunch of social pressure. OTOH, Eliezer has never asked me or demanded anything of me personally, and I’ve never really felt pressured to do anything in particular based on his writing. For the record, I am pretty happy with the equilibrium / norm where people are allowed to ask for things, and sometimes even exert social pressure about them when it’s important, and I think it’s counterproductive and misleading to label such asks as “demands” even when the stakes are high. Perhaps this point is mostly a dispute about language and terminology, heh.)
If, after we had tried to talk to him privately, Yudkowsky couldn’t be bothered to either live up to his own stated standards or withdraw his validation from the machine he built,
“couldn’t be bothered” is again making what look to me like unjustified / unsupported claims about another person’s mental state.
“and live up to his own stated standards or withdraw his validation...” is asking Eliezer (and your readers) to share your frame about all your object level claims.
The last several paragraphs of the post are where these mistakes seem most egregious, but I think they’re also present in the rest of the piece:
But the post is wrong in obvious ways.
“Obvious”, here and elsewhere seems trivially falsified by the amount of controversy and discussion that these posts have generated.
Given that, it’s hard to read the Tweets Yudkowsky published as anything other than an attempt to intimidate and delegitimize people who want to use language to reason about sex rather than gender identity.
Really? I had no problem not reading it that way, and still don’t.
It’s just not plausible that Yudkowsky was simultaneously savvy enough to choose to make these particular points while also being naïve enough to not understand the political context
Again, this seems like ruling out a bunch of other hypotheses. Also, I actually do find the one considered-and-rejected hypothesis at least somewhat plausible!
After Yudkowsky had stepped away from full-time writing, Alexander had emerged as our subculture’s preeminent writer. Most people in an intellectual scene “are writers” in some sense, but Alexander was the one “everyone” reads: you could often reference a Slate Star Codex post in conversation and expect people to be familiar with the idea, either from having read it, or by osmosis. The frequency with which ”… Not Man for the Categories” was cited at me seemed to suggest it had become our subculture’s party line on trans issues.
This entire paragraph describes a model of the rationalist community as having vastly more centralized leadership and more consensus than in my own model. “party line”? I’m not sure there is any issue or claim for which there is widespread consensus, even very basic ones. I’m not disputing your account of your personal experience, but I want to flag that many of the claims and assumptions in this post seem to rest on an assumption that that experience and model is shared, when it is in fact not shared by at least one other person.
Thanks!! Receipt details (to your selected charity) in PM yesterday.
Or …
(Another part of the problem here might be that I think the privacy norms let me report on my psychological speculations about named individuals, but not all of the evidence that supports them, which might have come from private conversations.)
Okay, so Yudkowsky had prevaricated about his own philosophy of language
[shaky premise]
But it’s not a premise being introduced suddenly out of nowhere; it’s a conclusion argued for at length earlier in the piece. Prevaricate, meaning, “To shift or turn from direct speech or behaviour [...] to waffle or be (intentionally) ambiguous.” “His own philosophy of language”, meaning, that he wrote a 30,000 word Sequence elaborating on 37 ways in which words can be wrong, including #30, “Your definition draws a boundary around things that don’t really belong together.”
When an author who wrote 30,000 words in 2008 pounding home over and over and over again that “words can be wrong”, then turns around in 2018 and says that “maybe as a matter of policy, you want to make a case for language being used a certain way [...] [b]ut you’re not making a stand for Truth in doing so” and that “you’re not standing in defense of truth if you insist on a word, brought explicitly into question, being used with some particular meaning”, about an issue where the opposing side’s view is precisely that the people bringing the word into question are using a definition that draws a boundary around things that don’t really belong together (because men aren’t women, and this continues to be the case even if you redefine some of the men as “trans women”), and the author doesn’t seem eager to clarify when someone calls him on the apparent reversal, I think it makes sense to describe that as the author being ambiguous, having shifted or turned from direct speech—in a word, prevaricating (although not “lying”).
I think I’ve made my case here. If you disagree with some part of the case, I’m eager to hear it! But I don’t think it’s fair for you to dimiss me for making a “shaky premise” when the premise is a conclusion that I argued for.
[unjustified assertion about someone else’s internal mental state that fails to consider alternate hypotheses]
On reflection, the word “transparently” definitely isn’t right here (thanks!), but I’m comfortable standing by “political reasons”. I think later events in the Whole Dumb Story bear me out.
you offered to pay me to respond, and I didn’t call it a “demand” or treat it that way, even though you exerted a bunch of social pressure
I think it would have been fine if you did call it a “demand”! Wiktionary’s first definition is “To request forcefully.” The grandparent is making a request, and the insults (“I don’t really expect you to have any good arguments”, &c.) make it “forceful”. Seems fine. Why did you expect me to object?
The last several paragraphs of the post are where these mistakes seem most egregious
Yeah, I definitely want to rewrite those to be clearer now (as alluded to in the last paragraph of the grandparent). Sorry.
“Obvious”, here and elsewhere seems trivially falsified by the amount of controversy and discussion that these posts have generated.
Sorry, let me clarify. “Obvious” is a 2 place word; there has to an implicit “to whom” even if not stated. I claim that the error in ”… Not Man for the Categories” is obvious to someone who understood the lessons in the “Human’s Guide to Words” Sequence, including the math. I agree that it’s not obvious to people in general, or self-identified “rationalists” in general.
I’m not disputing your account of your personal experience
Shouldn’t you, though? If my perception of “the community” was biased and crazy, it seems like you could totally embarrass me right now by pointing to evidence that my perceptions were biased and crazy.
For example, in part of the post (the paragraph starting with “Now, the memory of that social proof”), I link to comments from Yudkowsky, Alexander, Kelsey Piper, Ozy Brennan, and Rob Bensinger as evidence about the state of the “rationalist” zeitgiest. It seems like you could argue against this by saying something like, “Hey, what about authors X, Y, and Z, who are just as prominent in ‘the community’ as the people you named and are on the record saying that biological sex is immutable and supporting the integrity of female-only spaces?” Admittedly, this could get a little more complicated insofar as your claim is that I was overestimating the degree of consensus and centralization, because the less consensus and centralization there is, the less “What about this-and-such prominent ‘figure’” is a relevant consideration. I still feel like it should be possible to do better than my word against yours.
many of the claims and assumptions in this post seem to rest on an assumption that that experience and model is shared
What gave you that impression?! Would it help if I added the words “to me” or “I think” in key sentences? (An earlier draft said “I think” slightly more often, but my editor cut five instances of it.)
In general, I think a lot of the value proposition of my political writing is that I’m wreckless enough to write clearly about things that most people prefer not to be clear about—models that should be “obvious” but are not shared. I’m definitely not assuming my model is shared. If it were shared, I wouldn’t need to write so many words explaining it!
based on the claim that while nearly everyone else was criminally insane (causing huge amounts of damage due to disconnect from reality, in a way that would be criminal if done knowingly), he, almost uniquely, was not.
I don’t think Eliezer has ever made this claim in the sense that you connote [...]
I read Yudkowsky as asserting:
A very high estimate of his own intelligence, eg comparable to Feynman and Hofstadter.
A very high estimate of the value of intelligence in general, eg sufficient to takeover the world and tile the lightcone using only an internet connection.
A very high estimate of the damage caused on Earth by insufficient intelligence, eg human extinction.
In the fictional world of Dath Ilan where Yudkowsky is the median inhabitant, Yudkowsky says they are on track to solve the alignment (~95% confidence). Whereas in the actual world he says we are on track to go extinct (~100% confidence). Causing human extinction would be criminal if done knowingly, so this satisfies Zach’s claim as written.
I’m leaving this light on links, because I’m not sure what of the above you might object to. I realize that you had many other objections to Zach’s framing, but I thought this could be something to drill into.
Edit: I’m not offering any money to respond, and independently of that, it’s 100% fine if you don’t want to respond.
I mostly take issue with the phrases “criminally insane”, “causing enormous damage”, and “he, almost uniquely, was not” connoting a more unusual and more actionable view than Eliezer (or almost anyone else) actually holds or would agree with.
Lots of people in the world are doing things that are straightforwardly non-optimal, often not even in their own narrow self-interest. This is mostly just the mistake side of conflict vs. mistake theory though, which seems relatively uncontroversial, at least on LW.
Eliezer has pointed out some of those mistakes in the context of AGI and other areas, but so have many others (Scott Alexander, Zack himself, etc.), in a variety of areas (housing policy, education policy, economics, etc.). Such explanations often come (implicitly or explicitly) with a call for others to change their behavior if they accept such arguments, but Eliezer doesn’t seem particularly strident in making such calls, compared to e.g. ordinary politicians, public policy advocates, or other rationalists.
Note, I’m not claiming that Eliezer does not hold some object-level views considered weird or extreme by most, e.g. that sufficiently intelligent AGI could take over the world, or that establishing multinational agreements for control and use of GPUs would be good policy.
But I agree with most or all those views because they seem correct on the merits, not because Eliezer (or anyone else) happened to say them. Eliezer may have been the one to point them out, for which he deserves credit, but he’s always explained his reasoning (sometimes at extreme length) and never to my knowledge asked anyone to just trust him about something like that and start making drastic behavioral changes as a result.
In the fictional world of Dath Ilan where Yudkowsky is the median inhabitant, Yudkowsky says they are on track to solve the alignment (~95% confidence). Whereas in the actual world he says we are on track to go extinct (~100% confidence). Causing human extinction would be criminal if done knowingly, so this satisfies Zach’s claim as written.
Again, leaving aside whether Eliezer himself has actually claimed this or would agree with the sentiment, it seems straightforwardly true to me that a world where the median IQ was 140 would look a lot different (and be a lot better off) than the current Earth. Whether or not it would look exactly like Eliezer’s speculative fictional world, it seems strange and uncharitable to me to characterize that view (that the world would be much better off with more intelligence) as extreme, or to interpret it as a demand or call to action for anything in particular.
Zach (et al) are exaggerating how unusual/extreme/weird Yudkowsky’s positions are.
Zach (et al) are exaggerating how much Yudkowsky’s writings are an explicit call to action.
To the extent that Yudkowsky has unusual positions and calls for actions, you think he’s mostly correct on the merits.
Of these, I’d like to push on (1) a bit. However, I think this would probably work better as a new top-level post (working title “Yudkowsky on Yudkowsky”). To give a flavor, though, and because I’m quite likely to fail to write the top-level post, here’s an example. Shah and Yudkowsky on alignment failures.
This may, perhaps, be confounded by the phenomenon where I am one of the last living descendants of the lineage that ever knew how to say anything concrete at all. Richard Feynman—or so I would now say in retrospect—is noticing concreteness dying out of the world, and being worried about that, at the point where he goes to a college and hears a professor talking about “essential objects” in class, and Feynman asks “Is a brick an essential object?”—meaning to work up to the notion of the inside of a brick, which can’t be observed because breaking a brick in half just gives you two new exterior surfaces—and everybody in the classroom has a different notion of what it would mean for a brick to be an essential object.
I encourage you to follow the link to the rest of the conversation, which relates this to alignment work. So we have this phenomenon where one factor in humanity going extinct is that people don’t listen enough to Yudkowsky and his almost unique ability to speak concretely. This also supports (2) above—this isn’t an explicit call to action, he’s just observing a phenomenon.
A (1) take here is that the quote is cherry-picked, a joke, or an outlier, and his overall work implies a more modest self-assessment. A (3) take is that he really is almost uniquely able to speak concretely. My take (4) is that his self-assessment is positively biased. I interpret Zack’s “break-up” with Yudkowsky in the opening post as moving from a (3) model to a (4) model, and encouraging others to do the same.
Accepted, but I want to register that I am responding because you have successfully exerted social pressure, not because of any monetary incentive. I don’t mind the offer / ask (or the emotional appeals / explanations), but in the future I would prefer that you (or anyone) make such offers via PM.
It would hardly have been effective for Zack to make the offer via PM! In essence, you’re asking for Zack (or anyone) to act ineffectively, in order that you may avoid the inconvenience of having to publicly defend your claims against public disapprobation!
Financial incentives are ineffecitve if offered privately? That’s perhaps true for me personally at the level Zack is offering, but seems obviously false in general.
Offering money in private is maybe less effective than exerting social pressure in public (via publicly offering financial incentives, or other means). I merely pointed out that the two are entangled here, and that the pressure aspect is the one that actually motivates me in this case. I request that future such incentives be applied in a more disentangled way, but I’m not asking Zack to refrain from applying social pressure OR from offering financial incentives, just asking that those methods be explicitly disentangled. Zack is of course not obliged to comply with this request, but if he does not do so, I will continue flagging my actual motivations explicitly.
Financial incentives are ineffecitve if offered privately? That’s perhaps true for me personally at the level Zack is offering, but seems obviously false in general.
The financial incentive was clearly ineffective in this case, when offered publicly, so this is a red herring. (Really, who would’ve expected otherwise? $40, for the average Less Wrong reader? That’s a nominal amount, no more.)
No, what was effective was the social pressure—as you say!
I request that future such incentives be applied in a more disentangled way, but I’m not asking Zack to refrain from applying social pressure OR from offering financial incentives, just asking that those methods be explicitly disentangled. Zack is of course not obliged to comply with this request, but if he does not do so, I will continue flagging my actual motivations explicitly.
Disentangling these things as you describe would reduce the force of the social pressure, however.
I probably would have also responded if Zack had sent his comment verbatim as a PM. Maybe not as quickly or in exactly the same way, e.g. I wouldn’t have included the digression about incentives.
But anyway, I did in fact respond, so I don’t think it’s valid to conclude much about what would have been “clearly ineffective” in a counterfactual.
One other point that you seem to be missing is that it’s possible to exert social pressure via private channels, with or without financial incentives (and I’m also fine with Zack or others trying this, in general). Private might even be more effective at eliciting a response, in some cases.
In retrospect, I feel guilty about impulsively mixing the “cheerful price” mechanism and the “social pressure” mechanism. I suspect Said is right that the gimmick of the former added to the “punch” of the latter, but at the terrible cost of undermining the integrity of the former (it’s supposed to be cheerful!). I apologize for that.
Life is not graded on a curve and you can update yourself incrementally? I also hold all of those people in high regard—relative to the rest of the world. (And Anna is a personal friend.) I think all of those people hold me in high regard, relative to the rest of the world. Nevertheless, it seems like there ought to be a time and a place to talk about people having been culpably wrong about some things, even while the same people have also done a lot of things right?
(I think I apply this symmetrically; if someone wants to write a 22,000 word blog post about the ways in which my intellectual conduct failed to live up to standards, that’s fine with me.)
The thing me and my allies were hoping for a “court ruling” on was not about who should or shouldn’t be held in high regard, but about the philosophical claim that one “ought to accept an unexpected [X] or two deep inside the conceptual boundaries of what would normally be considered [Y] if [positive consequence]”. (I think this is false.) That’s what really matters, not who we should or shouldn’t hold in high regard.
People’s reputations only come into it because of considerations like, if you think Scott is right about the philosophical claim, that should be to his credit and to my detriment, and vice versa. The reason I’m telling this Whole Dumb Story about people instead of just making object-level arguments about ideas, is that I tried making object-level arguments about ideas first, for seven years, and it wasn’t very effective. At some point, I think it makes sense to jump up a meta level and try to reason about people to figure out why the ideas aren’t landing.
I understand what you’re trying to do. “Hold in high regard” is maybe the wrong choice of phrase since it connotes something more status-y than I intended; what I’m really saying is more general: this post failed to convince me of anything in particular, at any level of meta. I’m not inclined to wade into the actual arguments or explain why I feel that way, but I commented anyway since this post does impugn on various people’s internal motivations and reputations pretty directly, and because if my sentiment is widely shared, that might inform others’ decision about whether to respond or engage themselves.
I realize this might be frustrating and come across as intellectually rude or lazy to you or anyone who has a different assessment. Blunt / harsh / vague feedback still seemed better than no feedback, and I wanted to gauge sentiment and offer others a low-effort way to give slightly more nuanced feedback than a vote on the top-level post.
Sure, I’m generally fine with you or others doing this (about anyone), but I can’t imagine many people wanting to, and I’m hypothesizing above that the ROI would be low.
I think that’s a very good reason to go meta! And I sympathize deeply with the feeling of having your object-level ideas ignored or misunderstood no matter how many times you explain. But I also think reasoning about the internal motivations / sanity failures / rationality of others is very hard to do responsibly, and even harder to do correctly.
So, more blunt feedback: I think you have mostly succeeded in responsibly presenting your case. This post is a bit rambly, but otherwise clearly written and extremely detailed and… I wouldn’t exactly call it “objective”, but I mostly trust that is an honest and accurate depiction of the key facts, and that you are not close to impinging on any rights or expectations of privacy. And I have no problem with going meta and reasoning about others’ internal motivations and thoughts when done responsibly. It’s just that the reasoning also has to be correct, and I claim (again, rudely without explanation) that isn’t the case here.
Would you do it for $40? I can do PayPal or mail you a check.
Sorry, I know that’s not very much money. I’m low-balling this one partially because I’m trying to be more conservative with money since I’m not dayjobbing right now, and partially because I don’t really expect you to have any good arguments (and I feel OK about throwing away $40 to call your bluff, but I would feel bad spending more than that on lousy arguments).
Usually when people have good arguments and are motivated to comment at all, they use the comment to explain the arguments. So far, you’ve spent 475 words telling me nothing of substance except that you disagree, somehow, about something. This is not a promising sign.
I don’t know why you’re grouping these together. Blunt is great. Harsh is great. Vague is useless.
The reason that blunt is great and harsh is great is because detailed, intellectually substantive feedback phrased in a blunt or harsh manner can be evaluated on its intellectual merits. The merits can’t be faked—or at least, I think I’m pretty good at distinguishing between good and bad criticisms.
The reason vague is useless is because vague feedback is trivially faked. Anyone can just say “this post failed to convince me of anything in particular, at any level of meta” or “the reasoning also has to be correct, and I claim [...] that isn’t the case here.” Why should I trust you? How do I know you’re not bluffing?
If I got something wrong, I want to know about it! (I probably got lots of things wrong; humans aren’t good at writing 22,000 word documents without making any mistakes, and the Iceman/Said interaction already makes me think the last five paragraphs need a rewrite.) But I’m not a mind-reader: in order for me to become less wrong, someone does, actually, have to tell me what I got wrong, specifically. Seems like an easy way to earn forty bucks if you ask me!
Accepted, but I want to register that I am responding because you have successfully exerted social pressure, not because of any monetary incentive. I don’t mind the offer / ask (or the emotional appeals / explanations), but in the future I would prefer that you (or anyone) make such offers via PM.
Semi-relatedly, on cheerful prices, you wrote:
I don’t think there’s anything weird or coercive about offering to pay someone to respond or engage. But there are often preconditions that must be met between the buyer and seller for a particular cheerful price transaction to clear. For the seller to feel cheerful about a transaction at any price, the seller might need to be reasonably confident that (a) the buyer understands what they are buying (b) the buyer will not later regret having transacted (c) the seller will not later regret having transacted.
This requires a fair amount of trust between the buyer and seller, the buyer to have enough self-knowledge and stability, and for the seller to know enough about the buyer (or perform some due diligence) to be reasonably confident that all of these conditions obtain.
In the post, you recognize that someone might decline a cheerful price offer for reasons related to status, politics, or time / energy constraints. I think you’ve failed to consider a bunch of other reasons people might not want to engage, in public or in private.
Anyway, on to some specific objections:
Basically everywhere you speculate about someone’s internal motivations or thoughts, or claim that someone is doing something for political reasons or status reasons or energy reasons, I think you’re failing to consider alternate hypotheses adequately, starting from false or misleading premises, or drawing conclusions unjustifiably. Also, I disagree with your framing about just about everything, everywhere. I’ll go through a few examples to make my point:
Or… engaging with you and your allies is likely to be unpleasant. And empirically you don’t seem capable of gracefully handling a refusal to engage, without making potentially incorrect and public updates about the refuser’s internal state of mind based on that refusal. I guess that’s your right, but it increases the transaction costs of engaging with you, and others might choose to glomarize as a blanket policy in response.
[shaky premise]
[unjustified assertion about someone else’s internal mental state that fails to consider alternate hypotheses]
Loaded framing.
“demand”?
I don’t think Eliezer has ever made this claim in the sense that you connote, and is certainly not “making demands” based on it.
Again, I’m pretty sure Eliezer has never claimed anything like this, except in the literal sense that he has both claimed that AGI is likely to lead to human extinction, and presumably at some point asked people to work for or with him to prevent that. Eliezer is somewhat more certain of this claim than others, but it’s not exactly an abnormal or extreme view within this community on its own, and asking people to work for you for pay is a pretty fundamental and generally prosocial economic activity.
As written, this connotes both “work for me else the world ends badly” and “the world continues to exist iff everyone works for me”, both of which are untrue and I don’t think Eliezer has ever claimed.
(Also, you offered to pay me to respond, and I didn’t call it a “demand” or treat it that way, even though you exerted a bunch of social pressure. OTOH, Eliezer has never asked me or demanded anything of me personally, and I’ve never really felt pressured to do anything in particular based on his writing. For the record, I am pretty happy with the equilibrium / norm where people are allowed to ask for things, and sometimes even exert social pressure about them when it’s important, and I think it’s counterproductive and misleading to label such asks as “demands” even when the stakes are high. Perhaps this point is mostly a dispute about language and terminology, heh.)
“and live up to his own stated standards or withdraw his validation...” is asking Eliezer (and your readers) to share your frame about all your object level claims.
The last several paragraphs of the post are where these mistakes seem most egregious, but I think they’re also present in the rest of the piece:
“Obvious”, here and elsewhere seems trivially falsified by the amount of controversy and discussion that these posts have generated.
Really? I had no problem not reading it that way, and still don’t.
Again, this seems like ruling out a bunch of other hypotheses. Also, I actually do find the one considered-and-rejected hypothesis at least somewhat plausible!
This entire paragraph describes a model of the rationalist community as having vastly more centralized leadership and more consensus than in my own model. “party line”? I’m not sure there is any issue or claim for which there is widespread consensus, even very basic ones. I’m not disputing your account of your personal experience, but I want to flag that many of the claims and assumptions in this post seem to rest on an assumption that that experience and model is shared, when it is in fact not shared by at least one other person.
Thanks!! Receipt details (to your selected charity) in PM yesterday.
(Another part of the problem here might be that I think the privacy norms let me report on my psychological speculations about named individuals, but not all of the evidence that supports them, which might have come from private conversations.)
But it’s not a premise being introduced suddenly out of nowhere; it’s a conclusion argued for at length earlier in the piece. Prevaricate, meaning, “To shift or turn from direct speech or behaviour [...] to waffle or be (intentionally) ambiguous.” “His own philosophy of language”, meaning, that he wrote a 30,000 word Sequence elaborating on 37 ways in which words can be wrong, including #30, “Your definition draws a boundary around things that don’t really belong together.”
When an author who wrote 30,000 words in 2008 pounding home over and over and over again that “words can be wrong”, then turns around in 2018 and says that “maybe as a matter of policy, you want to make a case for language being used a certain way [...] [b]ut you’re not making a stand for Truth in doing so” and that “you’re not standing in defense of truth if you insist on a word, brought explicitly into question, being used with some particular meaning”, about an issue where the opposing side’s view is precisely that the people bringing the word into question are using a definition that draws a boundary around things that don’t really belong together (because men aren’t women, and this continues to be the case even if you redefine some of the men as “trans women”), and the author doesn’t seem eager to clarify when someone calls him on the apparent reversal, I think it makes sense to describe that as the author being ambiguous, having shifted or turned from direct speech—in a word, prevaricating (although not “lying”).
I think I’ve made my case here. If you disagree with some part of the case, I’m eager to hear it! But I don’t think it’s fair for you to dimiss me for making a “shaky premise” when the premise is a conclusion that I argued for.
On reflection, the word “transparently” definitely isn’t right here (thanks!), but I’m comfortable standing by “political reasons”. I think later events in the Whole Dumb Story bear me out.
I think it would have been fine if you did call it a “demand”! Wiktionary’s first definition is “To request forcefully.” The grandparent is making a request, and the insults (“I don’t really expect you to have any good arguments”, &c.) make it “forceful”. Seems fine. Why did you expect me to object?
Yeah, I definitely want to rewrite those to be clearer now (as alluded to in the last paragraph of the grandparent). Sorry.
Sorry, let me clarify. “Obvious” is a 2 place word; there has to an implicit “to whom” even if not stated. I claim that the error in ”… Not Man for the Categories” is obvious to someone who understood the lessons in the “Human’s Guide to Words” Sequence, including the math. I agree that it’s not obvious to people in general, or self-identified “rationalists” in general.
Shouldn’t you, though? If my perception of “the community” was biased and crazy, it seems like you could totally embarrass me right now by pointing to evidence that my perceptions were biased and crazy.
For example, in part of the post (the paragraph starting with “Now, the memory of that social proof”), I link to comments from Yudkowsky, Alexander, Kelsey Piper, Ozy Brennan, and Rob Bensinger as evidence about the state of the “rationalist” zeitgiest. It seems like you could argue against this by saying something like, “Hey, what about authors X, Y, and Z, who are just as prominent in ‘the community’ as the people you named and are on the record saying that biological sex is immutable and supporting the integrity of female-only spaces?” Admittedly, this could get a little more complicated insofar as your claim is that I was overestimating the degree of consensus and centralization, because the less consensus and centralization there is, the less “What about this-and-such prominent ‘figure’” is a relevant consideration. I still feel like it should be possible to do better than my word against yours.
What gave you that impression?! Would it help if I added the words “to me” or “I think” in key sentences? (An earlier draft said “I think” slightly more often, but my editor cut five instances of it.)
In general, I think a lot of the value proposition of my political writing is that I’m wreckless enough to write clearly about things that most people prefer not to be clear about—models that should be “obvious” but are not shared. I’m definitely not assuming my model is shared. If it were shared, I wouldn’t need to write so many words explaining it!
I read Yudkowsky as asserting:
A very high estimate of his own intelligence, eg comparable to Feynman and Hofstadter.
A very high estimate of the value of intelligence in general, eg sufficient to takeover the world and tile the lightcone using only an internet connection.
A very high estimate of the damage caused on Earth by insufficient intelligence, eg human extinction.
In the fictional world of Dath Ilan where Yudkowsky is the median inhabitant, Yudkowsky says they are on track to solve the alignment (~95% confidence). Whereas in the actual world he says we are on track to go extinct (~100% confidence). Causing human extinction would be criminal if done knowingly, so this satisfies Zach’s claim as written.
I’m leaving this light on links, because I’m not sure what of the above you might object to. I realize that you had many other objections to Zach’s framing, but I thought this could be something to drill into.
Edit: I’m not offering any money to respond, and independently of that, it’s 100% fine if you don’t want to respond.
I mostly take issue with the phrases “criminally insane”, “causing enormous damage”, and “he, almost uniquely, was not” connoting a more unusual and more actionable view than Eliezer (or almost anyone else) actually holds or would agree with.
Lots of people in the world are doing things that are straightforwardly non-optimal, often not even in their own narrow self-interest. This is mostly just the mistake side of conflict vs. mistake theory though, which seems relatively uncontroversial, at least on LW.
Eliezer has pointed out some of those mistakes in the context of AGI and other areas, but so have many others (Scott Alexander, Zack himself, etc.), in a variety of areas (housing policy, education policy, economics, etc.). Such explanations often come (implicitly or explicitly) with a call for others to change their behavior if they accept such arguments, but Eliezer doesn’t seem particularly strident in making such calls, compared to e.g. ordinary politicians, public policy advocates, or other rationalists.
Note, I’m not claiming that Eliezer does not hold some object-level views considered weird or extreme by most, e.g. that sufficiently intelligent AGI could take over the world, or that establishing multinational agreements for control and use of GPUs would be good policy.
But I agree with most or all those views because they seem correct on the merits, not because Eliezer (or anyone else) happened to say them. Eliezer may have been the one to point them out, for which he deserves credit, but he’s always explained his reasoning (sometimes at extreme length) and never to my knowledge asked anyone to just trust him about something like that and start making drastic behavioral changes as a result.
Again, leaving aside whether Eliezer himself has actually claimed this or would agree with the sentiment, it seems straightforwardly true to me that a world where the median IQ was 140 would look a lot different (and be a lot better off) than the current Earth. Whether or not it would look exactly like Eliezer’s speculative fictional world, it seems strange and uncharitable to me to characterize that view (that the world would be much better off with more intelligence) as extreme, or to interpret it as a demand or call to action for anything in particular.
I would summarize this as saying:
Zach (et al) are exaggerating how unusual/extreme/weird Yudkowsky’s positions are.
Zach (et al) are exaggerating how much Yudkowsky’s writings are an explicit call to action.
To the extent that Yudkowsky has unusual positions and calls for actions, you think he’s mostly correct on the merits.
Of these, I’d like to push on (1) a bit. However, I think this would probably work better as a new top-level post (working title “Yudkowsky on Yudkowsky”). To give a flavor, though, and because I’m quite likely to fail to write the top-level post, here’s an example. Shah and Yudkowsky on alignment failures.
I encourage you to follow the link to the rest of the conversation, which relates this to alignment work. So we have this phenomenon where one factor in humanity going extinct is that people don’t listen enough to Yudkowsky and his almost unique ability to speak concretely. This also supports (2) above—this isn’t an explicit call to action, he’s just observing a phenomenon.
A (1) take here is that the quote is cherry-picked, a joke, or an outlier, and his overall work implies a more modest self-assessment. A (3) take is that he really is almost uniquely able to speak concretely. My take (4) is that his self-assessment is positively biased. I interpret Zack’s “break-up” with Yudkowsky in the opening post as moving from a (3) model to a (4) model, and encouraging others to do the same.
It would hardly have been effective for Zack to make the offer via PM! In essence, you’re asking for Zack (or anyone) to act ineffectively, in order that you may avoid the inconvenience of having to publicly defend your claims against public disapprobation!
Financial incentives are ineffecitve if offered privately? That’s perhaps true for me personally at the level Zack is offering, but seems obviously false in general.
Offering money in private is maybe less effective than exerting social pressure in public (via publicly offering financial incentives, or other means). I merely pointed out that the two are entangled here, and that the pressure aspect is the one that actually motivates me in this case. I request that future such incentives be applied in a more disentangled way, but I’m not asking Zack to refrain from applying social pressure OR from offering financial incentives, just asking that those methods be explicitly disentangled. Zack is of course not obliged to comply with this request, but if he does not do so, I will continue flagging my actual motivations explicitly.
The financial incentive was clearly ineffective in this case, when offered publicly, so this is a red herring. (Really, who would’ve expected otherwise? $40, for the average Less Wrong reader? That’s a nominal amount, no more.)
No, what was effective was the social pressure—as you say!
Disentangling these things as you describe would reduce the force of the social pressure, however.
I probably would have also responded if Zack had sent his comment verbatim as a PM. Maybe not as quickly or in exactly the same way, e.g. I wouldn’t have included the digression about incentives.
But anyway, I did in fact respond, so I don’t think it’s valid to conclude much about what would have been “clearly ineffective” in a counterfactual.
One other point that you seem to be missing is that it’s possible to exert social pressure via private channels, with or without financial incentives (and I’m also fine with Zack or others trying this, in general). Private might even be more effective at eliciting a response, in some cases.
In retrospect, I feel guilty about impulsively mixing the “cheerful price” mechanism and the “social pressure” mechanism. I suspect Said is right that the gimmick of the former added to the “punch” of the latter, but at the terrible cost of undermining the integrity of the former (it’s supposed to be cheerful!). I apologize for that.
I have now rewritten that passage (now the third- and second-to-last paragraphs). Better?