I agree that most relevant bad behavior isn’t going to feel from the inside like an attempt to mislead, and I think that rationalists sometimes either ignore this or else have an unfounded optimism about nominal alignment.
It would be surprising, if bad intent were so rare in the relevant sense, that people would be so quick to jump to the conclusion that it is present. Why would that be adaptive?
In the evolutionary context, our utterances and conscious beliefs are optimized for their effects on others, and not merely for accuracy. Believing and claiming bad things about competitors is a typical strategy. Prima facie, accusations of bad faith are particularly attractive since they can be levied on sparse evidence yet are rationally compelling. Empirically, accusations of bad faith are particularly common.
Acting in bad faith doesn’t make you intrinsically a bad person, because there’s no such thing.
This makes an interesting contrast with the content of the post. The feeling that some people are bad is a strong and central social intuition. Do you think you’ve risen to the standard of evidence you are requesting here? It seems to me that you are largely playing the same game people normally play, and then trying to avoid norms that regulate the game by disclaiming “I’m not playing the game.”
For the most part these procedural issues seem secondary to disputes about facts on the ground. But all else equal they’re a reason to prefer object-level questions to questions about intent, logical argument and empirical data to intuition, and private discussion to public discussion.
The feeling that some people are bad is a strong and central social intuition. Do you think you’ve risen to the standard of evidence you are requesting here?
Nope! Good point.
It seems to me that you are largely playing the same game people normally play, and then trying to avoid norms that regulate the game by disclaiming “I’m not playing the game.”
Here’s a specific outcome I would like to avoid: ganging up on the individuals saying misleading things, replacing them with new individuals who have better track records, but doing nothing to alter the underlying incentives. That would be really bad. I think we actually have exceptionally high-integrity individuals in critical leadership positions now, in ways that make the problem of perceived incentive to distort much easier to solve than it might otherwise be.
I don’t actually know how not to play the same old game yet, but I am trying to construct a way.
I don’t actually know how not to play the same old game yet, but I am trying to construct a way.
I see you aiming to construct a way and making credible progress, but I worry that you’re trying to do to many things at once and are going to cause lasting damage by the time you figure it out.
Specifically, the “confidence game” framing of the previous post moved it from “making an earnest good faith effort to talk about things” to “the majority of the post’s content is making a status move”[1] (in particular in the context of your other recent posts, and is exacerbated by this one), and if I were using the framing of this current post I’d say both the previous post and this one have bad intent.
I don’t think that’s a good framing—I think it’s important that you (and folk at OpenPhil and at CEA) do not just have an internally positive narrative but are actually trying to do things that actually cache out to “help each other” (in a broad sense of “help each other”). But I’m worried that this will not remain the case much longer if you continue on your current trajectory.
A year ago, I was extremely impressed with the work you were doing and points you were making, and frustrated that those points were not having much impact.
My perception was “EA Has A Lying Problem” was an inflection point where a) yeah, people started actually paying attention to the class of criticism you’re doing, but the mechanism by which people started paying attention was by critics invoking rhetoric and courting controversy, which was approximately as bad as the problem it was trying to solve. (or at least, within an order of magnitude as bad)
[1] I realize there was a whole lot of other content of the Confidence Game post that was quite good. But, like, the confidence game part is the part I remember easily. Which is the problem.
Meta discussion of how to have conversations / high quality discourse / why this is important
Evaluating OpenPhil and CEA as institutions, in a manner that’s aiming to be evenhanded and fair
Making claims and discussing OpenPhil and CEA that seem pretty indistinguishable from “punishing them and building public animosity towards them.”
Because of #3, I think it’s a lot harder to have credibility when doing #1 or #2. I think there is now enough history with #3 (perceived or actual, whatever your intent), that if you want to be able to do #1 or #2 you need to signal pretty hard that you’re not doing #3 anymore, and specifically take actions aiming to rebuild trust. (And if you were doing #3 by accident, this includes figuring out why your process was outputting something that looked like 3)
I have thoughts about “what to do to cause OpenPhil and CEA to change their behavior” which’ll be a response to tristanm’s comment.
Making claims and discussing OpenPhil and CEA that seem pretty indistinguishable from “punishing them and building public animosity towards them.”
Um, this notion that publicly criticizing organizations such as OpenPhil and CEA amounts to unhelpfully “punishing them and building public animosity towards them”, and thus is per se something to be avoided, is exactly one of the glaring issues “EA has a Lying Problem” (specifically, the subsection Criticizing EA orgs is harmful to the movement) was about. Have we learned nothing since then?
I think I’m mostly going to have to retreat to “this is a very important conversation that I would very much like to have over skype but I think online text is not a good medium for it.”
But we’ve had this conversation online, when EA Has A Lying Problem was first posted. Some worthwhile points were raised that are quite close to your position here, such as the point that unrealistic standards of idealism/virtue, honesty and prompt response to criticism (that is, unrealistic for broadly any real-world institution) could undermine the very real progress that EA orgs are hopefully making, compared to most charitable organizations. This is very much true, but the supposed implication that any and all internal critiques are per se harmful simply doesn’t follow!
I wasn’t saying any and all critiques are harmful—the specific thing I was saying was “these are three things I see you doing right now, and I don’t think you can do all of those within a short timespan.”
Independently, I also think some-but-not-all of the specific critiques you are making are harmful, but that wasn’t the point I was making at the time.
The reason I’d much prefer to have the conversation in person is because by now the entire conversation is emotionally charged (at least for me, and it looks like for you), in a way that is counterproductive. Speaking only for myself, I know that in an in person conversation where I can read facial expressions, I can a) more easily maintain empathy throughout the process, b) as soon as I hit a point where either we disagree, or where the conversation is getting heated, it’s a lot easier to see that, step back and say “okay let’s stop drop and doublecrux.” (And, hopefully, often realize that something was a simple misunderstanding rather than a disagreement)
Online, there are two options at any interval: write out a short point, or write out a long point. If I write out a short point, it won’t actually address all the things I’m trying to point at. If I write a long point, at least one thing will probably be disagreed with or misunderstood, which will derail the whole post.
A) I think this is probably a good thing to do when an online conversation is accumulating drama and controversy.
B) Even if it’s not, I very much want to test it out and find out if it works.
Sometimes a just and accurate evaluation shows that someone’s not as great as they said they were. I’m not trying to be evenhanded in the sense of never drawing any conclusions, and I don’t see the value in that.
Overall, a lot of this feels to me like asking me to do more work, with no compensation, and no offers of concrete independent help, and putting the burden of making the interaction go well on the critic.
A year ago, I was extremely impressed with the work you were doing and points you were making, and frustrated that those points were not having much impact.
It would have been very, very helpful at that time to have public evidence that anyone at all agreed or at least thought that particular points I was making needed an answer. I’m getting that now, I wasn’t getting that then, so I find it hard to see the appeal in going back to a style that wasn’t working.
My perception was “EA Has A Lying Problem” was an inflection point where a) yeah, people started actually paying attention to the class of criticism you’re doing, but the mechanism by which people started paying attention was by critics invoking rhetoric and courting controversy, which was approximately as bad as the problem it was trying to solve. (or at least, within an order of magnitude as bad)
That was a blog post by Sarah Constantin. I am not Sarah Constantin. I wrote my own post in response and about the same things, which no one is bringing up here because no one remembers it. It got a bit of engagement at the time, but I think most of that was spillover from Sarah’s post.
If you want higher-quality discourse, you can engage more publicly with what you see as the higher-quality discourse. My older posts are still available to engage with on the public internet, and were written to raise points that would still be relevant in the future.
I agree that the “confidence game” framing, and particularly the comparison to a ponzi scheme seemed to me like surprisingly charged language, and not the kind of thing you would do if you wanted a productive dialogue with someone.
I’m not sure whether Benquo means for it to come across that way or not. (Pro: maybe he has in fact given up on direct communication with OpenPhil, and thinks his only method of influence is riling up their base. Con: maybe he just thought it was an apt metaphor and didn’t model it as a slap-in-the-face, like I did. Or maybe something else I’m missing.)
Just to add another datapoint, I read it as strongly hostile, more like aiming at delegitimizing the target in the eyes of others than at starting a constructive discussion with them.
If his goal is to actually convince EA organizations to change their behavior, then it could be argued that his rhetorical tactics are in fact likely to be the most effective way of actually achieving that. We should not underestimate the effectiveness of strategies that work by negative PR or by using rhetorical as opposed to strictly argumentative language. I would argue they actually have a pretty good track record of getting organizations to change, without completely destroying the organization (or an associated movement). Uber and United have probably just gone through some of the worst negative coverage it is possible to undergo, and yet the probability that either of them will be completely destroyed by that is almost negligible. On the other hand, the probability that they will improve due to the negative controversy is quite high by my estimation.
Noting the history of organizations that have been completely wiped out by scandal or controversy, it is usually the case that they failed to accomplish their primary goal (such as maximizing shareholder value), and typically in a catastrophic or permanent way that indicated almost beyond doubt that they would never be able to accomplish that goal. It is generally not enough that their leaders acted immorally or unethically (since they can usually be replaced), or that they fail at a subgoal (because subgoals tend to be easier to modify). And since EA is not a single organization, but is better understood as a movement, it is unlikely that the entire movement will be crippled by even a major controversy in one of its organizations. It’s really hard to destroy philosophies.
OpenPhil leadership stated that responding to criticisms and being transparent about their decision-making is a highly costly action to take. And I think it has been well-argued at this point (and not in a purely rhetorical way) that EA organizations are so strongly motivated against taking these actions (as judged by observation of their actions), that they may even occasionally act in the opposite direction. Therefore, if there exist convincing arguments that they are engaging in undesirable behavior, and given that we fairly well know that they are acting on strong incentives, then it follows that in order to change their behavior, they need to be strongly motivated in the other direction. It is not, in general, possible to modify an agent’s utility function by reasoning alone. All rational agents are instrumentally motivated to preserve their preferences and resist attempts at modification.
My argument is not that we need to resort to sensationalist tactics, but only that purely argumentative strategies that offer no negative cost to the organization in question are unlikely to be effective either. And additionally that actions that add this cost are unlikely to be so costly that they result in permanent or unrecoverable damage.
I agree that this is a big and complicated deal and “never resort to sensationalist tactics” isn’t a sufficient answer for reasons close to what you describe. I’m not sure what the answer is, but I’ve been thinking about ideas.
Basically, I think were automatically fail if we have no way to punish defectors, and we also automatically fail controversy/sensationalism-as-normally-practiced is our main tool of doing so.
I think the threat of sensationalist tactics needs to be real. But it needs to be more like Nuclear Deterrence than it is like tit-for-tat warfare.
We’ve seen where sensationalism/controversy leads—American journalism. It is a terrible race to the bottom of inducing as much outrage as you can. It is anti-epistemic, anti-instrumental, anti-everything. Once you start down the dark path, forever will it dominate your destiny.
I am very sympathetic to the fact that Ben tried NOT doing that, and it didn’t work.
Comments like this make me want to actually go nuclear, if I’m already not getting credit for avoiding doing so.
I haven’t really called anyone in the community names. I’ve worked hard to avoid singling people out, and instead tried to make the discussion about norms and actions, not persons. I haven’t tried to organize any material opposition to the interests of the organizations I’m criticizing. I haven’t talked to journalists about this. I haven’t made any efforts to widely publicize my criticisms outside of the community. I’ve been careful to bring up the good points as well as the bad of the people and institutions I’ve been criticizing.
I’d really, really like it if there were a way to get sincere constructive engagement with the tactics I’ve been using. They’re a much better fit for my personality than the other stuff. I’d like to save our community, not blow it up. But we are on a path towards enforcing norms to suppress information rather than disclose it, and if that keeps going, it’s simply going to destroy the relevant value.
(On a related note, I’m aware of exactly one individual who’s been accused of arguing in bad faith in the discourse around Nate’s post, and that individual is me.)
I’m not certain that there is, in fact, a nuclear option. Besides that, I believe there is still room for more of what you have been doing. In particular, I think there are a couple of important topics that have yet to be touched upon in depth, by anyone really.
The first is that the rationalist community has yet to be fully engaged with the conversation. I’ve been observing the level of activity here and on other blogs and see that the majority of conversation is conducted by a small number of people. In other locations, such as SSC, there is a high level of activity but there is also substantial overlap with groups not directly a part of the rationality community, and the conversations there aren’t typically on the topic of EA. Some of the more prominent people have not entered the conversation at all besides Nate. It would be nice if someone like Eliezer gave his two cents.
The second is that it has yet to be discussed in depth the fact that the rationality community and the EA community are, in fact, separate communities with differing goals and values that are more accurately said to have formed an alliance rather than actually merged into one group. They have different origin stories, different primary motivations, and have been focused on a different set of problems throughout their history. The re-focusing of EA towards AI safety occurred rather recently and I think that as their attention turned there, it become more obvious to the rationality community that there were significant differences in thought that were capable of causing conflict.
What I see as one of the main differences between the two cultures is that the rationality community is mostly concerned with accuracy of belief and methods of finding truth whereas the EA community is mostly concerned with being a real force for good in the world, achieved through utilitarian means. I think there is in fact a case to be made that we either need to find a way to reconcile these differences, or go our separate ways, but we certainly can’t pretend these differences don’t exist. One of my main problems with Nate’s post is that he appears to imply that there aren’t any genuine conflicts between the two communities, which I think is simply not the case. And I think this has caused some disappointing choices for MIRI in responding to criticisms. For example, it’s disappointing that MIRI has yet to publish a formal response to the critiques made by Open Phil. I think it’s basic PR 101 that if you’re going to link to or reference criticisms to your organization, you should be fully prepared to engage with each and every point made.
I think my overall point is that there is still room for you, and anyone else who wants to enter this conversation, to continue with the strategy you are currently using, because it does not seem to have fully permeated the rationality community. Some sort of critical mass of support has to be reached before progress can be made, I think.
reason to prefer object-level questions to questions about intent, logical argument and empirical data to intuition, and private discussion to public discussion
I think Nate is absolutely correct to note that if we just retreat to the object level and give up on implied trust, we lose something very valuable. We can’t each evaluate everything from scratch. If we’re going to make intellectual progress together, we need to be able to justifiably trust that people aren’t just trying to get us to do things that make sense to them, aren’t even just telling us things that happen to be literally true, but are making an honest good-faith attempt to give us the most decision-relevant information.
Discussions about what’s in good faith also seem hard to avoid when discussing things like standards of evidence, and how to evaluate summaries from outsiders.
But all else equal they’re a reason to prefer object-level questions to questions about intent, logical argument and empirical data to intuition, and private discussion to public discussion.
I agree with the first two items, but consider that the content of these private discussions, and whatever the conclusions that are being drawn from them, are probably only visible to the wider community in the form of the decisions being made at the highest levels. Therefore, how do you ensure that when these decisions are made, and the wider community is expected to support them, that there will not be disagreement or confusion? Especially since the reasoning behind them is probably highly complex.
Then this begs the question, what is the distribution of private / public discussion that is the most preferred? Certainly if all discussion was kept private, then the wider community (especially the EA community) would have no choice but to support decisions on faith alone. And at the other extreme, there is the high cost of writing and publishing documentation of reasoning, the risk of wide misunderstanding and confusion, and the difficulty associated with trying to retract or adjust statements that are no longer supported.
And if “private” discussion wasn’t constrained to just a small circle, but rather simply meant that you would have to communicate to each individual inquiry separately, than that may come at an even greater cost than that of simply publishing your thoughts openly, because it would require you to devote your attention and effort into multiple, possibly numerous individual discussions, that require modeling each person’s level of knowledge and understanding.
I essentially don’t think the answer is going to be as simple as “private” vs “public”, but I tend to err on the side of transparency, though this may reflect more of a value than a belief based on strong empirical data.
a. Private discussion is nearly as efficient as public discussion for information-transmission, but has way fewer political consequences. On top of that, the political context is more collaborative between participants and so it is less epistemically destructive.
b. I really don’t want to try to use collectively-enforced norms to guide epistemology and don’t think there are many examples of this working out well (whereas there seem to be examples of avoiding epistemically destructive norms by moving into private).
Can you define more precisely what you mean by “private discussion?” If by that you mean that all discourse is constrained to one-on-one conversations where the contents are not available to anyone else, I don’t intuitively see how this would be less destructive and more collaborative. It seems to require that a lot of interactions must occur before every person is up to date on the collective group knowledge, and also that for each conversation there is a lossy compression going on—it’s difficult for each conversation to carry the contents of each person’s history of previous conversations.
On the other hand, if you’re advocating for information to be filtered when transmitted beyond the trusted group, but flows freely within the trusted group, I believe that is less complicated and more efficient and I would have fewer objections to that.
By “private discussion” I mean discussions amongst small groups, in contrast with discussions amongst large groups. Both of them occur constantly. I’ve claimed that in general political considerations cut in favor of having private discussions more often than you otherwise would, I didn’t mean to be making a bold claim.
Can you define more precisely what you mean by “private discussion?” If by that you mean that all discourse is constrained to one-on-one conversations where the contents are not available to anyone else
I recently wanted to raise an issue with some possible controversy/politics in the main EA facebook group. Instead of approving the post I was told, “this post isn’t a good fit for the group, how about posting it instead in this secret facebook group.
That secret facebook group isn’t for one-on-one conversations but it’s still more private.
Private discussion is nearly as efficient as public discussion for information-transmission, but has way fewer political consequences.
If this is a categorical claim, then what are academic journals for? Should we ban the printing press?
If your claim is just that some public forums are too corrupted to be worth fixing, not a categorical claim, then the obvious thing to do is to figure out what went wrong, coordinate to move to an uncorrupted forum, and add the new thing to the set of things we filter out of our new walled garden.
I don’t believe that academic journals are an efficient form of information transmission. Academics support academic journals (when they support academic journals) because journals serve other useful purposes.
Often non-epistemic consequences of words are useful, and often they aren’t a big deal. I wouldn’t use the word “corrupted” to describe “having political consequences,” it’s the default state of human discussions.
Public discussion is sometimes much more efficient than private discussion. A central example is when the writer’s time is much more valuable than the reader’s time, or when it would be high-friction for the reader to buy off the writer’s time. (Though in this case, what’s occurring isn’t really discourse.) There are of course other examples.
Doing things like “writing down your thoughts carefully, and then reusing what you’ve written down” is important whether discussion occurs in public or private.
My intuition around whether some people are intrinsically bad (as opposed to bad at some things), is that it’s an artifact of systems of dominance like schools designed to create insecure attachment, and not a thing nonabused humans will think of on their own.
I think this would be valuable to work out eventually, but this probably isn’t the right time and place, and in the meantime I recognize that my position isn’t obviously true.
As far I remember, among the people (prison guards, psychiatrists) who work with… problematic humans the general consensus is that some small percentage of those they see (around 5% IIRC) are best described as irredeemably evil. Nothing works on them, they don’t become better with time or therapy or anything. There is no obvious cause either.
This seems completely false. Most people think that Hitler and Stalin were intrinsically bad, and they would be likely to think this with or without systems of dominance.
Kant and Thomas Aquinas explain it quite well: we call someone a “bad person” when we think they have bad will. And what does bad will mean? It means being willing to do bad things to bring about good things, rather than wanting to do good things period.
Do you think Nate’s claim was that we oughtn’t so often jump to the conclusion that people are willing to do bad things in order to bring about good things? That this is the accusation that’s burning the commons? I’m pretty sure many utilitarians would say that this is a fair description of their attitude at least in principle.
I would be a bit surprised if that was explicitly what Nate meant, but it is what we should be concerned about, in terms of being concerned about whether someone is a bad person.
To make my general claim clearer: “doing evil to bring about good, is still doing evil,” is necessarily true, for exactly the same reason that “blue objects touching white objects, are still blue objects,” is true.
I agree that many utilitarians understand their moral philosophy to recommend doing evil for the sake of good. To the extent that it does, their moral philosophy is mistaken. That does not necessarily mean that utilitarians are bad people, because you can be mistaken without being bad. But this is precisely the reason that when you present scenarios where you say, “would you be willing to do such and such a bad thing for the sake of good,” many utilitarians will reply, “No! That’s not the utilitarian thing to do!” And maybe it is the utilitarian thing, and maybe it isn’t. But the real reason they feel the impulse to say no, is that they are not bad people, and therefore they do not want to do bad things, even for the sake of good.
This also implies, however, that if someone understands utilitarianism in this way and takes it too seriously, they will indeed start down the road towards becoming a bad person. And that happened even in the context of the present discussion (understood more broadly to include its antecedents) when certain people insisted, saying in effect, “What’s so bad about lying and other deceitful tactics, as long as they advance my goals?”
I agree that most relevant bad behavior isn’t going to feel from the inside like an attempt to mislead, and I think that rationalists sometimes either ignore this or else have an unfounded optimism about nominal alignment.
In the evolutionary context, our utterances and conscious beliefs are optimized for their effects on others, and not merely for accuracy. Believing and claiming bad things about competitors is a typical strategy. Prima facie, accusations of bad faith are particularly attractive since they can be levied on sparse evidence yet are rationally compelling. Empirically, accusations of bad faith are particularly common.
This makes an interesting contrast with the content of the post. The feeling that some people are bad is a strong and central social intuition. Do you think you’ve risen to the standard of evidence you are requesting here? It seems to me that you are largely playing the same game people normally play, and then trying to avoid norms that regulate the game by disclaiming “I’m not playing the game.”
For the most part these procedural issues seem secondary to disputes about facts on the ground. But all else equal they’re a reason to prefer object-level questions to questions about intent, logical argument and empirical data to intuition, and private discussion to public discussion.
Nope! Good point.
Here’s a specific outcome I would like to avoid: ganging up on the individuals saying misleading things, replacing them with new individuals who have better track records, but doing nothing to alter the underlying incentives. That would be really bad. I think we actually have exceptionally high-integrity individuals in critical leadership positions now, in ways that make the problem of perceived incentive to distort much easier to solve than it might otherwise be.
I don’t actually know how not to play the same old game yet, but I am trying to construct a way.
I see you aiming to construct a way and making credible progress, but I worry that you’re trying to do to many things at once and are going to cause lasting damage by the time you figure it out.
Specifically, the “confidence game” framing of the previous post moved it from “making an earnest good faith effort to talk about things” to “the majority of the post’s content is making a status move”[1] (in particular in the context of your other recent posts, and is exacerbated by this one), and if I were using the framing of this current post I’d say both the previous post and this one have bad intent.
I don’t think that’s a good framing—I think it’s important that you (and folk at OpenPhil and at CEA) do not just have an internally positive narrative but are actually trying to do things that actually cache out to “help each other” (in a broad sense of “help each other”). But I’m worried that this will not remain the case much longer if you continue on your current trajectory.
A year ago, I was extremely impressed with the work you were doing and points you were making, and frustrated that those points were not having much impact.
My perception was “EA Has A Lying Problem” was an inflection point where a) yeah, people started actually paying attention to the class of criticism you’re doing, but the mechanism by which people started paying attention was by critics invoking rhetoric and courting controversy, which was approximately as bad as the problem it was trying to solve. (or at least, within an order of magnitude as bad)
[1] I realize there was a whole lot of other content of the Confidence Game post that was quite good. But, like, the confidence game part is the part I remember easily. Which is the problem.
Could you say more about which things you think I should be doing separately instead of together, and why?
Things I notice you doing:
Meta discussion of how to have conversations / high quality discourse / why this is important
Evaluating OpenPhil and CEA as institutions, in a manner that’s aiming to be evenhanded and fair
Making claims and discussing OpenPhil and CEA that seem pretty indistinguishable from “punishing them and building public animosity towards them.”
Because of #3, I think it’s a lot harder to have credibility when doing #1 or #2. I think there is now enough history with #3 (perceived or actual, whatever your intent), that if you want to be able to do #1 or #2 you need to signal pretty hard that you’re not doing #3 anymore, and specifically take actions aiming to rebuild trust. (And if you were doing #3 by accident, this includes figuring out why your process was outputting something that looked like 3)
I have thoughts about “what to do to cause OpenPhil and CEA to change their behavior” which’ll be a response to tristanm’s comment.
Um, this notion that publicly criticizing organizations such as OpenPhil and CEA amounts to unhelpfully “punishing them and building public animosity towards them”, and thus is per se something to be avoided, is exactly one of the glaring issues “EA has a Lying Problem” (specifically, the subsection Criticizing EA orgs is harmful to the movement) was about. Have we learned nothing since then?
I think I’m mostly going to have to retreat to “this is a very important conversation that I would very much like to have over skype but I think online text is not a good medium for it.”
But we’ve had this conversation online, when EA Has A Lying Problem was first posted. Some worthwhile points were raised that are quite close to your position here, such as the point that unrealistic standards of idealism/virtue, honesty and prompt response to criticism (that is, unrealistic for broadly any real-world institution) could undermine the very real progress that EA orgs are hopefully making, compared to most charitable organizations. This is very much true, but the supposed implication that any and all internal critiques are per se harmful simply doesn’t follow!
I wasn’t saying any and all critiques are harmful—the specific thing I was saying was “these are three things I see you doing right now, and I don’t think you can do all of those within a short timespan.”
Independently, I also think some-but-not-all of the specific critiques you are making are harmful, but that wasn’t the point I was making at the time.
The reason I’d much prefer to have the conversation in person is because by now the entire conversation is emotionally charged (at least for me, and it looks like for you), in a way that is counterproductive. Speaking only for myself, I know that in an in person conversation where I can read facial expressions, I can a) more easily maintain empathy throughout the process, b) as soon as I hit a point where either we disagree, or where the conversation is getting heated, it’s a lot easier to see that, step back and say “okay let’s stop drop and doublecrux.” (And, hopefully, often realize that something was a simple misunderstanding rather than a disagreement)
Online, there are two options at any interval: write out a short point, or write out a long point. If I write out a short point, it won’t actually address all the things I’m trying to point at. If I write a long point, at least one thing will probably be disagreed with or misunderstood, which will derail the whole post.
A) I think this is probably a good thing to do when an online conversation is accumulating drama and controversy.
B) Even if it’s not, I very much want to test it out and find out if it works.
Sometimes a just and accurate evaluation shows that someone’s not as great as they said they were. I’m not trying to be evenhanded in the sense of never drawing any conclusions, and I don’t see the value in that.
Overall, a lot of this feels to me like asking me to do more work, with no compensation, and no offers of concrete independent help, and putting the burden of making the interaction go well on the critic.
It would have been very, very helpful at that time to have public evidence that anyone at all agreed or at least thought that particular points I was making needed an answer. I’m getting that now, I wasn’t getting that then, so I find it hard to see the appeal in going back to a style that wasn’t working.
That was a blog post by Sarah Constantin. I am not Sarah Constantin. I wrote my own post in response and about the same things, which no one is bringing up here because no one remembers it. It got a bit of engagement at the time, but I think most of that was spillover from Sarah’s post.
If you want higher-quality discourse, you can engage more publicly with what you see as the higher-quality discourse. My older posts are still available to engage with on the public internet, and were written to raise points that would still be relevant in the future.
I agree that the “confidence game” framing, and particularly the comparison to a ponzi scheme seemed to me like surprisingly charged language, and not the kind of thing you would do if you wanted a productive dialogue with someone.
I’m not sure whether Benquo means for it to come across that way or not. (Pro: maybe he has in fact given up on direct communication with OpenPhil, and thinks his only method of influence is riling up their base. Con: maybe he just thought it was an apt metaphor and didn’t model it as a slap-in-the-face, like I did. Or maybe something else I’m missing.)
Just to add another datapoint, I read it as strongly hostile, more like aiming at delegitimizing the target in the eyes of others than at starting a constructive discussion with them.
If his goal is to actually convince EA organizations to change their behavior, then it could be argued that his rhetorical tactics are in fact likely to be the most effective way of actually achieving that. We should not underestimate the effectiveness of strategies that work by negative PR or by using rhetorical as opposed to strictly argumentative language. I would argue they actually have a pretty good track record of getting organizations to change, without completely destroying the organization (or an associated movement). Uber and United have probably just gone through some of the worst negative coverage it is possible to undergo, and yet the probability that either of them will be completely destroyed by that is almost negligible. On the other hand, the probability that they will improve due to the negative controversy is quite high by my estimation.
Noting the history of organizations that have been completely wiped out by scandal or controversy, it is usually the case that they failed to accomplish their primary goal (such as maximizing shareholder value), and typically in a catastrophic or permanent way that indicated almost beyond doubt that they would never be able to accomplish that goal. It is generally not enough that their leaders acted immorally or unethically (since they can usually be replaced), or that they fail at a subgoal (because subgoals tend to be easier to modify). And since EA is not a single organization, but is better understood as a movement, it is unlikely that the entire movement will be crippled by even a major controversy in one of its organizations. It’s really hard to destroy philosophies.
OpenPhil leadership stated that responding to criticisms and being transparent about their decision-making is a highly costly action to take. And I think it has been well-argued at this point (and not in a purely rhetorical way) that EA organizations are so strongly motivated against taking these actions (as judged by observation of their actions), that they may even occasionally act in the opposite direction. Therefore, if there exist convincing arguments that they are engaging in undesirable behavior, and given that we fairly well know that they are acting on strong incentives, then it follows that in order to change their behavior, they need to be strongly motivated in the other direction. It is not, in general, possible to modify an agent’s utility function by reasoning alone. All rational agents are instrumentally motivated to preserve their preferences and resist attempts at modification.
My argument is not that we need to resort to sensationalist tactics, but only that purely argumentative strategies that offer no negative cost to the organization in question are unlikely to be effective either. And additionally that actions that add this cost are unlikely to be so costly that they result in permanent or unrecoverable damage.
I agree that this is a big and complicated deal and “never resort to sensationalist tactics” isn’t a sufficient answer for reasons close to what you describe. I’m not sure what the answer is, but I’ve been thinking about ideas.
Basically, I think were automatically fail if we have no way to punish defectors, and we also automatically fail controversy/sensationalism-as-normally-practiced is our main tool of doing so.
I think the threat of sensationalist tactics needs to be real. But it needs to be more like Nuclear Deterrence than it is like tit-for-tat warfare.
We’ve seen where sensationalism/controversy leads—American journalism. It is a terrible race to the bottom of inducing as much outrage as you can. It is anti-epistemic, anti-instrumental, anti-everything. Once you start down the dark path, forever will it dominate your destiny.
I am very sympathetic to the fact that Ben tried NOT doing that, and it didn’t work.
Comments like this make me want to actually go nuclear, if I’m already not getting credit for avoiding doing so.
I haven’t really called anyone in the community names. I’ve worked hard to avoid singling people out, and instead tried to make the discussion about norms and actions, not persons. I haven’t tried to organize any material opposition to the interests of the organizations I’m criticizing. I haven’t talked to journalists about this. I haven’t made any efforts to widely publicize my criticisms outside of the community. I’ve been careful to bring up the good points as well as the bad of the people and institutions I’ve been criticizing.
I’d really, really like it if there were a way to get sincere constructive engagement with the tactics I’ve been using. They’re a much better fit for my personality than the other stuff. I’d like to save our community, not blow it up. But we are on a path towards enforcing norms to suppress information rather than disclose it, and if that keeps going, it’s simply going to destroy the relevant value.
(On a related note, I’m aware of exactly one individual who’s been accused of arguing in bad faith in the discourse around Nate’s post, and that individual is me.)
I’m not certain that there is, in fact, a nuclear option. Besides that, I believe there is still room for more of what you have been doing. In particular, I think there are a couple of important topics that have yet to be touched upon in depth, by anyone really.
The first is that the rationalist community has yet to be fully engaged with the conversation. I’ve been observing the level of activity here and on other blogs and see that the majority of conversation is conducted by a small number of people. In other locations, such as SSC, there is a high level of activity but there is also substantial overlap with groups not directly a part of the rationality community, and the conversations there aren’t typically on the topic of EA. Some of the more prominent people have not entered the conversation at all besides Nate. It would be nice if someone like Eliezer gave his two cents.
The second is that it has yet to be discussed in depth the fact that the rationality community and the EA community are, in fact, separate communities with differing goals and values that are more accurately said to have formed an alliance rather than actually merged into one group. They have different origin stories, different primary motivations, and have been focused on a different set of problems throughout their history. The re-focusing of EA towards AI safety occurred rather recently and I think that as their attention turned there, it become more obvious to the rationality community that there were significant differences in thought that were capable of causing conflict.
What I see as one of the main differences between the two cultures is that the rationality community is mostly concerned with accuracy of belief and methods of finding truth whereas the EA community is mostly concerned with being a real force for good in the world, achieved through utilitarian means. I think there is in fact a case to be made that we either need to find a way to reconcile these differences, or go our separate ways, but we certainly can’t pretend these differences don’t exist. One of my main problems with Nate’s post is that he appears to imply that there aren’t any genuine conflicts between the two communities, which I think is simply not the case. And I think this has caused some disappointing choices for MIRI in responding to criticisms. For example, it’s disappointing that MIRI has yet to publish a formal response to the critiques made by Open Phil. I think it’s basic PR 101 that if you’re going to link to or reference criticisms to your organization, you should be fully prepared to engage with each and every point made.
I think my overall point is that there is still room for you, and anyone else who wants to enter this conversation, to continue with the strategy you are currently using, because it does not seem to have fully permeated the rationality community. Some sort of critical mass of support has to be reached before progress can be made, I think.
I don’t have better description than “more like nuclear deterrence” for now, mulling it over.
I think Nate is absolutely correct to note that if we just retreat to the object level and give up on implied trust, we lose something very valuable. We can’t each evaluate everything from scratch. If we’re going to make intellectual progress together, we need to be able to justifiably trust that people aren’t just trying to get us to do things that make sense to them, aren’t even just telling us things that happen to be literally true, but are making an honest good-faith attempt to give us the most decision-relevant information.
Discussions about what’s in good faith also seem hard to avoid when discussing things like standards of evidence, and how to evaluate summaries from outsiders.
I agree with the first two items, but consider that the content of these private discussions, and whatever the conclusions that are being drawn from them, are probably only visible to the wider community in the form of the decisions being made at the highest levels. Therefore, how do you ensure that when these decisions are made, and the wider community is expected to support them, that there will not be disagreement or confusion? Especially since the reasoning behind them is probably highly complex.
Then this begs the question, what is the distribution of private / public discussion that is the most preferred? Certainly if all discussion was kept private, then the wider community (especially the EA community) would have no choice but to support decisions on faith alone. And at the other extreme, there is the high cost of writing and publishing documentation of reasoning, the risk of wide misunderstanding and confusion, and the difficulty associated with trying to retract or adjust statements that are no longer supported.
And if “private” discussion wasn’t constrained to just a small circle, but rather simply meant that you would have to communicate to each individual inquiry separately, than that may come at an even greater cost than that of simply publishing your thoughts openly, because it would require you to devote your attention and effort into multiple, possibly numerous individual discussions, that require modeling each person’s level of knowledge and understanding.
I essentially don’t think the answer is going to be as simple as “private” vs “public”, but I tend to err on the side of transparency, though this may reflect more of a value than a belief based on strong empirical data.
Why would these procedural issues be a reason to prefer private discourse?
a. Private discussion is nearly as efficient as public discussion for information-transmission, but has way fewer political consequences. On top of that, the political context is more collaborative between participants and so it is less epistemically destructive.
b. I really don’t want to try to use collectively-enforced norms to guide epistemology and don’t think there are many examples of this working out well (whereas there seem to be examples of avoiding epistemically destructive norms by moving into private).
Can you define more precisely what you mean by “private discussion?” If by that you mean that all discourse is constrained to one-on-one conversations where the contents are not available to anyone else, I don’t intuitively see how this would be less destructive and more collaborative. It seems to require that a lot of interactions must occur before every person is up to date on the collective group knowledge, and also that for each conversation there is a lossy compression going on—it’s difficult for each conversation to carry the contents of each person’s history of previous conversations.
On the other hand, if you’re advocating for information to be filtered when transmitted beyond the trusted group, but flows freely within the trusted group, I believe that is less complicated and more efficient and I would have fewer objections to that.
By “private discussion” I mean discussions amongst small groups, in contrast with discussions amongst large groups. Both of them occur constantly. I’ve claimed that in general political considerations cut in favor of having private discussions more often than you otherwise would, I didn’t mean to be making a bold claim.
I recently wanted to raise an issue with some possible controversy/politics in the main EA facebook group. Instead of approving the post I was told, “this post isn’t a good fit for the group, how about posting it instead in this secret facebook group.
That secret facebook group isn’t for one-on-one conversations but it’s still more private.
If this is a categorical claim, then what are academic journals for? Should we ban the printing press?
If your claim is just that some public forums are too corrupted to be worth fixing, not a categorical claim, then the obvious thing to do is to figure out what went wrong, coordinate to move to an uncorrupted forum, and add the new thing to the set of things we filter out of our new walled garden.
I don’t believe that academic journals are an efficient form of information transmission. Academics support academic journals (when they support academic journals) because journals serve other useful purposes.
Often non-epistemic consequences of words are useful, and often they aren’t a big deal. I wouldn’t use the word “corrupted” to describe “having political consequences,” it’s the default state of human discussions.
Public discussion is sometimes much more efficient than private discussion. A central example is when the writer’s time is much more valuable than the reader’s time, or when it would be high-friction for the reader to buy off the writer’s time. (Though in this case, what’s occurring isn’t really discourse.) There are of course other examples.
Doing things like “writing down your thoughts carefully, and then reusing what you’ve written down” is important whether discussion occurs in public or private.
My intuition around whether some people are intrinsically bad (as opposed to bad at some things), is that it’s an artifact of systems of dominance like schools designed to create insecure attachment, and not a thing nonabused humans will think of on their own.
I think this is very unlikely.
I think this would be valuable to work out eventually, but this probably isn’t the right time and place, and in the meantime I recognize that my position isn’t obviously true.
As far I remember, among the people (prison guards, psychiatrists) who work with… problematic humans the general consensus is that some small percentage of those they see (around 5% IIRC) are best described as irredeemably evil. Nothing works on them, they don’t become better with time or therapy or anything. There is no obvious cause either.
This is from memory, sorry, no links.
This seems completely false. Most people think that Hitler and Stalin were intrinsically bad, and they would be likely to think this with or without systems of dominance.
Kant and Thomas Aquinas explain it quite well: we call someone a “bad person” when we think they have bad will. And what does bad will mean? It means being willing to do bad things to bring about good things, rather than wanting to do good things period.
Do you think Nate’s claim was that we oughtn’t so often jump to the conclusion that people are willing to do bad things in order to bring about good things? That this is the accusation that’s burning the commons? I’m pretty sure many utilitarians would say that this is a fair description of their attitude at least in principle.
I would be a bit surprised if that was explicitly what Nate meant, but it is what we should be concerned about, in terms of being concerned about whether someone is a bad person.
To make my general claim clearer: “doing evil to bring about good, is still doing evil,” is necessarily true, for exactly the same reason that “blue objects touching white objects, are still blue objects,” is true.
I agree that many utilitarians understand their moral philosophy to recommend doing evil for the sake of good. To the extent that it does, their moral philosophy is mistaken. That does not necessarily mean that utilitarians are bad people, because you can be mistaken without being bad. But this is precisely the reason that when you present scenarios where you say, “would you be willing to do such and such a bad thing for the sake of good,” many utilitarians will reply, “No! That’s not the utilitarian thing to do!” And maybe it is the utilitarian thing, and maybe it isn’t. But the real reason they feel the impulse to say no, is that they are not bad people, and therefore they do not want to do bad things, even for the sake of good.
This also implies, however, that if someone understands utilitarianism in this way and takes it too seriously, they will indeed start down the road towards becoming a bad person. And that happened even in the context of the present discussion (understood more broadly to include its antecedents) when certain people insisted, saying in effect, “What’s so bad about lying and other deceitful tactics, as long as they advance my goals?”
I agree that this exists, and claim that it ought to be legitimate discourse to claim that someone else is doing it.