The feeling that some people are bad is a strong and central social intuition. Do you think you’ve risen to the standard of evidence you are requesting here?
Nope! Good point.
It seems to me that you are largely playing the same game people normally play, and then trying to avoid norms that regulate the game by disclaiming “I’m not playing the game.”
Here’s a specific outcome I would like to avoid: ganging up on the individuals saying misleading things, replacing them with new individuals who have better track records, but doing nothing to alter the underlying incentives. That would be really bad. I think we actually have exceptionally high-integrity individuals in critical leadership positions now, in ways that make the problem of perceived incentive to distort much easier to solve than it might otherwise be.
I don’t actually know how not to play the same old game yet, but I am trying to construct a way.
I don’t actually know how not to play the same old game yet, but I am trying to construct a way.
I see you aiming to construct a way and making credible progress, but I worry that you’re trying to do to many things at once and are going to cause lasting damage by the time you figure it out.
Specifically, the “confidence game” framing of the previous post moved it from “making an earnest good faith effort to talk about things” to “the majority of the post’s content is making a status move”[1] (in particular in the context of your other recent posts, and is exacerbated by this one), and if I were using the framing of this current post I’d say both the previous post and this one have bad intent.
I don’t think that’s a good framing—I think it’s important that you (and folk at OpenPhil and at CEA) do not just have an internally positive narrative but are actually trying to do things that actually cache out to “help each other” (in a broad sense of “help each other”). But I’m worried that this will not remain the case much longer if you continue on your current trajectory.
A year ago, I was extremely impressed with the work you were doing and points you were making, and frustrated that those points were not having much impact.
My perception was “EA Has A Lying Problem” was an inflection point where a) yeah, people started actually paying attention to the class of criticism you’re doing, but the mechanism by which people started paying attention was by critics invoking rhetoric and courting controversy, which was approximately as bad as the problem it was trying to solve. (or at least, within an order of magnitude as bad)
[1] I realize there was a whole lot of other content of the Confidence Game post that was quite good. But, like, the confidence game part is the part I remember easily. Which is the problem.
Meta discussion of how to have conversations / high quality discourse / why this is important
Evaluating OpenPhil and CEA as institutions, in a manner that’s aiming to be evenhanded and fair
Making claims and discussing OpenPhil and CEA that seem pretty indistinguishable from “punishing them and building public animosity towards them.”
Because of #3, I think it’s a lot harder to have credibility when doing #1 or #2. I think there is now enough history with #3 (perceived or actual, whatever your intent), that if you want to be able to do #1 or #2 you need to signal pretty hard that you’re not doing #3 anymore, and specifically take actions aiming to rebuild trust. (And if you were doing #3 by accident, this includes figuring out why your process was outputting something that looked like 3)
I have thoughts about “what to do to cause OpenPhil and CEA to change their behavior” which’ll be a response to tristanm’s comment.
Making claims and discussing OpenPhil and CEA that seem pretty indistinguishable from “punishing them and building public animosity towards them.”
Um, this notion that publicly criticizing organizations such as OpenPhil and CEA amounts to unhelpfully “punishing them and building public animosity towards them”, and thus is per se something to be avoided, is exactly one of the glaring issues “EA has a Lying Problem” (specifically, the subsection Criticizing EA orgs is harmful to the movement) was about. Have we learned nothing since then?
I think I’m mostly going to have to retreat to “this is a very important conversation that I would very much like to have over skype but I think online text is not a good medium for it.”
But we’ve had this conversation online, when EA Has A Lying Problem was first posted. Some worthwhile points were raised that are quite close to your position here, such as the point that unrealistic standards of idealism/virtue, honesty and prompt response to criticism (that is, unrealistic for broadly any real-world institution) could undermine the very real progress that EA orgs are hopefully making, compared to most charitable organizations. This is very much true, but the supposed implication that any and all internal critiques are per se harmful simply doesn’t follow!
I wasn’t saying any and all critiques are harmful—the specific thing I was saying was “these are three things I see you doing right now, and I don’t think you can do all of those within a short timespan.”
Independently, I also think some-but-not-all of the specific critiques you are making are harmful, but that wasn’t the point I was making at the time.
The reason I’d much prefer to have the conversation in person is because by now the entire conversation is emotionally charged (at least for me, and it looks like for you), in a way that is counterproductive. Speaking only for myself, I know that in an in person conversation where I can read facial expressions, I can a) more easily maintain empathy throughout the process, b) as soon as I hit a point where either we disagree, or where the conversation is getting heated, it’s a lot easier to see that, step back and say “okay let’s stop drop and doublecrux.” (And, hopefully, often realize that something was a simple misunderstanding rather than a disagreement)
Online, there are two options at any interval: write out a short point, or write out a long point. If I write out a short point, it won’t actually address all the things I’m trying to point at. If I write a long point, at least one thing will probably be disagreed with or misunderstood, which will derail the whole post.
A) I think this is probably a good thing to do when an online conversation is accumulating drama and controversy.
B) Even if it’s not, I very much want to test it out and find out if it works.
Sometimes a just and accurate evaluation shows that someone’s not as great as they said they were. I’m not trying to be evenhanded in the sense of never drawing any conclusions, and I don’t see the value in that.
Overall, a lot of this feels to me like asking me to do more work, with no compensation, and no offers of concrete independent help, and putting the burden of making the interaction go well on the critic.
A year ago, I was extremely impressed with the work you were doing and points you were making, and frustrated that those points were not having much impact.
It would have been very, very helpful at that time to have public evidence that anyone at all agreed or at least thought that particular points I was making needed an answer. I’m getting that now, I wasn’t getting that then, so I find it hard to see the appeal in going back to a style that wasn’t working.
My perception was “EA Has A Lying Problem” was an inflection point where a) yeah, people started actually paying attention to the class of criticism you’re doing, but the mechanism by which people started paying attention was by critics invoking rhetoric and courting controversy, which was approximately as bad as the problem it was trying to solve. (or at least, within an order of magnitude as bad)
That was a blog post by Sarah Constantin. I am not Sarah Constantin. I wrote my own post in response and about the same things, which no one is bringing up here because no one remembers it. It got a bit of engagement at the time, but I think most of that was spillover from Sarah’s post.
If you want higher-quality discourse, you can engage more publicly with what you see as the higher-quality discourse. My older posts are still available to engage with on the public internet, and were written to raise points that would still be relevant in the future.
I agree that the “confidence game” framing, and particularly the comparison to a ponzi scheme seemed to me like surprisingly charged language, and not the kind of thing you would do if you wanted a productive dialogue with someone.
I’m not sure whether Benquo means for it to come across that way or not. (Pro: maybe he has in fact given up on direct communication with OpenPhil, and thinks his only method of influence is riling up their base. Con: maybe he just thought it was an apt metaphor and didn’t model it as a slap-in-the-face, like I did. Or maybe something else I’m missing.)
Just to add another datapoint, I read it as strongly hostile, more like aiming at delegitimizing the target in the eyes of others than at starting a constructive discussion with them.
If his goal is to actually convince EA organizations to change their behavior, then it could be argued that his rhetorical tactics are in fact likely to be the most effective way of actually achieving that. We should not underestimate the effectiveness of strategies that work by negative PR or by using rhetorical as opposed to strictly argumentative language. I would argue they actually have a pretty good track record of getting organizations to change, without completely destroying the organization (or an associated movement). Uber and United have probably just gone through some of the worst negative coverage it is possible to undergo, and yet the probability that either of them will be completely destroyed by that is almost negligible. On the other hand, the probability that they will improve due to the negative controversy is quite high by my estimation.
Noting the history of organizations that have been completely wiped out by scandal or controversy, it is usually the case that they failed to accomplish their primary goal (such as maximizing shareholder value), and typically in a catastrophic or permanent way that indicated almost beyond doubt that they would never be able to accomplish that goal. It is generally not enough that their leaders acted immorally or unethically (since they can usually be replaced), or that they fail at a subgoal (because subgoals tend to be easier to modify). And since EA is not a single organization, but is better understood as a movement, it is unlikely that the entire movement will be crippled by even a major controversy in one of its organizations. It’s really hard to destroy philosophies.
OpenPhil leadership stated that responding to criticisms and being transparent about their decision-making is a highly costly action to take. And I think it has been well-argued at this point (and not in a purely rhetorical way) that EA organizations are so strongly motivated against taking these actions (as judged by observation of their actions), that they may even occasionally act in the opposite direction. Therefore, if there exist convincing arguments that they are engaging in undesirable behavior, and given that we fairly well know that they are acting on strong incentives, then it follows that in order to change their behavior, they need to be strongly motivated in the other direction. It is not, in general, possible to modify an agent’s utility function by reasoning alone. All rational agents are instrumentally motivated to preserve their preferences and resist attempts at modification.
My argument is not that we need to resort to sensationalist tactics, but only that purely argumentative strategies that offer no negative cost to the organization in question are unlikely to be effective either. And additionally that actions that add this cost are unlikely to be so costly that they result in permanent or unrecoverable damage.
I agree that this is a big and complicated deal and “never resort to sensationalist tactics” isn’t a sufficient answer for reasons close to what you describe. I’m not sure what the answer is, but I’ve been thinking about ideas.
Basically, I think were automatically fail if we have no way to punish defectors, and we also automatically fail controversy/sensationalism-as-normally-practiced is our main tool of doing so.
I think the threat of sensationalist tactics needs to be real. But it needs to be more like Nuclear Deterrence than it is like tit-for-tat warfare.
We’ve seen where sensationalism/controversy leads—American journalism. It is a terrible race to the bottom of inducing as much outrage as you can. It is anti-epistemic, anti-instrumental, anti-everything. Once you start down the dark path, forever will it dominate your destiny.
I am very sympathetic to the fact that Ben tried NOT doing that, and it didn’t work.
Comments like this make me want to actually go nuclear, if I’m already not getting credit for avoiding doing so.
I haven’t really called anyone in the community names. I’ve worked hard to avoid singling people out, and instead tried to make the discussion about norms and actions, not persons. I haven’t tried to organize any material opposition to the interests of the organizations I’m criticizing. I haven’t talked to journalists about this. I haven’t made any efforts to widely publicize my criticisms outside of the community. I’ve been careful to bring up the good points as well as the bad of the people and institutions I’ve been criticizing.
I’d really, really like it if there were a way to get sincere constructive engagement with the tactics I’ve been using. They’re a much better fit for my personality than the other stuff. I’d like to save our community, not blow it up. But we are on a path towards enforcing norms to suppress information rather than disclose it, and if that keeps going, it’s simply going to destroy the relevant value.
(On a related note, I’m aware of exactly one individual who’s been accused of arguing in bad faith in the discourse around Nate’s post, and that individual is me.)
I’m not certain that there is, in fact, a nuclear option. Besides that, I believe there is still room for more of what you have been doing. In particular, I think there are a couple of important topics that have yet to be touched upon in depth, by anyone really.
The first is that the rationalist community has yet to be fully engaged with the conversation. I’ve been observing the level of activity here and on other blogs and see that the majority of conversation is conducted by a small number of people. In other locations, such as SSC, there is a high level of activity but there is also substantial overlap with groups not directly a part of the rationality community, and the conversations there aren’t typically on the topic of EA. Some of the more prominent people have not entered the conversation at all besides Nate. It would be nice if someone like Eliezer gave his two cents.
The second is that it has yet to be discussed in depth the fact that the rationality community and the EA community are, in fact, separate communities with differing goals and values that are more accurately said to have formed an alliance rather than actually merged into one group. They have different origin stories, different primary motivations, and have been focused on a different set of problems throughout their history. The re-focusing of EA towards AI safety occurred rather recently and I think that as their attention turned there, it become more obvious to the rationality community that there were significant differences in thought that were capable of causing conflict.
What I see as one of the main differences between the two cultures is that the rationality community is mostly concerned with accuracy of belief and methods of finding truth whereas the EA community is mostly concerned with being a real force for good in the world, achieved through utilitarian means. I think there is in fact a case to be made that we either need to find a way to reconcile these differences, or go our separate ways, but we certainly can’t pretend these differences don’t exist. One of my main problems with Nate’s post is that he appears to imply that there aren’t any genuine conflicts between the two communities, which I think is simply not the case. And I think this has caused some disappointing choices for MIRI in responding to criticisms. For example, it’s disappointing that MIRI has yet to publish a formal response to the critiques made by Open Phil. I think it’s basic PR 101 that if you’re going to link to or reference criticisms to your organization, you should be fully prepared to engage with each and every point made.
I think my overall point is that there is still room for you, and anyone else who wants to enter this conversation, to continue with the strategy you are currently using, because it does not seem to have fully permeated the rationality community. Some sort of critical mass of support has to be reached before progress can be made, I think.
Nope! Good point.
Here’s a specific outcome I would like to avoid: ganging up on the individuals saying misleading things, replacing them with new individuals who have better track records, but doing nothing to alter the underlying incentives. That would be really bad. I think we actually have exceptionally high-integrity individuals in critical leadership positions now, in ways that make the problem of perceived incentive to distort much easier to solve than it might otherwise be.
I don’t actually know how not to play the same old game yet, but I am trying to construct a way.
I see you aiming to construct a way and making credible progress, but I worry that you’re trying to do to many things at once and are going to cause lasting damage by the time you figure it out.
Specifically, the “confidence game” framing of the previous post moved it from “making an earnest good faith effort to talk about things” to “the majority of the post’s content is making a status move”[1] (in particular in the context of your other recent posts, and is exacerbated by this one), and if I were using the framing of this current post I’d say both the previous post and this one have bad intent.
I don’t think that’s a good framing—I think it’s important that you (and folk at OpenPhil and at CEA) do not just have an internally positive narrative but are actually trying to do things that actually cache out to “help each other” (in a broad sense of “help each other”). But I’m worried that this will not remain the case much longer if you continue on your current trajectory.
A year ago, I was extremely impressed with the work you were doing and points you were making, and frustrated that those points were not having much impact.
My perception was “EA Has A Lying Problem” was an inflection point where a) yeah, people started actually paying attention to the class of criticism you’re doing, but the mechanism by which people started paying attention was by critics invoking rhetoric and courting controversy, which was approximately as bad as the problem it was trying to solve. (or at least, within an order of magnitude as bad)
[1] I realize there was a whole lot of other content of the Confidence Game post that was quite good. But, like, the confidence game part is the part I remember easily. Which is the problem.
Could you say more about which things you think I should be doing separately instead of together, and why?
Things I notice you doing:
Meta discussion of how to have conversations / high quality discourse / why this is important
Evaluating OpenPhil and CEA as institutions, in a manner that’s aiming to be evenhanded and fair
Making claims and discussing OpenPhil and CEA that seem pretty indistinguishable from “punishing them and building public animosity towards them.”
Because of #3, I think it’s a lot harder to have credibility when doing #1 or #2. I think there is now enough history with #3 (perceived or actual, whatever your intent), that if you want to be able to do #1 or #2 you need to signal pretty hard that you’re not doing #3 anymore, and specifically take actions aiming to rebuild trust. (And if you were doing #3 by accident, this includes figuring out why your process was outputting something that looked like 3)
I have thoughts about “what to do to cause OpenPhil and CEA to change their behavior” which’ll be a response to tristanm’s comment.
Um, this notion that publicly criticizing organizations such as OpenPhil and CEA amounts to unhelpfully “punishing them and building public animosity towards them”, and thus is per se something to be avoided, is exactly one of the glaring issues “EA has a Lying Problem” (specifically, the subsection Criticizing EA orgs is harmful to the movement) was about. Have we learned nothing since then?
I think I’m mostly going to have to retreat to “this is a very important conversation that I would very much like to have over skype but I think online text is not a good medium for it.”
But we’ve had this conversation online, when EA Has A Lying Problem was first posted. Some worthwhile points were raised that are quite close to your position here, such as the point that unrealistic standards of idealism/virtue, honesty and prompt response to criticism (that is, unrealistic for broadly any real-world institution) could undermine the very real progress that EA orgs are hopefully making, compared to most charitable organizations. This is very much true, but the supposed implication that any and all internal critiques are per se harmful simply doesn’t follow!
I wasn’t saying any and all critiques are harmful—the specific thing I was saying was “these are three things I see you doing right now, and I don’t think you can do all of those within a short timespan.”
Independently, I also think some-but-not-all of the specific critiques you are making are harmful, but that wasn’t the point I was making at the time.
The reason I’d much prefer to have the conversation in person is because by now the entire conversation is emotionally charged (at least for me, and it looks like for you), in a way that is counterproductive. Speaking only for myself, I know that in an in person conversation where I can read facial expressions, I can a) more easily maintain empathy throughout the process, b) as soon as I hit a point where either we disagree, or where the conversation is getting heated, it’s a lot easier to see that, step back and say “okay let’s stop drop and doublecrux.” (And, hopefully, often realize that something was a simple misunderstanding rather than a disagreement)
Online, there are two options at any interval: write out a short point, or write out a long point. If I write out a short point, it won’t actually address all the things I’m trying to point at. If I write a long point, at least one thing will probably be disagreed with or misunderstood, which will derail the whole post.
A) I think this is probably a good thing to do when an online conversation is accumulating drama and controversy.
B) Even if it’s not, I very much want to test it out and find out if it works.
Sometimes a just and accurate evaluation shows that someone’s not as great as they said they were. I’m not trying to be evenhanded in the sense of never drawing any conclusions, and I don’t see the value in that.
Overall, a lot of this feels to me like asking me to do more work, with no compensation, and no offers of concrete independent help, and putting the burden of making the interaction go well on the critic.
It would have been very, very helpful at that time to have public evidence that anyone at all agreed or at least thought that particular points I was making needed an answer. I’m getting that now, I wasn’t getting that then, so I find it hard to see the appeal in going back to a style that wasn’t working.
That was a blog post by Sarah Constantin. I am not Sarah Constantin. I wrote my own post in response and about the same things, which no one is bringing up here because no one remembers it. It got a bit of engagement at the time, but I think most of that was spillover from Sarah’s post.
If you want higher-quality discourse, you can engage more publicly with what you see as the higher-quality discourse. My older posts are still available to engage with on the public internet, and were written to raise points that would still be relevant in the future.
I agree that the “confidence game” framing, and particularly the comparison to a ponzi scheme seemed to me like surprisingly charged language, and not the kind of thing you would do if you wanted a productive dialogue with someone.
I’m not sure whether Benquo means for it to come across that way or not. (Pro: maybe he has in fact given up on direct communication with OpenPhil, and thinks his only method of influence is riling up their base. Con: maybe he just thought it was an apt metaphor and didn’t model it as a slap-in-the-face, like I did. Or maybe something else I’m missing.)
Just to add another datapoint, I read it as strongly hostile, more like aiming at delegitimizing the target in the eyes of others than at starting a constructive discussion with them.
If his goal is to actually convince EA organizations to change their behavior, then it could be argued that his rhetorical tactics are in fact likely to be the most effective way of actually achieving that. We should not underestimate the effectiveness of strategies that work by negative PR or by using rhetorical as opposed to strictly argumentative language. I would argue they actually have a pretty good track record of getting organizations to change, without completely destroying the organization (or an associated movement). Uber and United have probably just gone through some of the worst negative coverage it is possible to undergo, and yet the probability that either of them will be completely destroyed by that is almost negligible. On the other hand, the probability that they will improve due to the negative controversy is quite high by my estimation.
Noting the history of organizations that have been completely wiped out by scandal or controversy, it is usually the case that they failed to accomplish their primary goal (such as maximizing shareholder value), and typically in a catastrophic or permanent way that indicated almost beyond doubt that they would never be able to accomplish that goal. It is generally not enough that their leaders acted immorally or unethically (since they can usually be replaced), or that they fail at a subgoal (because subgoals tend to be easier to modify). And since EA is not a single organization, but is better understood as a movement, it is unlikely that the entire movement will be crippled by even a major controversy in one of its organizations. It’s really hard to destroy philosophies.
OpenPhil leadership stated that responding to criticisms and being transparent about their decision-making is a highly costly action to take. And I think it has been well-argued at this point (and not in a purely rhetorical way) that EA organizations are so strongly motivated against taking these actions (as judged by observation of their actions), that they may even occasionally act in the opposite direction. Therefore, if there exist convincing arguments that they are engaging in undesirable behavior, and given that we fairly well know that they are acting on strong incentives, then it follows that in order to change their behavior, they need to be strongly motivated in the other direction. It is not, in general, possible to modify an agent’s utility function by reasoning alone. All rational agents are instrumentally motivated to preserve their preferences and resist attempts at modification.
My argument is not that we need to resort to sensationalist tactics, but only that purely argumentative strategies that offer no negative cost to the organization in question are unlikely to be effective either. And additionally that actions that add this cost are unlikely to be so costly that they result in permanent or unrecoverable damage.
I agree that this is a big and complicated deal and “never resort to sensationalist tactics” isn’t a sufficient answer for reasons close to what you describe. I’m not sure what the answer is, but I’ve been thinking about ideas.
Basically, I think were automatically fail if we have no way to punish defectors, and we also automatically fail controversy/sensationalism-as-normally-practiced is our main tool of doing so.
I think the threat of sensationalist tactics needs to be real. But it needs to be more like Nuclear Deterrence than it is like tit-for-tat warfare.
We’ve seen where sensationalism/controversy leads—American journalism. It is a terrible race to the bottom of inducing as much outrage as you can. It is anti-epistemic, anti-instrumental, anti-everything. Once you start down the dark path, forever will it dominate your destiny.
I am very sympathetic to the fact that Ben tried NOT doing that, and it didn’t work.
Comments like this make me want to actually go nuclear, if I’m already not getting credit for avoiding doing so.
I haven’t really called anyone in the community names. I’ve worked hard to avoid singling people out, and instead tried to make the discussion about norms and actions, not persons. I haven’t tried to organize any material opposition to the interests of the organizations I’m criticizing. I haven’t talked to journalists about this. I haven’t made any efforts to widely publicize my criticisms outside of the community. I’ve been careful to bring up the good points as well as the bad of the people and institutions I’ve been criticizing.
I’d really, really like it if there were a way to get sincere constructive engagement with the tactics I’ve been using. They’re a much better fit for my personality than the other stuff. I’d like to save our community, not blow it up. But we are on a path towards enforcing norms to suppress information rather than disclose it, and if that keeps going, it’s simply going to destroy the relevant value.
(On a related note, I’m aware of exactly one individual who’s been accused of arguing in bad faith in the discourse around Nate’s post, and that individual is me.)
I’m not certain that there is, in fact, a nuclear option. Besides that, I believe there is still room for more of what you have been doing. In particular, I think there are a couple of important topics that have yet to be touched upon in depth, by anyone really.
The first is that the rationalist community has yet to be fully engaged with the conversation. I’ve been observing the level of activity here and on other blogs and see that the majority of conversation is conducted by a small number of people. In other locations, such as SSC, there is a high level of activity but there is also substantial overlap with groups not directly a part of the rationality community, and the conversations there aren’t typically on the topic of EA. Some of the more prominent people have not entered the conversation at all besides Nate. It would be nice if someone like Eliezer gave his two cents.
The second is that it has yet to be discussed in depth the fact that the rationality community and the EA community are, in fact, separate communities with differing goals and values that are more accurately said to have formed an alliance rather than actually merged into one group. They have different origin stories, different primary motivations, and have been focused on a different set of problems throughout their history. The re-focusing of EA towards AI safety occurred rather recently and I think that as their attention turned there, it become more obvious to the rationality community that there were significant differences in thought that were capable of causing conflict.
What I see as one of the main differences between the two cultures is that the rationality community is mostly concerned with accuracy of belief and methods of finding truth whereas the EA community is mostly concerned with being a real force for good in the world, achieved through utilitarian means. I think there is in fact a case to be made that we either need to find a way to reconcile these differences, or go our separate ways, but we certainly can’t pretend these differences don’t exist. One of my main problems with Nate’s post is that he appears to imply that there aren’t any genuine conflicts between the two communities, which I think is simply not the case. And I think this has caused some disappointing choices for MIRI in responding to criticisms. For example, it’s disappointing that MIRI has yet to publish a formal response to the critiques made by Open Phil. I think it’s basic PR 101 that if you’re going to link to or reference criticisms to your organization, you should be fully prepared to engage with each and every point made.
I think my overall point is that there is still room for you, and anyone else who wants to enter this conversation, to continue with the strategy you are currently using, because it does not seem to have fully permeated the rationality community. Some sort of critical mass of support has to be reached before progress can be made, I think.
I don’t have better description than “more like nuclear deterrence” for now, mulling it over.