Full disclosure: I am a MIRI Research Associate. This means that I receive funding from MIRI, but I am not a MIRI employee and I am not privy to its internal operation or secrets.
First of all, I am really sorry you had these horrible experiences.
A few thoughts:
Thought 1: I am not convinced the analogy between Leverage and MIRI/CFAR holds up to scrutiny. I think that Geoff Anders is most likely a bad actor, whereas MIRI/CFAR leadership is probably acting in good faith. There seems to be significantly more evidence of bad faith in Zoe’s account than in Jessica’s account, and the conclusion is reinforced by adding evidence from other accounts. In addition, MIRI definitely produced some valuable public research whereas I doubt the same can be said of Leverage, although I haven’t been following Leverage so I am not confident about the latter (ofc it’s in principle possible for a deeply unhealthy organization to produce some good outputs, and good outputs certainly don’t excuse abuse of personnel, but I do think good outputs provide some evidence against such abuse).
It is important not to commit the fallacy of gray: it would risk both judging MIRI/CFAR too harshly and judging Leverage insufficiently harshly. The comparison Jessica makes to “normal corporations” reinforces this impression: I have much experience in the industry, and although it’s possible I’ve been lucky in some ways, I still very much doubt the typical company is nearly as bad as Leverage.
Thought 2: From my experience, AI alignment is a domain of research that intrinsically comes with mental health hazards. First, the possibility of impending doom and the heavy sense of responsibility are sources of stress. Second, research inquiries often enough lead to “weird” metaphysical questions that risk overturning the (justified or unjustified) assumptions we implicitly hold to maintain a sense of safety in life. I think it might be the closest thing in real life to the Lovecraftian notion of “things that are best not to know because they will drive you mad”. Third, the sort of people drawn to the area and/or having the necessary talents seem to often also come with mental health issues (I am including myself in this group).
This might be regarded as an argument to blame MIRI less for the mental health fallout described by Jessica, but this is also an argument to pay more attention to the problem. It would be best if we could provide the people working in the area with the tools and environment to deal with these risks.
Thought 3: The part that concerned me the most in Jessica’s account (in part due to its novelty to me) is MIRI’s internal secrecy policy. While it might be justifiable to have some secrets to which only some employees are privy, it seems very extreme to require going through an executive because even the mere fact that a secret project exists is too dangerous. MIRI’s secrecy policy seemed questionable to me even before, but this new spin makes it even more dubious.
Overall, I wish MIRI was more transparent, so that for example its supporters would know about this internal policy. I realize there are tradeoffs involved, but I am not convinced MIRI chose the right balance. To me it feels like overconfidence about MIRI’s ability to steer the right way without the help of external critique.
Moreover, I’m a little worried that MIRI’s lack of transparency might pose a risk for the entire AI safety project. Tbh, one of my first thoughts when I saw the headline of the OP was “oh no, what if some scandal around MIRI blows up and the shockwave buries the entire community”. And I guess some people might think this is a reason for more secrecy. IMO it’s a reason for less secrecy (not necessarily less secrecy about technical AI stuff, but less secrecy about management and high-level plans). If we don’t have any skeletons in the closest, we don’t need to worry about the day they will come out. And eventually everything comes out, more or less. When most of everything is in the open, the community can find the right balance around it, and the reputation system is much more robust.
Thought 4: “Someone in the community told me that for me to think AGI probably won’t be developed soon, I must think I’m better at meta-rationality than Eliezer Yudkowsky, a massive claim of my own specialness.” I think (hope?) this is not at all a prevalent stance in the community (or at least in its leading echelons), but just for the record I want to note my strong position that the “someone” in this story is very misguided. Like I said, I don’t think community is currently comparable to Leverage, but this is the sort of thing that can push us in that direction.
Plus a million points for “IMO it’s a reason for less secrecy”!
If you put a lid on something you might contain it in the short term, but only at the cost of increasing the pressure: And pressure wants out, and the higher the pressure the more explosive it will be when it inevitably does come out.
I have heard too many accounts like this, in person and anecdotally, on the web and off for me to currently be interested in working or even getting to closely involved with any of the organizations in question. The only way to change this for me is to believably cultivate a healthy, transparent and supportive environment.
This made me go back and read “Every Cause wants to be a Cult” (Eliezer, 2007), which includes quotes like this one: ”Here I just want to point out that the worthiness of the Cause does not mean you can spend any less effort in resisting the cult attractor. And that if you can point to current battle lines, it does not mean you confess your Noble Cause unworthy. You might think that if the question were, “Cultish, yes or no?” that you were obliged to answer, “No,” or else betray your beloved Cause.”
Thought 2: From my experience, AI alignment is a domain of research that intrinsically comes with mental health hazards. First, the possibility of impending doom and the heavy sense of responsibility are sources of stress. Second, research inquiries often enough lead to “weird” metaphysical questions that risk overturning the (justified or unjustified) assumptions we implicitly hold to maintain a sense of safety in life. I think it might be the closest thing in real life to the Lovecraftian notion of “things that are best not to know because they will drive you mad”. Third, the sort of people drawn to the area and/or having the necessary talents seem to often also come with mental health issues (I am including myself in this group).
That sounds like MIRI should have a councillor on it’s staff.
That would make them more vulnerable to claims that they use organizational mind control on their employees, and at the same time make it more likely that they would actually use it.
You would likely hire someone who’s traditionally trained, credentialed and has work experience instead of doing a bunch of your own psych-experiments, likely in a tradition like gestalttherapy that focuses on being nonmanipulative.
There’s an easier solution that doesn’t run the risk of being or appearing manipulative. You can contract external and independent councillors and make them available to your staff anonymously. I don’t know if there’s anything comparable in the US, but in Australia they’re referred to as Employee Assistance Programs (EAPs). Nothing you discuss with the councillor can be disclosed to your workplace, although in rare circumstances there may be mandatory reporting to the police (e.g. if abuse or ongoing risk of a minor is involved).
This also goes a long way toward creating a place where employees can talk about things they’re worried will seem crazy in work contexts.
Solutions like that might work, but it’s worth noting that just having an average therapist likely won’t be enough.
If you actually care about a level of security that protects secrets against intelligence agencies, operational security of the office of the therapist is a concern.
Governments that have security clearances don’t want their employees to talk with therapists who don’t have the secuirty clearances about classified information.
Talking nonjudgmentally with someone who has reasonable fears that the humanity won’t survive the next ten years because of fast AI timelines is not easy.
As far as I can tell, normal corporate management is much worse than Leverage. The kind of people from that world will, sometimes when prompted in private conversations, say things like:
Standard practice is to treat negotiations with other parties as zero-sum games.
“If you look around the table and can’t tell who the sucker is, it’s you” is a description of a common, relevant social dynamic in corporate meetings.
They have PTSD symptoms from working in corporate management, and are very threat-sensitive in general.
They learned from experience to treat social reality in general as fake, everything as an act.
They learned to accept that “there’s no such thing as not being lost”, like they’ve lost the ability to self-locate in a global map (I’ve experienced losing this to a significant extent).
Successful organizations get to be where they are by committing crimes, so copying standard practices from them is copying practices for committing crimes.
This is, to a large extent, them admitting to being bad actors, them and others having been made so by their social context. (This puts the possibility of “Geoff Anders being a bad actor” into perspective)
MIRI is, despite the problems noted in the post, as far as I can tell the most high-integrity organization doing AI safety research. FHI contributes some, but overall lower-quality research; Paul Christiano does some relevant research; OpenAI’s original mission was actively harmful, and hasn’t done much relevant safety research as far as I can tell. MIRI’s public output in the past few years since I left has been low, which seems like bad sign for its future performance, but what it’s done so far has been quite a large portion of the relevant research. I’m not particularly worried about scandals sinking the overall non-MIRI AI safety world’s reputation, given the degree to which it is of mixed value.
As far as I can tell, normal corporate management is much worse than Leverage
Your original post drew a comparison between MIRI and Leverage, the latter of which has just been singled out for intense criticism.
If I take the quoted sentence literally, you’re saying that “MIRI was like Leverage” is a gentler critique than “MIRI is like your regular job”?
If the intended message was “my job was bad, although less bad than the jobs of many people reading this, and instead only about as bad as Leverage Research,” why release this criticism on the heels of a post condemning Leverage as an abusive cult? If you believe the normally-employed among LessWrong readers are being abused by sub-Leverage hellcults, all the time, that seems like quite the buried lede!
Sorry for the intense tone, it’s just … this sentence, if taken seriously, reframes the entire post for me in a big, weird, bad way.
I thought I was pretty clear, at the end of the post, that I wasn’t sad that I worked at MIRI instead of Google or academia. I’m glad I left when I did, though.
The conversations I’m mentioning with corporate management types were suprising to me, as were the contents of Moral Mazes, and Venkatesh Rao’s writing. So “like a regular job” doesn’t really communicate the magnitude of the harms to someone who doesn’t know how bad normal corporate management is. It’s hard for me to have strong opinions given that I haven’t worked in corporate management, though. Maybe a lot of places are pretty okay.
I’ve talked a lot with someone who got pretty high in Google’s management hierarchy, who seems really traumatized (and says she is) and who has a lot of physiological problems, which seem overall worse than mine. I wouldn’t trade places with her, mental health-wise.
MIRI wouldn’t make sense as a project if most regular jobs were fine, people who were really ok wouldn’t have reason to build unfriendly AI. I discussed with some friends about the benefits of working at Leverage vs. MIRI vs. the US Marines, and we agreed that Leverage and MIRI were probably overall less problematic, but the fact that the US marines signal that they’re going to dominate/abuse people is an important advantage relative to the alternatives, since it sets expectations more realistically.
MIRI wouldn’t make sense as a project if most regular jobs were fine, people who were really ok wouldn’t have reason to build unfriendly AI.
I just want to note that this is a contentious claim.
There is a competing story, and one much more commonly held among people who work for or support MIRI, that the world is heading towards an unaligned intelligence explosion due to the combination of a coordination problem and very normal motivated reasoning about the danger posed by lucrative and prestigious projects.
One could make the claim “healthy” people (whatever that means) wouldn’t exhibit those behaviors, ie that they would be able to coordinate and avoid rationalizing. But that’s a non-standard view.
I would prefer that you specifically flag it as a non-standard view, and then either make the argument for that view over the more common one, or highlight that you’re not going into detail on the argument and that you don’t expect others to accept the claim.
As it is, it feels a little like this is being slipped in as if it is a commonly accepted premise.
Yes, I would! Any pointers? (to avoid miscommunication I’m reading this to say that people are more likely to build UFAI because of traumatizing environment vs. normal reasons Eli mentioned)
Note that there’s an important distinction between “corporate management” and “corporate employment”—the thing where you say “yeesh, I’m glad I’m not a manager at Google” is substantially different from the thing where you say “yeesh, I’m glad I’m not a programmer at Google”, and the audience here has many more programmers than managers.
[And also Vanessa’s experience matches my impressions, tho I’ve spent less time in industry.]
[EDIT: I also thought it was clear that you meant this more as a “this is what MIRI was like” than “MIRI was unusually bad”, but I also think this means you’re open to nostalgebraist’s objection, that you’re ordering things pretty differently from how people might naively order them.]
My experience was that if you were T-5 (Senior), you had some overlap with PM and management games, and at T-6 (Staff), you were often in them. I could not handle the politics to get to T-7. Programmers below T-5 are expected to earn promotions or to leave.
Google’s a big company, so it might have been different elsewhere internally. My time at Google certainly traumatized me, but probably not to the point of anything in this or the Leverage thread.
Programmers below T-5 are expected to earn promotions or to leave.
This changed something like five years ago [edit: August 2017], to where people at level four (one level above new grad) no longer needed to get promoted to stay long term.
I think maybe a bit of the confusion here is nostalgebraist reading “corporate management” to mean something like “a regular job in industry”, whereas you’re pointing at “middle- or upper-management in sufficiently large or maze-like organizations”? Because those seem very different to me and I could imagine the second being much worse for people’s mental health than the first.
Separately I’m confused about the claim that “people who were really ok wouldn’t have reason to build unfriendly AI”; it sounds like you don’t agree that the idea that UFAI is the default outcome from building AFI without a specific effort to make it friendly? (This is probably a distraction from this threads’ subject but I’d be interested to read your thoughts on that if you’ve written them up somewhere.)
I think maybe a bit of the confusion here is nostalgebraist reading “corporate management” to mean something like “a regular job in industry”, whereas you’re pointing at “middle- or upper-management in sufficiently large or maze-like organizations”?
Yes, that seems likely. I did some interships at Google as a software engineer and they didn’t seem better than working at MIRI on average, although they had less intense psychological effects, as things didn’t break out in fractal betrayal during the time I was there.
Separately I’m confused about the claim that “people who were really ok wouldn’t have reason to build unfriendly AI”
People might think they “have to be productive” which points at increasing automation detached from human value, which points towards UFAI. Alternatively, they might think there isn’t a need to maximize productivity, and they can do things that would benefit their own values, which wouldn’t include UFAI. (I acknowledge there could be coordination problems where selfish behavior leads to cutting corners, but I don’t think that’s the main driver of existential risk failure modes)
I worked for 16 years in the industry, including management positions, including (briefly) having my own startup. I talked to many, many people who worked in many companies, including people who had their own startups and some with successful exits.
The industry is certainly not a rose garden. I encountered people who were selfish, unscrupulous, megalomaniac or just foolish. I’ve seen lies, manipulation, intrigue and plain incompetence. But, I also encountered people who were honest, idealistic, hardworking and talented. I’ve seen teams trying their best to build something actually useful for some corner of the world. And, it’s pretty hard to avoid reality checks when you need to deliver a real product for real customers (although some companies do manage to just get more and more investments without delivering anything until the eventual crash).
I honestly think most of them are not nearly as bad as Leverage.
Full disclosure: I am a MIRI Research Associate. This means that I receive funding from MIRI, but I am not a MIRI employee and I am not privy to its internal operation or secrets.
First of all, I am really sorry you had these horrible experiences.
A few thoughts:
Thought 1: I am not convinced the analogy between Leverage and MIRI/CFAR holds up to scrutiny. I think that Geoff Anders is most likely a bad actor, whereas MIRI/CFAR leadership is probably acting in good faith. There seems to be significantly more evidence of bad faith in Zoe’s account than in Jessica’s account, and the conclusion is reinforced by adding evidence from other accounts. In addition, MIRI definitely produced some valuable public research whereas I doubt the same can be said of Leverage, although I haven’t been following Leverage so I am not confident about the latter (ofc it’s in principle possible for a deeply unhealthy organization to produce some good outputs, and good outputs certainly don’t excuse abuse of personnel, but I do think good outputs provide some evidence against such abuse).
It is important not to commit the fallacy of gray: it would risk both judging MIRI/CFAR too harshly and judging Leverage insufficiently harshly. The comparison Jessica makes to “normal corporations” reinforces this impression: I have much experience in the industry, and although it’s possible I’ve been lucky in some ways, I still very much doubt the typical company is nearly as bad as Leverage.
Thought 2: From my experience, AI alignment is a domain of research that intrinsically comes with mental health hazards. First, the possibility of impending doom and the heavy sense of responsibility are sources of stress. Second, research inquiries often enough lead to “weird” metaphysical questions that risk overturning the (justified or unjustified) assumptions we implicitly hold to maintain a sense of safety in life. I think it might be the closest thing in real life to the Lovecraftian notion of “things that are best not to know because they will drive you mad”. Third, the sort of people drawn to the area and/or having the necessary talents seem to often also come with mental health issues (I am including myself in this group).
This might be regarded as an argument to blame MIRI less for the mental health fallout described by Jessica, but this is also an argument to pay more attention to the problem. It would be best if we could provide the people working in the area with the tools and environment to deal with these risks.
Thought 3: The part that concerned me the most in Jessica’s account (in part due to its novelty to me) is MIRI’s internal secrecy policy. While it might be justifiable to have some secrets to which only some employees are privy, it seems very extreme to require going through an executive because even the mere fact that a secret project exists is too dangerous. MIRI’s secrecy policy seemed questionable to me even before, but this new spin makes it even more dubious.
Overall, I wish MIRI was more transparent, so that for example its supporters would know about this internal policy. I realize there are tradeoffs involved, but I am not convinced MIRI chose the right balance. To me it feels like overconfidence about MIRI’s ability to steer the right way without the help of external critique.
Moreover, I’m a little worried that MIRI’s lack of transparency might pose a risk for the entire AI safety project. Tbh, one of my first thoughts when I saw the headline of the OP was “oh no, what if some scandal around MIRI blows up and the shockwave buries the entire community”. And I guess some people might think this is a reason for more secrecy. IMO it’s a reason for less secrecy (not necessarily less secrecy about technical AI stuff, but less secrecy about management and high-level plans). If we don’t have any skeletons in the closest, we don’t need to worry about the day they will come out. And eventually everything comes out, more or less. When most of everything is in the open, the community can find the right balance around it, and the reputation system is much more robust.
Thought 4: “Someone in the community told me that for me to think AGI probably won’t be developed soon, I must think I’m better at meta-rationality than Eliezer Yudkowsky, a massive claim of my own specialness.” I think (hope?) this is not at all a prevalent stance in the community (or at least in its leading echelons), but just for the record I want to note my strong position that the “someone” in this story is very misguided. Like I said, I don’t think community is currently comparable to Leverage, but this is the sort of thing that can push us in that direction.
Plus a million points for “IMO it’s a reason for less secrecy”!
If you put a lid on something you might contain it in the short term, but only at the cost of increasing the pressure: And pressure wants out, and the higher the pressure the more explosive it will be when it inevitably does come out.
I have heard too many accounts like this, in person and anecdotally, on the web and off for me to currently be interested in working or even getting to closely involved with any of the organizations in question. The only way to change this for me is to believably cultivate a healthy, transparent and supportive environment.
This made me go back and read “Every Cause wants to be a Cult” (Eliezer, 2007), which includes quotes like this one:
”Here I just want to point out that the worthiness of the Cause does not mean you can spend any less effort in resisting the cult attractor. And that if you can point to current battle lines, it does not mean you confess your Noble Cause unworthy. You might think that if the question were, “Cultish, yes or no?” that you were obliged to answer, “No,” or else betray your beloved Cause.”
That sounds like MIRI should have a councillor on it’s staff.
That would make them more vulnerable to claims that they use organizational mind control on their employees, and at the same time make it more likely that they would actually use it.
You would likely hire someone who’s traditionally trained, credentialed and has work experience instead of doing a bunch of your own psych-experiments, likely in a tradition like gestalttherapy that focuses on being nonmanipulative.
There’s an easier solution that doesn’t run the risk of being or appearing manipulative. You can contract external and independent councillors and make them available to your staff anonymously. I don’t know if there’s anything comparable in the US, but in Australia they’re referred to as Employee Assistance Programs (EAPs). Nothing you discuss with the councillor can be disclosed to your workplace, although in rare circumstances there may be mandatory reporting to the police (e.g. if abuse or ongoing risk of a minor is involved).
This also goes a long way toward creating a place where employees can talk about things they’re worried will seem crazy in work contexts.
Solutions like that might work, but it’s worth noting that just having an average therapist likely won’t be enough.
If you actually care about a level of security that protects secrets against intelligence agencies, operational security of the office of the therapist is a concern.
Governments that have security clearances don’t want their employees to talk with therapists who don’t have the secuirty clearances about classified information.
Talking nonjudgmentally with someone who has reasonable fears that the humanity won’t survive the next ten years because of fast AI timelines is not easy.
As far as I can tell, normal corporate management is much worse than Leverage. The kind of people from that world will, sometimes when prompted in private conversations, say things like:
Standard practice is to treat negotiations with other parties as zero-sum games.
“If you look around the table and can’t tell who the sucker is, it’s you” is a description of a common, relevant social dynamic in corporate meetings.
They have PTSD symptoms from working in corporate management, and are very threat-sensitive in general.
They learned from experience to treat social reality in general as fake, everything as an act.
They learned to accept that “there’s no such thing as not being lost”, like they’ve lost the ability to self-locate in a global map (I’ve experienced losing this to a significant extent).
Successful organizations get to be where they are by committing crimes, so copying standard practices from them is copying practices for committing crimes.
This is, to a large extent, them admitting to being bad actors, them and others having been made so by their social context. (This puts the possibility of “Geoff Anders being a bad actor” into perspective)
MIRI is, despite the problems noted in the post, as far as I can tell the most high-integrity organization doing AI safety research. FHI contributes some, but overall lower-quality research; Paul Christiano does some relevant research; OpenAI’s original mission was actively harmful, and hasn’t done much relevant safety research as far as I can tell. MIRI’s public output in the past few years since I left has been low, which seems like bad sign for its future performance, but what it’s done so far has been quite a large portion of the relevant research. I’m not particularly worried about scandals sinking the overall non-MIRI AI safety world’s reputation, given the degree to which it is of mixed value.
Your original post drew a comparison between MIRI and Leverage, the latter of which has just been singled out for intense criticism.
If I take the quoted sentence literally, you’re saying that “MIRI was like Leverage” is a gentler critique than “MIRI is like your regular job”?
If the intended message was “my job was bad, although less bad than the jobs of many people reading this, and instead only about as bad as Leverage Research,” why release this criticism on the heels of a post condemning Leverage as an abusive cult? If you believe the normally-employed among LessWrong readers are being abused by sub-Leverage hellcults, all the time, that seems like quite the buried lede!
Sorry for the intense tone, it’s just … this sentence, if taken seriously, reframes the entire post for me in a big, weird, bad way.
I thought I was pretty clear, at the end of the post, that I wasn’t sad that I worked at MIRI instead of Google or academia. I’m glad I left when I did, though.
The conversations I’m mentioning with corporate management types were suprising to me, as were the contents of Moral Mazes, and Venkatesh Rao’s writing. So “like a regular job” doesn’t really communicate the magnitude of the harms to someone who doesn’t know how bad normal corporate management is. It’s hard for me to have strong opinions given that I haven’t worked in corporate management, though. Maybe a lot of places are pretty okay.
I’ve talked a lot with someone who got pretty high in Google’s management hierarchy, who seems really traumatized (and says she is) and who has a lot of physiological problems, which seem overall worse than mine. I wouldn’t trade places with her, mental health-wise.
MIRI wouldn’t make sense as a project if most regular jobs were fine, people who were really ok wouldn’t have reason to build unfriendly AI. I discussed with some friends about the benefits of working at Leverage vs. MIRI vs. the US Marines, and we agreed that Leverage and MIRI were probably overall less problematic, but the fact that the US marines signal that they’re going to dominate/abuse people is an important advantage relative to the alternatives, since it sets expectations more realistically.
I just want to note that this is a contentious claim.
There is a competing story, and one much more commonly held among people who work for or support MIRI, that the world is heading towards an unaligned intelligence explosion due to the combination of a coordination problem and very normal motivated reasoning about the danger posed by lucrative and prestigious projects.
One could make the claim “healthy” people (whatever that means) wouldn’t exhibit those behaviors, ie that they would be able to coordinate and avoid rationalizing. But that’s a non-standard view.
I would prefer that you specifically flag it as a non-standard view, and then either make the argument for that view over the more common one, or highlight that you’re not going into detail on the argument and that you don’t expect others to accept the claim.
As it is, it feels a little like this is being slipped in as if it is a commonly accepted premise.
I agree this is a non-standard view.
Yes, I would! Any pointers?
(to avoid miscommunication I’m reading this to say that people are more likely to build UFAI because of traumatizing environment vs. normal reasons Eli mentioned)
Note that there’s an important distinction between “corporate management” and “corporate employment”—the thing where you say “yeesh, I’m glad I’m not a manager at Google” is substantially different from the thing where you say “yeesh, I’m glad I’m not a programmer at Google”, and the audience here has many more programmers than managers.
[And also Vanessa’s experience matches my impressions, tho I’ve spent less time in industry.]
[EDIT: I also thought it was clear that you meant this more as a “this is what MIRI was like” than “MIRI was unusually bad”, but I also think this means you’re open to nostalgebraist’s objection, that you’re ordering things pretty differently from how people might naively order them.]
My experience was that if you were T-5 (Senior), you had some overlap with PM and management games, and at T-6 (Staff), you were often in them. I could not handle the politics to get to T-7. Programmers below T-5 are expected to earn promotions or to leave.
Google’s a big company, so it might have been different elsewhere internally. My time at Google certainly traumatized me, but probably not to the point of anything in this or the Leverage thread.
This changed something like five years ago [edit: August 2017], to where people at level four (one level above new grad) no longer needed to get promoted to stay long term.
I think maybe a bit of the confusion here is nostalgebraist reading “corporate management” to mean something like “a regular job in industry”, whereas you’re pointing at “middle- or upper-management in sufficiently large or maze-like organizations”? Because those seem very different to me and I could imagine the second being much worse for people’s mental health than the first.
Separately I’m confused about the claim that “people who were really ok wouldn’t have reason to build unfriendly AI”; it sounds like you don’t agree that the idea that UFAI is the default outcome from building AFI without a specific effort to make it friendly? (This is probably a distraction from this threads’ subject but I’d be interested to read your thoughts on that if you’ve written them up somewhere.)
Yes, that seems likely. I did some interships at Google as a software engineer and they didn’t seem better than working at MIRI on average, although they had less intense psychological effects, as things didn’t break out in fractal betrayal during the time I was there.
People might think they “have to be productive” which points at increasing automation detached from human value, which points towards UFAI. Alternatively, they might think there isn’t a need to maximize productivity, and they can do things that would benefit their own values, which wouldn’t include UFAI. (I acknowledge there could be coordination problems where selfish behavior leads to cutting corners, but I don’t think that’s the main driver of existential risk failure modes)
I worked for 16 years in the industry, including management positions, including (briefly) having my own startup. I talked to many, many people who worked in many companies, including people who had their own startups and some with successful exits.
The industry is certainly not a rose garden. I encountered people who were selfish, unscrupulous, megalomaniac or just foolish. I’ve seen lies, manipulation, intrigue and plain incompetence. But, I also encountered people who were honest, idealistic, hardworking and talented. I’ve seen teams trying their best to build something actually useful for some corner of the world. And, it’s pretty hard to avoid reality checks when you need to deliver a real product for real customers (although some companies do manage to just get more and more investments without delivering anything until the eventual crash).
I honestly think most of them are not nearly as bad as Leverage.