I thought I was pretty clear, at the end of the post, that I wasn’t sad that I worked at MIRI instead of Google or academia. I’m glad I left when I did, though.
The conversations I’m mentioning with corporate management types were suprising to me, as were the contents of Moral Mazes, and Venkatesh Rao’s writing. So “like a regular job” doesn’t really communicate the magnitude of the harms to someone who doesn’t know how bad normal corporate management is. It’s hard for me to have strong opinions given that I haven’t worked in corporate management, though. Maybe a lot of places are pretty okay.
I’ve talked a lot with someone who got pretty high in Google’s management hierarchy, who seems really traumatized (and says she is) and who has a lot of physiological problems, which seem overall worse than mine. I wouldn’t trade places with her, mental health-wise.
MIRI wouldn’t make sense as a project if most regular jobs were fine, people who were really ok wouldn’t have reason to build unfriendly AI. I discussed with some friends about the benefits of working at Leverage vs. MIRI vs. the US Marines, and we agreed that Leverage and MIRI were probably overall less problematic, but the fact that the US marines signal that they’re going to dominate/abuse people is an important advantage relative to the alternatives, since it sets expectations more realistically.
MIRI wouldn’t make sense as a project if most regular jobs were fine, people who were really ok wouldn’t have reason to build unfriendly AI.
I just want to note that this is a contentious claim.
There is a competing story, and one much more commonly held among people who work for or support MIRI, that the world is heading towards an unaligned intelligence explosion due to the combination of a coordination problem and very normal motivated reasoning about the danger posed by lucrative and prestigious projects.
One could make the claim “healthy” people (whatever that means) wouldn’t exhibit those behaviors, ie that they would be able to coordinate and avoid rationalizing. But that’s a non-standard view.
I would prefer that you specifically flag it as a non-standard view, and then either make the argument for that view over the more common one, or highlight that you’re not going into detail on the argument and that you don’t expect others to accept the claim.
As it is, it feels a little like this is being slipped in as if it is a commonly accepted premise.
Yes, I would! Any pointers? (to avoid miscommunication I’m reading this to say that people are more likely to build UFAI because of traumatizing environment vs. normal reasons Eli mentioned)
Note that there’s an important distinction between “corporate management” and “corporate employment”—the thing where you say “yeesh, I’m glad I’m not a manager at Google” is substantially different from the thing where you say “yeesh, I’m glad I’m not a programmer at Google”, and the audience here has many more programmers than managers.
[And also Vanessa’s experience matches my impressions, tho I’ve spent less time in industry.]
[EDIT: I also thought it was clear that you meant this more as a “this is what MIRI was like” than “MIRI was unusually bad”, but I also think this means you’re open to nostalgebraist’s objection, that you’re ordering things pretty differently from how people might naively order them.]
My experience was that if you were T-5 (Senior), you had some overlap with PM and management games, and at T-6 (Staff), you were often in them. I could not handle the politics to get to T-7. Programmers below T-5 are expected to earn promotions or to leave.
Google’s a big company, so it might have been different elsewhere internally. My time at Google certainly traumatized me, but probably not to the point of anything in this or the Leverage thread.
Programmers below T-5 are expected to earn promotions or to leave.
This changed something like five years ago [edit: August 2017], to where people at level four (one level above new grad) no longer needed to get promoted to stay long term.
I think maybe a bit of the confusion here is nostalgebraist reading “corporate management” to mean something like “a regular job in industry”, whereas you’re pointing at “middle- or upper-management in sufficiently large or maze-like organizations”? Because those seem very different to me and I could imagine the second being much worse for people’s mental health than the first.
Separately I’m confused about the claim that “people who were really ok wouldn’t have reason to build unfriendly AI”; it sounds like you don’t agree that the idea that UFAI is the default outcome from building AFI without a specific effort to make it friendly? (This is probably a distraction from this threads’ subject but I’d be interested to read your thoughts on that if you’ve written them up somewhere.)
I think maybe a bit of the confusion here is nostalgebraist reading “corporate management” to mean something like “a regular job in industry”, whereas you’re pointing at “middle- or upper-management in sufficiently large or maze-like organizations”?
Yes, that seems likely. I did some interships at Google as a software engineer and they didn’t seem better than working at MIRI on average, although they had less intense psychological effects, as things didn’t break out in fractal betrayal during the time I was there.
Separately I’m confused about the claim that “people who were really ok wouldn’t have reason to build unfriendly AI”
People might think they “have to be productive” which points at increasing automation detached from human value, which points towards UFAI. Alternatively, they might think there isn’t a need to maximize productivity, and they can do things that would benefit their own values, which wouldn’t include UFAI. (I acknowledge there could be coordination problems where selfish behavior leads to cutting corners, but I don’t think that’s the main driver of existential risk failure modes)
I thought I was pretty clear, at the end of the post, that I wasn’t sad that I worked at MIRI instead of Google or academia. I’m glad I left when I did, though.
The conversations I’m mentioning with corporate management types were suprising to me, as were the contents of Moral Mazes, and Venkatesh Rao’s writing. So “like a regular job” doesn’t really communicate the magnitude of the harms to someone who doesn’t know how bad normal corporate management is. It’s hard for me to have strong opinions given that I haven’t worked in corporate management, though. Maybe a lot of places are pretty okay.
I’ve talked a lot with someone who got pretty high in Google’s management hierarchy, who seems really traumatized (and says she is) and who has a lot of physiological problems, which seem overall worse than mine. I wouldn’t trade places with her, mental health-wise.
MIRI wouldn’t make sense as a project if most regular jobs were fine, people who were really ok wouldn’t have reason to build unfriendly AI. I discussed with some friends about the benefits of working at Leverage vs. MIRI vs. the US Marines, and we agreed that Leverage and MIRI were probably overall less problematic, but the fact that the US marines signal that they’re going to dominate/abuse people is an important advantage relative to the alternatives, since it sets expectations more realistically.
I just want to note that this is a contentious claim.
There is a competing story, and one much more commonly held among people who work for or support MIRI, that the world is heading towards an unaligned intelligence explosion due to the combination of a coordination problem and very normal motivated reasoning about the danger posed by lucrative and prestigious projects.
One could make the claim “healthy” people (whatever that means) wouldn’t exhibit those behaviors, ie that they would be able to coordinate and avoid rationalizing. But that’s a non-standard view.
I would prefer that you specifically flag it as a non-standard view, and then either make the argument for that view over the more common one, or highlight that you’re not going into detail on the argument and that you don’t expect others to accept the claim.
As it is, it feels a little like this is being slipped in as if it is a commonly accepted premise.
I agree this is a non-standard view.
Yes, I would! Any pointers?
(to avoid miscommunication I’m reading this to say that people are more likely to build UFAI because of traumatizing environment vs. normal reasons Eli mentioned)
Note that there’s an important distinction between “corporate management” and “corporate employment”—the thing where you say “yeesh, I’m glad I’m not a manager at Google” is substantially different from the thing where you say “yeesh, I’m glad I’m not a programmer at Google”, and the audience here has many more programmers than managers.
[And also Vanessa’s experience matches my impressions, tho I’ve spent less time in industry.]
[EDIT: I also thought it was clear that you meant this more as a “this is what MIRI was like” than “MIRI was unusually bad”, but I also think this means you’re open to nostalgebraist’s objection, that you’re ordering things pretty differently from how people might naively order them.]
My experience was that if you were T-5 (Senior), you had some overlap with PM and management games, and at T-6 (Staff), you were often in them. I could not handle the politics to get to T-7. Programmers below T-5 are expected to earn promotions or to leave.
Google’s a big company, so it might have been different elsewhere internally. My time at Google certainly traumatized me, but probably not to the point of anything in this or the Leverage thread.
This changed something like five years ago [edit: August 2017], to where people at level four (one level above new grad) no longer needed to get promoted to stay long term.
I think maybe a bit of the confusion here is nostalgebraist reading “corporate management” to mean something like “a regular job in industry”, whereas you’re pointing at “middle- or upper-management in sufficiently large or maze-like organizations”? Because those seem very different to me and I could imagine the second being much worse for people’s mental health than the first.
Separately I’m confused about the claim that “people who were really ok wouldn’t have reason to build unfriendly AI”; it sounds like you don’t agree that the idea that UFAI is the default outcome from building AFI without a specific effort to make it friendly? (This is probably a distraction from this threads’ subject but I’d be interested to read your thoughts on that if you’ve written them up somewhere.)
Yes, that seems likely. I did some interships at Google as a software engineer and they didn’t seem better than working at MIRI on average, although they had less intense psychological effects, as things didn’t break out in fractal betrayal during the time I was there.
People might think they “have to be productive” which points at increasing automation detached from human value, which points towards UFAI. Alternatively, they might think there isn’t a need to maximize productivity, and they can do things that would benefit their own values, which wouldn’t include UFAI. (I acknowledge there could be coordination problems where selfish behavior leads to cutting corners, but I don’t think that’s the main driver of existential risk failure modes)