people usually think of corporations as either {advancing their own interests and also the public’s interests} or {advancing their own interests at cost to the public} — ime mostly the latter. what’s actually going on with AI frontier labs, i.e. {going against the interests of everyone including themselves}, is very un-memetic and very far from the overton window.
in fiction, the heads of big organizations are either good (making things good for everyone) or evil (worsening everyone else’s outcomes, but improving their own). most of the time, just evil. very rarely are they suicidal fools semi-unknowningly trying to kill everyone including themselves.
and the AI existential risk thing just doesn’t stick if you take it as a given that the organizations are acting in their own interest, because dying is not in their own interest.
the public systematically underestimates the foolishness of AI frontier labs.
There are plenty examples in fiction of greed and hubris leading to a disaster that takes down its own architects. The dwarves who mined too deep and awoke the Balrog, the creators of Skynet, Peter Isherwell in “Don’t Look Up”, Frankenstein and his Creature...
Moral Maze dynamics push corporations not just to pursue profit at all other costs, but also to be extremely myopic. As long as the death doesn’t happen before the end of the quarter, the big labs, being immoral mazes, have no reason to give a shit about x-risk. Of course, every individual member of a big lab has reason to care, but the organization as an egregore does not (and so there is strong selection pressure for these organizations to have people that have low P(doom) and/or don’t (think they) value the future lives of themselves and others).
there is strong selection pressure for these organizations to have people that have low P(doom) and/or don’t (think they) value the future lives of themselves and others
This is an important thing I didn’t realize. When I try to imagine the people who make decisions in organizations, my intuitive model would be somewhere between “normal people” and “greedy psychopaths”, depending on my mood, and how bad the organization seems.
But in addition to this, there is the systematic shift towards “people who genuinely believe things that happen to be convenient for the organization’s mission”, as a kind of cognitive bias on group scale. Not average people with average beliefs. Not psychopaths who prioritize profit above everything. But people who were selected from the pool of average by having their genuine beliefs aligned with what happens to be profitable in given organization.
I was already aware of similar things happening in “think tanks”, where producing beliefs is the entire point of the organization. Their collective beliefs are obviously biased, not primarily because the individuals are biased, but because the individuals were selected for having their genuine beliefs already extreme in a certain direction.
But I didn’t realize that the same is kinda true for every organization, because the implied belief is “this organization’s mission is good (or at least neutral, if I am merely doing it for money)”.
Would this mean that epistemically healthiest organizations are those whose employees don’t give a fuck about the mission and only do it for money?
same reason people make poor decisions all the time. if they had a clear head and hadn’t already sunk some cost into AI, they could see that working on AI might make them wealthy in the short term but it’ll increase {the risk that they die soon} enough that they go “not worth it”, as they should. but once you’re already working in AI stuff, it’s tempting and easy to retroactively justify why doing that is safe. or to just not worry about it and enjoy the money, even though if you thought about the impact of your actions on your own survival in the next few years you’d decide to quit.
people usually think of corporations as either {advancing their own interests and also the public’s interests} or {advancing their own interests at cost to the public} — ime mostly the latter. what’s actually going on with AI frontier labs, i.e. {going against the interests of everyone including themselves}, is very un-memetic and very far from the overton window.
in fiction, the heads of big organizations are either good (making things good for everyone) or evil (worsening everyone else’s outcomes, but improving their own). most of the time, just evil. very rarely are they suicidal fools semi-unknowningly trying to kill everyone including themselves.
and the AI existential risk thing just doesn’t stick if you take it as a given that the organizations are acting in their own interest, because dying is not in their own interest.
the public systematically underestimates the foolishness of AI frontier labs.
There are plenty examples in fiction of greed and hubris leading to a disaster that takes down its own architects. The dwarves who mined too deep and awoke the Balrog, the creators of Skynet, Peter Isherwell in “Don’t Look Up”, Frankenstein and his Creature...
Moral Maze dynamics push corporations not just to pursue profit at all other costs, but also to be extremely myopic. As long as the death doesn’t happen before the end of the quarter, the big labs, being immoral mazes, have no reason to give a shit about x-risk. Of course, every individual member of a big lab has reason to care, but the organization as an egregore does not (and so there is strong selection pressure for these organizations to have people that have low P(doom) and/or don’t (think they) value the future lives of themselves and others).
This is an important thing I didn’t realize. When I try to imagine the people who make decisions in organizations, my intuitive model would be somewhere between “normal people” and “greedy psychopaths”, depending on my mood, and how bad the organization seems.
But in addition to this, there is the systematic shift towards “people who genuinely believe things that happen to be convenient for the organization’s mission”, as a kind of cognitive bias on group scale. Not average people with average beliefs. Not psychopaths who prioritize profit above everything. But people who were selected from the pool of average by having their genuine beliefs aligned with what happens to be profitable in given organization.
I was already aware of similar things happening in “think tanks”, where producing beliefs is the entire point of the organization. Their collective beliefs are obviously biased, not primarily because the individuals are biased, but because the individuals were selected for having their genuine beliefs already extreme in a certain direction.
But I didn’t realize that the same is kinda true for every organization, because the implied belief is “this organization’s mission is good (or at least neutral, if I am merely doing it for money)”.
Would this mean that epistemically healthiest organizations are those whose employees don’t give a fuck about the mission and only do it for money?
“why would they be doing that?”
same reason people make poor decisions all the time. if they had a clear head and hadn’t already sunk some cost into AI, they could see that working on AI might make them wealthy in the short term but it’ll increase {the risk that they die soon} enough that they go “not worth it”, as they should. but once you’re already working in AI stuff, it’s tempting and easy to retroactively justify why doing that is safe. or to just not worry about it and enjoy the money, even though if you thought about the impact of your actions on your own survival in the next few years you’d decide to quit.
at least that’s my vague best guess.