Can you give some examples of organizations larger than a few dozen people, needing significant resources, with goals not aligned with wealth and power, which have good organizational incentives?
I don’t disagree that incentives matter, but I don’t see that there’s any way to radically change incentives without pretty structural changes across large swaths of society.
Nvidia, for example, has 26k employees, all incentivized to produce & sell marginally better GPUs, and possibly to sabotage others’ abilities to make and sell marginally better GPUs. They’re likely incentivized to do other things as well, like play politics, or spin off irrelevant side-projects. But for the most part I claim they end up contributing to producing marginally better GPUs.
You may complain that each individual in Nvidia is likely mostly chasing base-desires, and so is actually aligned with wealth & power, and it just so happens that in the situation they’re in, the best way of doing that is to make marginally better GPUs. But this is just my point! What you want is to position your company, culture, infrastructure, and friends such that the way for individuals to achieve wealth and power is to do good on your company’s goal.
I claim its in nobody’s interest & ability in or around Nvidia to make it produce marginally worse GPUs, or sabotage the company so that it instead goes all in on the TV business rather than the marginally better GPUs business.
Edit: Look at most any large company achieving consistent outcomes, and I claim its in everyone in that company’s interest or ability to help that company achieve those consistent outcomes.
I’m confused. NVidia (and most profit-seeking corporations) are reasonably aligned WRT incentives, because those are the incentives of the world around them.
I’m looking for examples of things like EA orgs, which have goals very different from standard capitalist structures, and how they can set up “good incentives” within this overall framework.
If there are no such examples, your complaint about ’strikingly bad at setting up good organizational incentives” is hard to understand. It may be more that the ENVIRONMENT in which they exist has competing incentives and orgs have no choice but to work within that.
You must misunderstand me. To what you say, I say that you don’t want your org to be fighting the incentives of the environment around it. You want to set up your org in a position in the environment where the incentives within the org correlate with doing good. If the founders of Nvidia didn’t want marginally better GPUs to be made, then they hired the wrong people, bought the wrong infrastructure, partnered with the wrong companies, and overall made the wrong organizational incentive structure for that job.
I would in fact be surprised if there were >1k worker sized orgs which consistently didn’t reward their workers for doing good according to the org’s values, was serving no demand present in the market, and yet were competently executing some altruistic goal.
Right now I feel like I’m just saying a bunch of obvious things which you should definitely agree with, yet you believe we have a disagreement. I do not understand what you think I’m saying. Maybe you could try restating what I originally said in your own words?
We absolutely agree that incentives matter. Where I think we disagree is on how much they matter and how controllable they are. Especially for orgs whose goals are orthogonal or even contradictory with the common cultural and environmental incentives outside of the org.
I’m mostly reacting to your topic sentence
EAs are, and I thought this even before the recent Altman situation, strikingly bad at setting up good organizational incentives.
And wondering if ‘strikingly bad’ is relative to some EA or non-profit-driven org that does it well,or if ‘strikingly bad’ is just acknowledgement that it may not be possible to do well.
By strikingly bad I mean there are easy changes EA can make to make it’s sponsored orgs have better incentives, and it has too much confidence that the incentives in the orgs it sponsors favor doing good above doing bad, politics, not doing anything, etc.
For example, nobody in Anthropic gets paid more if they follow their RSP and less of they don’t. Changing this isn’t sufficient for me to feel happy with Anthropic, but its one example among many for which Anthropic could be better.
When I think of an Anthropic I feel happy with I think of a formally defined balance of powers type situation with strong & public whistleblower protection and post-whistleblower reform processes, them hiring engineers loyal to that process (rather than building AGI), and them diversifying the sources for which they trade, such that its in none of their source’s interest to manipulate them.
I also claim marginal movements toward this target are often good.
As I said in the original shortform, I also think incentives are not all or nothing. Worse incentives just mean you need more upstanding workers & leaders.
Can you give some examples of organizations larger than a few dozen people, needing significant resources, with goals not aligned with wealth and power, which have good organizational incentives?
I don’t disagree that incentives matter, but I don’t see that there’s any way to radically change incentives without pretty structural changes across large swaths of society.
This is a great question. I can’t think of a good answer. Surely someone has done it on a large scale...
Nvidia, for example, has 26k employees, all incentivized to produce & sell marginally better GPUs, and possibly to sabotage others’ abilities to make and sell marginally better GPUs. They’re likely incentivized to do other things as well, like play politics, or spin off irrelevant side-projects. But for the most part I claim they end up contributing to producing marginally better GPUs.
You may complain that each individual in Nvidia is likely mostly chasing base-desires, and so is actually aligned with wealth & power, and it just so happens that in the situation they’re in, the best way of doing that is to make marginally better GPUs. But this is just my point! What you want is to position your company, culture, infrastructure, and friends such that the way for individuals to achieve wealth and power is to do good on your company’s goal.
I claim its in nobody’s interest & ability in or around Nvidia to make it produce marginally worse GPUs, or sabotage the company so that it instead goes all in on the TV business rather than the marginally better GPUs business.
Edit: Look at most any large company achieving consistent outcomes, and I claim its in everyone in that company’s interest or ability to help that company achieve those consistent outcomes.
I’m confused. NVidia (and most profit-seeking corporations) are reasonably aligned WRT incentives, because those are the incentives of the world around them.
I’m looking for examples of things like EA orgs, which have goals very different from standard capitalist structures, and how they can set up “good incentives” within this overall framework.
If there are no such examples, your complaint about ’strikingly bad at setting up good organizational incentives” is hard to understand. It may be more that the ENVIRONMENT in which they exist has competing incentives and orgs have no choice but to work within that.
You must misunderstand me. To what you say, I say that you don’t want your org to be fighting the incentives of the environment around it. You want to set up your org in a position in the environment where the incentives within the org correlate with doing good. If the founders of Nvidia didn’t want marginally better GPUs to be made, then they hired the wrong people, bought the wrong infrastructure, partnered with the wrong companies, and overall made the wrong organizational incentive structure for that job.
I would in fact be surprised if there were >1k worker sized orgs which consistently didn’t reward their workers for doing good according to the org’s values, was serving no demand present in the market, and yet were competently executing some altruistic goal.
Right now I feel like I’m just saying a bunch of obvious things which you should definitely agree with, yet you believe we have a disagreement. I do not understand what you think I’m saying. Maybe you could try restating what I originally said in your own words?
We absolutely agree that incentives matter. Where I think we disagree is on how much they matter and how controllable they are. Especially for orgs whose goals are orthogonal or even contradictory with the common cultural and environmental incentives outside of the org.
I’m mostly reacting to your topic sentence
And wondering if ‘strikingly bad’ is relative to some EA or non-profit-driven org that does it well,or if ‘strikingly bad’ is just acknowledgement that it may not be possible to do well.
By strikingly bad I mean there are easy changes EA can make to make it’s sponsored orgs have better incentives, and it has too much confidence that the incentives in the orgs it sponsors favor doing good above doing bad, politics, not doing anything, etc.
For example, nobody in Anthropic gets paid more if they follow their RSP and less of they don’t. Changing this isn’t sufficient for me to feel happy with Anthropic, but its one example among many for which Anthropic could be better.
When I think of an Anthropic I feel happy with I think of a formally defined balance of powers type situation with strong & public whistleblower protection and post-whistleblower reform processes, them hiring engineers loyal to that process (rather than building AGI), and them diversifying the sources for which they trade, such that its in none of their source’s interest to manipulate them.
I also claim marginal movements toward this target are often good.
As I said in the original shortform, I also think incentives are not all or nothing. Worse incentives just mean you need more upstanding workers & leaders.