My sense is that building functional institutions is something that has been done in history multiple times, and that the other things you list are extreme tail scenarios.
People are trying to build more functional institutions, or at least unintentionally experimenting with different institutional designs, all the time, so the fact that building functional institutions is something that has been done in history multiple times doesn’t imply that any particular attempt to build a particular kind of functional institutions has a high or moderate chance of success.
For some context, I recommend reading Samo’s blog
Can you recommend some specific articles?
It seems like something that works at a well-funded company (that has long time horizons, like SpaceX) would probably also work for a well-funded nonprofit
Are there any articles about what is different about SpaceX as far as institutional design? I think I tried to find some earlier but couldn’t. In my current state of knowledge I don’t see much reason to think that SpaceX’s success is best explained by it having made a lot of advances in institutional design. If it actually did create a lot of advances in institutional design, and they can be applied to a nonprofit working on AI alignment, wouldn’t it be high priority to write down what those advances are and how they can be applied to the nonprofit, so other people can critique those ideas?
Let’s reorganize the rest of our discussion by splitting your charity suggestion into two parts which we can consider separately:
Use your own or your friends’ evaluations of technical work instead of outsourcing evaluations to bigger / more distant / more formal organizations.
Give awards/tenure instead of grants.
(Is this a fair summary/decomposition of your suggestion?) I think aside from my previous objections which I’m still not sure about, perhaps my true rejection to 1 combined with 2 is that I don’t want to risk feeling duped and/or embarrassed if I help fund someone’s tenure and it turns out that I or my friends misjudged their technical abilities or motivation (i.e., they end up producing low quality work or suffer a large decrease in productivity). Yeah from an altruistic perspective I should be risk neutral but I don’t think I can override the social/face-saving part of my brain on this. To avoid this I could fund an organization instead of an individual but in that case it’s likely that most of the money would go towards expansion rather than increasing slack. I guess another problem is that if I start thinking about what AI alignment work to fund I get depressed thinking about the low chance of success of any particular approach, and it seems a lot easier to just donate to an evaluator charity and make it their problem.
It seems to me that 2 by itself is perhaps more promising and worth a try. In other words we could maybe convince a charity to reallocate some money from grants to awards/tenure and check if that improves outcomes, or create a new charity for this purpose. Is that something you’d support?
People are trying to build more functional institutions, or at least unintentionally experimenting with different institutional designs, all the time, so the fact that building functional institutions is something that has been done in history multiple times doesn’t imply that any particular attempt to build a particular kind of functional institutions has a high or moderate chance of success.
Agree, but this is a general reason why doing things is hard. Lots of people are working on philosophy too. I think my chance of success is way higher than the base rate, to the point where anchoring on the base rate does not make sense, but I can understand people not believing me on this. (Arguments of this form might fall under modest epistemology.)
In my current state of knowledge I don’t see much reason to think that SpaceX’s success is best explained by it having made a lot of advances in institutional design.
If one organization is much more effective than comparable ones, and it wasn’t totally by accident, then there are causal reasons for the difference in effectiveness. Even if it isn’t the formal structure of the organization, it could be properties of the people who seeded the organization, properties of the organization’s mission, etc. I am taking a somewhat broad view of “institutional design” that would include these things too. I am not actually saying anything less trivial than “some orgs work much better than other orgs and understanding why is important so this can be replicated and built upon”.
If it actually did create a lot of advances in institutional design, and they can be applied to a nonprofit working on AI alignment, wouldn’t it be high priority to write down what those advances
Yes, the general project of examining and documenting working institutions is high-priority. There will probably be multiple difficulties in directly applying SpaceX’s model (e.g. the fact that SpaceX is more engineering than research), so documenting multiple institutions would help. I am otherwise occupied right now, though.
(Is this a fair summary/decomposition of your suggestion?)
Yes, this would be my suggestion for donors.
I appreciate the introspection you’ve done on this, this is useful information. I think there’s a general issue where the AI alignment problem is really hard, so most people try to push the responsibility somewhere else, and those that do take responsibility usually end up feeling like they are highly constrained into doing the “responsible” thing (e.g. using defensible bureaucratic systems rather than their intuition), which is often at odds with the straightforward just-solve-it mental motion (which is used in, for example, playing video games), or curiosity in general (e.g. mathematical or scientific). I’ve experienced this personally. I recommend Ben’s post on this dynamic. This is part of why I exited the charity sector and don’t identify as an EA anymore. I don’t know how to fix this other than by taking the whole idea of non-local responsibility (i.e. beyond things like “I am responsible for driving safely and paying my bills on time”) less seriously, so I kind of do that.
In other words we could maybe convince a charity to reallocate some money from grants to awards/tenure and check if that improves outcomes, or create a new charity for this purpose. Is that something you’d support?
Yes, both awards and tenure seem like improvements, and in any case well worth experimenting with.
SpaceX might be more competitive than its competitors not because it’s particularly functional (i.e., compared to typical firms in other fields) but because its competitors are particularly non-functional.
If it is particularly effective but not due to the formal structure of the organization, then those informal things are likely to be hard to copy to another organization.
SpaceX can be very successful by having just marginally better institutional design than its competitors because they’re all trying to do the same things. A successful AI alignment organization however would have to be much more effective than organizations that are trying to build unaligned AI (or nominally trying to build aligned AI but cutting lots of corners to win the race, or following some AI alignment approach that seems easy but is actually fatally flawed).
those that do take responsibility usually end up feeling like they are highly constrained into doing the “responsible” thing (e.g. using defensible bureaucratic systems rather than their intuition), which is often at odds with the straightforward just-solve-it mental motion (which is used in, for example, playing video games), or curiosity in general (e.g. mathematical or scientific)
I don’t think that playing video games, math, and science are good models here because those all involve relatively fast feedback cycles which make it easy to build up good intuitions. It seems reasonable to not trust one’s intuitions in AI alignment, and the desire to appear defensible also seems understandable and hard to eliminate, but perhaps we can come up with better “defensible bureaucratic systems” than what exists today, i.e., systems that can still appear defensible but make better decisions than they currently do. I wonder if this problem has been addressed by anyone.
Yes, both awards and tenure seem like improvements, and in any case well worth experimenting with.
Ok, I made the suggestion to BERI since it seems like they might be open to this kind of thing.
ETA: Another consideration against individuals directly funding other individuals is that it wouldn’t be tax deductible. This could reduce the funding by up to 40%. If the funding is done through a tax-exempt non-profit, then IRS probably has some requirements about having formal procedures for deciding who/what to fund.
(Just got around to reading this. As a point of reference, it seems that at least Open Phil seems to have decided that tax-deductability is not more important than being able to give to things freely, which is why the Open Philanthropy Project is an LLC. I think this is at least slight evidence towards that tradeoff being worth it.)
There’s an enormous difference between having millions of dollars of operating expenditures in an LLC (so that an org is legally allowed to do things like investigate non-deductible activities like investment or politics), and giving up the ability to make billions of dollars of tax-deductible donations. Open Philanthropy being an LLC (so that its own expenses aren’t tax-deductible, but it has LLC freedom) doesn’t stop Good Ventures from making all relevant donations tax-deductible, and indeed the overwhelming majority of grants on its grants page are deductible.
Yep, sorry. I didn’t mean to imply that all of Open Phil’s funding is non-deductible, just that they decided that it was likely enough that they would find non-deductible opportunities that they went through the effort of restructuring their org to do so (and also gave up a bunch of other benefits like the ability to sponsor visas efficiently). My comment wasn’t very clear on that.
Here’s Open Phil’s blog post on why they decided to operate as an LLC. After reading it, I think their reasons are not very relevant to funding AI alignment research. (Mainly they want the freedom to recommend donations to non-501(c)(3) organizations like political groups.)
I am also pretty interested in 2 (ex-post giving). In 2015, there was impactpurchase.org. I got in contact with them about it, and the major updates Paul reported were a) being willing to buy partial contributions (not just for people who were claiming full responsibility for things) and b) more focused on what’s being funded (like for example, only asking for people to submit claims on blog posts and articles).
I realise that things like impactpurchase is possibly framed in terms of a slightly divergent reason for 2 (it seems more focused on changing the incentive landscape, whereas the posts above include thinking about whether giving slack to people with track records will lead those people to be counterfactually more effective in future).
People are trying to build more functional institutions, or at least unintentionally experimenting with different institutional designs, all the time, so the fact that building functional institutions is something that has been done in history multiple times doesn’t imply that any particular attempt to build a particular kind of functional institutions has a high or moderate chance of success.
Can you recommend some specific articles?
Are there any articles about what is different about SpaceX as far as institutional design? I think I tried to find some earlier but couldn’t. In my current state of knowledge I don’t see much reason to think that SpaceX’s success is best explained by it having made a lot of advances in institutional design. If it actually did create a lot of advances in institutional design, and they can be applied to a nonprofit working on AI alignment, wouldn’t it be high priority to write down what those advances are and how they can be applied to the nonprofit, so other people can critique those ideas?
Let’s reorganize the rest of our discussion by splitting your charity suggestion into two parts which we can consider separately:
Use your own or your friends’ evaluations of technical work instead of outsourcing evaluations to bigger / more distant / more formal organizations.
Give awards/tenure instead of grants.
(Is this a fair summary/decomposition of your suggestion?) I think aside from my previous objections which I’m still not sure about, perhaps my true rejection to 1 combined with 2 is that I don’t want to risk feeling duped and/or embarrassed if I help fund someone’s tenure and it turns out that I or my friends misjudged their technical abilities or motivation (i.e., they end up producing low quality work or suffer a large decrease in productivity). Yeah from an altruistic perspective I should be risk neutral but I don’t think I can override the social/face-saving part of my brain on this. To avoid this I could fund an organization instead of an individual but in that case it’s likely that most of the money would go towards expansion rather than increasing slack. I guess another problem is that if I start thinking about what AI alignment work to fund I get depressed thinking about the low chance of success of any particular approach, and it seems a lot easier to just donate to an evaluator charity and make it their problem.
It seems to me that 2 by itself is perhaps more promising and worth a try. In other words we could maybe convince a charity to reallocate some money from grants to awards/tenure and check if that improves outcomes, or create a new charity for this purpose. Is that something you’d support?
Agree, but this is a general reason why doing things is hard. Lots of people are working on philosophy too. I think my chance of success is way higher than the base rate, to the point where anchoring on the base rate does not make sense, but I can understand people not believing me on this. (Arguments of this form might fall under modest epistemology.)
On the Loss and Preservation of Knowledge
Live versus Dead Players
Functional Institutions are the Exception
Great Founder Theory
If one organization is much more effective than comparable ones, and it wasn’t totally by accident, then there are causal reasons for the difference in effectiveness. Even if it isn’t the formal structure of the organization, it could be properties of the people who seeded the organization, properties of the organization’s mission, etc. I am taking a somewhat broad view of “institutional design” that would include these things too. I am not actually saying anything less trivial than “some orgs work much better than other orgs and understanding why is important so this can be replicated and built upon”.
Yes, the general project of examining and documenting working institutions is high-priority. There will probably be multiple difficulties in directly applying SpaceX’s model (e.g. the fact that SpaceX is more engineering than research), so documenting multiple institutions would help. I am otherwise occupied right now, though.
Yes, this would be my suggestion for donors.
I appreciate the introspection you’ve done on this, this is useful information. I think there’s a general issue where the AI alignment problem is really hard, so most people try to push the responsibility somewhere else, and those that do take responsibility usually end up feeling like they are highly constrained into doing the “responsible” thing (e.g. using defensible bureaucratic systems rather than their intuition), which is often at odds with the straightforward just-solve-it mental motion (which is used in, for example, playing video games), or curiosity in general (e.g. mathematical or scientific). I’ve experienced this personally. I recommend Ben’s post on this dynamic. This is part of why I exited the charity sector and don’t identify as an EA anymore. I don’t know how to fix this other than by taking the whole idea of non-local responsibility (i.e. beyond things like “I am responsible for driving safely and paying my bills on time”) less seriously, so I kind of do that.
Yes, both awards and tenure seem like improvements, and in any case well worth experimenting with.
SpaceX might be more competitive than its competitors not because it’s particularly functional (i.e., compared to typical firms in other fields) but because its competitors are particularly non-functional.
If it is particularly effective but not due to the formal structure of the organization, then those informal things are likely to be hard to copy to another organization.
SpaceX can be very successful by having just marginally better institutional design than its competitors because they’re all trying to do the same things. A successful AI alignment organization however would have to be much more effective than organizations that are trying to build unaligned AI (or nominally trying to build aligned AI but cutting lots of corners to win the race, or following some AI alignment approach that seems easy but is actually fatally flawed).
I don’t think that playing video games, math, and science are good models here because those all involve relatively fast feedback cycles which make it easy to build up good intuitions. It seems reasonable to not trust one’s intuitions in AI alignment, and the desire to appear defensible also seems understandable and hard to eliminate, but perhaps we can come up with better “defensible bureaucratic systems” than what exists today, i.e., systems that can still appear defensible but make better decisions than they currently do. I wonder if this problem has been addressed by anyone.
Ok, I made the suggestion to BERI since it seems like they might be open to this kind of thing.
ETA: Another consideration against individuals directly funding other individuals is that it wouldn’t be tax deductible. This could reduce the funding by up to 40%. If the funding is done through a tax-exempt non-profit, then IRS probably has some requirements about having formal procedures for deciding who/what to fund.
(Just got around to reading this. As a point of reference, it seems that at least Open Phil seems to have decided that tax-deductability is not more important than being able to give to things freely, which is why the Open Philanthropy Project is an LLC. I think this is at least slight evidence towards that tradeoff being worth it.)
There’s an enormous difference between having millions of dollars of operating expenditures in an LLC (so that an org is legally allowed to do things like investigate non-deductible activities like investment or politics), and giving up the ability to make billions of dollars of tax-deductible donations. Open Philanthropy being an LLC (so that its own expenses aren’t tax-deductible, but it has LLC freedom) doesn’t stop Good Ventures from making all relevant donations tax-deductible, and indeed the overwhelming majority of grants on its grants page are deductible.
Yep, sorry. I didn’t mean to imply that all of Open Phil’s funding is non-deductible, just that they decided that it was likely enough that they would find non-deductible opportunities that they went through the effort of restructuring their org to do so (and also gave up a bunch of other benefits like the ability to sponsor visas efficiently). My comment wasn’t very clear on that.
Here’s Open Phil’s blog post on why they decided to operate as an LLC. After reading it, I think their reasons are not very relevant to funding AI alignment research. (Mainly they want the freedom to recommend donations to non-501(c)(3) organizations like political groups.)
I am also pretty interested in 2 (ex-post giving). In 2015, there was impactpurchase.org. I got in contact with them about it, and the major updates Paul reported were a) being willing to buy partial contributions (not just for people who were claiming full responsibility for things) and b) more focused on what’s being funded (like for example, only asking for people to submit claims on blog posts and articles).
I realise that things like impactpurchase is possibly framed in terms of a slightly divergent reason for 2 (it seems more focused on changing the incentive landscape, whereas the posts above include thinking about whether giving slack to people with track records will lead those people to be counterfactually more effective in future).