I greatly appreciate this post. I feel like “argh yeah it’s really hard to guarantee that actions won’t have huge negative consequences, and plenty of popular actions might actually be really bad, and the road to hell is paved with good intentions.” With that being said, I have some comments to consider.
The offices cost $70k/month on rent [1], and around $35k/month on food and drink, and ~$5k/month on contractor time for the office. It also costs core Lightcone staff time which I’d guess at around $75k/year.
That is ~$185k/month and ~$2.22m/year. I wonder if the cost has anything to do with the decision? There may be a tendency to say “an action is either extremely good or extremely bad because it either reduces x-risk or increases x-risk, so if I think it’s net positive I should be willing to spend huge amounts of money.” I think this framing neglects a middle ground of “an action could be somewhere in between extremely good and extremely bad.” Perhaps the net effects of the offices were “somewhat good, but not enough to justify the monetary cost.” I guess Ben sort of covers this point later (“Having two locations comes with a large cost”).
its value was substantially dependent on the existing EA/AI Alignment/Rationality ecosystem being roughly on track to solve the world’s most important problems, and that while there are issues, pouring gas into this existing engine, and ironing out its bugs and problems, is one of the most valuable things to do in the world.
Huh, it might be misleading to view the offices as “pouring gas into the engine of the entire EA/AI Alignment/Rationality ecosystem.” They contribute to some areas much more than others. Even if one thinks that the overall ecosystem is net harmful, there could still be ecosystem-building projects that are net helpful. It seems highly unlikely to me that all ecosystem-building projects are bad.
The Lighthouse system is going away when the leases end. Lighthouse 1 has closed, and Lighthouse 2 will continue to be open for a few more months.
These are group houses for members of the EA/AI Alignment/Rationality ecosystem, correct? Relating to the last point, I expect the effects of these to be quite different from the effects of the offices.
FTX is the obvious way in which current community-building can be bad, though in my model of the world FTX, while somewhat of outlier in scope, doesn’t feel like a particularly huge outlier in terms of the underlying generators.
I’m very unsure about this, because it seems plausible that SBF would have done something terrible without EA encouragement. Also, I’m confused about the detailed cause-and-effect analysis of how the offices will contribute to SBF-style catastrophes—is the idea that “people will talk in the offices and then get stupid ideas, and they won’t get equally stupid ideas without the offices?”
Overall I’m getting a sense of “look, there are bad things happening so the whole system must be bad.” Additionally, I think the negative impact of “mindkilly PR-stuff” is pretty insubstantial. On a related note, I somewhat agree with the idea that “most successful human ventures look—from up close—like dumpster fires.” It’s worth being wary of inferences resembling “X evokes a sense of disgust, so X is probably really harmful.”
I genuinely only have marginally better ability to distinguish the moral character of Anthropic’s leadership from the moral character of FTX’s leadership
Yeah this makes sense. I would really love to gain a clear understanding of who has power at the top AGI labs and what their views are on AGI risk. AFAIK nobody has done a detailed analysis of this?
Also, as in the case of RLHF, it’s worth noting that there are some reasonable arguments for Anthropic being helpful.
I think AI Alignment ideas/the EA community/the rationality community played a pretty substantial role in the founding of the three leading AGI labs (Deepmind, OpenAI, Anthropic)
Definitely true for Anthropic. For OpenAI I’m less sure; IIRC the argument is that there were lots of EA-related conferences that contributed to the formation of OpenAI, and I’d like to see more details than this; “there were EA events where key players talked” feels quite different from “without EA, OpenAI would not exist.” I feel similarly about DeepMind; IIRC Eliezer accidentally convinced one of the founders to work on AGI—are there other arguments?
And again, how do the Lightcone offices specifically contribute to the founding of more leading AGI labs? My impression is that the offices’ vibe conveyed a strong sense of “it’s bad to shorten timelines.”
It’s a bad idea to train models directly on the internet
I’m confused how the offices contribute to this.
The EA and AI Alignment community should probably try to delay AI development somehow, and this will likely include getting into conflict with a bunch of AI capabilities organizations, but it’s worth the cost
Again, I’m confused how the offices have a negative impact from this perspective. I feel this way about quite a few of the points in the list.
I do sure feel like a lot of AI alignment research is very suspiciously indistinguishable from capabilities research
...
It also appears that people who are concerned about AGI risk have been responsible for a very substantial fraction of progress towards AGI
...
A lot of people in AI Alignment I’ve talked to have found it pretty hard to have clear thoughts in the current social environment
To me these seem like some of the best reasons (among those in the list; I think Ben provides some more) to shut down the offices. The disadvantage of the list format is that it makes all the points seem equally important; it might be good to bold the points you see as most important or provide a numerical estimate for what percentage of the negative expect impact comes from each point.
The moral maze nature of the EA/longtermist ecosystem has increased substantially over the last two years, and the simulacra level of its discourse has notably risen too.
I feel similar to the way I felt about the “mindkilly PR-stuff”; I don’t think the negative impact is very high in magnitude.
the primary person taking orders of magnitudes more funding and staff talent (Dario Amodei) has barely explicated his views on the topic and appears (from a distance) to have disastrously optimistic views about how easy alignment will be and how important it is to stay competitive with state of the art models
Agreed. I’m confused about Dario’s views.
I recall at EAG in Oxford a year or two ago, people were encouraged to “list their areas of expertise” on their profile, and one person who works in this ecosystem listed (amongst many things) “Biorisk” even though I knew the person had only been part of this ecosystem for <1 year and their background was in a different field.
This seems very trivial to me. IIRC the Swapcard app just says “list your areas of expertise” or something, with very little details about what qualifies as expertise. Some people might interpret this as “list the things you’re currently working on.”
It also seems to me like people who show any intelligent thought or get any respect in the alignment field quickly get elevated to “great researchers that new people should learn from” even though I think that there’s less than a dozen people who’ve produced really great work
Could you please list the people who’ve produced really great work?
I similarly feel pretty worried by how (quite earnest) EAs describe people or projects as “high impact” when I’m pretty sure that if they reflected on their beliefs, they honestly wouldn’t know the sign of the person or project they were talking about, or estimate it as close-to-zero.
Strongly agree. Relatedly, I’m concerned that people might be exhibiting a lot of action bias.
Last point, unrelated to the quote: it feels like this post is entirely focused on the possible negative impacts of the offices, and that kind of analysis seems very likely to arrive at incorrect conclusions since it fails to consider the possible positive impacts. Granted, this post was a scattered collection of Slack messages, so I assume the Lightcone team has done more formal analyses internally.
That is ~$185k/month and ~$2.22m/year. I wonder if the cost has anything to do with the decision? There may be a tendency to say “an action is either extremely good or extremely bad because it either reduces x-risk or increases x-risk, so if I think it’s net positive I should be willing to spend huge amounts of money.”
I don’t think cost had that much to do with the decision, I expect that Open Philanthropy thought it was worth the money and would have been willing to continue funding at this price point.
In general I think the correct response to uncertainty is not half-speed. In my opinion it was the right call to spend this amount of funding on the office for the last ~6 months of its existence even when we thought we’d likely do something quite different afterwards, because it was still marginally worth doing it and the cost-effectiveness calculations for the use of billions of dollars of x-risk money on the current margin are typically quite extreme.
These are group houses for members of the EA/AI Alignment/Rationality ecosystem, correct?
Not quite. They were houses where people could book to visit for up to 3 weeks at a time, commonly used by people visiting the office or in town for a bit for another event/conference/retreat. Much more like AirBnbs than group houses.
I’m confused about the detailed cause-and-effect analysis of how the offices will contribute to SBF-style catastrophes—is the idea that “people will talk in the offices and then get stupid ideas, and they won’t get equally stupid ideas without the offices?”
I think the most similar story is “A smart, competent, charismatic, person with horrible ethics will enter the office because they’ve managed to get good standing in the EA/longtermist ecosystem, cause a bunch of other very smart and competent people to work for them on the basis of expecting to do good in the world, and then do something corrupting and evil with them instead.”
I don’t think cost had that much to do with the decision, I expect that Open Philanthropy thought it was worth the money and would have been willing to continue funding at this price point.
In general I think the correct response to uncertainty is not half-speed. In my opinion it was the right call to spend this amount of funding on the office for the last ~6 months of its existence even when we thought we’d likely do something quite different afterwards, because it was still marginally worth doing it and the cost-effectiveness calculations for the use of billions of dollars of x-risk money on the current margin are typically quite extreme.
You’re probably not the one to rant to about funding but I guess while the conversation is open I could use additional feedback and some reasons for why OpenPhil wouldn’t be irresponsible in spending the money that way. (I only talk about OpenPhil and not particularly Lightcone, maybe you couldn’t think of better ways to spend the money and didn’t have other options)
Cost effectiveness calculations for reducing x-risk kinda always favor x-risk reduction so looking at it in the absolute isn’t relevant. Currently AI x-risk reduction work is limited because of severe funding restrictions (there are many useful things to do that no one is doing for lack of money) which should warrant carefully done triage (and in particular considering the counterfactual).
I assume the average Lightcone office resident would be doing the same work with slightly reduced productivity (let’s say 1⁄3) if they didn’t have that office space (notably because many are rich enough to get other shared office space from their own pocket). Assuming 30 full time workers in the office, that’s 10 man months per month of extra x-risk reduction work.
For contrast, on the same time period, $185k/month could provide for salary, lodging and office space for 50 people in Europe, all who counterfactually would not be doing that work otherwise, for which I claim 50 man months per month of extra x-risk reduction work. The biggest difference I see is incubation time would be longer than for the Lightcone offices, but if I start now with $20k/month I’d find 5 people and scale it up to 50 by the end of the year.
For contrast, on the same time period, $185k/month could provide for salary, lodging and office space for 50 people in Europe, all who counterfactually would not be doing that work otherwise, for which I claim 50 man months per month of extra x-risk reduction work.
The default outcome of giving people money, is either nothing, noise, or the resources getting captured by existing incentive gradients. In my experience, if you give people free money, they will take it, and they will nominally try to please you with it, so it’s not that surprising if you can find 50 people to take your free money, but causing such people to do specific and hard things is a much higher level of challenge.
I had some hope that “just write good LessWrong posts” is sufficient incentive to get people to do useful stuff, but the SERI MATS scholars have tried this and only a few have produced great LessWrong posts, and otherwise there was a lot of noise. Perhaps it’s worth it in expected value but my guess is that you could do much more selection and save a lot of the money and still get 80% of the value.
I think free office spaces of the sort we offered are only worthwhile inside an ecosystem where there are teams already working on good projects, and already good incentive gradients to climb, such that pouring in resources get invested well even with little discernment from those providing them. In contrast, simply creating free resources and having people come for those with the label of your goal on them, sounds like a way to get all the benefits of goodharting and none of the benefits of the void.
In my opinion it was the right call to spend this amount of funding on the office for the last ~6 months of its existence even when we thought we’d likely do something quite different afterwards
This is confusing to me. Why not do “something quite different” from the start?
I’m trying to point at opportunity costs more than “gee, that’s a lot of money, the outcome had better be good!” There are many other uses for that money besides the Lightcone offices.
A smart, competent, charismatic, person with horrible ethics will enter the office because they’ve managed to get good standing in the EA/longtermist ecosystem
My current understanding is that Sam gained good standing as a result of having lots of money for EA causes, not as a result of being charismatic in EA spaces? My sense is that the person you mentioned would struggle to gain good standing in the Lightcone offices without any preexisting money or other power.
My current understanding is that Sam gained good standing as a result of having lots of money for EA causes, not as a result of being charismatic in EA spaces? My sense is that the person you mentioned would struggle to gain good standing in the Lightcone offices without any preexisting money or other power.
No, he gained good standing from being around the EA community for so many years and having sophisticated ethical views (veganism, a form of utilitarianism, etc) and convincing well-respected EAs to work with him and fund him, as well as from having a lot of money and donating it to these spaces. Had the Lightcone Offices existed 5 years ago I expect he would have been invited to work from there (had he asked to), and that was at the start of Alameda.
Sorry if I wrote unclearly. For most of the time (even in the last 6 months) I thought it was worth continuing to support the ecosystem, and certainly to support the people in the office, even if I was planning later to move on. I wanted to move on primarily because of the opportunity cost — I thought we could do something greater. But I believe Habryka wanted to separate from the whole ecosystem and question whether the resources we were providing were actually improving the world at all, and at that point it’s not simply a question of opportunity cost but of whether you’re helping or hurting. If you’re worried that you’re not even helping but just making the problem worse, then it’s a much stronger reason to stop.
You seem to think it wasn’t worth it because of opportunity costs alone? I have been used to the world for a while now where there are two multi-billion dollar funders who are interested in funding x-risk work who don’t have enough things to spend their money on, so I didn’t feel like this was really competing with much else. Just because Lightcone was spending that money didn’t mean another project didn’t get money, none of the major funders were (or are) spending close to their maximum burn rate.
I greatly appreciate this post. I feel like “argh yeah it’s really hard to guarantee that actions won’t have huge negative consequences, and plenty of popular actions might actually be really bad, and the road to hell is paved with good intentions.” With that being said, I have some comments to consider.
That is ~$185k/month and ~$2.22m/year. I wonder if the cost has anything to do with the decision? There may be a tendency to say “an action is either extremely good or extremely bad because it either reduces x-risk or increases x-risk, so if I think it’s net positive I should be willing to spend huge amounts of money.” I think this framing neglects a middle ground of “an action could be somewhere in between extremely good and extremely bad.” Perhaps the net effects of the offices were “somewhat good, but not enough to justify the monetary cost.” I guess Ben sort of covers this point later (“Having two locations comes with a large cost”).
Huh, it might be misleading to view the offices as “pouring gas into the engine of the entire EA/AI Alignment/Rationality ecosystem.” They contribute to some areas much more than others. Even if one thinks that the overall ecosystem is net harmful, there could still be ecosystem-building projects that are net helpful. It seems highly unlikely to me that all ecosystem-building projects are bad.
These are group houses for members of the EA/AI Alignment/Rationality ecosystem, correct? Relating to the last point, I expect the effects of these to be quite different from the effects of the offices.
I’m very unsure about this, because it seems plausible that SBF would have done something terrible without EA encouragement. Also, I’m confused about the detailed cause-and-effect analysis of how the offices will contribute to SBF-style catastrophes—is the idea that “people will talk in the offices and then get stupid ideas, and they won’t get equally stupid ideas without the offices?”
Worth noting that there is plenty of room for debate on the impacts of RLHF, including the discussion in the linked post.
Overall I’m getting a sense of “look, there are bad things happening so the whole system must be bad.” Additionally, I think the negative impact of “mindkilly PR-stuff” is pretty insubstantial. On a related note, I somewhat agree with the idea that “most successful human ventures look—from up close—like dumpster fires.” It’s worth being wary of inferences resembling “X evokes a sense of disgust, so X is probably really harmful.”
Yeah this makes sense. I would really love to gain a clear understanding of who has power at the top AGI labs and what their views are on AGI risk. AFAIK nobody has done a detailed analysis of this?
Also, as in the case of RLHF, it’s worth noting that there are some reasonable arguments for Anthropic being helpful.
Definitely true for Anthropic. For OpenAI I’m less sure; IIRC the argument is that there were lots of EA-related conferences that contributed to the formation of OpenAI, and I’d like to see more details than this; “there were EA events where key players talked” feels quite different from “without EA, OpenAI would not exist.” I feel similarly about DeepMind; IIRC Eliezer accidentally convinced one of the founders to work on AGI—are there other arguments?
And again, how do the Lightcone offices specifically contribute to the founding of more leading AGI labs? My impression is that the offices’ vibe conveyed a strong sense of “it’s bad to shorten timelines.”
I’m confused how the offices contribute to this.
Again, I’m confused how the offices have a negative impact from this perspective. I feel this way about quite a few of the points in the list.
To me these seem like some of the best reasons (among those in the list; I think Ben provides some more) to shut down the offices. The disadvantage of the list format is that it makes all the points seem equally important; it might be good to bold the points you see as most important or provide a numerical estimate for what percentage of the negative expect impact comes from each point.
I feel similar to the way I felt about the “mindkilly PR-stuff”; I don’t think the negative impact is very high in magnitude.
Agreed. I’m confused about Dario’s views.
This seems very trivial to me. IIRC the Swapcard app just says “list your areas of expertise” or something, with very little details about what qualifies as expertise. Some people might interpret this as “list the things you’re currently working on.”
Could you please list the people who’ve produced really great work?
Strongly agree. Relatedly, I’m concerned that people might be exhibiting a lot of action bias.
Last point, unrelated to the quote: it feels like this post is entirely focused on the possible negative impacts of the offices, and that kind of analysis seems very likely to arrive at incorrect conclusions since it fails to consider the possible positive impacts. Granted, this post was a scattered collection of Slack messages, so I assume the Lightcone team has done more formal analyses internally.
A few replies:
I don’t think cost had that much to do with the decision, I expect that Open Philanthropy thought it was worth the money and would have been willing to continue funding at this price point.
In general I think the correct response to uncertainty is not half-speed. In my opinion it was the right call to spend this amount of funding on the office for the last ~6 months of its existence even when we thought we’d likely do something quite different afterwards, because it was still marginally worth doing it and the cost-effectiveness calculations for the use of billions of dollars of x-risk money on the current margin are typically quite extreme.
Not quite. They were houses where people could book to visit for up to 3 weeks at a time, commonly used by people visiting the office or in town for a bit for another event/conference/retreat. Much more like AirBnbs than group houses.
I think the most similar story is “A smart, competent, charismatic, person with horrible ethics will enter the office because they’ve managed to get good standing in the EA/longtermist ecosystem, cause a bunch of other very smart and competent people to work for them on the basis of expecting to do good in the world, and then do something corrupting and evil with them instead.”
There are other stories too.
You’re probably not the one to rant to about funding but I guess while the conversation is open I could use additional feedback and some reasons for why OpenPhil wouldn’t be irresponsible in spending the money that way. (I only talk about OpenPhil and not particularly Lightcone, maybe you couldn’t think of better ways to spend the money and didn’t have other options)
Cost effectiveness calculations for reducing x-risk kinda always favor x-risk reduction so looking at it in the absolute isn’t relevant. Currently AI x-risk reduction work is limited because of severe funding restrictions (there are many useful things to do that no one is doing for lack of money) which should warrant carefully done triage (and in particular considering the counterfactual).
I assume the average Lightcone office resident would be doing the same work with slightly reduced productivity (let’s say 1⁄3) if they didn’t have that office space (notably because many are rich enough to get other shared office space from their own pocket). Assuming 30 full time workers in the office, that’s 10 man months per month of extra x-risk reduction work.
For contrast, on the same time period, $185k/month could provide for salary, lodging and office space for 50 people in Europe, all who counterfactually would not be doing that work otherwise, for which I claim 50 man months per month of extra x-risk reduction work. The biggest difference I see is incubation time would be longer than for the Lightcone offices, but if I start now with $20k/month I’d find 5 people and scale it up to 50 by the end of the year.
The default outcome of giving people money, is either nothing, noise, or the resources getting captured by existing incentive gradients. In my experience, if you give people free money, they will take it, and they will nominally try to please you with it, so it’s not that surprising if you can find 50 people to take your free money, but causing such people to do specific and hard things is a much higher level of challenge.
I had some hope that “just write good LessWrong posts” is sufficient incentive to get people to do useful stuff, but the SERI MATS scholars have tried this and only a few have produced great LessWrong posts, and otherwise there was a lot of noise. Perhaps it’s worth it in expected value but my guess is that you could do much more selection and save a lot of the money and still get 80% of the value.
I think free office spaces of the sort we offered are only worthwhile inside an ecosystem where there are teams already working on good projects, and already good incentive gradients to climb, such that pouring in resources get invested well even with little discernment from those providing them. In contrast, simply creating free resources and having people come for those with the label of your goal on them, sounds like a way to get all the benefits of goodharting and none of the benefits of the void.
This is confusing to me. Why not do “something quite different” from the start?
I’m trying to point at opportunity costs more than “gee, that’s a lot of money, the outcome had better be good!” There are many other uses for that money besides the Lightcone offices.
My current understanding is that Sam gained good standing as a result of having lots of money for EA causes, not as a result of being charismatic in EA spaces? My sense is that the person you mentioned would struggle to gain good standing in the Lightcone offices without any preexisting money or other power.
No, he gained good standing from being around the EA community for so many years and having sophisticated ethical views (veganism, a form of utilitarianism, etc) and convincing well-respected EAs to work with him and fund him, as well as from having a lot of money and donating it to these spaces. Had the Lightcone Offices existed 5 years ago I expect he would have been invited to work from there (had he asked to), and that was at the start of Alameda.
Sorry if I wrote unclearly. For most of the time (even in the last 6 months) I thought it was worth continuing to support the ecosystem, and certainly to support the people in the office, even if I was planning later to move on. I wanted to move on primarily because of the opportunity cost — I thought we could do something greater. But I believe Habryka wanted to separate from the whole ecosystem and question whether the resources we were providing were actually improving the world at all, and at that point it’s not simply a question of opportunity cost but of whether you’re helping or hurting. If you’re worried that you’re not even helping but just making the problem worse, then it’s a much stronger reason to stop.
You seem to think it wasn’t worth it because of opportunity costs alone? I have been used to the world for a while now where there are two multi-billion dollar funders who are interested in funding x-risk work who don’t have enough things to spend their money on, so I didn’t feel like this was really competing with much else. Just because Lightcone was spending that money didn’t mean another project didn’t get money, none of the major funders were (or are) spending close to their maximum burn rate.