What kind of cooperation is OpenAI supposed to have “trashed”? Did they make AI R&D more competitive? Did they prevent AI safety researchers from talking to each other?
There is a “AI-arms-race-o’meter” and as a result of OpenAI the meter has gone up. This is real bad, because when we start getting close to AGI/APS-AI/etc. it’s gonna be real important that we slow down and go carefully, but we won’t do that if we are racing.
May I suggest that for an organization which really could create superhuman general AI, more important than a vague vow to go slow and be careful, would be a concrete vow to try to solve the problem of friendliness/safety/alignment. (And I’ll promote June Ku’s https://metaethical.ai as the best model we have of what that would look like. Not yet practical, but a step in the right direction.)
Perhaps it is a result of my being located on the margins, but I have never seen slowing the overall AI race as a tactic worth thinking about. That it is already an out-of-control race in which no one even knows all the competitors, has been my model for a long time. I also think one should not proceed on the assumption that there are years (let alone decades) to spare. For all we know, a critical corner could be turned at any moment.
Perhaps it feels different if you’re DeepMind or MIRI; you may feel that you know most of the competitors, and have some chance of influencing them in a collegial way. But was OpenAI’s announcement of GPT-3 in 2020, a different kind of escalation than DeepMind’s announcement of AlphaGo in 2016? OpenAI has been concerned with AI safety from the beginning; they had Paul Christiano on their alignment team for years.
I am not exactly here to say that DeepMind is that much better! :) One thing I dislike about the OP is that it makes it seem like the problem is specifically with OpenAI compared to other companies. If OpenAI came first and then Elon went and founded DeepMind that would approximately just as bad, or even slightly worse.
I agree that maybe an arms race was inevitable, in which case founding OpenAI maybe wasn’t a bad thing after all. Maybe. But maybe not.
It’s true that OpenAI had some great safety researchers. Now most of them have quit. (There are still some that remain). But they probably could have got jobs at DeepMind, so this isn’t relevant to evaluating Elon’s decision.
Also, there’s the whole openness ideal/norm. Terrible idea, for reasons various people (e.g. Scott Alexander) said at the time. (I can try to remember what the post was called if you like… it made the same point as Yudkowsky here, if we haven’t solved alignment yet and we give AI to everyone then we are killing ourselves. If we have solved alignment, great, but that’s the difficult part and we haven’t done that yet. That point and a few others.)
For the sake of discussion, let’s suppose that the next big escalation in AI power is the final one, and that it’s less than five years away. Any thoughts on what the best course of action is?
Warm take: Our main hope lies in the EA/longtermist/AI-safety community. I say this because my first-order answer to your question is “no idea,” so instead I go meta and say “Rally the community. Get organized. Create common knowledge. Get the wisest people in a room together to discuss the problem and make a plan. Everybody stick to the plan.” (The plan will of course be a living document that updates over time in response to new info. The point is that for a crisis, you need organization; you need a chain of command. It seems to be the main way that humans are able to get large numbers of people working effectively together on short notice. The main problem with organizations is that they tend to become corrupt and decay over time, hence the importance of competition/markets/independence. However this is less of a problem on short timescales and anyway what choice do we have?)
We have good-enough alignment for the AIs we have. We don’t have a general solution to alignment that will work for the ASIs we don’t have. We also don’t know whether we we need one, ie. we don’t know that we need to solve ASI alignment beyond getting ASIs to work acceptably.
I constantly see conflations of AI and ASI. It doesn’t give me much faith in amateur (unrelated to industry) efforts at AI safety.
I don’t know what you mean by skipped. Here’s some more concreteness though:
--Thanks to OpenAI, there is more of an “AI research should be made available to everyone” ethos, more of a “Boo anyone who does AI research and doesn’t tell the world what they did or how they did it or even decides not to share the weights!” Insofar as this ethos persists during the crucial period, whichever labs are building AGI will be under more internal and external pressure to publish/share. This makes it harder for them to go slow and be cautious when the stakes are high.
--Thanks to OpenAI, there were two world-leading AGI labs, not one. Obviously it’s a lot harder to coordinate two than one. This is not as bad as it sounds because plausibly before the crucial period more AGI labs would have appeared anyway. But still.
--Thanks to OpenAI, scaling laws and GPT tech are public knowledge now. This is a pretty big deal because it’s motivating lots of other players to start building AGI or AGI-like things, and because it seems to be opening up lots of new profit opportunities for the AI industry which will encourage further investment, shortening timelines and increasing the number of actors that need to coordinate. Again, presumably this would have happened eventually anyway. But OpenAI made it happen faster.
Ironically, I am a believer in FOSS AI models, and I find OpenAI’s influence anything but encouraging in this regard. The only thing they are publicly releasing is marketing nowadays.
Yep! :) But the damage is done; thanks to OpenAI there is now a large(r) group of people who believe in FOSS AI than there otherwise would be, and there are various new actors who have entered the race who wouldn’t have if not for OpenAI publications.
To be clear, I’m not confident FOSS AI is bad. I just think it probably is, for reasons mentioned. I won’t be too surprised if I turn out to be wrong and actually (e.g. for reasons Dirichlet-to-Neumann mentioned, or because FOSS AI is harder to make a profit on and therefore will get less investment and therefore will be developed later, buying more time for safety research) the FOSS AI ethos was net-positive.
I’d be interested to hear your perspective on FOSS AI.
Epistemic status: I am not an expert on this debate, I have not thought very deeply about it, etc.
I am fairly certain that as long as we don’t fail miserably (i.e., a loose misaligned super AI that collapses our civilization), FOSS AI is extremely preferable to proprietary software. The reasons are common to other software projects, though the usefulness and blackboxy-ness of AGI make this particularly important.
I am skeptical of “conspiracies.” I think a publicly auditable, transparent process with frequent peer feedback on a global scale is much more likely to result in trustable results with fewer unforeseen consequences/edge cases.
I am extremely skeptical of the human incentives that a monopoly on AGI encourages. E.g., when was the single time atomic bombs were used? Exactly when there was a monopoly on them.
I don’t see the current DL approaches as at all near achieving efficient AGI that would be dangerous. AI alignment probably needs more concrete capability research IMO. (At least, more capability research is likely to contribute to safety research as well.) I like the world to enjoy having better narrow AI sooner, and I am not convinced delaying things is buying all that much. (Full disclosure: I am weighing the lives of my social bubble and contemporaries more than random future lives. Though if I were to not do this, then it’s likely that intelligence will evolve again in the universe anyhow, and so humanity failing is not that big of a deal? None of our civilization is built that long-termist, too, so it’s pretty out of distribution for me to think about. Related point: I have an unverified impression that the people who advocate slowing capability research are already well off and healthy, so they don’t particularly need technological progress. Perhaps this is an unfair/false intuition I have, but I do have it, and disabusing me of it will change my opinion a bit.)
In a slow takeoff scenario, my intuition is that multiple competing super intelligences will leave us more leverage. (I assume in a fast takeoff scenario the first such intelligence will crush the others in their infancy.)
Safety research seems to be more aligned with academic incentives than business incentives. Proprietary research is less suited to academia though.
Thanks! In case you are interested in my opinion: I think I agree with 1 (but I expect us to fail miserably) and 2 and 6. I disagree with 3 (AGI, unlike nukes, can be used in ways that aren’t threatening and don’t hurt anyone and just make loads of money and save loads of lives. So people will find it hard to resist using it. So the more people have access to it, the more likely it is it’ll be used before it is safe.) My timelines are about 50% by 10 years; I gather from point 4 that yours are longer. I think 5 might be true but might not be; history is full of examples of different groups fighting each other yet still managing to conquer and crush some third group. For example, the Spanish conquistadors were fighting each other even as they conquered Mexico and Peru. Maybe humans will be clever enough to play the AIs off against each other in a way that lets us maintain control until we solve alignment… but I wouldn’t bet on it.
It seems to me that being open about what you are working on, and having a proven record of publishing/sharing critical informations, including weights, is a very good way to fight the arm race.
If you don’t know where your concurrent are, it is much more difficult to stop to think about alignment than to rush toward capacity first. If you know where your concurrent are, and if you know that you will be at worst a couple weeks or months late because they always publish and you will thus be able to catch up, you have much more slack to pursue alignment (or speculative research in general).
For the strategic arms reduction treaties signed between Russia and the USA, verification tools were a crucial part of the process, because you need to know what the other is doing to disarm: https://en.wikipedia.org/wiki/START_I#Verification_tools
Yes, when we are getting really close to AGI it will be good for the leading contenders to share info with each other. Even then it won’t be a good idea for the leading contenders to publish publicly, because then there’ll be way more contenders! And now, when we are not really close to AGI, public publication accelerates research in general and thus shortens timelines, while also bringing more actors into the race.
Trust between partners do not happen overnight. You don’t suddenly begin sharing information with concurrents when the prize is in sight. We need a history of shared information to build upon, and now—when, as you said, AGI is not really close—is the good time to build it. Because if you don’t trust someone with GPT-3, you are certainly not going to trust them with an AGI.
Because if you don’t trust someone with GPT-3, you are certainly not going to trust them with an AGI.
Choosing to not release GPT-3′s weights to the whole world doesn’t imply that you don’t trust DeepMind or Anthropic or whoever. It just implies that there exists at least one person in the world you don’t trust.
I agree that releasing everything publicly would make it easier/more likely to release crucial things to key competitors when the time comes. Alas, the harms are big enough to outweigh this benefit, I think.
What kind of cooperation is OpenAI supposed to have “trashed”? Did they make AI R&D more competitive? Did they prevent AI safety researchers from talking to each other?
There is a “AI-arms-race-o’meter” and as a result of OpenAI the meter has gone up. This is real bad, because when we start getting close to AGI/APS-AI/etc. it’s gonna be real important that we slow down and go carefully, but we won’t do that if we are racing.
May I suggest that for an organization which really could create superhuman general AI, more important than a vague vow to go slow and be careful, would be a concrete vow to try to solve the problem of friendliness/safety/alignment. (And I’ll promote June Ku’s https://metaethical.ai as the best model we have of what that would look like. Not yet practical, but a step in the right direction.)
Perhaps it is a result of my being located on the margins, but I have never seen slowing the overall AI race as a tactic worth thinking about. That it is already an out-of-control race in which no one even knows all the competitors, has been my model for a long time. I also think one should not proceed on the assumption that there are years (let alone decades) to spare. For all we know, a critical corner could be turned at any moment.
Perhaps it feels different if you’re DeepMind or MIRI; you may feel that you know most of the competitors, and have some chance of influencing them in a collegial way. But was OpenAI’s announcement of GPT-3 in 2020, a different kind of escalation than DeepMind’s announcement of AlphaGo in 2016? OpenAI has been concerned with AI safety from the beginning; they had Paul Christiano on their alignment team for years.
I am not exactly here to say that DeepMind is that much better! :) One thing I dislike about the OP is that it makes it seem like the problem is specifically with OpenAI compared to other companies. If OpenAI came first and then Elon went and founded DeepMind that would approximately just as bad, or even slightly worse.
I agree that maybe an arms race was inevitable, in which case founding OpenAI maybe wasn’t a bad thing after all. Maybe. But maybe not.
It’s true that OpenAI had some great safety researchers. Now most of them have quit. (There are still some that remain). But they probably could have got jobs at DeepMind, so this isn’t relevant to evaluating Elon’s decision.
Also, there’s the whole openness ideal/norm. Terrible idea, for reasons various people (e.g. Scott Alexander) said at the time. (I can try to remember what the post was called if you like… it made the same point as Yudkowsky here, if we haven’t solved alignment yet and we give AI to everyone then we are killing ourselves. If we have solved alignment, great, but that’s the difficult part and we haven’t done that yet. That point and a few others.)
For the sake of discussion, let’s suppose that the next big escalation in AI power is the final one, and that it’s less than five years away. Any thoughts on what the best course of action is?
Warm take: Our main hope lies in the EA/longtermist/AI-safety community. I say this because my first-order answer to your question is “no idea,” so instead I go meta and say “Rally the community. Get organized. Create common knowledge. Get the wisest people in a room together to discuss the problem and make a plan. Everybody stick to the plan.” (The plan will of course be a living document that updates over time in response to new info. The point is that for a crisis, you need organization; you need a chain of command. It seems to be the main way that humans are able to get large numbers of people working effectively together on short notice. The main problem with organizations is that they tend to become corrupt and decay over time, hence the importance of competition/markets/independence. However this is less of a problem on short timescales and anyway what choice do we have?)
https://slatestarcodex.com/2015/12/17/should-ai-be-open/
We have good-enough alignment for the AIs we have. We don’t have a general solution to alignment that will work for the ASIs we don’t have. We also don’t know whether we we need one, ie. we don’t know that we need to solve ASI alignment beyond getting ASIs to work acceptably.
I constantly see conflations of AI and ASI. It doesn’t give me much faith in amateur (unrelated to industry) efforts at AI safety.
Taboo “racing”? I don’t understand what concrete actions were thought to have been skipped.
I don’t know what you mean by skipped. Here’s some more concreteness though:
--Thanks to OpenAI, there is more of an “AI research should be made available to everyone” ethos, more of a “Boo anyone who does AI research and doesn’t tell the world what they did or how they did it or even decides not to share the weights!” Insofar as this ethos persists during the crucial period, whichever labs are building AGI will be under more internal and external pressure to publish/share. This makes it harder for them to go slow and be cautious when the stakes are high.
--Thanks to OpenAI, there were two world-leading AGI labs, not one. Obviously it’s a lot harder to coordinate two than one. This is not as bad as it sounds because plausibly before the crucial period more AGI labs would have appeared anyway. But still.
--Thanks to OpenAI, scaling laws and GPT tech are public knowledge now. This is a pretty big deal because it’s motivating lots of other players to start building AGI or AGI-like things, and because it seems to be opening up lots of new profit opportunities for the AI industry which will encourage further investment, shortening timelines and increasing the number of actors that need to coordinate. Again, presumably this would have happened eventually anyway. But OpenAI made it happen faster.
Ironically, I am a believer in FOSS AI models, and I find OpenAI’s influence anything but encouraging in this regard. The only thing they are publicly releasing is marketing nowadays.
Yep! :) But the damage is done; thanks to OpenAI there is now a large(r) group of people who believe in FOSS AI than there otherwise would be, and there are various new actors who have entered the race who wouldn’t have if not for OpenAI publications.
To be clear, I’m not confident FOSS AI is bad. I just think it probably is, for reasons mentioned. I won’t be too surprised if I turn out to be wrong and actually (e.g. for reasons Dirichlet-to-Neumann mentioned, or because FOSS AI is harder to make a profit on and therefore will get less investment and therefore will be developed later, buying more time for safety research) the FOSS AI ethos was net-positive.
I’d be interested to hear your perspective on FOSS AI.
Epistemic status: I am not an expert on this debate, I have not thought very deeply about it, etc.
I am fairly certain that as long as we don’t fail miserably (i.e., a loose misaligned super AI that collapses our civilization), FOSS AI is extremely preferable to proprietary software. The reasons are common to other software projects, though the usefulness and blackboxy-ness of AGI make this particularly important.
I am skeptical of “conspiracies.” I think a publicly auditable, transparent process with frequent peer feedback on a global scale is much more likely to result in trustable results with fewer unforeseen consequences/edge cases.
I am extremely skeptical of the human incentives that a monopoly on AGI encourages. E.g., when was the single time atomic bombs were used? Exactly when there was a monopoly on them.
I don’t see the current DL approaches as at all near achieving efficient AGI that would be dangerous. AI alignment probably needs more concrete capability research IMO. (At least, more capability research is likely to contribute to safety research as well.) I like the world to enjoy having better narrow AI sooner, and I am not convinced delaying things is buying all that much. (Full disclosure: I am weighing the lives of my social bubble and contemporaries more than random future lives. Though if I were to not do this, then it’s likely that intelligence will evolve again in the universe anyhow, and so humanity failing is not that big of a deal? None of our civilization is built that long-termist, too, so it’s pretty out of distribution for me to think about. Related point: I have an unverified impression that the people who advocate slowing capability research are already well off and healthy, so they don’t particularly need technological progress. Perhaps this is an unfair/false intuition I have, but I do have it, and disabusing me of it will change my opinion a bit.)
In a slow takeoff scenario, my intuition is that multiple competing super intelligences will leave us more leverage. (I assume in a fast takeoff scenario the first such intelligence will crush the others in their infancy.)
Safety research seems to be more aligned with academic incentives than business incentives. Proprietary research is less suited to academia though.
Thanks! In case you are interested in my opinion: I think I agree with 1 (but I expect us to fail miserably) and 2 and 6. I disagree with 3 (AGI, unlike nukes, can be used in ways that aren’t threatening and don’t hurt anyone and just make loads of money and save loads of lives. So people will find it hard to resist using it. So the more people have access to it, the more likely it is it’ll be used before it is safe.) My timelines are about 50% by 10 years; I gather from point 4 that yours are longer. I think 5 might be true but might not be; history is full of examples of different groups fighting each other yet still managing to conquer and crush some third group. For example, the Spanish conquistadors were fighting each other even as they conquered Mexico and Peru. Maybe humans will be clever enough to play the AIs off against each other in a way that lets us maintain control until we solve alignment… but I wouldn’t bet on it.
It seems to me that being open about what you are working on, and having a proven record of publishing/sharing critical informations, including weights, is a very good way to fight the arm race.
If you don’t know where your concurrent are, it is much more difficult to stop to think about alignment than to rush toward capacity first. If you know where your concurrent are, and if you know that you will be at worst a couple weeks or months late because they always publish and you will thus be able to catch up, you have much more slack to pursue alignment (or speculative research in general).
For the strategic arms reduction treaties signed between Russia and the USA, verification tools were a crucial part of the process, because you need to know what the other is doing to disarm:
https://en.wikipedia.org/wiki/START_I#Verification_tools
https://en.wikipedia.org/wiki/New_START#Monitoring_and_verification
Yes, when we are getting really close to AGI it will be good for the leading contenders to share info with each other. Even then it won’t be a good idea for the leading contenders to publish publicly, because then there’ll be way more contenders! And now, when we are not really close to AGI, public publication accelerates research in general and thus shortens timelines, while also bringing more actors into the race.
Trust between partners do not happen overnight. You don’t suddenly begin sharing information with concurrents when the prize is in sight. We need a history of shared information to build upon, and now—when, as you said, AGI is not really close—is the good time to build it.
Because if you don’t trust someone with GPT-3, you are certainly not going to trust them with an AGI.
Choosing to not release GPT-3′s weights to the whole world doesn’t imply that you don’t trust DeepMind or Anthropic or whoever. It just implies that there exists at least one person in the world you don’t trust.
I agree that releasing everything publicly would make it easier/more likely to release crucial things to key competitors when the time comes. Alas, the harms are big enough to outweigh this benefit, I think.
Literally?
Not as far as I’m aware. I don’t think anyone is quantitatively tracking this phenomenon.