Akash

Karma: 6,232

Akash 22 Nov 2024 20:24 UTC
4 points
0
in reply to: Bogdan Ionut Cirstea’s comment on: Bogdan Ionut Cirstea’s Shortform
Thanks for spelling it out. I agree that more people should think about these scenarios. I could see something like this triggering central international coordination (or conflict).
(I still don’t think this would trigger the USG to take different actions in the near-term, except perhaps “try to be more secret about AGI development” and maybe “commission someone to do some sort of study or analysis on how we would handle these kinds of dynamics & what sorts of international proposals would advance US interests while preventing major conflict.” The second thing is a bit optimistic but maybe plausible.)

Akash 22 Nov 2024 18:11 UTC
4 points
2
in reply to: Bogdan Ionut Cirstea’s comment on: Bogdan Ionut Cirstea’s Shortform
What kinds of conflicts are you envisioning?
I think if the argument is something along the lines of “maybe at some point other countries will demand that the US stop AI progress”, then from the perspective of the USG, I think it’s sensible to operate under the perspective of “OK so we need to advance AI progress as much as possible and try to hide some of it, and if at some future time other countries are threatening us we need to figure out how to respond.” But I don’t think it justifies anything like “we should pause or start initiating international agreements.”
(Separately, whether or not it’s “truer” depends a lot on one’s models of AGI development. Most notably: (a) how likely is misalignment and (b) how slow will takeoff be//will it be very obvious to other nations that super advanced AI is about to be developed, and (c) how will governments and bureaucracies react and will they be able to react quickly enough.)
(Also separately– I do think more people should be thinking about how these international dynamics might play out & if there’s anything we can be doing to prepare for them. I just don’t think they naturally lead to a “oh, so we should be internationally coordinating” mentality and instead lead to much more of a “we can do whatever we want unless/until other countries get mad at us & we should probably do things more secretly” mentality.)

Akash 22 Nov 2024 15:51 UTC
2 points
0
in reply to: davekasten’s comment on: Akash’s Shortform
@davekasten I know you posed this question to us, but I’ll throw it back on you :) what’s your best-guess answer?
Or perhaps put differently: What do you think are the factors that typically influence whether the cautious folks or the non-cautious folks end up in charge? Are there any historical or recent examples of these camps fighting for power over an important operation?

Akash 21 Nov 2024 16:28 UTC
2 points
0
in reply to: tlevin’s comment on: Akash’s Shortform
it’s less clear that a non-centralized situation inevitably leads to a decisive strategic advantage for the leading project
Can you say more about what has contributed to this update?

Akash 21 Nov 2024 16:27 UTC
4 points
2
in reply to: Richard_Ngo’s comment on: Akash’s Shortform
Can you say more about scenarios where you envision a later project happening that has different motivations?
I think in the current zeitgeist, such a project would almost definitely be primarily motivated by beating China. It doesn’t seem clear to me that it’s good to wait for a new zeitgeist. Reasons:
- A company might develop AGI (or an AI system that is very good at AI R&D that can get to AGI) before a major zeitgeist change.
- The longer we wait, the more capable the “most capable model that wasn’t secured” is. So we could risk getting into a scenario where people want to pause but since China and the US both have GPT-Nminus1, both sides feel compelled to race forward (whereas this wouldn’t have happened if security had kicked off sooner.)

Akash 20 Nov 2024 20:07 UTC
4 points
0
in reply to: habryka’s comment on: Akash’s Shortform
If you could only have “partial visibility”, what are some of the things you would most want the government to be able to know?

Akash 20 Nov 2024 19:58 UTC
4 points
2
in reply to: MondSemmel’s comment on: Akash’s Shortform
Another frame: If alignment turns out to be easy, then the default trajectory seems fine (at least from an alignment POV. You might still be worried about EG concentration of power).
If alignment turns out to be hard, then the policy decisions we make to affect the default trajectory matter a lot more.
This means that even if misalignment risks are relatively low, a lot of value still comes from thinking about worlds where misalignment is hard (or perhaps “somewhat hard but not intractably hard”).

Akash 20 Nov 2024 17:34 UTC
5 points
0
in reply to: Daniel Kokotajlo’s comment on: Akash’s Shortform
What do you think are the most important factors for determining if it results in them behaving responsibly later?
For instance, if you were in charge of designing the AI Manhattan Project, are there certain things you would do to try to increase the probability that it leads to the USG “behaving more responsibly later?”

Akash 20 Nov 2024 17:18 UTC
4 points
2
in reply to: habryka’s comment on: Akash’s Shortform
Good points. Suppose you were on a USG taskforce that had concluded they wanted to go with the “subsidy model”, but they were willing to ask for certain concessions from industry.
Are there any concessions/arrangements that you would advocate for? Are there any ways to do the “subsidy model” well, or do you think the model is destined to fail even if there were a lot of flexibility RE how to implement it?

Akash 20 Nov 2024 17:13 UTC
33 points
−4
in reply to: Akash’s comment on: Akash’s Shortform
My own impression is that this would be an improvement over the status quo. Main reasons:
- A lot of my P(doom) comes from race dynamics.
- Right now, if a leading lab ends up realizing that misalignment risks are super concerning, they can’t do much to end the race. Their main strategy would be to go to the USG.
- If the USG runs the Manhattan Project (or there’s some sort of soft nationalization in which the government ends up having a much stronger role), it’s much easier for the USG to see that misalignment risks are concerning & to do something about it.
- A national project would be more able to slow down and pursue various kinds of international agreements (the national project has more access to POTUS, DoD, NSC, Congress, etc.)
- I expect the USG to be stricter on various security standards. It seems more likely to me that the USG would EG demand a lot of security requirements to prevent model weights or algorithmic insights from leaking to China. One of my major concerns is that people will want to pause at GPT-X but they won’t feel able to because China stole access to GPT-Xminus1 (or maybe even a slightly weaker version of GPT-X).
- In general, I feel like USG natsec folks are less “move fast and break things” than folks in SF. While I do think some of the AGI companies have tried to be less “move fast and break things” than the average company, I think corporate race dynamics & the general cultural forces have been the dominant factors and undermined a lot of attempts at meaningful corporate governance.
(Caveat that even though I see this as a likely improvement over status quo, this doesn’t mean I think this is the best thing to be advocating for.)
(Second caveat that I haven’t thought about this particular question very much and I could definitely be wrong & see a lot of reasonable counterarguments.)

Akash 20 Nov 2024 17:00 UTC
4 points
0
in reply to: Akash’s comment on: Akash’s Shortform
@davekasten @Zvi @habryka @Rob Bensinger @ryan_greenblatt @Buck @tlevin @Richard_Ngo @Daniel Kokotajlo I suspect you might have interesting thoughts on this. (Feel free to ignore though.)

Akash 20 Nov 2024 16:58 UTC
37 points
0
on: Akash’s Shortform
Suppose the US government pursued a “Manhattan Project for AGI”. At its onset, it’s primarily fuelled by a desire to beat China to AGI. However, there’s some chance that its motivation shifts over time (e.g., if the government ends up thinking that misalignment risks are a big deal, its approach to AGI might change.)
Do you think this would be (a) better than the current situation, (b) worse than the current situation, or (c) it depends on XYZ factors?

Akash 19 Nov 2024 23:11 UTC
7 points
4
in reply to: Sodium’s comment on: Bogdan Ionut Cirstea’s Shortform
We’re not going to be bottlenecked by politicians not caring about AI safety. As AI gets crazier and crazier everyone would want to do AI safety, and the question is guiding people to the right AI safety policies
I think we’re seeing more interest in AI, but I think interest in “AI in general” and “AI through the lens of great power competition with China” has vastly outpaced interest in “AI safety”. (Especially if we’re using a narrow definition of AI safety; note that people in DC often use the term “AI safety” to refer to a much broader set of concerns than AGI safety/misalignment concerns.)
I do think there’s some truth to the quote (we are seeing more interest in AI and some safety topics), but I think there’s still a lot to do to increase the salience of AI safety (and in particular AGI alignment) concerns.

Akash 16 Nov 2024 16:06 UTC
39 points
11
on: OpenAI Email Archives (from Musk v. Altman)
A few quotes that stood out to me:
Greg:
I hope for us to enter the field as a neutral group, looking to collaborate widely and shift the dialog towards being about humanity winning rather than any particular group or company.
Greg and Ilya (to Elon):
The goal of OpenAI is to make the future good and to avoid an AGI dictatorship. You are concerned that Demis could create an AGI dictatorship. So do we. So it is a bad idea to create a structure where you could become a dictator if you chose to, especially given that we can create some other structure that avoids this possibility.
Greg and Ilya (to Altman):
But we haven’t been able to fully trust your judgements throughout this process, because we don’t understand your cost function.
We don’t understand why the CEO title is so important to you. Your stated reasons have changed, and it’s hard to really understand what’s driving it.
Is AGI truly your primary motivation? How does it connect to your political goals? How has your thought process changed over time?

Akash 15 Nov 2024 20:43 UTC
10 points
14
in reply to: MondSemmel’s comment on: Lao Mein’s Shortform
and recently founded another AI company
Potentially a hot take, but I feel like xAI’s contributions to race dynamics (at least thus far) have been relatively trivial. I am usually skeptical of the whole “I need to start an AI company to have a seat at the table”, but I do imagine that Elon owning an AI company strengthens his voice. And I think his AI-related comms have mostly been used to (a) raise awareness about AI risk, (b) raise concerns about OpenAI/Altman, and (c) endorse SB1047 [which he did even faster and less ambiguously than Anthropic].
The counterargument here is that maybe if xAI was in 1st place, Elon’s positions would shift. I find this plausible, but I also find it plausible that Musk (a) actually cares a lot about AI safety, (b) doesn’t trust the other players in the race, and (c) is more likely to use his influence to help policymakers understand AI risk than any of the other lab CEOs.

Akash 15 Nov 2024 20:35 UTC
45 points
27
on: Making a conservative case for alignment
I agree with many points here and have been excited about AE Studio’s outreach. Quick thoughts on China/international AI governance:
- I think some international AI governance proposals have some sort of “kum ba yah, we’ll all just get along” flavor/tone to them, or some sort of “we should do this because it’s best for the world as a whole” vibe. This isn’t even Dem-coded so much as it is naive-coded, especially in DC circles.
- US foreign policy is dominated primarily by concerns about US interests. Other considerations can matter, but they are not the dominant driving force. My impression is that this is true within both parties (with a few exceptions).
- I think folks interested in international AI governance should study international security agreements and try to get a better understanding of relevant historical case studies. Lots of stuff to absorb from the Cold War, the Iran Nuclear Deal, US-China relations over the last several decades, etc. (I’ve been doing this & have found it quite helpful.)
- Strong Republican leaders can still engage in bilateral/multilateral agreements that serve US interests. Recall that Reagan negotiated arms control agreements with the Soviet Union, and the (first) Trump Administration facilitated the Abraham Accords. Being “tough on China” doesn’t mean “there are literally no circumstances in which I would be willing to sign a deal with China.” (But there likely does have to be a clear case that the deal serves US interests, has appropriate verification methods, etc.)

Akash 12 Nov 2024 22:43 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
Did they have any points that you found especially helpful, surprising, or interesting? Anything you think folks in AI policy might not be thinking enough about?
(Separately, I hope to listen to these at some point & send reactions if I have any.)

Akash 2 Nov 2024 21:01 UTC
2 points
0
in reply to: dirk’s comment on: dirk’s Shortform
you have to spend several years resume-building before painstakingly convincing people you’re worth hiring for paid work
For government roles, I think “years of experience” is definitely an important factor. But I don’t think you need to have been specializing for government roles specifically.
Especially for AI policy, there are several programs that are basically like “hey, if you have AI expertise but no background in policy, we want your help.” To be clear, these are often still fairly competitive, but I think it’s much more about being generally capable/competent and less about having optimized your resume for policy roles.

Akash 1 Nov 2024 14:32 UTC
18 points
14
on: The Compendium, A full argument about extinction risk from AGI
I like the section where you list out specific things you think people should do. (One objection I sometimes hear is something like “I know that [evals/RSPs/if-then plans/misc] are not sufficient, but I just don’t really know what else there is to do. It feels like you either have to commit to something tangible that doesn’t solve the whole problem or you just get lost in a depressed doom spiral.”)
I think your section on suggestions could be stronger by presenting more ambitious/impactful stories of comms/advocacy. I think there’s something tricky about a document that has the vibe “this is the most important issue in the world and pretty much everyone else is approaching it the wrong way” and then pivots to “and the right way to approach it is to post on Twitter and talk to your friends.”
My guess is that you prioritized listing things that were relatively low friction and accessible. (And tbc I do think that the world would be in better shape if more people were sharing their views and contributing to the broad discourse.)
But I think when you’re talking to high-context AIS people who are willing to devote their entire career to work on AI Safety, they’ll be interested in more ambitious/sexy/impactful ways of contributing.
Put differently: Should I really quit my job at [fancy company or high-status technical safety group] to Tweet about my takes, talk to my family/friends, and maybe make some website? Or are there other paths I could pursue?
As I wrote here, I think we have some of those ambitious/sexy/high-impact role models that could be used to make this pitch stronger, more ambitious, and more inspiring. EG:
One possible critique is that their suggestions are not particularly ambitious. This is likely because they’re writing for a broader audience (people who haven’t been deeply engaged in AI safety).
For people who have been deeply engaged in AI safety, I think the natural steelman here is “focus on helping the public/government better understand the AI risk situation.”
There are at least some impactful and high-status examples of this (e.g., Hinton, Bengio, Hendrycks). I think in the last few years, for instance, most people would agree that Hinton/Bengio/Hendrycks have had far more impact in their communications/outreach/policy work than their technical research work.
And it’s not just the famous people– I can think of ~10 junior or mid-career people who left technical research in the last year to help policymakers better understand AI progress and AI risk, and I think their work is likely far more impactful than if they had stayed in technical research. (And I think is true even if I exclude technical people who are working on evals/if-then plans in govt. Like, I’m focusing on people who see their primary purpose as helping the public or policymakers develop “situational awareness”, develop stronger models of AI progress and AI risk, understand the conceptual arguments for misalignment risk, etc.)
I’d also be curious to hear what your thoughts are on people joining government organizations (like the US AI Safety Institute, UK AI Safety Institute, Horizon Fellowship, etc.) Most of your suggestions seem to involve contributing from outside government, and I’d be curious to hear more about your suggestions for people who are either working in government or open to working in government.

Akash 1 Nov 2024 14:01 UTC
36 points
16
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
I think there’s something about Bay Area culture that can often get technical people to feel like the only valid way to contribute is through technical work. It’s higher status and sexier and there’s a default vibe that the best way to understand/improve the world is through rigorous empirical research.
I think this an incorrect (or at least incomplete) frame, and I think on-the-margin it would be good for more technical people to spend 1-5 days seriously thinking about what alternative paths they could pursue in comms/policy.
I also think there are memes spreading around that you need to be some savant political mastermind genius to do comms/policy, otherwise you will be net negative. The more I meet policy people (including successful policy people from outside the AIS bubble), the more I think this narrative was, at best, an incorrect model of the world. At worst, a take that got amplified in order to prevent people from interfering with the AGI race (e.g., by granting excess status+validity to people/ideas/frames that made it seem crazy/unilateralist/low-status to engage in public outreach, civic discourse, and policymaker engagement.)
(Caveat: I don’t think the adversarial frame explains everything, and I do think there are lots of people who were genuinely trying to reason about a complex world and just ended up underestimating how much policy interest there would be and/or overestimating the extent to which labs would be able to take useful actions despite the pressures of race dynamics.)