tlevin

• tlevinFeb 28, 2025, 3:23 AM

  1 point

  0

  in reply to: Anthony DiGiovanni’s comment on: tlevin’s Shortform

  Sequence thinking can totally generate that, but it seems like it is also prone to this kind of stylized simple model where you wind up with too few arrows in your causal graph and then inaccurately conclude that some parts are necessary and others aren’t helpful.

• tlevinFeb 25, 2025, 4:44 AM

  31 points

  3

  on: tlevin’s Shortform

  Biggest disagreement between the average worldview of people I met with at EAG and my own is something like “cluster thinking vs sequence thinking,” where people at EAG were often like “but even if we get this specific policy/​technical win, doesn’t it not matter unless you also have this other, harder thing?” and I was often more like, “Well, very possibly we won’t get that other, harder thing, but still seems really useful to get that specific policy/​technical win, here’s a story where we totally fail on that first thing and the second thing turns out to matter a ton!”

Skep­ti­cism to­wards claims about the views of pow­er­ful institutions

• tlevinDec 3, 2024, 4:31 PM

  3 points

  0

  in reply to: Zac Hatfield-Dodds’s comment on: A case for donat­ing to AI risk re­duc­tion (in­clud­ing if you work in AI)

  Agreed, I think people should apply a pretty strong penalty when evaluating a potential donation that has or worsens these dynamics. There are some donation opportunities that still have the “major donors won’t [fully] fund it” and “I’m advantaged to evaluate it as an AIS professional” without the “I’m personal friends with the recipient” weirdness, though—e.g. alignment approaches or policy research/​advocacy directions you find promising that Open Phil isn’t currently funding that would be executed thousands of miles away.

A case for donat­ing to AI risk re­duc­tion (in­clud­ing if you work in AI)

• tlevinNov 20, 2024, 8:22 PM

  5 points

  2

  in reply to: Orpheus16’s comment on: Akash’s Shortform

  Depends on the direction/​magnitude of the shift!

  I’m currently feeling very uncertain about the relative costs and benefits of centralization in general. I used to be more into the idea of a national project that centralized domestic projects and thus reduced domestic racing dynamics (and arguably better aligned incentives), but now I’m nervous about the secrecy that would likely entail, and think it’s less clear that a non-centralized situation inevitably leads to a decisive strategic advantage for the leading project. Which is to say, even under pretty optimistic assumptions about how much such a project invests in alignment, security, and benefit-sharing, I’m pretty uncertain that this would be good, and with more realistic assumptions I probably lean towards it being bad. But it super depends on the governance, the wider context, how a “Manhattan Project” would affect domestic companies and China’s policymaking, etc.

  (I think a great start would be not naming it after the Manhattan Project, though. It seems path dependent, and that’s not a great first step.)

• tlevinAug 30, 2024, 12:27 AM

  3 points

  2

  in reply to: habryka’s comment on: tlevin’s Shortform

  It’s not super clear whether from a racing perspective having an equal number of nukes is bad. I think it’s genuinely messy (and depends quite sensitively on how much actors are scared of losing vs. happy about winning vs. scared of racing).

  
  Importantly though, once you have several thousand nukes the strategic returns to more nukes drop pretty close to zero, regardless of how many your opponents have, while if you get the scary model’s weights and then don’t use them to push capabilities even more, your opponent maybe gets a huge strategic advantage over you. I think this is probably true, but the important thing is whether the actors think it might be true.

  In-general I think it’s very hard to predict whether people will overestimate or underestimate things. I agree that literally right now countries are probably underestimating it, but an overreaction in the future also wouldn’t surprise me very much (in the same way that COVID started with an underreaction, and then was followed by a massive overreaction).

  Yeah, good point.

• tlevinAug 29, 2024, 11:58 PM

  3 points

  0

  in reply to: habryka’s comment on: tlevin’s Shortform

  Yeah doing it again it works fine, but it was just creating a long list of empty bullet points (I also have this issue in GDocs sometimes)

• tlevinAug 29, 2024, 11:56 PM

  3 points

  2

  in reply to: habryka’s comment on: tlevin’s Shortform

  Gotcha. A few disanalogies though—the first two specifically relate to the model theft/​shared access point, the latter is true even if you had verifiable API access:

  1. Me verifying how many nukes you have doesn’t mean I suddenly have that many nukes, unlike AI model capabilities, though due to compute differences it does not mean we suddenly have the same time distance to superintelligence.

  2. Me having more nukes only weakly enables me to develop more nukes faster, unlike AI that can automate a lot of AI R&D.

  3. This model seems to assume you have an imprecise but accurate estimate of how many nukes I have, but companies will probably be underestimating the proximity of each other to superintelligence, for the same reason that they’re underestimating their own proximity to superintelligence, until it’s way more salient/​obvious.

• tlevinAug 29, 2024, 11:49 PM

  6 points

  1

  on: Monthly Roundup #21: Au­gust 2024

  In general, we should be wary of this sort of ‘make things worse in order to make things better.’ You are making all conversations of all sizes worse in order to override people’s decisions.

  Glad to be included in the roundup, but two issues here.

  First, it’s not about overriding people’s decisions; it’s a collective action problem. When the room is silent and there’s a single group of 8, I don’t actually face a choice of a 2- or 3-person conversation; it doesn’t exist! The music lowers the costs for people to split into smaller conversations, so the people who prefer those now have better choices, not worse.

  Second, this is a Simpson’s Paradox-related fallacy: you are indeed making all conversations more difficult, but in my model, smaller conversations are much better, so by making conversations of all sizes slightly to severely worse but moving the population to smaller conversations, you’re still improving the conversations on net.

• tlevinAug 29, 2024, 11:39 PM

  1 point

  0

  in reply to: habryka’s comment on: tlevin’s Shortform

  Also—I’m not sure I’m getting the thing where verifying that your competitor has a potentially pivotal model reduces racing?

• tlevinAug 29, 2024, 11:37 PM

  3 points

  0

  in reply to: habryka’s comment on: tlevin’s Shortform

  The “how do we know if this is the most powerful model” issue is one reason I’m excited by OpenMined, who I think are working on this among other features of external access tools

• tlevinAug 29, 2024, 11:35 PM

  3 points

  0

  in reply to: habryka’s comment on: tlevin’s Shortform

  If probability of misalignment is low, probability of human+AI coups (including e.g. countries invading each other) is high, and/​or there aren’t huge offense-dominant advantages to being somewhat ahead, you probably want more AGI projects, not fewer. And if you need a ton of compute to go from an AI that can do 99% of AI R&D tasks to an AI that can cause global catastrophe, then model theft is less of a factor. But the thing I’m worried about re: model theft is a scenario like this, which doesn’t seem that crazy:

  • Company/​country X has an AI agent that can do 99% [edit: let’s say “automate 90%”] of AI R&D tasks, call it Agent-GPT-7, and enough of a compute stock to have that train a significantly better Agent-GPT-8 in 4 months at full speed ahead, which can then train a basically superintelligent Agent-GPT-9 in another 4 months at full speed ahead. (Company/​country X doesn’t know the exact numbers, but their 80% CI is something like 2-8 months for each step; company/​country Y has less info, so their 80% CI is more like 1-16 months for each step.)

  • The weights for Agent-GPT-7 are available (legally or illegally) to company/​country Y, which is known to company/​country X.

  • Y has, say, a fifth of the compute. So each of those steps will take 20 months. Symmetrically, company/​country Y thinks it’ll take 10-40 months and company/​country X thinks it’s 5-80.

  • Once superintelligence is in sight like this, both company/​country X and Y become very scared of the other getting it first—in the country case, they are worried it will undermine nuclear deterrence, upend their political system, basically lead to getting taken over by the other. The relevant decisionmakers think this outcome is better than extinction, but maybe not by that much, whereas getting superintelligence before the other side is way better. In the company case, it’s a lot less intense, but they still would much rather get superintelligence than their arch-rival CEO.

  • So, X thinks they have anywhere from 5-80 months before Y has superintelligence, and Y thinks they have 1-16 months. So X and Y both think it’s easily possible, well within their 80% CI, that Y beats X.

  • X and Y have no reliable means of verifying a commitment like “we will spend half our compute on safety testing and alignment research.”

  • If these weights were not available, Y would have a similarly good system in 18 months, 80% CI 12-24.

  So, had the weights not been available to Y, X would be confident that it had 12 + 5 months to manage a capabilities explosion that would have happened in 8 months at full speed; it can spend >half of its compute on alignment/​safety/​etc, and it has 17 rather than 5 months of serial time to negotiate with Y, possibly develop some verification methods and credible mechanisms for benefit/​power-sharing, etc. If various transparency reforms have been implemented, such that the world is notified in ~real-time that this is happening, there would be enormous pressure to do so; I hope and think it will seem super illegitimate to pursue this kind of power without these kinds of commitments. I am much more worried about X not doing this and instead just trying to grab enormous amounts of power if they’re doing it all in secret.

  [Also: I just accidentally went back a page by command-open bracket in an attempt to get my text out of bullet format and briefly thought I lost this comment; thank you in your LW dev capacity for autosave draft text, but also it is weirdly hard to get out of bullets]

• tlevinAug 29, 2024, 9:43 PM

  16 points

  −9

  on: tlevin’s Shortform

  [reposting from Twitter, lightly edited/​reformatted] Sometimes I think the whole policy framework for reducing catastrophic risks from AI boils down to two core requirements—transparency and security—for models capable of dramatically accelerating R&D.

  If you have a model that could lead to general capabilities much stronger than human-level within, say, 12 months, by significantly improving subsequent training runs, the public and scientific community have a right to know this exists and to see at least a redacted safety case; and external researchers need to have some degree of red-teaming access. Probably various other forms of transparency would be useful too. It feels like this is a category of ask that should unite the “safety,” “ethics,” and “accelerationist” communities?

  And the flip side is that it’s very important that a model capable of kicking off that kind of dash to superhuman capabilities not get stolen/​exfiltrated, such that you don’t wind up with multiple actors facing enormous competitive tradeoffs to rush through this process.

  These have some tradeoffs, especially as you approach AGI—e.g. if you develop a system that can do 99% of foundation model training tasks and your security is terrible you do have some good reasons not to immediately announce it—but not if we make progress on either of these before then, IMO. What the Pareto Frontier of transparency and security looks like, and where we should land on that curve, seems like a very important research agenda.

  If you’re interested in moving the ball forward on either of these, my colleagues and I would love to see your proposal and might fund you to work on it!

• tlevinJul 31, 2024, 7:10 AM

  3 points

  2

  in reply to: habryka’s comment on: tlevin’s Shortform

  Seems cheap to get the info value, especially for quieter music? Can be expensive to set up a multi-room sound system, but it’s probably most valuable in the room that is largest/​most prone to large group formation, so maybe worth experimenting with a speaker playing some instrumental jazz or something. I do think the architecture does a fair bit of work already.

• tlevinJul 31, 2024, 1:31 AM

  94 points

  25

  on: tlevin’s Shortform

  I’m confident enough in this take to write it as a PSA: playing music at medium-size-or-larger gatherings is a Chesterton’s Fence situation.

  It serves the very important function of reducing average conversation size: the louder the music, the more groups naturally split into smaller groups, as people on the far end develop a (usually unconscious) common knowledge that it’s too much effort to keep participating in the big one and they can start a new conversation without being unduly disruptive.

  If you’ve ever been at a party with no music where people gravitate towards a single (or handful of) group of 8+ people, you’ve experienced the failure mode that this solves: usually these conversations are then actually conversations of 2-3 people with 5-6 observers, which is usually unpleasant for the observers and does not facilitate close interactions that easily lead to getting to know people.

  By making it hard to have bigger conversations, the music naturally produces smaller ones; you can modulate the volume to have the desired effect on a typical discussion size. Quiet music (e.g. at many dinner parties) makes it hard to have conversations bigger than ~4-5, which is already a big improvement. Medium-volume music (think many bars) facilitates easy conversations of 2-3. The extreme end of this is dance clubs, where very loud music (not coincidentally!) makes it impossible to maintain conversations bigger than 2.

  I suspect that high-decoupler hosts are just not in the habit of thinking “it’s a party, therefore I should put music on,” or even actively think “music makes it harder to talk and hear each other, and after all isn’t that the point of a party?” But it’s a very well-established cultural practice to play music at large gatherings, so, per Chesterton’s Fence, you need to understand what function it plays. The function it plays is to stop the party-destroying phenomenon of big group conversations.

  What links here?

  • Monthly Roundup #21: Au­gust 2024 by Zvi (Aug 20, 2024, 12:20 AM; 22 points)

• tlevinJul 28, 2024, 11:24 PM

  4 points

  3

  in reply to: Orpheus16’s comment on: How the AI safety tech­ni­cal land­scape has changed in the last year, ac­cord­ing to some practitioners

  I agree that that’s the most important change and that there’s reason to think people in Constellation/​the Bay Area in general might systematically under-attend to policy developments, but I think the most likely explanation for the responses concentrating on other things is that I explicitly asked about technical developments that I missed because I wasn’t in the Bay, and the respondents generally have the additional context that I work in policy and live in DC, so responses that centered policy change would have been off-target.

How the AI safety tech­ni­cal land­scape has changed in the last year, ac­cord­ing to some practitioners

• tlevinMay 17, 2024, 6:16 PM

  25 points

  8

  in reply to: habryka’s comment on: William_S’s Shortform

  Kelsey Piper now reports: “I have seen the extremely restrictive off-boarding agreement that contains nondisclosure and non-disparagement provisions former OpenAI employees are subject to. It forbids them, for the rest of their lives, from criticizing their former employer. Even acknowledging that the NDA exists is a violation of it.”

• tlevinMay 2, 2024, 1:29 AM

  5 points

  −4

  in reply to: Orpheus16’s comment on: tlevin’s Shortform

  Quick reactions:

  1. Re: how over-emphasis on “how radical is my ask” vs “what my target audience might find helpful” and generally the importance of making your case well regardless of how radical it is, that makes sense. Though notably the more radical your proposal is (or more unfamiliar your threat models are), the higher the bar for explaining it well, so these do seem related.

  2. Re: more effective actors looking for small wins, I agree that it’s not clear, but yeah, seems like we are likely to get into some reference class tennis here. “A lot of successful organizations that take hard-line positions and (presumably) get a lot of their power/​influence from the ideological purity that they possess & communicate”? Maybe, but I think of like, the agriculture lobby, who just sort of quietly make friends with everybody and keep getting 11-figure subsidies every year, in a way that (I think) resulted more from gradual ratcheting than making a huge ask. “Pretty much no group– whether radical or centrist– has had tangible wins” seems wrong in light of the EU AI Act (where I think both a “radical” FLI and a bunch of non-radical orgs were probably important) and the US executive order (I’m not sure which strategy is best credited there, but I think most people would have counted the policies contained within it as “minor asks” relative to licensing, pausing, etc). But yeah I agree that there are groups along the whole spectrum that probably deserve credit.

  3. Re: poisoning the well, again, radical-ness and being dumb/​uninformed are of course separable but the bar rises the more radical you get, in part because more radical policy asks strongly correlate with more complicated procedural asks; tweaking ECRA is both non-radical and procedurally simple, creating a new agency to license training runs is both outside the DC Overton Window and very procedurally complicated.

  4. Re: incentives, I agree that this is a good thing to track, but like, “people who oppose X are incentivized to downplay the reasons to do X” is just a fully general counterargument. Unless you’re talking about financial conflicts of interest, but there are also financial incentives for orgs pursuing a “radical” strategy to downplay boring real-world constraints, as well as social incentives (e.g. on LessWrong IMO) to downplay boring these constraints and cognitive biases against thinking your preferred strategy has big downsides.

  5. I agree that the CAIS statement, Hinton leaving Google, and Bengio and Hogarth’s writing have been great. I think that these are all in a highly distinct category from proposing specific actors take specific radical actions (unless I’m misremembering the Hogarth piece). Yudkowsky’s TIME article, on the other hand, definitely counts as an Overton Window move, and I’m surprised that you think this has had net positive effects. I regularly hear “bombing datacenters” as an example of a clearly extreme policy idea, sometimes in a context that sounds like it maybe made the less-radical idea seem more reasonable, but sometimes as evidence that the “doomers” want to do crazy things and we shouldn’t listen to them, and often as evidence that they are at least socially clumsy, don’t understand how politics works, etc, which is related to the things you list as the stuff that actually poisons the well. (I’m confused about the sign of the FLI letter as we’ve discussed.)

  6. I’m not sure optimism vs pessimism is a crux, except in very short, like, 3-year timelines. It’s true that optimists are more likely to value small wins, so I guess narrowly I agree that a ratchet strategy looks strictly better for optimists, but if you think big radical changes are needed, the question remains of whether you’re more likely to get there via asking for the radical change now or looking for smaller wins to build on over time. If there simply isn’t time to build on these wins, then yes, better to take a 2% shot at the policy that you actually think will work; but even in 5-year timelines I think you’re better positioned to get what you ultimately want by 2029 if you get a little bit of what you want in 2024 and 2026 (ideally while other groups also make clear cases for the threat models and develop the policy asks, etc.). Another piece this overlooks is the information and infrastructure built by the minor policy changes. A big part of the argument for the reporting requirements in the EO was that there is now going to be an office in the US government that is in the business of collecting critical information about frontier AI models and figuring out how to synthesize it to the rest of government, that has the legal authority to do this, and both the office and the legal authority can now be expanded rather than created, and there will now be lots of individuals who are experienced in dealing with this information in the government context, and it will seem natural that the government should know this information. I think if we had only been developing and advocating for ideal policy, this would not have happened (though I imagine that this is not in fact what you’re suggesting the community do!).