MichaelA

• MichaelAMay 27, 2023, 9:05 AM

  2 points

  0

  in reply to: Olli Järviniemi’s comment on: Sur­vey on in­ter­me­di­ate goals in AI governance

  Glad to hear that!

  I do feel excited about this being used as a sort of “201 level” overview of AI strategy and what work it might be useful to do. And I’m aware of the report being included in the reading lists /​ curricula for two training programs for people getting into AI governance or related work, which was gratifying.

  Unfortunately we did this survey before ChatGPT and various other events since then, which have majorly changed the landscape of AI governance work to be done, e.g. opening various policy windows. So I imagine people reading this report today may feel it has some odd omissions /​ vibes. But I still think it serves as a good 201 level overview despite that. Perhaps we’ll run a followup in a year or two to provide an updated version.

• MichaelAMar 25, 2023, 9:28 AM

  4 points

  in reply to: Maxime Riché’s comment on: Database of ex­is­ten­tial risk estimates

  I’d consider those to be “in-scope” for the database, so the database would include any such estimates that I was aware of and that weren’t too private to share in the database.

  If I recall correctly, some estimates in the database are decently related to that, e.g. are framed as “What % of the total possible moral value of the future will be realized?” or “What % of the total possible moral value of the future is lost in expectation due to AI risk?”

  But I haven’t seen many estimates of that type, and I don’t remember seeing any that were explicitly framed as “What fraction of the accessible universe’s resources will be used in a way optimized for ‘the correct moral theory’?”

  If you know of some, feel free to comment in the database to suggest they be added :)

• MichaelAMar 17, 2023, 1:20 PM

  1 point

  0

  on: Sur­vey on in­ter­me­di­ate goals in AI governance

  ...and while I hopefully have your attention: My team is currently hiring for a Research Manager! If you might be interested in managing one or more researchers working on a diverse set of issues relevant to mitigating extreme risks from the development and deployment of AI, please check out the job ad!

  The application form should take <2 hours. The deadline is the end of the day on March 21. The role is remote and we’re able to hire in most countries.

  People with a wide range of backgrounds could turn out to be the best fit for the role. As such, if you’re interested, please don’t rule yourself out due to thinking you’re not qualified unless you at least read the job ad first!

Sur­vey on in­ter­me­di­ate goals in AI governance

• MichaelAJan 7, 2023, 1:06 PM

  2 points

  2

  in reply to: habryka’s comment on: Let’s think about slow­ing down AI

  I found this thread interesting and useful, but I feel a key point has been omitted thus far (from what I’ve read):

  • Public, elite, and policymaker beliefs and attitudes related to AI risk aren’t just a variable we (members of the EA/​longtermist/​AI safety communities) have to bear in mind and operate in light of, but instead also a variable we can intervene on.

  • And so far I’d say we have (often for very good reasons) done significantly less to intervene on that variable than we could’ve or than we could going forward.

  • So it seems plausible that actually these people are fairly convincible if exposed to better efforts to really explain the arguments in a compelling way.

  We’ve definitely done a significant amount of this kind of work, but I think we’ve often (a) deliberately held back on doing so or on conveying key parts of the arguments, due to reasonable downside risk concerns, and (b) not prioritized this. And I think there’s significantly more we could do if we wanted to, especially after a period of actively building capacity for this.

  Important caveats /​ wet blankets:

  • I think there are indeed strong arguments against trying to shift relevant beliefs and attitudes in a more favorable direction, including not just costs and plausibly low upside but also multiple major plausible downside risks.^[1]

  • So I wouldn’t want anyone to take major steps in this direction without checking in with multiple people working on AI safety/​governance first.

  • And it’s not at all obvious to me we should be doing more of that sort of work. (Though I think whether, how, & when we should is an important question and I’m aware of and excited about a couple small research projects that are happening on that.)

  All I really want to convey in this comment is what I said in my first paragraph: we may be able to significantly push beliefs and opinions in favorable directions relative to where they are now or would be n future by default.

  1. ^

     Due to time constraints, I’ll just point to this vague overview.

• MichaelAJan 7, 2023, 12:47 PM

  3 points

  2

  in reply to: Jan_Kulveit’s comment on: Let’s think about slow­ing down AI

  Personally I haven’t thought about how strong the analogy to GoF is, but another thing that feels worth noting is that there may be a bunch of other cases where the analogy is similarly strong and where major government efforts aimed at risk-reduction have occurred. And my rough sense is that that’s indeed the case, e.g. some of the examples here.

  In general, at least for important questions worth spending time on, it seems very weird to say “You think X will happen, but we should be very confident it won’t because in analogous case Y it didn’t”, without also either (a) checking for other analogous cases or other lines of argument or (b) providing an argument for why this one case is far more relevant evidence than any other available evidence. I do think it totally makes sense to flag the analogous case and to update in light of it, but stopping there and walking away feeling confident in the answer seems very weird.

  I haven’t read any of the relevant threads in detail, so perhaps the arguments made are stronger than I imply here, but my guess is they weren’t. And it seems to me that it’s unfortunately decently common for AI risk discussions on LessWrong to involve this pattern I’m sketching here.

  (To be clear, all I’m arguing here is that these arguments often seem weak, not that their conclusions are false.)

  (This comment is raising an additional point to Jan’s, not disagreeing.)

  Update: Oh, I just saw Steve Byrnes also the following in this thread, which I totally agree with:

  “[Maybe one could argue] “It’s all very random—who happens to be in what position of power and when, etc.—and GoF is just one example, so we shouldn’t generalize too far from it” (OK maybe, but if so, then can we pile up more examples into a reference class to get a base rate or something? and what are the interventions to improve the odds, and can we also try those same interventions on GoF?)”

• MichaelAApr 30, 2022, 7:20 PM

  1 point

  on: LessWrong is pro­vid­ing feed­back and proofread­ing on drafts as a service

  Cool!

  Two questions:

  1. Is it possible to also get something re-formatted via this service? (E.g., porting a Google Doc with many footnotes and tables to LessWrong or the EA Forum.)

  2. Is it possible to get feedback, proofreading, etc. via this service for things that won’t be posts?

     • E.g. mildly infohazardous research outputs that will just be shared in the relevant research & policy community but not made public

  (Disclaimer: I only skimmed this post, having landed here from Habryka’s comment on It could be useful if someone ran a copyediting service. Apologies if these questions are answered already in the post.)

• MichaelAFeb 23, 2022, 12:57 PM

  7 points

  on: Re­ward Good Bets That Had Bad Outcomes

  Thanks for this post! This seems like good advice to me.

  I made an Anki card on your three “principles that stand out” so I can retain those ideas. (Mainly for potentially suggesting to people I manage or other people I know—I think I already have roughly the sort of mindset this post encourages, but I think many people don’t and that me suggesting these techniques sometimes could be helpful.)

• MichaelAFeb 13, 2022, 3:14 PM

  1 point

  on: Com­men­tary on AGI Safety from First Principles

  It’s not sufficient to argue that taking over the world will improve prediction accuracy. You also need to argue that during the training process (in which taking over the world wasn’t possible), the agent acquired a set of motivations and skills which will later lead it to take over the world. And I think that depends a lot on the training process.

  [...] if during training the agent is asked questions about the internet, but has no ability to edit the internet, then maybe it will have the goal of “predicting the world”, but maybe it will have the goal of “understanding the world”. The former incentivises control, the latter doesn’t.

  I agree with your key claim that it’s not obvious/​guaranteed that an AI system that has faced some selection pressure in favour of predicting/​understanding the world accurately would then want to take over the world. I also think I agree that a goal of “understanding the world” is a somewhat less dangerous goal in this context than a goal of “predicting the world”. But it seems to me that a goal of “understanding the world” could still be dangerous for basically the same reason as why “predicting the world” could be dangerous. Namely, some world states are easier to understand than others, and some trajectories of the world are easier to maintain an accurate understanding of than others.

  E.g., let’s assume that the “understanding” is meant to be at a similar level of analysis to that which humans typically use (rather than e.g., being primarily focused at the level of quantum physics), and that (as in humans) the AI sees it as worse to have a faulty understanding of “the important bits” than “the rest”. Given that, I think:

  • a world without human civilization or with far more homogeneity of its human civilization seems to be an easier world to understand

  • a world that stays pretty similar in terms of “the important bits” (not things like distant stars coming into/​out of existence), rather than e.g. having humanity spread through the galaxy creating massive structures with designs influenced by changing culture, requires less further effort to maintain an understanding of and has less risk of later being understood poorly

  I’d be interested in whether you think I’m misinterpreting your statement or missing some important argument.

  (Though, again, I see this just as pushback against one particular argument of yours, and I think one could make a bunch of other arguments for the key claim that was in question.)

• MichaelAFeb 13, 2022, 2:59 PM

  2 points

  on: AI safety from first prin­ci­ples: Conclusion

  Thanks for this series! I found it very useful and clear, and am very likely to recommend it to various people.

  Minor comment: I think “latter” and “former” are the wrong way around in the following passage?

  By contrast, I think the AI takeover scenarios that this report focuses on have received much more scrutiny—but still, as discussed previously, have big question marks surrounding some of the key premises. However, it’s important to distinguish the question of how likely it is that the second species argument is correct, from the question of how seriously we should take it. Often people with very different perspectives on the latter actually don’t disagree very much on the former.

  (I.e., I think you probably mean that, of people who’ve thought seriously about the question, probability estimates vary wildly but (a) tend to be above (say) 1 percentage point of x-risk from a second species risk scenario and (b) thus tend to suffice to make the people think humanity should put a lot more resources into understanding and mitigating the risk than we currently do. Rather than that people tend to wildly disagree on how much effort to put into this risk yet agree on how likely the risk is. Though I’m unsure, since I’m just guessing from context that “how seriously we should take it” means “how much resources should be spent on this issue”, but in other contexts it’d mean “how likely is this to be correct” or “how big a deal is this”, which people obviously disagree on a lot.)

• MichaelAJan 6, 2022, 11:37 AM

  3 points

  2

  on: Ob­ject level and Meta level

  FWIW, I feel that this entry doesn’t capture all/​most of how I see “meta-level” used.

  Here’s my attempted description, which I wrote for another purpose. Feel free to draw on it here and/​or to suggest ways it could be improved.

  • Meta-level and object-level = typically, “object-level” means something like “Concerning the actual topic at hand” while “Meta-level” means something like “Concerning how the topic is being tackled/​researched/​discussed, or concerning more general principles/​categories related to this actual topic”

    • E.g., “Meta-level: I really appreciate this style of comment; I think you having a policy of making this sort of comment is quite useful in expectation. Object-level: I disagree with your argument because [reasons]”

• MichaelAJan 3, 2022, 11:43 AM

  4 points

  on: Dist­in­guish­ing AI takeover scenarios

  Thanks for writing this. The summary table is pretty blurry /​ hard to read for me—do you think you could upload a higher resolution version? Or if for some reason that doesn’t work on LessWrong, could you link to a higher resolution version stored elsewhere?

• MichaelADec 31, 2021, 11:48 AM

  1 point

  on: Good and bad ways to think about down­side risks

  This section of a new post may be more practically useful than this post was: https://​​forum.effectivealtruism.org/​​posts/​​4T887bLdupiNyHH6f/​​six-takeaways-from-ea-global-and-ea-retreats#Takeaway__2__Take_more__calculated__risks

• MichaelADec 28, 2021, 12:45 PM

  3 points

  in reply to: MichaelA’s comment on: My Overview of the AI Align­ment Land­scape: A Bird’s Eye View

  My Anki cards

  Nanda broadly sees there as being 5 main types of approach to alignment research.

  Addressing threat models: We keep a specific threat model in mind for how AGI causes an existential catastrophe, and focus our work on things that we expect will help address the threat model.
  
  Agendas to build safe AGI: Let’s make specific plans for how to actually build safe AGI, and then try to test, implement, and understand the limitations of these plans. With an emphasis on understanding how to build AGI safely, rather than trying to do it as fast as possible.
  
  Robustly good approaches: In the long-run AGI will clearly be important, but we’re highly uncertain about how we’ll get there and what, exactly, could go wrong. So let’s do work that seems good in many possible scenarios, and doesn’t rely on having a specific story in mind. Interpretability work is a good example of this.
  
  De-confusion: Reasoning about how to align AGI involves reasoning about complex concepts, such as intelligence, alignment and values, and we’re pretty confused about what these even mean. This means any work we do now is plausibly not helpful and definitely not reliable. As such, our priority should be to do some conceptual work on how to think about these concepts and what we’re aiming for, and trying to become less confused.
  I consider the process of coming up with each of the research motivations outlined in this post to be examples of good de-confusion work
  
  Field-building: One of the biggest factors in how much Alignment work gets done is how many researchers are working on it, so a major priority is building the field. This is especially valuable if you think we’re confused about what work needs to be done now, but will eventually have a clearer idea once we’re within a few years of AGI. When this happens, we want a large community of capable, influential and thoughtful people doing Alignment work.

  ---

  Nanda focuses on three threat models that he thinks are most prominent and are addressed by most current research:

  Power-Seeking AI
  
  You get what you measure
  [The case given by Paul Christiano in What Failure Looks Like (Part 1)]
  
  AI Influenced Coordination Failures
  [The case put forward by Andrew Critch, eg in What multipolar failure looks like. Many players get AGI around the same time. They now need to coordinate and cooperate with each other and the AGIs, but coordination is an extremely hard problem. We currently deal with this with a range of existing international norms and institutions, but a world with AGI will be sufficiently different that many of these will no longer apply, and we will leave our current stable equilibrium. This is such a different and complex world that things go wrong, and humans are caught in the cross-fire.]

  ---

  Nanda considers three agendas to build safe AGI to be most prominent:

  Iterated Distillation and Amplification (IDA)
  
  AI Safety via Debate
  
  Solving Assistance Games
  [This is Stuart Russell’s agenda, which argues for a perspective shift in AI towards a more human-centric approach.]

  ---

  Nanda highlights 3 “robustly good approaches” (in the context of AGI risk):

  Interpretability
  
  Robustness
  
  Forecasting
  
  [I doubt he sees these as exhaustive—though that’s possible—and I’m not sure if he sees them as the most important/​prominent/​most central examples.]

• MichaelADec 28, 2021, 12:43 PM

  LW: 3 AF: 3

  AF

  on: My Overview of the AI Align­ment Land­scape: A Bird’s Eye View

  Thanks for this! I found it interesting and useful.

  I don’t have much specific feedback, partly because I listened to this via Nonlinear Library while doing other things rather than reading it, but I’ll share some thoughts anyway since you indicated being very keen for feedback.

  • I in general think this sort of distillation work is important and under-supplied

  • This seems like a good example of what this sort of distillation work should be like—broken into different posts that can be read separately, starting with an overall overview, each post is broken down into clear and logical sections and subsections, use of bold, clarity about terms, addition of meta notes where relevant

  • Maybe it would’ve been useful to just name & link to sources on threat models, agendas to build safe AGI, and robustly good approaches that you don’t discuss in any further detail? Rathe than not mentioning them at all.

    • That could make it easier for people to dive deeper if they want, could help avoid giving the impression that the things you list are the only things in those categories, and could help people understand what you mean by the overall categories by seeing more examples of things within the categories.

    • This is assuming you think there are other discernible nameable constituents of those categories which you didn’t name—I guess it’s possible that you don’t think that.

  • I’ll put in a reply to this comment the Anki cards I made, on the off chance that that’s of interest to you as oblique feedback or of interest to other people so they can use the same cards themselves

• MichaelADec 10, 2021, 4:33 PM

  1 point

  in reply to: MichaelA’s comment on: MichaelA’s Shortform

  Adam Binks replied to this list on the EA Forum with:

  To add to your list—Subjective Logic represents opinions with three values: degree of belief, degree of disbelief, and degree of uncertainty. One interpretation of this is as a form of second-order uncertainty. It’s used for modelling trust. A nice summary here with interactive tools for visualising opinions and a trust network.

• MichaelAJun 18, 2021, 8:31 AM

  1 point

  in reply to: M. Y. Zuo’s comment on: An­nounc­ing the Nu­clear Risk Fore­cast­ing Tournament

  Not sure what you mean by that being unverifiable? The question says:

  This question resolves as the total number of nuclear weapons (fission or thermonuclear) reported to be possessed across all states on December 31, 2022. This includes deployed, reserve/​ nondeployed, and retired (but still intact) warheads, and both strategic and nonstrategic weapons.

  Resolution criteria will come from the Federation of American Scientists (FAS). If they cease publishing such numbers before resolution, resolution will come from the Arms Control Association or any other similar platform.

  FAS update their estimates fairly regularly—here are their estimates as of May (that link is also provided earlier in the question text).

  Though I do realise now that they’re extremely unlikely to update their numbers on December 31 specifically, and maybe not even in December 2022 at all. I’ll look into the best way to tweak the question in light of that. If that’s what you meant, thanks for the feedback!

  (I do expect there’ll be various minor issues like that, and we hope the community catches them quickly so we can tweak the questions to fix them. This was also one reason for showing some questions before they “open”.)

An­nounc­ing the Nu­clear Risk Fore­cast­ing Tournament

Overview of Re­think Pri­ori­ties’ work on risks from nu­clear weapons

• MichaelAApr 8, 2021, 12:50 AM

  1 point

  in reply to: ozziegooen’s comment on: [Part 1] Am­plify­ing gen­er­al­ist re­search via fore­cast­ing – Models of im­pact and challenges

  That makes sense to me.

  But it seems like you’re just saying the issue I’m gesturing at shouldn’t cause mis-calibration or overconfidence, rather than that it won’t reduce the resolution/​accuracy or the practical usefulness of a system based on X predicting what Y will think?