MichaelA

Karma: 837

MichaelA May 27, 2023, 9:05 AM
2 points
0
in reply to: Olli Järviniemi’s comment on: Survey on intermediate goals in AI governance
Glad to hear that!
I do feel excited about this being used as a sort of “201 level” overview of AI strategy and what work it might be useful to do. And I’m aware of the report being included in the reading lists / curricula for two training programs for people getting into AI governance or related work, which was gratifying.
Unfortunately we did this survey before ChatGPT and various other events since then, which have majorly changed the landscape of AI governance work to be done, e.g. opening various policy windows. So I imagine people reading this report today may feel it has some odd omissions / vibes. But I still think it serves as a good 201 level overview despite that. Perhaps we’ll run a followup in a year or two to provide an updated version.

MichaelA Mar 25, 2023, 9:28 AM
4 points
in reply to: Maxime Riché’s comment on: Database of existential risk estimates
I’d consider those to be “in-scope” for the database, so the database would include any such estimates that I was aware of and that weren’t too private to share in the database.
If I recall correctly, some estimates in the database are decently related to that, e.g. are framed as “What % of the total possible moral value of the future will be realized?” or “What % of the total possible moral value of the future is lost in expectation due to AI risk?”
But I haven’t seen many estimates of that type, and I don’t remember seeing any that were explicitly framed as “What fraction of the accessible universe’s resources will be used in a way optimized for ‘the correct moral theory’?”
If you know of some, feel free to comment in the database to suggest they be added :)

MichaelA Mar 17, 2023, 1:20 PM
1 point
0
on: Survey on intermediate goals in AI governance
...and while I hopefully have your attention: My team is currently hiring for a Research Manager! If you might be interested in managing one or more researchers working on a diverse set of issues relevant to mitigating extreme risks from the development and deployment of AI, please check out the job ad!
The application form should take <2 hours. The deadline is the end of the day on March 21. The role is remote and we’re able to hire in most countries.
People with a wide range of backgrounds could turn out to be the best fit for the role. As such, if you’re interested, please don’t rule yourself out due to thinking you’re not qualified unless you at least read the job ad first!

MichaelA Jan 7, 2023, 1:06 PM
2 points
2
in reply to: habryka’s comment on: Let’s think about slowing down AI
I found this thread interesting and useful, but I feel a key point has been omitted thus far (from what I’ve read):
- Public, elite, and policymaker beliefs and attitudes related to AI risk aren’t just a variable we (members of the EA/longtermist/AI safety communities) have to bear in mind and operate in light of, but instead also a variable we can intervene on.
- And so far I’d say we have (often for very good reasons) done significantly less to intervene on that variable than we could’ve or than we could going forward.
- So it seems plausible that actually these people are fairly convincible if exposed to better efforts to really explain the arguments in a compelling way.
We’ve definitely done a significant amount of this kind of work, but I think we’ve often (a) deliberately held back on doing so or on conveying key parts of the arguments, due to reasonable downside risk concerns, and (b) not prioritized this. And I think there’s significantly more we could do if we wanted to, especially after a period of actively building capacity for this.
Important caveats / wet blankets:
- I think there are indeed strong arguments against trying to shift relevant beliefs and attitudes in a more favorable direction, including not just costs and plausibly low upside but also multiple major plausible downside risks.^[1]
- So I wouldn’t want anyone to take major steps in this direction without checking in with multiple people working on AI safety/governance first.
- And it’s not at all obvious to me we should be doing more of that sort of work. (Though I think whether, how, & when we should is an important question and I’m aware of and excited about a couple small research projects that are happening on that.)
All I really want to convey in this comment is what I said in my first paragraph: we may be able to significantly push beliefs and opinions in favorable directions relative to where they are now or would be n future by default.
1. ^
  Due to time constraints, I’ll just point to this vague overview.

MichaelA Jan 7, 2023, 12:47 PM
3 points
2
in reply to: Jan_Kulveit’s comment on: Let’s think about slowing down AI
Personally I haven’t thought about how strong the analogy to GoF is, but another thing that feels worth noting is that there may be a bunch of other cases where the analogy is similarly strong and where major government efforts aimed at risk-reduction have occurred. And my rough sense is that that’s indeed the case, e.g. some of the examples here.
In general, at least for important questions worth spending time on, it seems very weird to say “You think X will happen, but we should be very confident it won’t because in analogous case Y it didn’t”, without also either (a) checking for other analogous cases or other lines of argument or (b) providing an argument for why this one case is far more relevant evidence than any other available evidence. I do think it totally makes sense to flag the analogous case and to update in light of it, but stopping there and walking away feeling confident in the answer seems very weird.
I haven’t read any of the relevant threads in detail, so perhaps the arguments made are stronger than I imply here, but my guess is they weren’t. And it seems to me that it’s unfortunately decently common for AI risk discussions on LessWrong to involve this pattern I’m sketching here.
(To be clear, all I’m arguing here is that these arguments often seem weak, not that their conclusions are false.)
(This comment is raising an additional point to Jan’s, not disagreeing.)
Update: Oh, I just saw Steve Byrnes also the following in this thread, which I totally agree with:
“[Maybe one could argue] “It’s all very random—who happens to be in what position of power and when, etc.—and GoF is just one example, so we shouldn’t generalize too far from it” (OK maybe, but if so, then can we pile up more examples into a reference class to get a base rate or something? and what are the interventions to improve the odds, and can we also try those same interventions on GoF?)”

MichaelA Apr 30, 2022, 7:20 PM
1 point
on: LessWrong is providing feedback and proofreading on drafts as a service
Cool!
Two questions:
1. Is it possible to also get something re-formatted via this service? (E.g., porting a Google Doc with many footnotes and tables to LessWrong or the EA Forum.)
2. Is it possible to get feedback, proofreading, etc. via this service for things that won’t be posts?
  - E.g. mildly infohazardous research outputs that will just be shared in the relevant research & policy community but not made public
(Disclaimer: I only skimmed this post, having landed here from Habryka’s comment on It could be useful if someone ran a copyediting service. Apologies if these questions are answered already in the post.)

MichaelA Feb 23, 2022, 12:57 PM
7 points
on: Reward Good Bets That Had Bad Outcomes
Thanks for this post! This seems like good advice to me.
I made an Anki card on your three “principles that stand out” so I can retain those ideas. (Mainly for potentially suggesting to people I manage or other people I know—I think I already have roughly the sort of mindset this post encourages, but I think many people don’t and that me suggesting these techniques sometimes could be helpful.)

MichaelA Feb 13, 2022, 3:14 PM
1 point
on: Commentary on AGI Safety from First Principles
It’s not sufficient to argue that taking over the world will improve prediction accuracy. You also need to argue that during the training process (in which taking over the world wasn’t possible), the agent acquired a set of motivations and skills which will later lead it to take over the world. And I think that depends a lot on the training process.
[...] if during training the agent is asked questions about the internet, but has no ability to edit the internet, then maybe it will have the goal of “predicting the world”, but maybe it will have the goal of “understanding the world”. The former incentivises control, the latter doesn’t.
I agree with your key claim that it’s not obvious/guaranteed that an AI system that has faced some selection pressure in favour of predicting/understanding the world accurately would then want to take over the world. I also think I agree that a goal of “understanding the world” is a somewhat less dangerous goal in this context than a goal of “predicting the world”. But it seems to me that a goal of “understanding the world” could still be dangerous for basically the same reason as why “predicting the world” could be dangerous. Namely, some world states are easier to understand than others, and some trajectories of the world are easier to maintain an accurate understanding of than others.
E.g., let’s assume that the “understanding” is meant to be at a similar level of analysis to that which humans typically use (rather than e.g., being primarily focused at the level of quantum physics), and that (as in humans) the AI sees it as worse to have a faulty understanding of “the important bits” than “the rest”. Given that, I think:
- a world without human civilization or with far more homogeneity of its human civilization seems to be an easier world to understand
- a world that stays pretty similar in terms of “the important bits” (not things like distant stars coming into/out of existence), rather than e.g. having humanity spread through the galaxy creating massive structures with designs influenced by changing culture, requires less further effort to maintain an understanding of and has less risk of later being understood poorly
I’d be interested in whether you think I’m misinterpreting your statement or missing some important argument.
(Though, again, I see this just as pushback against one particular argument of yours, and I think one could make a bunch of other arguments for the key claim that was in question.)

MichaelA Feb 13, 2022, 2:59 PM
2 points
on: AI safety from first principles: Conclusion
Thanks for this series! I found it very useful and clear, and am very likely to recommend it to various people.
Minor comment: I think “latter” and “former” are the wrong way around in the following passage?
By contrast, I think the AI takeover scenarios that this report focuses on have received much more scrutiny—but still, as discussed previously, have big question marks surrounding some of the key premises. However, it’s important to distinguish the question of how likely it is that the second species argument is correct, from the question of how seriously we should take it. Often people with very different perspectives on the latter actually don’t disagree very much on the former.
(I.e., I think you probably mean that, of people who’ve thought seriously about the question, probability estimates vary wildly but (a) tend to be above (say) 1 percentage point of x-risk from a second species risk scenario and (b) thus tend to suffice to make the people think humanity should put a lot more resources into understanding and mitigating the risk than we currently do. Rather than that people tend to wildly disagree on how much effort to put into this risk yet agree on how likely the risk is. Though I’m unsure, since I’m just guessing from context that “how seriously we should take it” means “how much resources should be spent on this issue”, but in other contexts it’d mean “how likely is this to be correct” or “how big a deal is this”, which people obviously disagree on a lot.)

MichaelA Jan 6, 2022, 11:37 AM
3 points
2
on: Object level and Meta level
FWIW, I feel that this entry doesn’t capture all/most of how I see “meta-level” used.
Here’s my attempted description, which I wrote for another purpose. Feel free to draw on it here and/or to suggest ways it could be improved.
- Meta-level and object-level = typically, “object-level” means something like “Concerning the actual topic at hand” while “Meta-level” means something like “Concerning how the topic is being tackled/researched/discussed, or concerning more general principles/categories related to this actual topic”
  - E.g., “Meta-level: I really appreciate this style of comment; I think you having a policy of making this sort of comment is quite useful in expectation. Object-level: I disagree with your argument because [reasons]”

MichaelA Jan 3, 2022, 11:43 AM
4 points
on: Distinguishing AI takeover scenarios
Thanks for writing this. The summary table is pretty blurry / hard to read for me—do you think you could upload a higher resolution version? Or if for some reason that doesn’t work on LessWrong, could you link to a higher resolution version stored elsewhere?

MichaelA Dec 31, 2021, 11:48 AM
1 point
on: Good and bad ways to think about downside risks
This section of a new post may be more practically useful than this post was: https://forum.effectivealtruism.org/posts/4T887bLdupiNyHH6f/six-takeaways-from-ea-global-and-ea-retreats#Takeaway__2__Take_more__calculated__risks

MichaelA Dec 28, 2021, 12:45 PM
3 points
in reply to: MichaelA’s comment on: My Overview of the AI Alignment Landscape: A Bird’s Eye View
My Anki cards
Nanda broadly sees there as being 5 main types of approach to alignment research.
Addressing threat models: We keep a specific threat model in mind for how AGI causes an existential catastrophe, and focus our work on things that we expect will help address the threat model.

Agendas to build safe AGI: Let’s make specific plans for how to actually build safe AGI, and then try to test, implement, and understand the limitations of these plans. With an emphasis on understanding how to build AGI safely, rather than trying to do it as fast as possible.

Robustly good approaches: In the long-run AGI will clearly be important, but we’re highly uncertain about how we’ll get there and what, exactly, could go wrong. So let’s do work that seems good in many possible scenarios, and doesn’t rely on having a specific story in mind. Interpretability work is a good example of this.

De-confusion: Reasoning about how to align AGI involves reasoning about complex concepts, such as intelligence, alignment and values, and we’re pretty confused about what these even mean. This means any work we do now is plausibly not helpful and definitely not reliable. As such, our priority should be to do some conceptual work on how to think about these concepts and what we’re aiming for, and trying to become less confused.
I consider the process of coming up with each of the research motivations outlined in this post to be examples of good de-confusion work

Field-building: One of the biggest factors in how much Alignment work gets done is how many researchers are working on it, so a major priority is building the field. This is especially valuable if you think we’re confused about what work needs to be done now, but will eventually have a clearer idea once we’re within a few years of AGI. When this happens, we want a large community of capable, influential and thoughtful people doing Alignment work.
Nanda focuses on three threat models that he thinks are most prominent and are addressed by most current research:
Power-Seeking AI

You get what you measure
[The case given by Paul Christiano in What Failure Looks Like (Part 1)]

AI Influenced Coordination Failures
[The case put forward by Andrew Critch, eg in What multipolar failure looks like. Many players get AGI around the same time. They now need to coordinate and cooperate with each other and the AGIs, but coordination is an extremely hard problem. We currently deal with this with a range of existing international norms and institutions, but a world with AGI will be sufficiently different that many of these will no longer apply, and we will leave our current stable equilibrium. This is such a different and complex world that things go wrong, and humans are caught in the cross-fire.]
Nanda considers three agendas to build safe AGI to be most prominent:
Iterated Distillation and Amplification (IDA)

AI Safety via Debate

Solving Assistance Games
[This is Stuart Russell’s agenda, which argues for a perspective shift in AI towards a more human-centric approach.]
Nanda highlights 3 “robustly good approaches” (in the context of AGI risk):
Interpretability

Robustness

Forecasting

[I doubt he sees these as exhaustive—though that’s possible—and I’m not sure if he sees them as the most important/prominent/most central examples.]

MichaelA Dec 28, 2021, 12:43 PM
LW: 3 AF: 3
AF
on: My Overview of the AI Alignment Landscape: A Bird’s Eye View
Thanks for this! I found it interesting and useful.
I don’t have much specific feedback, partly because I listened to this via Nonlinear Library while doing other things rather than reading it, but I’ll share some thoughts anyway since you indicated being very keen for feedback.
- I in general think this sort of distillation work is important and under-supplied
- This seems like a good example of what this sort of distillation work should be like—broken into different posts that can be read separately, starting with an overall overview, each post is broken down into clear and logical sections and subsections, use of bold, clarity about terms, addition of meta notes where relevant
- Maybe it would’ve been useful to just name & link to sources on threat models, agendas to build safe AGI, and robustly good approaches that you don’t discuss in any further detail? Rathe than not mentioning them at all.
  - That could make it easier for people to dive deeper if they want, could help avoid giving the impression that the things you list are the only things in those categories, and could help people understand what you mean by the overall categories by seeing more examples of things within the categories.
  - This is assuming you think there are other discernible nameable constituents of those categories which you didn’t name—I guess it’s possible that you don’t think that.
- I’ll put in a reply to this comment the Anki cards I made, on the off chance that that’s of interest to you as oblique feedback or of interest to other people so they can use the same cards themselves

MichaelA Dec 10, 2021, 4:33 PM
1 point
in reply to: MichaelA’s comment on: MichaelA’s Shortform
Adam Binks replied to this list on the EA Forum with:
To add to your list—Subjective Logic represents opinions with three values: degree of belief, degree of disbelief, and degree of uncertainty. One interpretation of this is as a form of second-order uncertainty. It’s used for modelling trust. A nice summary here with interactive tools for visualising opinions and a trust network.

MichaelA Jun 18, 2021, 8:31 AM
1 point
in reply to: M. Y. Zuo’s comment on: Announcing the Nuclear Risk Forecasting Tournament
Not sure what you mean by that being unverifiable? The question says:
This question resolves as the total number of nuclear weapons (fission or thermonuclear) reported to be possessed across all states on December 31, 2022. This includes deployed, reserve/ nondeployed, and retired (but still intact) warheads, and both strategic and nonstrategic weapons.
Resolution criteria will come from the Federation of American Scientists (FAS). If they cease publishing such numbers before resolution, resolution will come from the Arms Control Association or any other similar platform.
FAS update their estimates fairly regularly—here are their estimates as of May (that link is also provided earlier in the question text).
Though I do realise now that they’re extremely unlikely to update their numbers on December 31 specifically, and maybe not even in December 2022 at all. I’ll look into the best way to tweak the question in light of that. If that’s what you meant, thanks for the feedback!
(I do expect there’ll be various minor issues like that, and we hope the community catches them quickly so we can tweak the questions to fix them. This was also one reason for showing some questions before they “open”.)

MichaelA Apr 8, 2021, 12:50 AM
1 point
in reply to: ozziegooen’s comment on: [Part 1] Amplifying generalist research via forecasting – Models of impact and challenges
That makes sense to me.
But it seems like you’re just saying the issue I’m gesturing at shouldn’t cause mis-calibration or overconfidence, rather than that it won’t reduce the resolution/accuracy or the practical usefulness of a system based on X predicting what Y will think?

MichaelA Mar 28, 2021, 11:38 AM
3 points
in reply to: MichaelA’s comment on: Introduction To Lesswrong Subculture
(Update: I just saw the post Welcome to LessWrong!, and I think that that serves my needs well.)

MichaelA Mar 28, 2021, 4:26 AM
7 points
on: Introduction To Lesswrong Subculture
I think it’s good that a page like this exists; I’d want to be able to use it as a go-to link when suggesting people engage with or post on LessWrong, e.g. in my post on Notes on EA-related research, writing, testing fit, learning, and the Forum.
Unfortunately, it seems to me that this page isn’t well suited to that purpose. Here are some things that seem like key issues to me (maybe other people would disagree):
- This introduction seems unnecessarily intimidating, non-welcoming, and actually (in my perception) somewhat arrogant. For example:
  - “If you have no familiarity with the cultural articles and other themes before you begin interacting, your social experiences are likely to be highly awkward. The rationalist way of thinking and subculture is extremely, extremely complex. To give you a gist of how complex it is and what kind of complexity you’ll encounter:”
    This feels to me like saying “We’re very special and you need to do your homework to deeply understand us before interacting at all with us, or you’re just wasting our time and we’ll want you to go away.”
    I do agree that the rationalist culture can take some getting used to, but I don’t think it’s far more complex or unusual than the cultures in a wide range of other subcultures, and I think it’s very often easiest to get up to speed with a culture partly just by interacting with it.
    I do agree that reading parts of the Sequences is useful, and that it’s probably good to gently encourage new users to do that. But I wouldn’t want to make it sound like it’s a hard requirement or like they have to read the whole thing. And this passage will probably cause some readers to infer that, even if it doesn’t outright say it. (A lot of people lurk more than they should, have imposter syndrome, etc.)
    I started interacting on LessWrong before having finished the Sequences (though I’d read some), and I think I both got and provided value from those interactions.
    Part of this is just my visceral reaction to any group saying their way of thinking and subculture is “extremely, extremely complex”, rather than me having explicit reasons to think that that’s bad.
- I wrote all of that before reading the next paragraphs, and the next paragraphs very much intensified my emotional feeling of “These folks seem really arrogant and obnoxious and I don’t want to ever hang out with them”
  - This is despite the fact that I’ve actually engaged a lot on LessWrong, really value a lot about it, rank the Sequences and HPMOR as among my favourite books, etc.
- Maybe part of this is that this is describing what rationalists aim to be as if all rationalists always hit that mark.
  - Rationalists and the rationalist community often do suffer from the same issues other people and communities do. This was in fact one of the really valuable things Eliezer’s posts pointed out (e.g., being wary of trending towards cult-hood).
Again, these are just my perceptions. But FWIW, I do feel these things quite strongly.
Here are a couple much less important issues:
- I don’t think I’d characterise the Sequences as “mostly like Kahneman, but more engaging, and I guess with a bit of AI etc.” From memory, a quite substantial chunk of the sequences—and quite a substantial chunk of their value—had to do with things other than cognitive biases, e.g. what goals one should form, why, how to act on them, etc. Maybe this is partly a matter of instrumental rather than just epistemic rationality.
  - Relatedly, I think this page presents a misleading or overly narrow picture of what’s distinctive (and good!) about rationalist approaches to forming beliefs and choosing decisions when it says “There are over a hundred cognitive biases that humans are affected by that rationalists aim to avoid. Imagine you added over one hundred improvements to your way of thinking.”
- “Kahneman is notoriously dry” feels like an odd thing to say. Maybe he is, but I’ve never actually heard anyone say this, and I’ve read one of his books and papers and watched one of his talks and found them all probably somewhat more engaging than similar things from the average scientist. (Though maybe this was more the ideas themselves, rather than the presentation.)
(I didn’t read “Website Participation Intro or “Why am I being downvoted?”″, because it was unfortunately already clear that I wouldn’t want to link to this page when aiming to introduce people to LessWrong and encourage them to read, comment, and/or post there.)

MichaelA Mar 18, 2021, 2:13 AM
2 points
in reply to: Gentzel’s comment on: Modernization and arms control don’t have to be enemies.
Authoritarian closed societies probably have an advantage at covert racing, at devoting a larger proportion of their economic pie to racing suddenly, and at artificially lowering prices to do so. Open societies have probably a greater advantage at discovery/the cutting edge and have a bigger pie in the first place (though better private sector opportunities compete up the cost of defense engineering talent).
These are interesting points which I hadn’t considered—thanks!
(Your other point also seems interesting and plausible, but I feel I lack the relevant knowledge to immediately evaluate it well myself.)