MichaelA

Karma: 838

MichaelA Mar 18, 2021, 2:11 AM
2 points
on: Epistemic Warfare
Interesting post.
You or other readers might also find the idea of epistemic security interesting, as discussed in the report “Tackling threats to informed decisionmaking in democratic societies: Promoting epistemic security in a technologically-advanced world”. The report is by researchers at CSER and some other institutions. I’ve only read the executive summary myself.
There’s also a BBC Futures article on the topic by some of the same authors.

MichaelA Mar 17, 2021, 9:38 AM
1 point
on: Modernization and arms control don’t have to be enemies.
While I am not sure I agree fully with the panel, an implication to be drawn from their arguments is that from an equilibrium of treaty compliance, maintaining the ability to race can disincentivize the other side from treaty violation: it increases the cost to the other side of gaining advantage, and that can be especially decisive if your side has an economic advantage.
This is an idea/argument I hadn’t encountered before, and seems plausible, so it seems valuable that you shared it.
But it seems to me that there’s probably an effect pushing in the opposite direction:
- Even from an equilibrium of treaty compliance, if one state has the ability to race, that might incentivise the other side to develop the ability to race as well. That wouldn’t necessarily require treaty violation.
- Either or especially both sides having the ability to race can increase risks if they could race covertly until they have gained an advantage, or race so quickly that they gain an advantage before the other side can get properly started, or if the states don’t always act as rational cohesive entities (e.g., if leaders are more focused on preventing regime change than preventing millions of deaths in their own country), or probably under other conditions.
  - I think the term “arms race stability” captures the sort of thing I’m referring to, though I haven’t yet looked into the relevant theoretical work much.
- In contrast, if we could reach a situation where neither side currently had the ability to race, that might be fairly stable. This could be true if building up that ability would take some time and be detectable early enough to be responded to (by sanctions, a targeted strike, the other side building up their own ability, or whatever).
Does this seem accurate to you?
I guess an analogy could be to whether you’d rather be part of a pair of cowboys who both have guns but haven’t drawn them (capability but not yet racing), or part of a pair who don’t have guns but could go buy one. It seems like we’d have more opportunities for de-escalation, less risk from nerves and hair-triggers, etc. in the latter scenario than the former.
I think this overlaps with some of Schelling’s points in The Strategy of Conflict (see also my notes on that), but I can’t remember for sure.

MichaelA Mar 17, 2021, 8:42 AM
1 point
on: The Future of Nuclear Arms Control?
Thanks for this thought-provoking post. I found the discussion of how political warfare may have influenced nuclear weapons activism particularly interesting.
Since large yield weapons can loft dust straight to the stratosphere, they don’t even have to produce firestorms to start contributing to nuclear winter: once you get particles that block sunlight to an altitude that heating by the sun can keep them lofted, you’ll block sunlight a very long time and start harming crop yields.
I think it’s true that this could “contribute” to nuclear winter, but I don’t think I’ve seen this mentioned as a substantial concern in the nuclear winter papers I’ve read. E.g., I don’t think I’ve seen any papers suggest that nuclear winter could occur solely due to that effect, without there being any firestorms, or that that effect could make the climate impacts 20% worse than would occur with firestorms alone. Do you have any citations on hand for this claim?

MichaelA Mar 2, 2021, 12:44 AM
1 point
on: Notes on “Bioterror and Biowarfare” (2006)
Final thoughts on whether you should read this book
- I found the book useful
  - The parts I found most useful were (a) the early chapters on the history of biowarfare and bioterrorism and (b) the later chapters on attempts to use international law to reduce risks from bioterror and biowarfare
- I found parts of the book hard to pay attention to and remember information from
  - In particular, the middle chapters on various types and examples of pathogens
    But this might just be a “me problem”. Ever since high school, I’ve continually noticed that I seem to have a harder time paying attention to and remembering information about biology than information from other disciplines. (I don’t understand what that would be the case, and I’m not certain it’s actually true, but it has definitely seemed true.)
- I’m not sure how useful this book would be to someone who already knows a lot about bioterror, biowarfare, and/or chemical weapons
- I’m not sure how useful this book would be to someone who doesn’t have much interest in the topics of bioterror, biowarfare, and/or chemical weapons
  - But I’m inclined to think most longtermists should read consume at least one book’s worth of content from experts on those topics
  - And I think the book could be somewhat useful for understanding WMDs, international relations, and international law more generally
- There might be better books on the topic
  - In particular, it’s possible a more recent book would be better?

MichaelA Mar 2, 2021, 12:43 AM
1 point
on: Notes on “Bioterror and Biowarfare” (2006)
My Anki cards
Note that:
- It’s possible that some of these cards include mistakes, or will be confusing or misleading out of context.
- I haven’t fact-checked Dando on any of these points.
- Some of these cards are just my own interpretations—rather than definitely 100% parroting what the book is saying
- The indented parts are the questions, the answers are in “spoiler blocks” (hover over them to reveal the text), and the parts in square brackets are my notes-to-self.
Dando says ___ used biological weapons in WW1, but seemingly only against ___.
the Germans and perhaps the French;
draft animals (e.g. horses), not humans
[This was part of sabotage operations, seemingly only/especially in the US, Romania, Norway, and Argentina. The US and Romania were neutral at the time; not sure whether Norway and Argentina were.]
Dando says the 1925 Geneva Protocol prohibits ___, but not ___, of chemical and biological weapons, and that many of the parties to the Protocol entered reservations to their agreement to make it clear that ___.
Use;
Development and stockpiling;
Although they would not use such weapons first, they were prepared to use them in retaliation if such weapons were used first against them
[And a number of offensive bio weapons programs were undertaken by major states in the interwar period. Only later in the 20th century were further arms control restrictions placed on chem and bio weapons.]
Japan’s offensive biological warfare program was unique in that ___. The program probably caused ___.
It used human experimentation to test biological agents;
The deaths of thousands of Chinese people
[This program ran from 1931-1945]
Dando mentions 6 countries as having had “vigorous” offensive biological weapons programs during WW2:
Japan, The Soviet Union, France, the UK, the US, Canada
[He doesn’t explicitly say these were the only countries with such programs, but does seem to imply that, or at least that no other countries had similarly large programs
He notes that Germany didn’t have such a program.
France’s program was interrupted by the German invasion in 1940, but was resumed after WW2.]
Dando suggests that the main or most thoroughly prepared type of British WW2 biological warfare weapon/plan was...
To drop millions of cattle cakes infected with anthrax spores onto German fields, to wipe out cattle and thus deal an economic blow to Germany’s overstretched agricultural system
[The British did make 5 million of these cakes.]
Dando says that there are 7 countries which definitely had offensive biological weapons programs in the second half of the 20th century:
The US, the UK, the Soviet Union, Canada, France, South Africa, Iraq
[He also says there’ve been numerous accusations that other countries had such programs as well, but that there isn’t definite information about them.]
Dando says that 3 countries continued to have offensive biological weapons after becoming the depository for, ratifying, and/or signing the BTWC:
Soviet Union, South Africa, and Iraq
[This was then illegal under international law. Prior to the BTWC, having such a program wasn’t illegal—only the use of bioweapons was.
I think the other 4 states that had had such programs between WW2 and 1972 stopped at that point or before then.]
During WW2, the US offensive biological weapons program was developing anti-___, anti-___, and anti-___ weapons.
personnel; animal; plant
[And the US was considering using anti-plant weapons against Japanese rice production.]
What major change in high-level US policy regarding chemical and biological weapons does Dando suggest occurred around 1956?
What does he suggest this was partly a reaction to?
Changing from a retaliation-only policy for BW and CW to a policy stating that the US would be prepared to use BW or CW in a general war for the purposes of enhancing military effectiveness [and the decision would be reserved for the president];
Soviet statements in 1956 that chemical and biological weapons would be used in future wars for the purposes of mass destruction
[Dando notes that the retaliation-only policy was in line with the US’s signature of the 1925 Geneva protocol, but also that the US didn’t actually ratify the Geneva protocol till 1975; until then it was only a signatory.]
Dando says an army report says the origin of the US’s shift (under Nixon) to renouncing biological and chemical weapons dates from...
Criticism of US application of chemical herbicides and riot control agent(s) in Vietnam starting in the 1960s
[I think this means criticism/opposition by the public.]
The UK’s work on an offensive biological weapons capability had been abandoned by...
1957
[According to a report cited by Dando.
Though Dando later indicates the UK restarted some of this work in 1961, I think particularly/only to find a nonlethal incapacitating chemical weapon.]
Dando says that, at the end of WW2, the UK viewed biological weapons as...
On a par with nuclear weapons
[“Only when the UK obtained its own nuclear systems did interest in biological weapons decline.”
I don’t know precisely what Dando means by this.]
South Africa had an offensive biological weapons program during...
The later stages of the Apartheid regime
[But it was terminated before the regime change.]
What was the scale of South Africa’s offensive biological weapons program? What does its main purpose seem to have been?
Relatively small (e.g. smaller than Iraq’s program)
Finding means of assassinating the Apartheid regime’s enemies
[Elsewhere, Dando suggests that original motivations for the program—or perhaps for some chemical weapons work? - also included the Angola war and a desire to find crowd control agents.]
What has Iraq stated about authority (as of ~1991) to launch its chemical and biological weapons?
Authority was pre-delegated to regional commanders if Baghdad was hit with nuclear weapons
[UNSCOM has noted that that doesn’t exclude other forms of use, and doesn’t constitute a proof of a retaliation-only policy.]
The approach to chemical weapons that Iraq pursued was ___, in contrast to a Western approach of ___.
Production and rapid use;
Production and stockpiling
[I’m guessing that this means that Iraq pursued the ability to produce chemical weapons shortly before they were needed, rather than having a pre-made, long-lasting stockpile of more stable versions.
Dando says a similar approach could’ve been taken towards biological weapons.]
Dando says that the main lesson from the Iraqi biological weapons program is that...
A medium-sized country without great scientific and technical resources was, within a few years, able to reach the stage of weaponising a range of deadly biological agents
What kind of vaccine does Dando say South Africa’s biological weapons program tried to find? What does someone who had knowledge of the program say the vaccine might’ve been used for, if it had been found?
An anti-fertility vaccine;
Administering to black women without their knowledge
Dando lists 6 different types of biological agents that could be used for biological weapons:
Bacteria; Viruses; Toxins; Bioregulators; Protozoa; Fungi
[I’m not sure whether this was meant to be exhaustive, nor whether I’m right to say these are “different types of biological agents”.
There’s also a chance I forgot one of the types he mentioned.]
Dando says that vaccination during a plague epidemic would not be of much help, because...
Immunity takes a month to build up
[Note that I haven’t fact-checked this, and that, for all I know, the situation may be different with other pathogens or newer vaccines.]
In the mid twentieth century, ___ tried to use plague-infected fleas to cause an outbreak among ___.
Japan; the Chinese
Dando notes at least 3 factors that could make the option of biowarfare or bioterrorism against animal agriculture attractive:
1. The animals are densely packed in confined areas
2. The animals reared are often from very limited genetic stock (so that a large percentage of them could succumb to a single strain of a pathogen)
3. Many/all pathogens that would be used don’t infect humans (reducing risks to the people involved in producing and using the pathogens)
[Dando implies that that third point is more relevant to bioterrorism than biowarfare, but doesn’t say why. I assume it’s because terrorists will tend to have fewer skills and resources than military programs, making them more vulnerable to accidents.]
What proportion of state-level offensive biological weapons programs (of which we have knowledge) “carefully investigated anti-plant attacks”?
Nearly all
In the 1990s, the US OTA concluded that the cheapest overt production route for 1 nuclear bomb per year, with no international controls, would cost __.
They also concluded that a chemical weapons arsenal for substantial military capability would cost __.
They concluded that a large biological weapons arsenal may cost __.
~$200 million;
$10s of millions;
Less than $10 million
[I’m unsure precisely what this meant.
I assume the OTA thought a covert route for nuclear weapons, with international controls, would be more expensive than the overt route with no international controls.]
Efforts in the 1990s to strengthen the BWC through agreement of a verification protocol eventually failed in 2001 due to the opposition from which country?
The United States
The BTWC was opened for signature in __, and entered into force in __.
1972; 1975
Dando highlights two key deficiencies of the BTWC (at least as of it entering into force in 1975):
1. There was a lack of verification measures
2. No organisation had been put in place to take care of the convention, of its effective implementation, and of its development between review conferences
[Dando notes that, in contrast to 2, there was a large organisation associated with the Chemical Weapons Convention.
Wikipedia suggests that a (very small) Implementation Support Unit for the BTWC was finally created in 2006.]
Dando highlights a US-based stakeholder as being vocally opposed to the ideas that were proposed for verifying compliance with the BTWC:
The huge US pharmaceutical industry and its linked trade associations
[I think Dando might’ve been talking about opposition to inspections in particular
Dando implies that this contributed to US executive branch being lukewarm on or sort-of opposed to these verification ideas.]

MichaelA Mar 2, 2021, 12:43 AM
2 points
on: Notes on “Bioterror and Biowarfare” (2006)
See also:

MichaelA Feb 28, 2021, 8:08 AM
3 points
on: [Part 1] Amplifying generalist research via forecasting – Models of impact and challenges
A final thought that came to mind, regarding the following passage:
It seems possible for person X to predict a fair number of a more epistemically competent person Y’s beliefs—even before person X is as epistemically competent as Y. And in that case, doing so is evidence that person X is moving in the right direction.
I think that that’s is a good and interesting point.
But I imagine there would also be many cases in which X develops an intuitive ability to predict Y’s beliefs quite well in a given set of domains, but in which that ability doesn’t transferring to new domains. It’s possible that this would be because X’s “black box” simulation of Y’s beliefs is more epistemically competent than Y in this new domain. But it seems more likely that Y is somewhat similarly epistemically competent in this new domain as in the old domain, but has to draw on different reasoning processes, knowledge, theories, intuitions, etc., and X’s intuitions aren’t calibrated for how Y is now thinking.
I think we could usefully think of this issue as a question of robustness to distributional shift.
I think the same issue could probably also occur even if X has a more explicit process for predicting Y’s beliefs. E.g., even if X believes they understand what sort of sources of information Y considers and how Y evaluates it and X tries to replicate that (rather than just trying to more intuitively guess what Y will say), the process X uses may not be robust to distributional shift.
But I’d guess that more explicit, less “black box” approaches for predicting what Y will say will tend to either be more robust to distributional shift or more able to fail gracefully, such as recognising that uncertainty is now much higher and there’s a need to think more carefully.
(None of this means I disagree with the quoted passage; I’m just sharing some additional thoughts that came to mind when I read it, which seem relevant and maybe useful.)

MichaelA Feb 28, 2021, 7:59 AM
3 points
on: [Part 1] Amplifying generalist research via forecasting – Models of impact and challenges
Here’s a second thought that came to mind, which again doesn’t seem especially critical to this post’s aims...
You write:
Someone who can both predict my beliefs and disagrees with me is someone I should listen to carefully. They seem to both understand my model and still reject it, and this suggests they know something I don’t.
I think I understand the rationale for this statement (though I didn’t read the linked Science article), and I think it will sometimes be true and important. But I think that those sentences might overstate the point. In particular, I think that those sentences implicitly presume that this other person is genuinely primarily trying to form accurate beliefs, and perhaps also that they’re doing so in a way that’s relatively free from bias.
But (almost?) everyone is at least sometimes primarily aiming (perhaps unconsciously) at something other than forming accurate beliefs, even when it superficially looks like they’re aiming at forming accurate beliefs. For example, they may be engaging in “ideologically motivated cognition[, i.e.] a form of information processing that promotes individuals’ interests in forming and maintaining beliefs that signify their loyalty to important affinity groups”. The linked study also notes that “subjects who scored highest in cognitive reflection were the most likely to display ideologically motivated cognition”.
So I think it might be common for people to be able to predict my beliefs and disagree with me, but with their disagreement not being based on knowing more or having better reasoning process but rather finding ways to continue to hold beliefs that they’re (in some sense) “motivated” to hold for some other reason.
Additionally, some people may genuinely be trying to form accurate beliefs, but with unusually bad epistemics / unusually major bias. If so, they may be able to predict my beliefs and disagree with me, but with their disagreement not being based on knowing more or having better reasoning process but rather being a result of their bad epistemics / biases.
Of course, we should be very careful with assuming that any of the above is why a person disagrees with us! See also this and this.
The claims I’d more confidently agree with are:
Someone who can both predict my beliefs and disagrees with me might be someone I should listen to carefully. They seem to both understand my model and still reject it, and this suggests they might know something I don’t (especially if they seem to be genuinely trying to form accurate beliefs and to do so via a reasonable process).
(Or maybe having that parenthetical at the end would be bad via making people feel licensed to dismiss people who disagree with them as just biased.)

MichaelA Feb 28, 2021, 7:44 AM
1 point
on: [Part 1] Amplifying generalist research via forecasting – Models of impact and challenges
Thanks for this and its companion post; I found the two posts very interesting, and I think they’ll usefully inform some future work for me.
A few thoughts came to mind as I read, some of which can sort-of be seen as pushing back against some claims, but in ways that I think aren’t very important and that I expect you’ve already thought about. I’ll split these into separate comments.
Firstly, as you note, what you’re measuring is how well predictions match a proxy for the truth (the proxy being Elizabeth’s judgement), rather than the truth itself. Something I think you don’t explicitly mention is that:
1. Elizabeth’s judgement may be biased in some way (rather than just randomly erring), and
2. The network-based forecasters’ judgements may be biased in a similar way, and therefore
3. This may “explain away” part of the apparent value of the network-based forecasters’ predictions, along with part of their apparent superiority over the online crowdworkers’ predictions.
E.g., perhaps EA-/rationality-adjacent people are biased towards disagreeing with “conventional wisdom” on certain topics, and this bias is somewhat shared between Elizabeth and the network-based forecasters. (I’m not saying this is actually the case; it’s just an example.)
You make a somewhat similar point in the Part 2 post, when you say that the online crowdworkers:
were operating under a number of disadvantages relative to other participants, which means we should be careful when interpreting their performance. [For example, the online crowdworkers] did not know that Elizabeth was the researcher who created the claims and would resolve them, and so they had less information to model the person whose judgments would ultimately decide the questions.
But that is about participants’ ability to successfully focus on predicting what Elizabeth will say, rather than their ability to accidentally be biased in the same way as Elizabeth when both are trying to make judgements about the ground truth.
In any case, I don’t think this matters much. One reason is that this “shared bias” issue probably at most “explains away” a relatively small fraction of the apparent value of the network-adjacent forecasters’ predictions, probably without tipping the balance of whether this sort of set-up is worthwhile. Another reason is that there may be ways to mitigate this “shared bias” issue.

MichaelA Feb 12, 2021, 9:26 AM
3 points
in reply to: Raemon’s comment on: Notes on Schelling’s “Strategy of Conflict” (1960)
Good idea! I didn’t know about that feature.
I’ve now edited the post to use spoiler-blocks (though a bit messily, as I wanted to do it quickly), and will use them for future lazy-Anki-card-notes-posts as well.

MichaelA Feb 12, 2021, 9:17 AM
2 points
in reply to: Jon Zero’s comment on: Notes on Schelling’s “Strategy of Conflict” (1960)
I didn’t add that tag; some other reader did.
And any reader can indeed downvote any tag, so if you feel that that tag shouldn’t be there, you could just downvote it.
Unless you feel that the tag shouldn’t be there but aren’t very confident about that, and thus wanted to just gently suggest that maybe the tag should be removed—like putting in a 0.5 vote rather than a full one. But that doesn’t seem to match the tone of your comment.
That said, it actually does seem to me that this post fairly clearly does match the description for that tag; the exercise is using these Anki cards as Anki cards. People can find a link to download these cards in the Anki card file format here. (I’ve now added that link in the body of the post itself; I guess I should’ve earlier.)
---
As a meta comment: For what it’s worth, I feel like your comment had an unnecessarily snarky tone, at least to my eye. I think you could’ve either just downvoted the tag, or said the same thing in a way that sounds less snarky. That said:
- It’s very possible (even probable?) that you didn’t intend to be snarky, and tht this is just a case of tone getting misread on the internet
- And in any case, this didn’t personally bug me, partly because I’ve posted on LessWrong and the EA Forum a lot.
  - But I think if I was newer to the sites or to posting, this might leave a bad taste in my mouth and make me less inclined to post in future. (Again, I’m not at all trying to say this was your intent!)
(Edited to add: Btw, I wasn’t the person who downvoted your comment, so that appears to be slightly more evidence that your comment was at least liable to be interpreted as unnecessary and snarky—although again I know that that may not have been your intention.)

MichaelA Feb 11, 2021, 11:53 PM
3 points
in reply to: chasmani’s comment on: Notes on Schelling’s “Strategy of Conflict” (1960)
Yeah, I definitely agree that that’s a good idea with any initialisations that won’t already be known to the vast majority of one’s readers (e.g., I wouldn’t bother with US or UK, but would with APA). In this case, I just copied and pasted the post from the EA Forum, where I do think the vast majority of readers would know what “EA” means—but I should’ve used the expanded form “effective altruism” the first time in the LessWrong version. I’ve now edited that.

MichaelA Feb 10, 2021, 2:54 AM
3 points
on: Notes on Schelling’s “Strategy of Conflict” (1960)
Here’s a comment I wrote on the EA Forum version of this post, which I’m copying here as I’d be interested on people’s thoughts on the equivalent questions in the context of LessWrong:
Meta: Does this sort of post seem useful? Should there be more posts like this?
I previously asked Should pretty much all content that’s EA-relevant and/or created by EAs be (link)posted to the Forum? I found Aaron Gertler’s response interesting and useful. Among other things, he said:
Eventually, we’d like it to be the case that almost all well-written EA content exists on the Forum somewhere.
[...]
I meant “quite EA-relevant and well-written”. I don’t especially care whether the content is written by community members, though I suppose that’s slightly preferable (as community members are much more likely to respond to comments on their work).
[...]
A single crosspost with a bit of context from the author—e.g. a few sentences each of summary/highlights, commentary, and action items/takeaways—seems better to me than three or four crossposts with no context at all. In my view, the best Forum content tends to give busy people a quick way to decide whether to read further.
And I read a lot of stuff that I think it could be useful for at least some other EAs to read, and that isn’t (link)posted to the Forum. So Aaron’s comments, combined with my own thinking and some comments from other people, make me think it’d be good for me to make linkposts for lots of that stuff if there was a way to do it that took up very little of my time.
Unfortunately, writing proper book reviews, or even just notes that are geared for public consumption, for all of those things I read would probably take a while.
But, starting about a month ago, I now make Anki cards for myself anyway during most of the reading I do. So maybe I should just make posts sort-of like this one for most particularly interesting things I read? And maybe other people could start doing that too?
A big uncertainty I have is how often the cards I make myself would be able to transmit useful ideas even to people who (a) aren’t me and (b) didn’t read the thing I read, and how often they’d do that with an efficiency comparable to people just finding and reading useful sources themselves directly. Another, related uncertainty is whether there’d be any demand for posts like this.
So I’d be interested in people’s thoughts on the above.

MichaelA Feb 10, 2021, 2:54 AM
5 points
on: Notes on Schelling’s “Strategy of Conflict” (1960)
Note: If you found this post interesting, you may also be interested in my Notes on “The Bomb: Presidents, Generals, and the Secret History of Nuclear War” (2020), or (less likely) Notes on The WEIRDest People in the World: How the West Became Psychologically Peculiar and Particularly Prosperous. (The latter book has a very different topic; I just mention it as the style of post is the same.)

MichaelA Jan 25, 2021, 12:27 AM
2 points
in reply to: Ben Cottier’s comment on: Clarifying some key hypotheses in AI alignment
To your first point...
My impression is that there is indeed substantially less literature on misuse risk and structural risk, compared to accident risk, in relation to AI x-risk. (I’m less confident when it comes to a broader set of negative outcomes, not just x-risks, but that’s also less relevant here and less important to me.) I do think that that might the sort of work this post does less interesting if done in relation to those less-discussed types of risks, since there fewer disagreements have been revealed, so there’s less to analyse and summarise.
That said, I still expect interesting stuff along these lines could be done on those topics. It just might be a quicker job with a smaller output than this post.
I collected a handful of relevant sources and ideas here. I think someone reading those things and providing a sort of summary, analysis, and/or mapping could be pretty handy, and might even be doable in just a day or so of work. It might also be relatively easy to provide more “novel ideas” in the course of that work that it would’ve been for your post, since misuse/structural risks seem like less charted territory.
(Unfortunately I’m unlikely to do this myself, as I’m currently focused on nuclear war risk.)
---
A separate point is that I’d guess that one reason why there’s less work on misuse/structural AI x-risk than on accidental AI x-risk is that a lot of people aren’t aware of those other categories of risks, or rarely think about them, or assume the risks are much smaller. And I think one reason for that is that people often write or talk about “AI x-risk” while actually only mentioning accidental AI x-risk. That’s part of why I say “So, personally, I think I’d have made that choice of scope even more explicit.”
(But again, I do very much like this post overall. And as a target of this quibble of mine, you’re in good company—I have the same quibble with The Precipice. I think one of the quibbles I most often have with posts I like is “This post seems to imply, or could be interpreted as implying, that it covers [topic]. But really it covers [some subset of that topic]. That’s fair enough and still very useful, but I think it’d be good to be clearer about what the scope is.”)
---
I know some people working on expanded and more in-depth models like this post. It would be great to get your thoughts when they’re ready.
Sounds very cool! Yeah, I’d be happy to have a look at that work when it’s ready.

MichaelA Jan 14, 2021, 3:00 AM
4 points
on: The “Commitment Races” problem
Thanks for this post; this does seem like a risk worth highlighting.
I’ve just started reading Thomas Schelling’s 1960 book The Strategy of Conflict, and noticed a lot of ideas in chapter 2 that reminded me of many of the core ideas in this post. My guess is that that sentence is an uninteresting, obvious observation, and that Daniel and most readers were already aware (a) that many of the core ideas here were well-trodden territory in game theory and (b) that this post’s objectives were to:
- highlight these ideas to people on LessWrong
- highlight their potential relevance to AI risk
- highlight how this interacts with updateless decision theory and acausal trade
But maybe it’d be worth people who are interested in this problem reading that chapter of The Strategy of Conflict, or other relevant work in standard academic game theory, to see if there are additional ideas there that could be fruitful here.

MichaelA Jan 2, 2021, 8:39 AM
9 points
on: MichaelA’s Shortform
Problems in AI risk that economists could potentially contribute to
List(s) of relevant problems
- What can the principal-agent literature tell us about AI risk? (and this comment)
- Many of the questions in Technical AGI safety research outside AI
- Many of the questions in The Centre for the Governance of AI’s research agenda
- Many of the questions in Cooperation, Conflict, and Transformative Artificial Intelligence (a research agenda of the Center on Long-Term Risk)
- At least a couple of the questions in 80,000 Hours’ Research questions that could have a big social impact, organised by discipline
- Longtermist AI policy projects for economists (this doc was originally just made for Risto Uuk’s own use, so the ideas shouldn’t be taken as high-confidence recommendations to anyone else)
Context
I intend for this to include both technical and governance problems, and problems relevant to a variety of AI risk scenarios (e.g., AI optimising against humanity, AI misuse by humans, AI extinction risk, AI dystopia risk...)
Wei Dai’s list of Problems in AI Alignment that philosophers could potentially contribute to made me think that it could be useful to have a list of problems in AI risk that economists could potentially contribute to. So I began making such a list.
But:
- I’m neither an AI researcher nor an economist
- I spent hardly any time on this, and just included things I’ve stumbled upon, rather than specifically searching for these things
So I’m sure there’s a lot I’m missing.
Please comment if you know of other things worth mentioning here. Or if you’re better placed to make a list like this than I am, feel very free to do so; you could take whatever you want from this list and then comment here to let people know where to find your better thing.
(It’s also possible another list like this already exists. And it’s also possible that economists could contribute to such a large portion of AI risk problems that there’s no added value in making a separate list for economists specifically. If you think either of those things are true, please comment to say so!)
What links here?

MichaelA Jan 2, 2021, 7:38 AM
3 points
on: Clarifying some key hypotheses in AI alignment
It occurs to me that all of the hypotheses, arguments, and approaches mentioned here (though not necessarily the scenarios) seem to be about the “technical” side of things. There are two main things I mean by that statement:
First, this post seems to be limited to explaining something along the lines of “x-risks from AI accidents”, rather than “x-risks from misuse of AI”, or “x-risk from AI as a risk factor” (e.g., how AI could potentially increase risks of nuclear war).
I do think it makes sense to limit the scope that way, because:
- no one post can cover everything
- you don’t want to make the diagram overwhelming
- there’s a relatively clear boundary between what you’re covering and what you’re not
- what you’re covering seems like the most relevant thing for technical AI safety researchers, whereas the other parts are perhaps more relevant for people working on AI strategy/governance/policy
And the fact that this post’s scope is limited in that way seems somewhat highlighted by saying this is about AI alignment (whereas misuse could occur even with a system aligned to some human’s goals), and by saying “The idea is closely connected to the problem of artificial systems optimizing adversarially against humans.”
But I think misuse and “risk factor”/“structural risk” issues are also quite important, that they should be on technical AI safety researchers’ radars to some extent, and that they probably interact in some ways with technical AI safety/alignment. So, personally, I think I’d have made that choice of scope even more explicit.
I’d also be really excited to see a post that takes the same approach as this one, but for those other classes of AI risks.
---
The second thing I mean by the above statement is that this post seems to exclude non-technical factors that seem like they’d also impact the technical side or the AI accident risks.
One crux of this type would be “AI researchers will be cautious/sensible/competent “by default””. Here are some indications that that’s an “important and controversial hypothes[is] for AI alignment”:
- AI Impacts summarised some of Rohin’s comments as “AI researchers will in fact correct safety issues rather than hacking around them and redeploying. Shah thinks that institutions developing AI are likely to be careful because human extinction would be just as bad for them as for everyone else.”
- But my impression is that many people at MIRI would disagree with that, and are worried that people will merely “patch” issues in ways that don’t adequately address the risks.
- And I think many would argue that institutions won’t be careful enough, because they only pay a portion of the price of extinction; reducing extinction risk is a transgenerational global public good (see Todd and this comment).
- And I think views on these matters influence how much researchers would be happy with the approach of “Use feedback loops to course correct as we go”. I think the technical things influence how easily we theoretically could do that, while the non-technical things influence how much we realistically can rely on people to do that.
So it seems to me that a crux like that could perhaps fit well in the scope of this post. And I thus think it’d be cool if someone could either (1) expand this post to include cruxes like that, or (2) make another post with a similar approach, but covering non-technical cruxes relevant to AI safety.

MichaelA Jan 2, 2021, 7:35 AM
2 points
on: Clarifying some key hypotheses in AI alignment
Thanks for this post! This seems like a really great way of visually representing how these different hypotheses, arguments, approaches, and scenarios interconnect. (I also think it’d be cool to see posts on other topics which use a similar approach!)
It seems that AGI timelines aren’t explicitly discussed here. (“Discontinuity to AGI” is mentioned, but I believe that’s a somewhat distinct matter.) Was that a deliberate choice?
It does seem like several of the hypotheses/arguments mentioned here would feed into or relate to beliefs about timelines—in particular, Discontinuity to AGI, Discontinuity from AGI, and Recursive self-improvement, ML scales to AGI, and Deep insights needed (or maybe not that last one, as that means “needed” for alignment in particular). But I don’t think beliefs about timelines would be fully accounted for by those hypotheses/arguments—beliefs about timelines could also involve cruxes like whether “Intelligence is a huge collection of specific things”) or whether “There’ll be another AI winter before AGI” could also play a role.
I’m not sure to what extent beliefs about timelines (aside from beliefs about discontinuity) would influence which of the approaches people should/would take, out of the approaches you list. But I imagine that beliefs that timelines are quite short might motivate work on ML or prosaic alignment rather than (Near) proof-level assurance of alignment or Foundational or “deconfusion” research. This would be because people might then think the latter approaches would take too long, such that our only shot (given these people’s beliefs) is doing ML or prosaic alignment and hoping that’s enough. (See also.)
And it seems like beliefs about timelines would feed into decisions about other approaches you don’t mention, like opting for investment or movement-building rather than direct, technical work. (That said, it seems reasonable for this post’s scope to just be what a person should do once they have decided to work on AI alignment now.)

MichaelA Jan 1, 2021, 8:03 AM
2 points
on: Does the US nuclear policy still target cities?
Thanks for this post; I found it useful.
The US policy has never ruled out the possibility of escalation to full countervalue targeting and is unlikely to do so.
But the 2013 DoD report says “The United States will not intentionally target civilian populations or civilian objects”. That of course doesn’t prove that the US actually wouldn’t engage in countervalue targeting, but doesn’t it indicate that US policy at that time ruled out engaging in countervalue targeting?
This is a genuine rather than rhetorical question. I feel I might be just missing something, because, as you note, the paper you cited says:
Did this mean that the United States was discarding its ultimate assured destruction threat for deterring nuclear war? Clearly not. The guidance was carefully drafted. Does not rely on is different from will not resort to
...and yet, as far as I can see, the paper just doesn’t address the “will not intentionally target” line. So I feel confused by the paper’s analysis. (Though I haven’t read the paper in full.)

MichaelA

Final thoughts on whether you should read this book

My Anki cards

Problems in AI risk that economists could potentially contribute to