The definitions follow the format recommended in The Craft of
Research:
‘I’m studying X, because I want to find out Y, so that I (and you) can better
understand Z.’ The quality of language and content varies.
An unaligned benchmark
----------------------
1
I'm studying mild optimization,
because I want to understand what its problems and limitations
are,
because I want to decide whether or not to work on one of
them,
because well-functioning mild optimization appears to
be close to what humans and organisations safely do
today, so basing an AI on it might have useful
outcomes.
- This is more a study project than a research project.
2
I'm studying how image classifiers can recognize new types of
images,
because I want to find out how to detect out-of-distribution
inputs,
in order to help my reader understand how to make ML
systems that don't confidently make wrong predictions.
Approval-directed agents/bootstrapping
--------------------------------------
3
I'm studying Bayesian machine learning,
because I want to understand how to make ML systems that
notice when they are confused
in order to help my reader understand how to make ML
systems that will ask the overseer for input when doing
otherwise would lead to failure.
4
I'm studying possible structures of approval-directed agents
because I want to understand how much human thought and input
they would require
in order to help my reader understand whether
approval-directed agents are feasible.
Humans Consulting HCH
---------------------
5
I'm studying the articles linked from Humans Consulting HCH,
because I want to understand the section ‘Hope’ from Humans
Consulting HCH
in order be able to think about the tension between the
system being capable and reflecting the human's judgment.
Corrigibility (Christiano 2017)
-------------------------------
6
I'm studying Omohundro's preferences-about-your-utility-function
case,
because I want to implement it
in order to help my reader understand how it can be
implemented and whether it is unstable.
7
I'm studying the change in prediction failures as a predictor
becomes stronger,
because I want to find out whether the failures not only
become fewer, but also harder to detect over time,
in order to help my reader understand whether systems
built on such predictors will remain corrigible as they
become stronger.
Iterated Distillation and Amplification
---------------------------------------
8
I'm studying the capability and safety-relevant properties of
imitation learning,
because I want to find out whether it can produce aligned
agents,
in order to help my reader understand how to even get to
the base case of iterated distillation and amplification.
Benign model-free RL
--------------------
9
I'm studying the performance of benign model-free RL,
because I want to know whether the claim that it will achieve
state-of-the-art performance is true,
in order to help my reader understand how useful benign
model-free RL will be.
10
I'm studying reward learning and robustness,
because I want to know whether they can achieve competitive
agents without malign behaviour,
in order to help my reader understand whether benign
model-free RL will be possible.
11
I'm implementing (parts of) benign model-free RL,
because I want to know what works and doesn't work in
practice,
in order to help my reader understand which parts of the
scheme need further conceptual research.
Supervising strong learners by amplifying weak experts
------------------------------------------------------
12
I'm studying ways to improve the sample efficiency of a supervised
learner,
because I want to know how to reduce the number of calls to H
in CSASupAmp,
in order to help my reader understand how we can adapt
that proof-of-concept for solving real world tasks that
require even more
training data.
13
I'm studying the effects of how CSASupAmp samples questions,
because I want to know how to sample questions in a way that
improves the scheme's learning performance,
in order to help my reader understand how we can adapt
that proof-of-concept for solving real world tasks that
require even more training data.
Machine Learning Projects for Iterated Distillation and Amplification
--------------------------------------------------------------------
14
❇ Any of the projects there. At first glance Adaptive Computation
is most interesting, but perhaps also requires most studying. I
would ask Owain to find out what people are already working on or
what has received most interest, then work on the least crowded
one.
Directions and desiderata for AI alignment
------------------------------------------
15
I'm studying integrating models of known human heuristics and
biases into IRL systems,
because I want to improve the performance of IRL in a domain
where it is hindered by the discrepancies between the existing
error models and actual human irrationality
in order to help my readers understand how to get IRL
systems to infer true human values despite Stuart
Armstrong's impossibility result.
The reward engineering problem
------------------------------
16
I'm experimenting with semi-supervised reinforcement learning,
because I want to find out how humans can supervise machine
learning with reasonably small effort,
in order to help my reader understand how to avoid
optimizing proxy objectives that we have to use because
the sample hunger of current ML algorithms is so great.
17
I'm studying the use of a discriminator in imitation learning,
because I want to find out how to help humans produce
demonstrations that the agent can imitate,
in order to help my reader understand how we might use
imitation learning to solve the reward engineering
problem.
Capability amplification (Christiano 2016)
------------------------------------------
18
I'm studying cognitive tasks and how to decompose them into ever
simpler steps,
because I want to find algorithms for capability amplification
in order to help my reader understand the nature of
obstacles to capability amplification. – What is the
obstacle? How exactly is it an obstacle? Make it simple.
Learning with catastrophes
--------------------------
19
❇ I'm studying the development of the performance of an adversary
in adversarial training,
because I want to find out whether the adversary gets worse as
the primary agent becomes more robust, or whether the
adversary gets traction at all when the primary agent is
already quite robust,
in order to help my reader understand how confident we
could be in a red team to find all relevant catastrophic
situations.
Thoughts on reward engineering
------------------------------
20
I'm studying the effects of importance sampling on the behaviour
that an RL agent learns,
because I want to find out whether it can lead to undesirable
outcomes
in order to help my reader understand whether importance
sampling can solve the problem of widely varying rewards
in reward engineering.
21
I'm studying the effects of an inconsistent comparison function on
optimizing with comparisons,
because I want to know whether it prevents the two agents from
converging on a desirable equilibrium quickly enough
inorder to help my reader understand whether optimizing
with comparisons can solve the problem of inconsistency
and unreliability in reward engineering.
Techniques for optimizing worst-case performance
------------------------------------------------
22
❇ I'm studying transparency in the service of adversarial training
(using transparency to ease finding adversarial examples, or to
detect adversarial success earlier/more often),
because I want to make the adversary ten times more effective
in order to help my reader understand how to build ML
systems that never fail catastrophically.
Reliability ampflification
--------------------------
23
I'm studying the impact of overseer failure on RL-based IDA,
because I want to know under what conditions the amplification
increases or decreases the failure rate,
in order to help my reader understand whether we need to
combine capability amplification with explicit reliability
amplification in all cases.
Security amplification
----------------------
24
I'm studying adversarial examples for meta-execution with ML-based
sub-agents,
because I want to find a case where security amplification by
meta-execution fails to amplify security,
in order to help my reader understand what obstructions to
security amplification there are.
Meta-execution
--------------
No project idea from the article directly. Ought has put forth
some open issues (cf.
https://docs.google.com/document/d/1xzFuDD1xiG-oe750MYrP9PEgwEXtxbfSBnHrgrCRnhY/edit#),
but that might be outdated and would be too closely tied to Ought.
Twenty-three AI alignment research project definitions
I came up with these research project definitions when I read the iterated amplification sequence. Last year I put five of them up for voting (see Which of these five AI alignment research projects ideas are no good?) and chose no. 23 to work on (see IDA with RL and overseer failures). But I didn’t think of publishing all of them until JJ Hepburn gave me the idea that they might be useful to others.
The definitions follow the format recommended in The Craft of Research: ‘I’m studying X, because I want to find out Y, so that I (and you) can better understand Z.’ The quality of language and content varies.