Chris Olah wrote this topic prompt (with some feedback from me (Asya) and Nick Beckstead). We didn’t want to commit him to being responsible for this post or responding to comments on it, so we submitted this on his behalf. (I’ve changed the by-line to be more explicit about this.)
abergal
Karma: 755
Updates to Open Phil’s career development and transition funding program
The Long-Term Future Fund is looking for a full-time fund chair
Long-Term Future Fund Ask Us Anything (September 2023)
Long-Term Future Fund: April 2023 grant recommendations
[AMA] Announcing Open Phil’s University Group Organizer and Century Fellowships [x-post]
Truthful and honest AI
Interpretability
Techniques for enhancing human feedback
Measuring and forecasting risks
Request for proposals for projects in AI alignment that work with deep learning systems
Thanks for writing this! Would “fine-tune on some downstream task and measure the accuracy on that task before and after fine-tuning” count as measuring misalignment as you’re imagining it? My sense is that there might be a bunch of existing work like that.
This RFP is an experiment for us, and we don’t yet know if we’ll be doing more of them in the future. I think we’d be open to including research directions we think that are promising that apply equally well to both DL and non-DL systems—I’d be interested in hearing any particular suggestions you have.
(We’d also be happy to fund particular proposals in the research directions we’ve already listed that apply to both DL and non-DL systems, though we will be evaluating them on how well they address the DL-focused challenges we’ve presented.)
Getting feedback in the next week would be ideal; September 15th will probably be too late.
Different request for proposals!
Provide feedback on Open Philanthropy’s AI alignment RFP
Open Philanthropy is seeking proposals for outreach projects
Thank you so much for writing this! I’ve been confused about this terminology for a while and I really like your reframing.
An additional terminological point that I think it would be good to solidify is what people mean when they refer to “inner alignment” failures. As you alude to, my impression is that some people use it to refer to objective robustness failures, broadly, whereas others (e.g. Evan) use it to refer to failures that involve mesa optimization. There is then additional confusion around whether we should think “inner alignment” failures that don’t involve mesa optimization will be catastrophic and, relatedly, around whether humans count as mesa optimizers.
I think I’d advocate for letting “inner alignment” failures refer to objective robustness failures broadly, talking about “mesa optimization failures” as such, and then leaving the question about whether there are problematic inner alignment failures that aren’t mesa optimization-related on the table.- Refactoring Alignment (attempt #2) by (26 Jul 2021 20:12 UTC; 46 points)
- Collection of arguments to expect (outer and inner) alignment failure? by (28 Sep 2021 16:55 UTC; 20 points)
- Collection of arguments to expect (outer and inner) alignment failure? by (28 Sep 2021 16:55 UTC; 20 points)
I feel pretty bad about both of your current top two choices (Bellingham or Peekskill) because they seem too far from major cities. I worry this distance will seriously hamper your ability to hire good people, which is arguably the most important thing MIRI needs to be able to do. [Speaking personally, not on behalf of Open Philanthropy.]
Announcement: “How much hardware will we need to create AGI?” was actually inspired by a conversation I had with Ronny Fernandez and forgot about, credit goes to him for the original idea of using ‘the weights of random objects’ as a reference class.
https://i.imgflip.com/1xvnfi.jpg
I think your first point basically covers why—people are worried about alignment difficulties in superhuman systems, in particular (because those are the dangerous systems which can cause existential failures). I think a lot of current RLHF work is focused on providing reward signals to current systems in ways that don’t directly address the problem of “how do we reward systems with behaviors that have consequences that are too complicated for humans to understand”.