(To clarify: I co-founded and led PIBBSS since 2021, butstepped down from leadership in June this year to work with davidad’s on the Safeguarded AI programme. This means I’m no longer in charge of executive & day-to-day decisions at PIBBSS. As such, nothing of what I say below should be taking as authoritative source about what PIBBSS is going to do. I do serve on the board.)
Ryan - I appreciate the donation, and in particular you sharing your reasoning here.
I agree with a lot of what you write. Especially “connectors” (point 2) and bringing in relatively more senior academics from non-CS/non-ML fields (point 3) are IMO things that are valuable and PIBBSS has a good track record for delivering on.
Regarding both your point 1 and reservation 1 (while trying to abstract a bit from the fact that terms like ‘blue sky’ research are somewhat fuzzy, and that I expect at least some parts of a disagreement here might turn out to disappear when considering concrete examples), I do think there has been some change in PIBBSS’ research thinking & prioritization which has been unfolding in my head since summer 2023, and finding its way more concretely into PIBBSS’ strategy since the start of 2024. Lucas is the best person to talk to this in more detail, but I’ll still share a few thoughts that were on my mind back when I was still leading PIBBSS.
I continue to believe that there is a lot of value to be head in investigating the underlying principles of intelligent behaviour (what one might refer to as blue sky or basic research), and to do so with a good dose of epistemic pluralism (studying such intelligent behaviour from/across a range of different systems, substrates and perspectives). I think this is a or the core aspect of the PIBBSS spirit. However, after the first 2 years of PIBBSS, I also wasn’t entirely happy with our concrete research outputs. I thought a lot about what’s up here (first mainly by myself, and later together with Lucas) and about how to do better—all the while staying true to the roots/generators/principles of PIBBSS, as I see them.
One of the key axis of improvement we’ve grown increasingly confident about is what we sometimes refer to as “bridging the theory-practice gap”. I’m pretty bullish on theory(!) -- but theory alone is not the answer. Theory on its own isn’t in a great position to know where to go next/what to prioritise, or whether it’s making progress at all. I have seen many theoretical threads that felt intriguing/compelling, but failed to bottom out in something tangibly productive because they were lacking feedback loops that help guide them, and that would force them to operationalise abstract notions into something of concrete empirical and interventionist value. (EDIT: ‘interventionist’ is me trying to point to when a theory is good enough to allow you to intervene in the world or design artifacts in such a way that they reliably lead to what you intended.)
This is not an argument against theory, in my eyes, but an argument that theorizing about intelligent behaviour (or any given phenomena) will benefit a lot from having productive feedback loops with the empirical. As such, what we’ve come to want to foster at PIBBSS is (a) ambitious, ‘theory-first’ AI safety research, (b) with an interdisciplinary angle, (c) that is able to find meaningful and iterative empirical feedback loops. It’s not so much that a theoretical project that doesn’t yet have a meaningful way of making contact with reality should be disregarded—and more than a key source of progress for said project will be to find ways of articulating/operationalising that theory, so that it starts making actual (testable) predictions about a real system and can come to inform design choices.
These updates in our thinking led to a few different downstream decisions. One of them was trying to have our fellowship cohort have include some empirical ML profiles/projects (I endorse roughly a 10-15% fraction, similar to what Joseph suggests). Reasons for this are, both, that we think this work is likely to be useful, and also because it changes (and IMO improves) the cohort dynamics (compared to, say, 0-5% ML). That said, I agree that once going above 20%, I would start to worry that something essential about the PIBBSS spirit might get lost, and I’m not excited about that from an ecosystem perspective (given that e.g. MATS is doing what it’s doing).
Another downstream implication (though it’s somewhat earlier days for PIBBSS on that one) is that I’ve become pretty excited about trying to help move ambitious ideas from (what I call) an ‘Idea Readiness Level’ (IDL; borrowing from the notion of ‘Technology Readiness Levels’) 1-3, to an IDL of 5-7. My thinking here is that once an idea/research agenda is at IDLs 5-7, it typically has been able to enter the broader epistemic discourse, it has some initial legible evidence/output it can rely on to making its own case—and at that point I would say it no longer is in the area where PIBBSS has the greatest comparative advantage to support it. On the other hand, I think there isn’t much (if any) obvious places where IDL 1-3 ideas get a chance to get iterate on quickly and stress-tested to develop into a more mature & empirically grounded agenda. (I think the way we were able to support Adam Shai & co in developing the computational mechanics agenda is a pretty great example of this use case I’m painting here -though notably their ideas were already relatively mature compared to other things PIBBSS might ambitiously help to mature.)
I’d personally be very excited for a PIBBSS that becomes excellent at filling that gap, and think PIBBSS has a bunch of the necessary ingredients for that already. I see this as a potentially critical investment in medium term robustness of the research ecosystem, and into what i think is an undeniable need to come to base AI Safety on rigorous scientific understanding. (Though notably Lucas/Dusan, PIBBSS’ new leadership, might disagree & should have a chance to speak for themselves here.)
[Edited a bit for clafity]
(To clarify: I co-founded and led PIBBSS since 2021, but stepped down from leadership in June this year to work with davidad’s on the Safeguarded AI programme. This means I’m no longer in charge of executive & day-to-day decisions at PIBBSS. As such, nothing of what I say below should be taking as authoritative source about what PIBBSS is going to do. I do serve on the board.)
Ryan - I appreciate the donation, and in particular you sharing your reasoning here.
I agree with a lot of what you write. Especially “connectors” (point 2) and bringing in relatively more senior academics from non-CS/non-ML fields (point 3) are IMO things that are valuable and PIBBSS has a good track record for delivering on.
Regarding both your point 1 and reservation 1 (while trying to abstract a bit from the fact that terms like ‘blue sky’ research are somewhat fuzzy, and that I expect at least some parts of a disagreement here might turn out to disappear when considering concrete examples), I do think there has been some change in PIBBSS’ research thinking & prioritization which has been unfolding in my head since summer 2023, and finding its way more concretely into PIBBSS’ strategy since the start of 2024. Lucas is the best person to talk to this in more detail, but I’ll still share a few thoughts that were on my mind back when I was still leading PIBBSS.
I continue to believe that there is a lot of value to be head in investigating the underlying principles of intelligent behaviour (what one might refer to as blue sky or basic research), and to do so with a good dose of epistemic pluralism (studying such intelligent behaviour from/across a range of different systems, substrates and perspectives). I think this is a or the core aspect of the PIBBSS spirit. However, after the first 2 years of PIBBSS, I also wasn’t entirely happy with our concrete research outputs. I thought a lot about what’s up here (first mainly by myself, and later together with Lucas) and about how to do better—all the while staying true to the roots/generators/principles of PIBBSS, as I see them.
One of the key axis of improvement we’ve grown increasingly confident about is what we sometimes refer to as “bridging the theory-practice gap”. I’m pretty bullish on theory(!) -- but theory alone is not the answer. Theory on its own isn’t in a great position to know where to go next/what to prioritise, or whether it’s making progress at all. I have seen many theoretical threads that felt intriguing/compelling, but failed to bottom out in something tangibly productive because they were lacking feedback loops that help guide them, and that would force them to operationalise abstract notions into something of concrete empirical and interventionist value. (EDIT: ‘interventionist’ is me trying to point to when a theory is good enough to allow you to intervene in the world or design artifacts in such a way that they reliably lead to what you intended.)
This is not an argument against theory, in my eyes, but an argument that theorizing about intelligent behaviour (or any given phenomena) will benefit a lot from having productive feedback loops with the empirical. As such, what we’ve come to want to foster at PIBBSS is (a) ambitious, ‘theory-first’ AI safety research, (b) with an interdisciplinary angle, (c) that is able to find meaningful and iterative empirical feedback loops. It’s not so much that a theoretical project that doesn’t yet have a meaningful way of making contact with reality should be disregarded—and more than a key source of progress for said project will be to find ways of articulating/operationalising that theory, so that it starts making actual (testable) predictions about a real system and can come to inform design choices.
These updates in our thinking led to a few different downstream decisions. One of them was trying to have our fellowship cohort have include some empirical ML profiles/projects (I endorse roughly a 10-15% fraction, similar to what Joseph suggests). Reasons for this are, both, that we think this work is likely to be useful, and also because it changes (and IMO improves) the cohort dynamics (compared to, say, 0-5% ML). That said, I agree that once going above 20%, I would start to worry that something essential about the PIBBSS spirit might get lost, and I’m not excited about that from an ecosystem perspective (given that e.g. MATS is doing what it’s doing).
Another downstream implication (though it’s somewhat earlier days for PIBBSS on that one) is that I’ve become pretty excited about trying to help move ambitious ideas from (what I call) an ‘Idea Readiness Level’ (IDL; borrowing from the notion of ‘Technology Readiness Levels’) 1-3, to an IDL of 5-7. My thinking here is that once an idea/research agenda is at IDLs 5-7, it typically has been able to enter the broader epistemic discourse, it has some initial legible evidence/output it can rely on to making its own case—and at that point I would say it no longer is in the area where PIBBSS has the greatest comparative advantage to support it. On the other hand, I think there isn’t much (if any) obvious places where IDL 1-3 ideas get a chance to get iterate on quickly and stress-tested to develop into a more mature & empirically grounded agenda. (I think the way we were able to support Adam Shai & co in developing the computational mechanics agenda is a pretty great example of this use case I’m painting here -though notably their ideas were already relatively mature compared to other things PIBBSS might ambitiously help to mature.)
I’d personally be very excited for a PIBBSS that becomes excellent at filling that gap, and think PIBBSS has a bunch of the necessary ingredients for that already. I see this as a potentially critical investment in medium term robustness of the research ecosystem, and into what i think is an undeniable need to come to base AI Safety on rigorous scientific understanding. (Though notably Lucas/Dusan, PIBBSS’ new leadership, might disagree & should have a chance to speak for themselves here.)