I definitely am not quite sure what the epistemic state of the paper is, or even its goal. Bostrom, Dafoe and Flynn keep mentioning that this paper is not a complete list of desiderata, but I don’t know what portion of key desiderata they think they’ve hit, or why they think it’s worthwhile at this stage to pre-emptively list the desiderata that currently seem important.
(Added: My top hypothesis is that Bostrom was starting a policy group with Dafoe as its head, and thought to himself “What are the actual policy implications of the work in my book?” and then wrote them down, without expecting it to be complete, just an obvious starting point.)
As to my thoughts on whether the recommendations in the paper seem good… to be honest, it all felt so reasonable and simple (added: this is a good thing). There were not big leaps of inference. It didn’t feel surprising to me. But here’s a few updates/reflections.
I have previously run the thought experiment “What would I do if I were at the start, or just before the start, of the industrial revolution?” Thought pertaining to massive turbulence, redistribution, and concentration, and adaptability, seemed natural focal concerns to me, but I had not made them as precise or as clear as the paper had. Then again I’d been thinking more about what I as an individual should do, not how a government or larger organisation should approach the problem. I definitely hadn’t thought about population dynamics in that context (which were also a big deal after the industrial revolution—places like England scaled by an order of magnitude, requiring major infrastructural changes in politics, education, industry, and elsewhere).
I think that the technical details of AI are most important in the sections on Efficiency and Population. The sections on Allocation and Process I would expect to apply to any technological revolution (industrial, agricultural, etc).
I’m not sure that this is consistent with his actions, but I think it’s likely that Ben from yesterday would’ve said the words “In order to make sensible progress on AI policy you require a detailed understanding of the new technology”. I realise now that, while it is indeed required to get the overall picture right, there is progress to be made that merely takes heed of this being a technological revolution of historic proportions, and does not need to matter too much which particular technological revolution we’re going through.
I’ve seen another discussion here, along with the Vulnerable World Hypothesis paper (LW discussion here), for the need for the ability to execute a massive coordination increase. I’m going to definitely think more about ‘conditional stabilization’, how exactly it follows from the conceptual space of thinking about singletons and coordination, and what possible things it might look like (global surveillance seems terrible on the face of it, I wonder if moving straight to that is premature. I think there’s probably a lot more granular ways of thinking about surveillance).
In general this paper is full of very cautious and careful conceptual work, based on simple arguments and technical understandings of AI and coordination. In general I don’t trust many people to do this without vetting the ideas in depth myself or without seeing a past history of their success. Bostrom certainly ticks the latter box and weakly ticks the former box for me (I’ve yet to personally read enough of his writings to say anything stronger there), and given that he’s a primary author on this paper, I feel epistemically safe taking on these framings without 30-100 hours of further examination.
I hope to be able to spend a similar effort summarising the many other strategic papers Bostrom and others at the FHI have produced.
Feedback
For future posts of a similar nature, please PM me if you have any easy changes that would’ve made this post more useful to you / made it easier to get the info you needed (I will delete public comments on that topic). It’d also be great to (publicly) hear that someone else actually read the paper and checked whether my notes missed something important or are inaccurate.
It’d also be great to (publicly) hear that someone else actually read the paper and checked whether my notes missed something important or are inaccurate.
I read the paper over a year ago (before the update), and reviewing my notes, they look similar to yours (but less detailed).
Reflections
I definitely am not quite sure what the epistemic state of the paper is, or even its goal. Bostrom, Dafoe and Flynn keep mentioning that this paper is not a complete list of desiderata, but I don’t know what portion of key desiderata they think they’ve hit, or why they think it’s worthwhile at this stage to pre-emptively list the desiderata that currently seem important.
(Added: My top hypothesis is that Bostrom was starting a policy group with Dafoe as its head, and thought to himself “What are the actual policy implications of the work in my book?” and then wrote them down, without expecting it to be complete, just an obvious starting point.)
As to my thoughts on whether the recommendations in the paper seem good… to be honest, it all felt so reasonable and simple (added: this is a good thing). There were not big leaps of inference. It didn’t feel surprising to me. But here’s a few updates/reflections.
I have previously run the thought experiment “What would I do if I were at the start, or just before the start, of the industrial revolution?” Thought pertaining to massive turbulence, redistribution, and concentration, and adaptability, seemed natural focal concerns to me, but I had not made them as precise or as clear as the paper had. Then again I’d been thinking more about what I as an individual should do, not how a government or larger organisation should approach the problem. I definitely hadn’t thought about population dynamics in that context (which were also a big deal after the industrial revolution—places like England scaled by an order of magnitude, requiring major infrastructural changes in politics, education, industry, and elsewhere).
I think that the technical details of AI are most important in the sections on Efficiency and Population. The sections on Allocation and Process I would expect to apply to any technological revolution (industrial, agricultural, etc).
I’m not sure that this is consistent with his actions, but I think it’s likely that Ben from yesterday would’ve said the words “In order to make sensible progress on AI policy you require a detailed understanding of the new technology”. I realise now that, while it is indeed required to get the overall picture right, there is progress to be made that merely takes heed of this being a technological revolution of historic proportions, and does not need to matter too much which particular technological revolution we’re going through.
I’ve seen another discussion here, along with the Vulnerable World Hypothesis paper (LW discussion here), for the need for the ability to execute a massive coordination increase. I’m going to definitely think more about ‘conditional stabilization’, how exactly it follows from the conceptual space of thinking about singletons and coordination, and what possible things it might look like (global surveillance seems terrible on the face of it, I wonder if moving straight to that is premature. I think there’s probably a lot more granular ways of thinking about surveillance).
In general this paper is full of very cautious and careful conceptual work, based on simple arguments and technical understandings of AI and coordination. In general I don’t trust many people to do this without vetting the ideas in depth myself or without seeing a past history of their success. Bostrom certainly ticks the latter box and weakly ticks the former box for me (I’ve yet to personally read enough of his writings to say anything stronger there), and given that he’s a primary author on this paper, I feel epistemically safe taking on these framings without 30-100 hours of further examination.
I hope to be able to spend a similar effort summarising the many other strategic papers Bostrom and others at the FHI have produced.
Feedback
For future posts of a similar nature, please PM me if you have any easy changes that would’ve made this post more useful to you / made it easier to get the info you needed (I will delete public comments on that topic). It’d also be great to (publicly) hear that someone else actually read the paper and checked whether my notes missed something important or are inaccurate.
I read the paper over a year ago (before the update), and reviewing my notes, they look similar to yours (but less detailed).