I put up a market, and then started looking. Click through to his post for the theses. I will be quoting a few of them in full, but not most of them.
I am not trying to be exact with these probabilities when the question calls for them, nor am I being super careful to make them consistent, so errors and adjustments are inevitable.
There are few things more important to U.S. national interest than close monitoring of frontier model capabilities, and also the ability to intervene.
Indeed, I believe one should be at best skeptical or ambivalent about most potential forms of regulation of anything, AI included. Yet I think the case for ‘oversight of the frontier labs’ is overwhelming.
Shout it from the rooftops: “As a temporary measure, using compute thresholds to pick out the AGI labs for safety-testing and disclosures is as light-touch and well-targeted as it gets.” It would be so helpful if more people understood this, and more others stopped pretending they did not understand it.
This as well. When you regulate ‘use’ or ‘risk’ you need to check on everyone’s ‘use’ of everything, and you make a lot of detailed micro interventions, and everyone has to file lots of paperwork and do lots of dumb things, and the natural end result is universal surveillance and a full ‘that which is not compulsory is forbidden’ regime across much of existence. Whereas a technology-focused approach can be entirely handled by the lab or manufacturer, then you are free.
Exactly. Compute is an imperfect proxy, but it is remarkably simple and robust. When it makes mistakes, they are false positives, where someone uses compute poorly and gets poor results. That is a small (measurement) mistake. Certainly compute is vastly better than all proposed alternative metrics.
It is highly reasonable to invoke the Defense Production Act regarding frontier AI as an actual bone fide national security situation where defense is a key concern. It is a far better justification than the median invocation of the act. The better reason to use the DPA is that it is currently the only mechanism available to the executive and our Congress is for now incapable of legislative action.
It does not require AGI or ASI to be near for us to get great value out of visibility into the frontier labs, and without that visibility the government cannot be confident that AGI or ASI is not near. I would prefer a different mechanism, but that would require a new law or counterfactual voluntary cooperation.
Shout it from the rooftops, seriously everyone stop pretending otherwise in the default case: “Requiring safety testing and disclosures for the outputs of $100 million-plus training runs is not an example of regulatory capture nor a meaningful barrier to entry relative to the cost of compute.” Yes, obviously one could eventually in theory ramp up those safety testing requirements sufficiently that they start to cost tens of millions or lots of specialized expertise and become a real barrier, and in theory that could scale faster than the training costs, but it is bizarre to think this is any kind of default. What you should worry about is not the cost of the test, it is that you might fail the test, at which point we ask why.
Section 2 is Most proposed ‘AI regulations’ are ill-conceived or premature.
My guess is that depends how we weigh the various proposals?
Yes. The government will need the ability to flexibly react quickly to events.
‘It is unwise to craft comprehensive statutory regulation at a technological inflection point, as the basic ontology of what is being regulated is in flux.’ I do think this is a good general principle, and would agree with it strongly if it said (e.g.) ‘typically unwise.’ And indeed, I would avoid committing to as many details as we can avoid, again especially with respect to mundane considerations. But also life is about to come at us fast and our government is slow, so we cannot afford to wait too long. So overall I will say agree (but not strongly).
Shout it from the rooftops: “The optimal policy response to AI likely combines targeted regulation with comprehensive deregulation across most sectors.” So does the optimal policy response to a lack of AI.
Yes, we can all agree that many regulation details will become obsolete even if they start out right at the time. So will many decisions to leave some area alone.
Even the static gains from deregulation tend to be a good deal, but yes I would say that in general the ability to adapt tends to be the bigger benefit. Certainly that is true in the AI case.
In the commercial space I strongly agree that legacy legal requirements are going to likely be much greater barriers than anything new we throw up any time soon. Indeed, I expect new laws to net enable AI adaptation, not prevent it.
This is highlighting common sense. If impact is sooner brace for it sooner.
Yes. The alternative path does not seem viable.
Shout it from the rooftops in all domains: “Existing laws and regulations are calibrated with the expectation of imperfect enforcement.”
I strongly agree that AI will enable more stringent law enforcement across the board. It is an important and under considered point. AI will often remove the norms and frictions that are load-bearing in prevent various problems, including in law enforcement. All of our laws, even those that have nothing to do with AI, will need to adjust to the new equilibrium, even if the world relatively ‘looks normal.’
I mostly agree that it is first best for states to avoid AI regulations, especially excluding California. For mundane AI they should very much avoid butting in. I do think there is a strong second-best ‘someone has to and no one else yet will’ argument for a bill like CA’s SB 1047, given the Congress we have. My biggest practical concern is exactly that California might not step aside and let itself be superseded when the time for that arrives, and the biggest advantage is it could be a template for the federal level.
Section 3 claims AI progress is accelerating, not plateauing.
I think this is probably right as a thesis statement, but definitely ‘too soon to tell’ applies. Here, it is less whether I agree, and more what probability I assign.
I would say something like 85% that the last 12 months were the slowest progress we’ll see in AI for the next let’s say 5 years (or until a potential post-singularity stabilization, which would not be foreseeable), in terms of publicly available capabilities. We started out with GPT-4, and ended with GPT-4-Turbo, Claude Opus and Gemini Advanced, all of which are only a little better, and didn’t see much else done. Yet. Buckle up. Strongly agree.
I notice I am confused on this one. Minimizing cross-entropy loss over human-generated text should converge to the abilities necessary to predict all human-generated text, which requires at least maximum-human intelligence to do? But in pure terms, if you literally could do nothing but scale LLMs and not improve your process, then my gut says yes, this would indeed converge, but I am only maybe 75% confident in that, and I note that it excludes a bunch of not so difficult to implement scaffolding capabilities, and also that ‘upper-human-level’ would likely allow bootstrapping.
This is a very similar and highly correlated prediction with 2, so 75% again.
I am not sure how exactly to interpret the claim here, but I think that RL-based threat models are being less than fully discounted, and reasonably so, but perhaps too much and I would not count them out? Maybe 40%? So disagree. Weird one.
Could be is weasel territory that implies 100%, however in terms of ‘will be’ I do expect this to be true in practice, something like 80% to be importantly true.
I agree with the first half and think that is a gimme as written, maybe another 80% zone. For the second half, it depends on fast something would count as a ‘foom.’ If it’s the traditional ‘in an hour or a day’ and requires ‘god-like ASI’ as is implied by the context then I’m reasonably confident here that the restrictions apply, and would be in the 90% zone, so ~70% compounded (to avoid implying false precision).
Again I think the ‘may’ clause is fully true, and this is even more likely to happen in practice, so let’s say 85%.
Yes, this is a strong agree, 95%.
Section 4 says open source is mostly a red herring.
Let’s see what he means by that. In some senses I might agree.
I expect [an expanding delta between closed and open models at the top end] to be true (75%) because I expect companies like Meta to realize the financial folly of giving away their work for free, and also for governments like America’s to not be keen on letting them do that for national security reasons, and also safety issues.
This is my first strong disagreement, because I expect ‘open source advocates’ to not come around until the actual catastrophe happens, at a minimum. Potential capabilities, I predict, will not convince them. I created a market for this one. Before any trading on it I would have put this rather low, something like 25% if we think ‘many’ means about half.
I strongly agree as written, as in it does not apply to Llama-3 400B. That release I do not expect to be dangerous directly either, but I would have caveats, as I have previously discussed.
Well, yes. I have long worried open weights is a no-good, very bad middle ground.
Yes.
I strongly disagree here. Open source advocates are not doing this because they love Meta, and they very much have deep philosophical views. Give them credit where credit is due, and also they hope to one day themselves catch up somehow. Right now Meta is the only one crazy enough and rich enough to plausibly do something hugely damaging, but that could change. A lot of the concerns of both sides are quite reasonably with what happens ‘at the limit.’
Well, yes, obviously, but that has little to do with how Meta operates. So I am not onboard with ‘the implication’ but I do agree as written.
I strongly disagree here as well. Why should Zuck’s Meta shares make him more concerned? Why would him drawing a salary matter? Altman is plenty rich already and this is him avoiding tying his wealth to OpenAI. As for the non-profit board, yeah, I am confused how one could think that, although of course a given board can care about anything at all.
I would be cautious about what counts as ‘lower-tier,’ and it is not obvious that such even properly mitigating these issues leads to great outcomes in some cases, but I would weakly agree as written.
Technically yes because of wording, certainly they have some of that effect as one thing they do, but mostly no, in the intended meaningful sense I disagree. I do not think being open is so important for defensive purposes, certainly far less so than offensive ones, although of course that too is ‘undermining adaptation’ in some sense. The primary ways restricting open sourcing ‘undermines adaptation’ I think would be (1) people who wanted to do various open things that the closed model owners won’t allow or that require privacy or data issues be solved, and (2) those restrictions will slow down offensive capabilities, and the offensive capabilities would otherwise force adaptation for defensive purposes to not get wiped out.
I mostly agree for sufficiently broad values of the terms widely available and cheap, for capabilities that would not be catastrophic to allow, and if we are ruling out ways to make them not widely available or not cheap. I think I more agree than disagree as written. But see #12, and also many other things that are cheap or easy to do that we make illegal, or that would be cheap or easy to do but we do our best to make expensive and difficult, because we believe the alternative is worse. Sometimes, although less than half the time, we are wise to do that.
True. And I do not especially want such laws repealed in most cases.
Section 5 claims accelerate versus decelerate is a false dichotomy.
This might be a bell curve meme situation? Yes, in important senses of course it is not so simple and a false dichotomy, but also in at least one important sense it is a real dichotomy.
That’s an interesting question. Will this be the most important decade for decisions? There have been some historical moments that seem highly contingent. The most obvious alternative candidate period is the decade leading up to World War 2, if one means decisions broadly. In terms of total impact, I can see pointing to crises in the Cold War that almost went nuclear, or certain key moments in religious history. Also, on the flip side, if you think the die is already cast, you could argue that the key moments were in the last decade or earlier, and what plays out now is incentives no one can stop. But I think I mostly agree with Hammond.
I like to think I am an existence proof of this, and I know many others.
This is strong enough that I disagree with it. yes, technology involves branching paths and things are nonlinear and the Civilization tech tree is a simplification and all that. But also there is a single light of science, and accelerating key developments in AI will tend to accelerate future key such developments, although I think at this point most AI activities do not meaningfully accelerate us further. Acceleration is a useful fake framework.
I think both matter. The speed we go down paths matters for shifting paths, including shifting among subpaths and branches, and also impacts what happens along even the mainline of those paths, for better and also worse. Also we do not only lose time to shift paths but to learn what paths might exist. But overall I do have to agree that as written the path we choose is the more important question.
This gets into what ‘AGI’ means. For sufficiently strong definitions, yes.
Yep.
Strongly disagree. Effective Altruism is not a bunch of virtue ethicists in disguise, they say they are utilitarians and when people tell you who they are believe them. I should know because I am a virtue ethicist who gets mad at them about this. e/acc is not about Nietzschean anything, he would write a highly entertaining rant if he saw you claiming that. Nor are they meaningfully atheists. They are the Waluigi of EA, and playing with memes and vibes. If you think EAs are metaphorical or spiritual Christians, then e/acc is not atheist, it is satanic.
Yes, of course the ‘accelerationism’ lobby outrstrips and outspends the safety lobby. Shout it from the rooftops, and roll your eyes if anyone tells you different.
There is high uncertainty, but in expectation I disagree and think Biden is better, given that Biden issued the executive order and Trump has pledged to repeal the executive order, I presume mostly because Biden issued it. I do think that Trump is in essentially all ways ‘high variance’ so if you think we are super doomed in the baseline scenarios then I can see an argument the other way.
Agreed.
I mean, consider the baseline of the average progressive. So yes, very much so, I only wish such voices were as loud in all the places where they are right.
Yep, exactly, so much so I noted this in #9. One can generalize this beyond AI.
I assume these are true statements. I do not think Bannon has any influence on Trump. But Hannity also thinks AI is crazy dangerous, and he might.
Section 6 is The AI wave is inevitable, superintelligence isn’t.
I don’t know what the ‘tech tree’ looks like for superintelligence, but under my baseline scenario it seems extremely difficult to avoid entirely, although we have a lot of control still over what form it would take.
I agree it is not a fait accompli. Like almost anything it can be an ideological goal, but I do not think it is right to say it is primarily that. So I think I weakly disagree.
Right now I strongly agree. The question is how long this will remain true as the pressures mount, or how long it would remain true if those three companies used their degrees of freedom.
Yes, shout it from the rooftops: “Creating a superintelligence is inherently dangerous and destabilizing, independent of the hardness of alignment.”
Yes, we could, but can we make this choice in practice? That is the question.
Yes, especially the childless part, but you could still do so much worse.
I disagree that SBF and Altman are more alike than different, but not so strongly, and I see from context that Hammond knows what he is claiming here.
This is a true statement, and he is making his full claims very clear.
I laid out my view in the Moral Mazes sequence. I think we disagree here more than we agree, but Hammond’s view here is more accurate than the median one.
Section 7 says technological transitions cause regime changes.
Why yes, they do.
Yes, even the best case scenarios are going to be dicey, move fast and break things.
Yes, along with everything else. I’m not quite going to disagree but I think this is severely underselling what is coming.
Congress has been unacceptably unproductive, well, since FDR, but also that has protected us from, well, the kinds of things done under FDR. I think I disagree that it will be important to have Congress keep up, we do not have a Congress capable of keeping up. They will need to get a few big things right and enable the state to react largely without them otherwise, and I think this could work. No, that is not ideal in many senses, but I do not see any practical alternative. We cannot expect miracles. Although with AI to help, productivity could get much higher very quickly.
What are we comparing this to? Adaptation of AI willy nilly? Using the standard practices whatever they are? I don’t even know, this is not a strong area for me. Obviously every time you slow things down for non-critical concerns you raise possibility of systemic failure, so some of this is net harmful in that sense. But I think without any such policies at all systemic failure is inevitable, so I disagree.
Shout it from the rooftops, only even more generalized and unhedged: ‘The rapid diffusion of AI agents with approximately human-level reasoning and planning abilities is likely sufficient to destabilize most existing U.S. institutions.’
Yes, and indeed so did past cognitive transitions that might otherwise look small.
Yes, although I doubt that this is the scenario we will land ourselves in.
Section 8 says institutional regime changes are packaged deals.
This does seem to historically be true.
Yes, liberal democratic capitalism is a technologically-contingent equilibrium, and also contingent on other things, it could still have fallen during the 20th century on multiple occasions if things had been not so different, and replaced by one of two much, much worse alternatives. But the key thing here is that liberal democratic capitalism works because it happens to work best in the technological settings we have had in the past. We hope this will continue to be true, but it might not be, and our fertility problems are also a big hint that it might not be such a stable equilibrium even without AI.
I see why one would say that, and I would confirm that when conditions change in some ways this often requires or suggests other adjustments, but mostly I think I disagree and that people are being too cute by at least half here.
This does seem like the default if AI advances sufficiently, and this would likely be the least of our transformations and problems. Our institutions are based on various assumptions and intuitions that will stop making any sense, and there will be various things they will not know how to handle.
Yes. Maximally ‘democratized’ AI, or giving everyone access to similarly powerful AI, would force much more oppressive interventions, both to maintain civilization and to satisfy public demands. If you have empowered even the smallest computing devices in ways the public cannot abide, then even if this does not fully cause collapse, catastrophe, loss of control or extinction, you are not going to get a crypto libertarian paradise. You are going to, at best, get full universal surveillance and social control, at least of electronics.
Yes, and people are sleeping on this.
Yes, versus the alternative.
So do periods that lack technological change. Our recent past is no exception.
I am definitely not going to go full Robin Hanson here. Do not presume your property rights will protect you under explosive growth. But I still disagree with Hammond here, because I do not think this rises to the level of imply. Your property rights might be less violated than they are rendered not so relevant.
Note that this is an extremely optimistic future for regular humans, where demand for labor keeps rising because humans become more productive on the margin, not less. Should we expect this scenario? It is a kind of middle path, where AI is mostly complementary to humans and thus demand for labor goes up rather than down. I disagree, because I do not see this as likely. I expect AI to make us more productive, but to primarily turn out to be a substitute more than a compliment in the areas it greatly advances. Nor do I think we will need any such incentive to deploy AI to places it can work, there will likely only be a small window where AI policeman versus human policeman is a close comparison.
I even more strongly disagree here. Technological unemployment happens, essentially, when the AI takes both your job and the job that would replace your job under past technological employment shifts. At some point, what is there left for you to do? And why should we assume this involves a collapse of capitalism? To some extent, yes, there will be ‘demand for humans as humans,’ but even here one should expect limits.
Section 9 says dismissing AGI risks as ‘sci-fi’ is a failure of imagination.
That is one of the things it at least sometimes is.
Yes. Even AI-Fizzle world looks like sci-fi.
Yes. Dismissing things as ‘sci-fi’ is unserious. Talk about physical possibility.
There are smart terminator analogies and also dumb ones. The problem is that the most basic ones are some mix of dumb and easy to mock and portray as dumb. And there are also many ways these analogies can mislead. And of course, you don’t want your examples to involve time travel, even if we all agree the time travel has nothing to do with anything. The actual movies are much smarter than they look, and actually raise good points, but analogies care about what people can point to and how people associate and vibe. So on net I think I disagree that terminator analogies are underrated in practice, we go to discourse with the associations we have. Alas. But I could be wrong.
I don’t even know what we mean by consciousness. I notice I am confused and suspect others are confused as well and can see this either way, so I’m going to neither agree nor disagree.
Obviously consciousness is scale-dependent on some lower bound, but I presume that is not what he means here. The theory here is that it also might have an upper bound, or no longer be needed then? I think I am going to disagree here with the central intent, because I doubt scaling up would make consciousness become inefficient, even though technically this is a ‘may’ statement.
I have not taken the time to look in depth, but for now I disagree, this does not seem right or promising to me.
I strongly disagree here, assuming this is ‘in the eyes of humans.’ I notice that if you tell me humans were demoted as moral persons, I am highly confident artificial minds got promoted to moral persons instead. I do not see a plausible future of humans thinking there are zero moral persons. Of course, if all the humans die and only AIs remain, then in some sense humans have been demoted as moral persons and AIs might not be moral persons to each other, and that future seems highly plausible to me, but I would not consider this humans being demoted in this sense, and I do not think this is what Hammond meant?
I think it’s pretty much nonsense to talk about ‘thermodynamics favors’ anything, but certainly I think that unconscious replicators are a likely outcome. I think that counts as agreement here.
I think this is probably right, although this still seems rather bona fide to me.
Interesting set of choices you gave us there. I am confident it would be a much bigger deal than the printing press, or else it wouldn’t count and AI has fizzled, but in the spirit intended I agree that this is up for grabs.
Yes, this seems right enough to go with, if loose and imprecise.
Sure, why not?
I do not think ‘IQ of 1,000’ is a meaningful thing given how I think the scale works, but to the extent it is, then yes, so I think I agree with the intent.
I disagree after reading the Wikipedia definition of anticommons. I do agree we could probably do it if we cared enough, and it should be a top priority and a top social good, but I don’t see why it is an anticommons situation.
Shout territory: “There are more ways for a post-human transition to go poorly than to go well.” Indeed. Anyone who says ‘particular bad scenario X is unlikely therefore things will go well’ is not addressing the actual situation. Conditional on transitioning to something in any sense ‘post-human’ that is vastly more true.
I’ve made related points often, that ‘who can be blamed’ is a key aspect of any situation, and often ‘no one’ is the ideal answer.
One can never be fully sure, but I am confident one should act as if this is true.
Tallying Up the Points
So in total, that’s 23 disagreements and 1 where I don’t feel I can either agree or disagree, which leaves 71 agreements out of 95. There is a bit of ‘cheating’ in the sense that some of these are essentially facts and others us words like ‘may,’ but I think we are still looking at about 60% agreement on non-trivial statements.
Conclusion
I very much appreciated the format of the 95 theses as concrete taking off points. This seems like a highly valuable exercise, perhaps I should try to do a version as well, and I encourage others to do so. It is good to be explicit and concrete. I now feel I have a much better idea of where Hammond stands than most others out there.
I Got 95 Theses But a Glitch Ain’t One
Link post
Or rather Samuel Hammond does. Tyler Cowen finds it interesting but not his view.
I put up a market, and then started looking. Click through to his post for the theses. I will be quoting a few of them in full, but not most of them.
I am not trying to be exact with these probabilities when the question calls for them, nor am I being super careful to make them consistent, so errors and adjustments are inevitable.
Section 1 is Oversight of AGI labs is prudent.
I do tend to say that.
There are few things more important to U.S. national interest than close monitoring of frontier model capabilities, and also the ability to intervene.
Indeed, I believe one should be at best skeptical or ambivalent about most potential forms of regulation of anything, AI included. Yet I think the case for ‘oversight of the frontier labs’ is overwhelming.
Shout it from the rooftops: “As a temporary measure, using compute thresholds to pick out the AGI labs for safety-testing and disclosures is as light-touch and well-targeted as it gets.” It would be so helpful if more people understood this, and more others stopped pretending they did not understand it.
This as well. When you regulate ‘use’ or ‘risk’ you need to check on everyone’s ‘use’ of everything, and you make a lot of detailed micro interventions, and everyone has to file lots of paperwork and do lots of dumb things, and the natural end result is universal surveillance and a full ‘that which is not compulsory is forbidden’ regime across much of existence. Whereas a technology-focused approach can be entirely handled by the lab or manufacturer, then you are free.
Exactly. Compute is an imperfect proxy, but it is remarkably simple and robust. When it makes mistakes, they are false positives, where someone uses compute poorly and gets poor results. That is a small (measurement) mistake. Certainly compute is vastly better than all proposed alternative metrics.
It is highly reasonable to invoke the Defense Production Act regarding frontier AI as an actual bone fide national security situation where defense is a key concern. It is a far better justification than the median invocation of the act. The better reason to use the DPA is that it is currently the only mechanism available to the executive and our Congress is for now incapable of legislative action.
It does not require AGI or ASI to be near for us to get great value out of visibility into the frontier labs, and without that visibility the government cannot be confident that AGI or ASI is not near. I would prefer a different mechanism, but that would require a new law or counterfactual voluntary cooperation.
Shout it from the rooftops, seriously everyone stop pretending otherwise in the default case: “Requiring safety testing and disclosures for the outputs of $100 million-plus training runs is not an example of regulatory capture nor a meaningful barrier to entry relative to the cost of compute.” Yes, obviously one could eventually in theory ramp up those safety testing requirements sufficiently that they start to cost tens of millions or lots of specialized expertise and become a real barrier, and in theory that could scale faster than the training costs, but it is bizarre to think this is any kind of default. What you should worry about is not the cost of the test, it is that you might fail the test, at which point we ask why.
Section 2 is Most proposed ‘AI regulations’ are ill-conceived or premature.
My guess is that depends how we weigh the various proposals?
Yes. The government will need the ability to flexibly react quickly to events.
‘It is unwise to craft comprehensive statutory regulation at a technological inflection point, as the basic ontology of what is being regulated is in flux.’ I do think this is a good general principle, and would agree with it strongly if it said (e.g.) ‘typically unwise.’ And indeed, I would avoid committing to as many details as we can avoid, again especially with respect to mundane considerations. But also life is about to come at us fast and our government is slow, so we cannot afford to wait too long. So overall I will say agree (but not strongly).
Shout it from the rooftops: “The optimal policy response to AI likely combines targeted regulation with comprehensive deregulation across most sectors.” So does the optimal policy response to a lack of AI.
Yes, we can all agree that many regulation details will become obsolete even if they start out right at the time. So will many decisions to leave some area alone.
Even the static gains from deregulation tend to be a good deal, but yes I would say that in general the ability to adapt tends to be the bigger benefit. Certainly that is true in the AI case.
In the commercial space I strongly agree that legacy legal requirements are going to likely be much greater barriers than anything new we throw up any time soon. Indeed, I expect new laws to net enable AI adaptation, not prevent it.
This is highlighting common sense. If impact is sooner brace for it sooner.
Yes. The alternative path does not seem viable.
Shout it from the rooftops in all domains: “Existing laws and regulations are calibrated with the expectation of imperfect enforcement.”
I strongly agree that AI will enable more stringent law enforcement across the board. It is an important and under considered point. AI will often remove the norms and frictions that are load-bearing in prevent various problems, including in law enforcement. All of our laws, even those that have nothing to do with AI, will need to adjust to the new equilibrium, even if the world relatively ‘looks normal.’
I mostly agree that it is first best for states to avoid AI regulations, especially excluding California. For mundane AI they should very much avoid butting in. I do think there is a strong second-best ‘someone has to and no one else yet will’ argument for a bill like CA’s SB 1047, given the Congress we have. My biggest practical concern is exactly that California might not step aside and let itself be superseded when the time for that arrives, and the biggest advantage is it could be a template for the federal level.
Section 3 claims AI progress is accelerating, not plateauing.
I think this is probably right as a thesis statement, but definitely ‘too soon to tell’ applies. Here, it is less whether I agree, and more what probability I assign.
I would say something like 85% that the last 12 months were the slowest progress we’ll see in AI for the next let’s say 5 years (or until a potential post-singularity stabilization, which would not be foreseeable), in terms of publicly available capabilities. We started out with GPT-4, and ended with GPT-4-Turbo, Claude Opus and Gemini Advanced, all of which are only a little better, and didn’t see much else done. Yet. Buckle up. Strongly agree.
I notice I am confused on this one. Minimizing cross-entropy loss over human-generated text should converge to the abilities necessary to predict all human-generated text, which requires at least maximum-human intelligence to do? But in pure terms, if you literally could do nothing but scale LLMs and not improve your process, then my gut says yes, this would indeed converge, but I am only maybe 75% confident in that, and I note that it excludes a bunch of not so difficult to implement scaffolding capabilities, and also that ‘upper-human-level’ would likely allow bootstrapping.
This is a very similar and highly correlated prediction with 2, so 75% again.
I am not sure how exactly to interpret the claim here, but I think that RL-based threat models are being less than fully discounted, and reasonably so, but perhaps too much and I would not count them out? Maybe 40%? So disagree. Weird one.
Could be is weasel territory that implies 100%, however in terms of ‘will be’ I do expect this to be true in practice, something like 80% to be importantly true.
I agree with the first half and think that is a gimme as written, maybe another 80% zone. For the second half, it depends on fast something would count as a ‘foom.’ If it’s the traditional ‘in an hour or a day’ and requires ‘god-like ASI’ as is implied by the context then I’m reasonably confident here that the restrictions apply, and would be in the 90% zone, so ~70% compounded (to avoid implying false precision).
Again I think the ‘may’ clause is fully true, and this is even more likely to happen in practice, so let’s say 85%.
Yes, this is a strong agree, 95%.
Section 4 says open source is mostly a red herring.
Let’s see what he means by that. In some senses I might agree.
I expect [an expanding delta between closed and open models at the top end] to be true (75%) because I expect companies like Meta to realize the financial folly of giving away their work for free, and also for governments like America’s to not be keen on letting them do that for national security reasons, and also safety issues.
This is my first strong disagreement, because I expect ‘open source advocates’ to not come around until the actual catastrophe happens, at a minimum. Potential capabilities, I predict, will not convince them. I created a market for this one. Before any trading on it I would have put this rather low, something like 25% if we think ‘many’ means about half.
I strongly agree as written, as in it does not apply to Llama-3 400B. That release I do not expect to be dangerous directly either, but I would have caveats, as I have previously discussed.
Well, yes. I have long worried open weights is a no-good, very bad middle ground.
Yes.
I strongly disagree here. Open source advocates are not doing this because they love Meta, and they very much have deep philosophical views. Give them credit where credit is due, and also they hope to one day themselves catch up somehow. Right now Meta is the only one crazy enough and rich enough to plausibly do something hugely damaging, but that could change. A lot of the concerns of both sides are quite reasonably with what happens ‘at the limit.’
Well, yes, obviously, but that has little to do with how Meta operates. So I am not onboard with ‘the implication’ but I do agree as written.
I strongly disagree here as well. Why should Zuck’s Meta shares make him more concerned? Why would him drawing a salary matter? Altman is plenty rich already and this is him avoiding tying his wealth to OpenAI. As for the non-profit board, yeah, I am confused how one could think that, although of course a given board can care about anything at all.
I would be cautious about what counts as ‘lower-tier,’ and it is not obvious that such even properly mitigating these issues leads to great outcomes in some cases, but I would weakly agree as written.
Technically yes because of wording, certainly they have some of that effect as one thing they do, but mostly no, in the intended meaningful sense I disagree. I do not think being open is so important for defensive purposes, certainly far less so than offensive ones, although of course that too is ‘undermining adaptation’ in some sense. The primary ways restricting open sourcing ‘undermines adaptation’ I think would be (1) people who wanted to do various open things that the closed model owners won’t allow or that require privacy or data issues be solved, and (2) those restrictions will slow down offensive capabilities, and the offensive capabilities would otherwise force adaptation for defensive purposes to not get wiped out.
I mostly agree for sufficiently broad values of the terms widely available and cheap, for capabilities that would not be catastrophic to allow, and if we are ruling out ways to make them not widely available or not cheap. I think I more agree than disagree as written. But see #12, and also many other things that are cheap or easy to do that we make illegal, or that would be cheap or easy to do but we do our best to make expensive and difficult, because we believe the alternative is worse. Sometimes, although less than half the time, we are wise to do that.
True. And I do not especially want such laws repealed in most cases.
Section 5 claims accelerate versus decelerate is a false dichotomy.
This might be a bell curve meme situation? Yes, in important senses of course it is not so simple and a false dichotomy, but also in at least one important sense it is a real dichotomy.
That’s an interesting question. Will this be the most important decade for decisions? There have been some historical moments that seem highly contingent. The most obvious alternative candidate period is the decade leading up to World War 2, if one means decisions broadly. In terms of total impact, I can see pointing to crises in the Cold War that almost went nuclear, or certain key moments in religious history. Also, on the flip side, if you think the die is already cast, you could argue that the key moments were in the last decade or earlier, and what plays out now is incentives no one can stop. But I think I mostly agree with Hammond.
I like to think I am an existence proof of this, and I know many others.
This is strong enough that I disagree with it. yes, technology involves branching paths and things are nonlinear and the Civilization tech tree is a simplification and all that. But also there is a single light of science, and accelerating key developments in AI will tend to accelerate future key such developments, although I think at this point most AI activities do not meaningfully accelerate us further. Acceleration is a useful fake framework.
I think both matter. The speed we go down paths matters for shifting paths, including shifting among subpaths and branches, and also impacts what happens along even the mainline of those paths, for better and also worse. Also we do not only lose time to shift paths but to learn what paths might exist. But overall I do have to agree that as written the path we choose is the more important question.
This gets into what ‘AGI’ means. For sufficiently strong definitions, yes.
Yep.
Strongly disagree. Effective Altruism is not a bunch of virtue ethicists in disguise, they say they are utilitarians and when people tell you who they are believe them. I should know because I am a virtue ethicist who gets mad at them about this. e/acc is not about Nietzschean anything, he would write a highly entertaining rant if he saw you claiming that. Nor are they meaningfully atheists. They are the Waluigi of EA, and playing with memes and vibes. If you think EAs are metaphorical or spiritual Christians, then e/acc is not atheist, it is satanic.
Yes, of course the ‘accelerationism’ lobby outrstrips and outspends the safety lobby. Shout it from the rooftops, and roll your eyes if anyone tells you different.
There is high uncertainty, but in expectation I disagree and think Biden is better, given that Biden issued the executive order and Trump has pledged to repeal the executive order, I presume mostly because Biden issued it. I do think that Trump is in essentially all ways ‘high variance’ so if you think we are super doomed in the baseline scenarios then I can see an argument the other way.
Agreed.
I mean, consider the baseline of the average progressive. So yes, very much so, I only wish such voices were as loud in all the places where they are right.
Yep, exactly, so much so I noted this in #9. One can generalize this beyond AI.
I assume these are true statements. I do not think Bannon has any influence on Trump. But Hannity also thinks AI is crazy dangerous, and he might.
Section 6 is The AI wave is inevitable, superintelligence isn’t.
I don’t know what the ‘tech tree’ looks like for superintelligence, but under my baseline scenario it seems extremely difficult to avoid entirely, although we have a lot of control still over what form it would take.
I agree it is not a fait accompli. Like almost anything it can be an ideological goal, but I do not think it is right to say it is primarily that. So I think I weakly disagree.
Right now I strongly agree. The question is how long this will remain true as the pressures mount, or how long it would remain true if those three companies used their degrees of freedom.
Yes, shout it from the rooftops: “Creating a superintelligence is inherently dangerous and destabilizing, independent of the hardness of alignment.”
Yes, we could, but can we make this choice in practice? That is the question.
Understatement of the year. If an ASI exists and it isn’t you? Look at me. I’m the sovereign now.
Yes, especially the childless part, but you could still do so much worse.
I disagree that SBF and Altman are more alike than different, but not so strongly, and I see from context that Hammond knows what he is claiming here.
This is a true statement, and he is making his full claims very clear.
I laid out my view in the Moral Mazes sequence. I think we disagree here more than we agree, but Hammond’s view here is more accurate than the median one.
Section 7 says technological transitions cause regime changes.
Why yes, they do.
Yes, even the best case scenarios are going to be dicey, move fast and break things.
Yes, along with everything else. I’m not quite going to disagree but I think this is severely underselling what is coming.
Congress has been unacceptably unproductive, well, since FDR, but also that has protected us from, well, the kinds of things done under FDR. I think I disagree that it will be important to have Congress keep up, we do not have a Congress capable of keeping up. They will need to get a few big things right and enable the state to react largely without them otherwise, and I think this could work. No, that is not ideal in many senses, but I do not see any practical alternative. We cannot expect miracles. Although with AI to help, productivity could get much higher very quickly.
What are we comparing this to? Adaptation of AI willy nilly? Using the standard practices whatever they are? I don’t even know, this is not a strong area for me. Obviously every time you slow things down for non-critical concerns you raise possibility of systemic failure, so some of this is net harmful in that sense. But I think without any such policies at all systemic failure is inevitable, so I disagree.
Shout it from the rooftops, only even more generalized and unhedged: ‘The rapid diffusion of AI agents with approximately human-level reasoning and planning abilities is likely sufficient to destabilize most existing U.S. institutions.’
Yes, and indeed so did past cognitive transitions that might otherwise look small.
Yes, although I doubt that this is the scenario we will land ourselves in.
Section 8 says institutional regime changes are packaged deals.
This does seem to historically be true.
Yes, liberal democratic capitalism is a technologically-contingent equilibrium, and also contingent on other things, it could still have fallen during the 20th century on multiple occasions if things had been not so different, and replaced by one of two much, much worse alternatives. But the key thing here is that liberal democratic capitalism works because it happens to work best in the technological settings we have had in the past. We hope this will continue to be true, but it might not be, and our fertility problems are also a big hint that it might not be such a stable equilibrium even without AI.
I see why one would say that, and I would confirm that when conditions change in some ways this often requires or suggests other adjustments, but mostly I think I disagree and that people are being too cute by at least half here.
This does seem like the default if AI advances sufficiently, and this would likely be the least of our transformations and problems. Our institutions are based on various assumptions and intuitions that will stop making any sense, and there will be various things they will not know how to handle.
Yes. Maximally ‘democratized’ AI, or giving everyone access to similarly powerful AI, would force much more oppressive interventions, both to maintain civilization and to satisfy public demands. If you have empowered even the smallest computing devices in ways the public cannot abide, then even if this does not fully cause collapse, catastrophe, loss of control or extinction, you are not going to get a crypto libertarian paradise. You are going to, at best, get full universal surveillance and social control, at least of electronics.
Yes, and people are sleeping on this.
Yes, versus the alternative.
So do periods that lack technological change. Our recent past is no exception.
I am definitely not going to go full Robin Hanson here. Do not presume your property rights will protect you under explosive growth. But I still disagree with Hammond here, because I do not think this rises to the level of imply. Your property rights might be less violated than they are rendered not so relevant.
Note that this is an extremely optimistic future for regular humans, where demand for labor keeps rising because humans become more productive on the margin, not less. Should we expect this scenario? It is a kind of middle path, where AI is mostly complementary to humans and thus demand for labor goes up rather than down. I disagree, because I do not see this as likely. I expect AI to make us more productive, but to primarily turn out to be a substitute more than a compliment in the areas it greatly advances. Nor do I think we will need any such incentive to deploy AI to places it can work, there will likely only be a small window where AI policeman versus human policeman is a close comparison.
I even more strongly disagree here. Technological unemployment happens, essentially, when the AI takes both your job and the job that would replace your job under past technological employment shifts. At some point, what is there left for you to do? And why should we assume this involves a collapse of capitalism? To some extent, yes, there will be ‘demand for humans as humans,’ but even here one should expect limits.
Section 9 says dismissing AGI risks as ‘sci-fi’ is a failure of imagination.
That is one of the things it at least sometimes is.
Yes. Even AI-Fizzle world looks like sci-fi.
Yes. Dismissing things as ‘sci-fi’ is unserious. Talk about physical possibility.
There are smart terminator analogies and also dumb ones. The problem is that the most basic ones are some mix of dumb and easy to mock and portray as dumb. And there are also many ways these analogies can mislead. And of course, you don’t want your examples to involve time travel, even if we all agree the time travel has nothing to do with anything. The actual movies are much smarter than they look, and actually raise good points, but analogies care about what people can point to and how people associate and vibe. So on net I think I disagree that terminator analogies are underrated in practice, we go to discourse with the associations we have. Alas. But I could be wrong.
I don’t even know what we mean by consciousness. I notice I am confused and suspect others are confused as well and can see this either way, so I’m going to neither agree nor disagree.
Obviously consciousness is scale-dependent on some lower bound, but I presume that is not what he means here. The theory here is that it also might have an upper bound, or no longer be needed then? I think I am going to disagree here with the central intent, because I doubt scaling up would make consciousness become inefficient, even though technically this is a ‘may’ statement.
I have not taken the time to look in depth, but for now I disagree, this does not seem right or promising to me.
I strongly disagree here, assuming this is ‘in the eyes of humans.’ I notice that if you tell me humans were demoted as moral persons, I am highly confident artificial minds got promoted to moral persons instead. I do not see a plausible future of humans thinking there are zero moral persons. Of course, if all the humans die and only AIs remain, then in some sense humans have been demoted as moral persons and AIs might not be moral persons to each other, and that future seems highly plausible to me, but I would not consider this humans being demoted in this sense, and I do not think this is what Hammond meant?
I think it’s pretty much nonsense to talk about ‘thermodynamics favors’ anything, but certainly I think that unconscious replicators are a likely outcome. I think that counts as agreement here.
I think this is probably right, although this still seems rather bona fide to me.
Interesting set of choices you gave us there. I am confident it would be a much bigger deal than the printing press, or else it wouldn’t count and AI has fizzled, but in the spirit intended I agree that this is up for grabs.
Finally, Section 10 says biology is an information technology.
Yes, this seems right enough to go with, if loose and imprecise.
Sure, why not?
I do not think ‘IQ of 1,000’ is a meaningful thing given how I think the scale works, but to the extent it is, then yes, so I think I agree with the intent.
I disagree after reading the Wikipedia definition of anticommons. I do agree we could probably do it if we cared enough, and it should be a top priority and a top social good, but I don’t see why it is an anticommons situation.
Shout territory: “There are more ways for a post-human transition to go poorly than to go well.” Indeed. Anyone who says ‘particular bad scenario X is unlikely therefore things will go well’ is not addressing the actual situation. Conditional on transitioning to something in any sense ‘post-human’ that is vastly more true.
I’ve made related points often, that ‘who can be blamed’ is a key aspect of any situation, and often ‘no one’ is the ideal answer.
One can never be fully sure, but I am confident one should act as if this is true.
Tallying Up the Points
So in total, that’s 23 disagreements and 1 where I don’t feel I can either agree or disagree, which leaves 71 agreements out of 95. There is a bit of ‘cheating’ in the sense that some of these are essentially facts and others us words like ‘may,’ but I think we are still looking at about 60% agreement on non-trivial statements.
Conclusion
I very much appreciated the format of the 95 theses as concrete taking off points. This seems like a highly valuable exercise, perhaps I should try to do a version as well, and I encourage others to do so. It is good to be explicit and concrete. I now feel I have a much better idea of where Hammond stands than most others out there.