Director at AI Impacts.
Richard Korzekwa
Karma: 1,442
The only lesson to learn from history is to not learn lessons from it, especially when something as freaky and unprecedented as AGI is concerned.
This seems like a pretty wild claim to me, even as someone who agrees that AGI is freaky and unprecedented, possibly to the point that we should expect it to depart drastically from past experience.
Also, this is reference class tennis. If the rules allow for changing the metric in the middle of the debate, I shoot back with “the first telegraph cable improved transatlantic communication latency more than ten-million-fold the instant it was turned on; how’s that for a discontinuity”.
To be clear, I’m not saying “there’s this iron law about technology and you might thing nuclear weapons disprove it, but they don’t because <reasons>” (I’m not claiming there are any laws or hard rules about anything at all). What I’m saying is that there’s a thing that usually happens, but it didn’t happen with nuclear weapons, and I think we can see why. Nuclear weapons absolutely do live in the relevant reference class, and I think the way their development happened should make us more worried about AGI.
Little Boy was a gun-type device with hardly any moving parts; it was the “larger and heavier” and inefficient and impractical prototype and it still absolutely blew every conventional bomb out of the water.
It was, and this is a fair point. But Little Boy used like a billion dollars worth of HEU, which provided a very strong incentive not to approach the design process in the usual iterative way.
For contrast, the laser’s basic physics advantage over other light sources (in coherence length and intensity) is at least as big as the nuclear weapons advantage over conventional explosives, and the first useful laser was approximately as simple as Little Boy, but the first laser was still not very useful. My claim is that this is because the cost of iterating was very low and there was no need to make it useful on the first try.
The less useful device would (probably) not have been (much) lower yield, it would have been much larger and heavier. For example, part of what led to the implosion device was the calculation for how long a gun-type plutonium weapon would need to be, which showed it would not fit on an aircraft. I agree that the scarcity of the materials is likely sufficient to limit the kind of iterated “let’s make sure we understand how this works in principle before we try to make something useful” process that normally goes into making new things (and that was part of “these constraints” you quote, though maybe I didn’t write it very clearly).
Edited to add:
Also, my phrasing “scarcity of materials” may be downplaying the extent to which scaling up uranium and plutonium production was part of the technological progress necessary for making a nuclear weapon. But I sometimes see people attribute the impressive and scary suddenness of deployable nuclear weapons entirely to the physics of energy release from a supercritical mass, and I think this is a mistake.
Yes, sorry, I somehow missed your reply until now. I probably meant this: https://store.yujiintl.com/collections/high-cri-white-led-strips/products/yujileds-high-cri-95-dynamic-tunable-white-multirow-led-flexible-strip
(That link I did share is pretty interesting, BTW. It describes some stuff during the Wild West days of medical research, including the use of a frighteningly primitive laser to kill a tumor in a kid’s eye)
Observed patterns around major technological advancements
Shortly before this went up, I made a spreadsheet to do “various mathematical assessments” (brier scores in particular) on your predictions and Scott’s. This was purely to satisfy my own curiosity, and to see if my very rough impression of which predictions were faring better was accurate. I did it in a pretty quick-and-dirty way, so it seems likely that I made mistakes. But if anyone else is curious, I’m sharing it here. Feel free leave comments or copy the sheet and do whatever with it.
For what it’s worth, what you’re describing at Google is consistent with my reading of the prediction. I read it as “Google continues to widely allow remote work, no questions asked”. If, as of the resolution date, Google was still allowing people to work from home without special approval, that sounds like “allowing remote work, no questions asked”, even if it is not a permanent state of affairs. If there’s some process for officially requesting permission to work from home, but it is approved by default, that still seems positive to me but not as clearly positive.
It is ambiguously-worded, so I can see why people are saying it’s wrong, but to me the default reading resolves positive based on what Google employees are saying.
Yeah, this is part of what I was getting at. The narrowness of the task “write a set of instructions for a one-off victory against a particular player” is a crucial part of what makes it seem not-obviously-impossible to me. Fully simulating Magnus should be adequate, but then obviously you’re invoking a reasoner. What I’m uncertain about is if you can write such instructions without invoking a reasoner.
Like, you can’t make an “oracle chess AI” that tells you at the beginning of the game what moves to play, because even chess is too chaotic for that game tree to be feasibly representable. You’ve gotta keep running your chess AI on each new observation, to have any hope of getting the fragment of the game tree that you consider down to a managable size.
It’s not obvious to me how generally true this is. You can’t literally specify every move at the beginning of the game, but it seems like there could be instructions that work for more specified chess tasks. Like, I imagine a talented human chess coach could generate a set of instructions in English that would work well for defeating me at chess at least once (maybe there already exist “how to beat noobs at chess” instructions that will work for this). I would be unsurprised if there exists a set of human-readable instructions of human-readable length that would give me better-than-even odds of defeating a pre-specified chess expert at least once, that can be generated just by noticing and exploiting as-yet-unnoticed regularities in either that expert’s play in particular or human-expert-level chess in general.
It’s possible my intuition here is related to my complete lack of expertise in chess, and I would not be surprised if Magnus-Carlsen-defeating instructions do not exist (at least, not without routing through a reasoner). Still, I think I assign greater credence to shallow-pattern-finding AI enabling a pivotal act than you do, and I’m wondering if the chess example is probing this difference in intuition.
Two views that I have seen from the AI risk community on perverse incentives within academia are:
-
Incentives within the academic community are such that researchers are unable to pursue or publish work that is responsible, high-quality, and high-value. With few exceptions, anyone who attempts to do so will be outcompeted and miss out on funding, tenure, and social capital, which will eventually lead to their exit from academia.
-
Money, effort, and attention within the academic community are allocated by a process that is only loosely aligned with the goal of producing good research. Some researchers are able to avoid the trap and competently pursue valuable projects, but many will not, and a few will even go after projects that are harmful.
I think 1 is overplayed. There may be fields/subfields that are like that, but I think there is room in most fields for the right kind of people to pursue high-value research while succeeding in academia. I think 2 is a pretty big deal, though. I’m not too worried about all the people who will get NSF grants for unimportant research, though I am a little concerned that a flood of papers that are missing the point will go against the goal of legitimizing AI risk research for policy impact.
What I’m more worried about is research that is actively harmful. For example, my understanding is that a substantial portion of gain-of-function research has been funded by the federal government. This strikes me as frighteningly analogous to the kind of work that we should be concerned about in AI risk. I think this was mostly NIH, not NSF, so maybe there are good reasons for thinking the NSF is less likely to support dangerous work? Is there a strategy or an already-in-place mechanism for preventing people from using NSF funds for high-risk work? Or maybe there’s an important difference in incentives here that I’m not seeing?
For what it’s worth, I’m mostly agnostic on this, with a slight lean toward NSF attention being bad. Many of the people I most admire for their ability to solve difficult problems are academics, and I’m excited about the prospect of getting more people like that working on these problems. I really don’t want to dismiss it unfairly. I find it pretty easy to imagine worlds in which this goes very badly, but I think the default outcome is probably that a bunch of money goes to pointless stuff, a smaller amount goes to very valuable work, the field grows and diversifies, and (assuming timelines are long enough) the overall result is a reduction in AI risk. But I’m not very confident of this, and the downsides seem much larger than the potential benefits.
-
As an elaboration on 5:
If a very minimal answer is helpful, emphasize that this is the case. For example, if I’m emailing an expert on some topic I don’t know very much about, I will add something like “I would be grateful for any direction on this, even if it’s just the names of some relevant authors”
In my experience, people are often pretty good at putting together a low-cost-to-them, but highly-valuable-for-me answer. For example, they’ll say something like “This was well-studied by <names> several years ago, but basically the answer is <one or two sentence response that lacks details and might be a bit high-context for me, but is nonetheless helpful>”
If there’s a succinct way to convey your background, this can make it easier for the person to write a short response that is useful to you (though this can quickly go against 8, if you’re not careful). For example, I once emailed a guy who studies bird aerodynamics and after explaining my question, I added the context that I have a PhD in physics. He gave me a very short, but very helpful answer that was only possible by using some physics jargon.
It seems likely to me that, when evaluating the impact of this, changes in available funding are a smaller consideration than changes in the field’s status with the government and the academic community. NSF grants carry different obligations from the most prominent funding streams for AI safety research, and they function as a credential of sorts for being a Legitimate Line of Inquiry.
I’m pretty uncertain about how this plays out. I would expect policy work to benefit from research that is more legibly credible to people outside the community, and NSF support should help with that. On the other hand, the more traditional scientific community is full of perverse incentives and it may be bad to get tangled up in it. I imagine there are other considerations I’m not aware of.
How much AI safety work is already receiving federal funding? Maybe there’s already some evidence about how this is likely to go?
There’s the study mentioned in another comment that shows promising-but-suspicious evidence that vaccines more-or-less eliminate long covid risk.
Katja may be able to tell you if she knows more about the particular studies she cites, but I’ve found that there is very little good research on long covid that is not mostly or entirely looking at unvaccinated cases. The Norwegian pre-print that Katja mentions explicitly excludes vaccinated people. I don’t see stats on the dates for the papers in the 81 study review article she cites, nor did I look at the dates on all 81 studies, but I did look at the ones with controls and they were all from 2020 or the first half of 2021, so pretty clearly too early to look at vaccinated covid. All of the high-quality studies I’m aware of are too early to include any substantial number of vaccinated cases.
This is unfortunate and somewhat preventable, but not too surprising. There were not many breakthrough cases before the delta waves, which didn’t take off until about 7 months ago. This doesn’t leave much time for researchers to track people who recover from acute cases and show months of persistent symptoms, then write a paper and get it published.
If half of my friends are getting covid right now, there seems to be massive value in waiting another three months before reassessing precautions, so I can see how it goes for them. While noisy, I expect to get a better sense of the distribution of outcomes among people in my reference class—and especially whether there is a macroscopic chance of really dire outcomes—from this (I think regrettable) experiment than I get from cobbling together different studies about vaguely defined symptoms among ambiguous or not-quite-relevant demographics, with various guesstimated adjustments.
I recommend registering in advance how you will evaluate the new information you get over the next several months. In particular, it would be valuable to write down your credence on various outcomes, conditional on covid being something that is worth paying a substantial cost to avoid.
My guess is that you and I will disagree on what we’re likely to see in low vs high risk worlds. For example, I think that A and B in your analysis are almost guaranteed, whether long covid risk is large or small. Also, C, F, G, H, and L are entirely consistent with low risk, and only modestly more likely in worlds where long covid is a major concern.
Yeah, those or these: https://vdoc.pub/documents/lasers-in-opthalmology-basics-diagnostics-and-surgical-aspects-a-review-3ha5mu7ureog
Or the long 2700K/6500K ribbons.
They’re not as bad to setup as I’d feared, though they are a bit of a hassle. I’m experimenting with them now, and I will write about it if I come up with a good way to build a light fixture with them.
First, some meta-level things I’ve learned since writing this:
-
What people crave most is very practical advice on what to buy. In retrospect this should have been more obvious to me. When I look for help from others on how to solve a problem I do not know much about, the main thing I want is very actionable advice, like “buy this thing”, “use this app”, or “follow this Twitter account”.
-
Failing that, what people want is legible, easy-to-use criteria for making decisions on their own. Advice like “Find something with CRI>90, and more CRI is better” is better than “Here’s a big, complex description of the criteria you should try to optimize for that will inevitably tradeoff against each other”.
-
The technical background is important, but in a somewhat different way than I’d thought when I wrote it. When I was writing it, I was hoping to help transmit my model of how things work so that people could use it to make their own decisions. I still think it’s good to try to do this, however imperfectly it might happen in practice. But I think the main reason it is important is because people want to know where I’m coming from, what kinds of things I considered, and how deeply I have investigated the matter.
On the object level:
-
As far as I know, the post does not contain any major errors in the technical background.
-
Some of the practical advice is probably skewed too far toward my personal preferences. A lot of people seem to prefer lower color temperature light than I do, for example.
-
I now think that getting LEDs with a high-quality spectrum is fairly easy with the right budget, and the harder part is figuring out where to put them to illuminate your visual field without doing something annoying like having a bright cornbulb at eye-level. The Lightcone team seems like they’re making good progress on this and doing good experiments.
-
My impression from talking to people over the last 14 months is that it would be a pretty huge public service for someone to keep an up-to-date list of what the best lights are on the market(s) for various budgets and circumstances, as well as a bunch of guides/photographs for helping replicate specific solutions people have found that work for them.
-
By far my favorite new-to-me lighting product is the Yuji adjustable-color-temperature LED strips/panels. I’m excited to experiment with them and hopefully publish some useful results.
-
I have thought about it, and I am somewhat considering one for my office. Heat and power consumption used to be a concern, but now that there are good LED floodlights available, the main issue for me, personally, is that in the spaces where I work they will tend to cast sharper shadows that I want, since most floodlights behave like small sources and you only need one or maybe two to light an area.
I’d be interested to hear about any particular arrangements you’ve considered!
The second one is interesting to me because if you increase weight by caking it in mud, the mud will break/fall/rub off, and the rock will return to its previous weight. But if you break off a piece, it will generally not return to its previous weight. Maybe a version of this that returns to equilibrium from both directions is a car? If you break a reasonable number of pieces off or put wear on the tires or burn some gas or oil, it will return to its ‘equilibrium’ weight via maintenance?
Yeah, I can’t really tell how much to conclude from the examples I give on this. The problem is that “uncertainty” is both hard to specify in a way that makes for good comparisons and hard to evaluate in retrospect.
I’m glad you brought up flight, because I think it may be a counterexample to my claim that uncertain communities have produced important advances, but confused communities have not. My impression is that everyone was pretty confused about flight in 1903, but I don’t know that much about it. There may also be a connection between level of confusion and ability to make the first version less terrible or improve on it quickly (for example, I think the Manhattan Project scientists were less confused than the scientists working on early lasers).
I think this objection is basically right, in that this sample (and arguably the entire reference class) relies heavily on discreteness in a way that may ultimately be irrelevant to TAI. Like, maybe there will be no clear “first version” of an AI that deeply and irrevocably changes the world. Still, it may be worth mentioning that some of the members of this reference class, such as penicillin and the Haber process, turned out not to be discontinuities (according to our narrow definition).
This doesn’t seem crazy. I think the lesson from historical building sizes is “Whoa, building height really didn’t track underlying tech at all”. If for some reason AI performance tracks limits of underlying technology very badly, we might expect the first version of a scary thing to conform badly to these patterns.
I would guess this is not what will happen, though, since most scary AI capabilities that we worry about are much more valuable than building height. Still, penicillin was a valuable technology that sat around for stupid reasons for a decade, so who knows.