I agree that AI capabilities are spiky and developed in an unusual order. And I agree that because of this, the single-variable representation of intelligence is not very useful for understanding the range of abilities of current frontier models.

At the same time, I expect the jump from “Worse than humans at almost everything” to “Better than humans at almost everything” will be <5 years, which would make the single-variable representation work reasonably well for the purposes of the graph.

I think these “examples of silly mistakes” have not held up well at all. This was often blamed on “training around the limitations”; however, in the case of the linked post, we got a model the next day that performed much better.

And almost every benchmark and measurable set of capabilities has rapidly improved (in some cases beyond human experts).

”We too often give wrong answers to questions ourselves to be justified in being very pleased at such evidence of fallibility on the part of the machines. Further, our superiority can only be felt on such an occasion in relation to the one machine over which we have scored our petty triumph.”
Alan Turing, Computing Machinery and Intelligence
1950

I think Sam Bowman’s The Checklist: What Succeeding at AI Safety Will Involve is a pretty good list and I’m glad it exists. Unfortunately, I think its very unlikely that we will manage to complete this list, given my guess at timelines. It seems very likely that the large majority of important interventions on this list will go basically unsolved.
I might go through The Checklist at some point and give my guess at success for each of the items.

Consider that there are people with high P(doom) who don’t have any depression or anxiety. Emotions are not as much caused by our beliefs as we tend to assume. A therapist might be able to teach more productive thought patterns and behaviors, but they are unlikely to speak with competence on the object level issue of AI doom.

Independently I recommend trying to get a prescription for SSRIs. Most probably won’t help, but some might, and they tend to not have strong side effects in my experience, so trying them doesn’t hurt.

Only problem is that trying different SSRIs can take a very long time: usually you take one for several weeks, nothing happens, the doctor says “up the dosage”, weeks pass, still no effect, and the doctor might increase the dosage again. Only then may they switch you to a different SSRI, and the whole process begins anew. So persistence is required.

Well, if we extrapolate from the current progress, soon AI will be superhumanly good at complex analysis and group theory while only being moderately good at ordering pizza.

That’s why I think that comparing AI to humans on a one-dimensional scale doesn’t work well.

Alternatively or in addition to this, you can embrace AI and design new tasks and assignments that cause students to learn together with the AI.

This magical suggestion needs explication.

From what I’ve seen via Ethan Mollick, instead of copy-pasting, the new assignments that would be effective are the same as the usual—just “do the work,” but at the AI. Enter a simulation, but please don’t dual-screen the task. Teach the AI (I guess the benefit here is immediate feedback, as if you couldn’t use yourself or friend as a sounding board), but please don’t dual-screen the task. Have a conversation (again, not in class or on a discussion board or among friends), but please don’t dual-screen the task. Then show us you “did it.” You could of course do these things without AI, though maybe AI makes a better (and certainly faster) partner. But the crux is that you have to do the task yourself. Also note that this admits the pre-existing value of these kinds of tasks.

Students who will good-faith do the work and leverage AI for search, critique, and tutoring are...doing the work and getting the value, like (probably more efficiently than, possibly with higher returns than) those who do the work without AI. Students who won’t...are not doing the work and not getting the value, aside from the signaling value from passing the class. So there you have it—educators can be content that not doing the assignment delivers worse results for the student, but the student doesn’t mind as long as they get their grade, which is problematic. Thus, educators are not going quietly and are in fact very concerned about AI-proofing the work, including shifts to in-person testing and tasks.

However, that only preserves the benefit of the courses and in turn degree (I’m not saying pure signaling value doesn’t exist, I’m just saying human capital development value is non-zero and under threat). It does not insulate the college graduate from competition in knowledge work from AI (here’s the analogy: it would obviously be bad for the Ford brand to send lemons into the vehicle market, but even if they are sending decent cars out, they should still be worried about new entrants).

Yep, but my understanding is that the time associated with marginal scraping, sensor data, and physical components don’t matter much when talking about AI progress which is on the order of a year. Or honestly, maybe marginal improvements in these sorts of components don’t matter that much at all over this time scale (like freezing all these things for a year wouldn’t be much tax if you prepped in advance). Not super sure about situation with scrapping though.

>instead of “AI is 10 times worse than humans at everything”, it’s “AI is roughly as good as expert humans at X and useless at Y”.

How long before it’s “AI is out of sight of expert humans at X and merely far above them at Y”?

Well but also kind of yes? Like agreed with what you said, but also the hypothesis is that there’s a certain kind of depression-manifestation which is somewhat atypical and that we’ve seen bupropion work magic on.

*And that this sounds a lot like that manifestation. So it might be particularly good at giving John in particular (and me, and others) the Wizard spirit back.

Disclaimer: I am not a doctor and this is not medical advice. Do your own research.

In short: I experienced something similar. Garrett and I call it “Rat(ionalist) Depression.” It manifested as similar to a loss/​lessening of Will To Wizard Power as John uses the term here. Importantly: I wasn’t “sad”, or pessimistic about the future (AI risk aside,) or most other classical signs of depression; I was considered pretty well emotionally put-together by myself and my friends (throughout, and this has never stopped being true.) But at some point for reasons unclear to me, I became listless. The many projects of a similar flavor to things John points at above, which I used do to in spades, lost their visceral appeal (though they kept their cognitive/​aesthetic/​non-visceral appeal and so compelled me to force myself now and then to some success but also some discomfort and cognitive dissonance)-- and it happened gradually so that it seemed like a natural development over a year or two.

My girlfriend, who is on Bupropion for regular physician-recognized depression, encouraged me to try it just to see. So I did. And it worked.

And it kicks in very quickly. There was a honeymoon phase during the first ~8 days it takes for all of the long half-lived active metabolites to reach equilibrium concentrations, during which I and others I know have reported feeling mild euphoria along with the other benefits. After that subsides, it’s a background thing where mostly you look back on your day/​week and realize you just got things done and did more things. And it’s been consistently helpful ever since. (4-6 months for me, ~7 years for my girlfriend, years for some family members and somewhat less time so far for others I know personally.)

Oh and my social battery is way larger. I used to get introvert-exhaustion in a way that ~basically doesn’t happen anymore. Parties are more often fun than not, now.

Further nice-to-haves:

• It’s not an SSRI, it’s an NDRI, so it doesn’t do the terrible SSRI things. Side effects may include decreased mental fog, increased libido, decreased appetite, and a renewed will to Wizard Power.

• You’ll “feel it” right away (~same day) even though it takes a ~week to settle in to equilibrium concentrations (and, anecdotally from others, possibly up to month to feel it’s final form?)

• It’s fairly easy to get. Go to your psychiatrist and ask for it (XR, extended release to be taken in the morning) or trade time/​convenience for money and go online to a site like Nurx.com and if, upon completing their intake survey, they consider you to have mild depression (not severe or you’ll scare them off) they’ll start mailing you bupropion once a month!

It doesn’t work for literally everyone. If you have bad anxiety, or if you have mania, be warned. But for the large handful of people around me who are now on it, they’ve reported fast and significant positive effects, including at least one other “Rat Depression” case.

That’s most of the pitch.

https://​​samuelshadrach.com/​​raw/​​text_english_html/​​unimportant/​​therapy_effectiveness_personal.html

Try @Kaj_Sotala or another therapist /​ coach from the community. It’s easier when you don’t have to doompill your psychological support person.

Also SlowCorp has magically 50x better networking equipment than NormalCorp, and 50x higher rate limits on every site they’re trying to scrape, and 50x as much sensor data from any process in the world, and 50x faster shipping on any physical components they need, etc etc (and AutomatedCorp has magically 50x worse of all of those things).

But yeah, agreed that you should ignore all of those intuitions when considering the “1 week” scenario—I just found that I couldn’t actually turn all of those intuitions off when considering the scenario.

I reckon I might be pretty good(ish) at coming up with good concepts, but given my downvotes, it seems I’m not good at presenting said concepts. Does anyone have any advice for that?

Yeah, I discuss this here:

The way I set up the analogy makes it seem like AutomatedCorp has a serial compute advantage: because they have 50 years they can run things that take many serial years while NormalCorp can’t. As in, the exact analogy implies that they could use a tenth of their serial time to run a 5 year long training run on 50k H100s, while they could actually only do this if the run was sufficiently parallelizable such that it could be done on 2.5 million H100s in a tenth of a year. So, you should ignore any serial compute advantage. Similarly, you should ignore difficulties that SlowCorp might have in parallelizing things sufficiently etc.

You can also imagine that SlowCorp has 10 million magically good GPUs (and CPUs etc) which are like H100s but 50x serially faster (but still only has 1 week) while AutomatedCorp has 10 million much worse versions of H100s (and CPUs etc) which are 50x serially slower but otherwise the same (and has 50 years still).

Is there a list of rationality one-liners that can possibly actually make you more rational? Just barely enough words to be saying something rather than nothing, and to have it maybe land. Eg

• Wide mugs are always bigger than tall mugs.

• newyep:newnope = oldyep * chanceifyep : oldnope * chanceifnope (assuming independent events!)

• Tasks take sample(lognormal(median)) hours to complete.

• If it took you a long time to think of something, then figure out how you coulda figured it out faster. Then next time figure fast.

• People care very much what words you choose to convey the same meaning. They really really care.

Of course, if the questions on which we need to use AI advice force those distributions to skew too much, and there’s no way for debaters to adapt and bootstrap from on-distribution human data, that will mean our protocol isn’t competitive.

This is my concern, and I’m glad it’s at least on your radar. How do you /​ your team think about competitiveness in general? (I did a simple search and the word doesn’t appear in this post or the previous one.) How much competitiveness are you aiming for? Will there be a “competitiveness case” later in this sequence, or later in the project? Etc.?

But generally this requires you to have some formal purchase on the philosophical aspects where humans are off distribution, which may be rough.

Because of the “slowness of philosophy” issue I talked about in my post, we have no way of quickly reaching high confidence that any such formalization is correct, and we have a number of negative examples where a proposed formal solution to some philosophical problem that initially looked good turned out to be flawed upon deeper examination. (See decision theory and Solomonoff induction.) AFAIK we don’t really have any positive examples of such formalizations that have stood the test of time. So I feel like this is basically not a viable approach.

I do write sometimes! Actually, two of my stories won a prize at the EA Forum’s AI fables contest. See here https://​​forum.effectivealtruism.org/​​posts/​​EAmfYSBaJsMzHY2cW/​​ai-fables-writing-contest-winners

I also write rationalist fanfiction on AO3.

To be clear, I’m not claiming that the “write a text with precisely X words” task is super-duper-mega-hard, and I wouldn’t be surprised if a new frontier model was much better at it than Gemini. I have a very similar opinion to the author of this post: I’m saying that given what the models currently can do, it’s surprising that they also currently can’t (reliably) do a lot of things. I’m saying that there are very sharp edges in models’ capabilities, much sharper than I expected. And the existence of very sharp edges makes it very difficult to compare AI to humans on a one-dimensional intelligence scale, because instead of “AI is 10 times worse than humans at everything”, it’s “AI is roughly as good as expert humans at X and useless at Y”.

FYI (cc @Gram_Stone) the 2023 course website has (poor quality edit:nevermind I was accessing them wrong) video lectures.

Edit 2: For future (or present) folks, I’ve also downloaded local mp4s of the slideshow versions of the videos here, and can share privately with those who dm, in case you want them too or the site goes down.

I have very different intuitions about 50M GPUs for 1 week vs 200k GPUs for with 200 hours of work spread evenly across 50 years.

I think SlowCorp-v2 would get a lot more done than SlowCorp-v1 (though obviously still a lot less than AutomatedCorp). And also SlowCorp-v2 seems to be a closer analogy than SlowCorp-v1 - both corporations have the same amount of serial time, and my intuition is that you generally can’t make a training run go 10x faster just by throwing 10x as many GPUs at it, because you’ll be bottlenecked by IO.

And I know “SlowCorp is bottlenecked by IO” is not what the point of this intuition pump was supposed to be, but at least for me, it ended up being the main consideration pumping my intuition.