I’m a rising sophomore at the University of Chicago studying philosophy and economics/math. I’m largely interested in formal epistemology, metaethics, formal ethics, decision theory, and I have minor interests in a few other areas—I think LessWrong ideas are heavily underrated in philosophy academia, though I have some contentions. I also have a blog where I post about philosophy (and other stuff sometimes) here: https://substack.com/@irrationalitycommunity?utm_source=user-menu.
Noah Birnbaum
Karma: 263
Thanks! Honestly, I think this kind of project needs to get much more appreciation and should be done more by those who are very confident in their positions and would like to steelman the other side. I also often hear people very confident about their beliefs and truly have no idea what the bets counterarguments are—maybe this uncommon, but I went to an in-person rationalist meetup like last week, and the people were really confident but haven’t heard of a bunch of these counterarguments, which I though is not at all in the LessWrong spirit. That interaction was one of my inspirations for the post.
I think I agree, but I’m having a bit of trouble understanding how you would evaluate arguments so much differently than I am now. I would say my method is pretty different than that of twitter debates (in many ways, I am very sympathetic and influenced by the LessWrong approach). I think I could have made a list of cruxes of each argument, but I didn’t want the post to be too long—much fewer would read it which is why I recommended that people first get a grasp on the general arguments for AI being an existential risk right at the beginning (adding a credence or range, i think, is pretty silly given that people should be able to assign their own, and I’m just some random undergrad on the internet).
Yep—I totally agree. I don’t personally take the argument super seriously (though I attempted to steelman what that argument as I think other people take it very seriously). I was initially going to respond to every argument, but I didn’t want to make a 40+ minute post. I also did qualify that claim a bunch (as I did with others like the intractability argument)
Fair point. I do think the LeCun argument misunderstands a bunch about different aspects of the debate, but he’s probably smarter than me.
I think I’m gonna have to just disagree here. While I defintelely think finding cruxes are extremely important (and this sometimes requires much back and forth), there is a certain type of way arguments can go back and forth that I tend to think has little (and should have little) influence on beliefs—I’m open to being wrong, though!
Different but related point:
I think, generally, I largely agree with you on many things you’ve said and just appreciate the outside view more. A modest epistemology of sorts. Even if I don’t find an argument super compelling, if a bunch of people that I think are pretty smart do (Yann LeCun has done some groundbreaking work in AI stuff, so that seems like a reason to take him seriously), I’m still gonna write about it. This is another reason why I didn’t put credences on these arguments—let the people decide!
I think Eliezer briefly responds to this in his podcast with Dwarkesh Patel — satisfactorily is pretty subjective. https://youtu.be/41SUp-TRVlg?si=hE3gcWxjDtl1-j14
At about 24:40.
Thanks for the comment; I appreciate the response! One thing: I would say generally that people should avoid assuming other motives/ bad epistemics (i.e. motivated reasoming) unless pretty obvious (which I don’t think is the case here) and can be resolved by pointing it out. This usually doesn’t help anyone any parties get any closer to the truth, and if anything, it creates bad faith among people leading them to actually have other motives (which is bad, I think).
I also would be interested in what you think of my response to the argument that the commenter made.
I laughed at your first line, so thank you for that lol. I would love to hear more about why you prefer to collect models over arguments because i don’t think I intuitively get the reasons for why this would be better—to be fair, I haven’t spent enough time thinking about it probably. Any references you like on arguments for this would be super helpful!
I agree that many (even simple) arguments can be split up into many pieces—this is a good point. I would however say that there are still more and less complicated (ie more premises with lower probabilities) arguments that should receive lower credences. I assume that you will agree with this, and maybe you can let me know precisely the way to get the AGI doom argument off the ground with the least amount of contingent propositions—that would be really helpful.
Totally agree—it’s pretty hard to determine what will be necessary and this could lead to argument sloppiness. Though, I don’t think we should throw our hands in the air, say the argument is sloppy, and ignore it (I am not saying that you are or plan to do this for the record) -- I only mean to say that it should count for something, and I leave it up to the reader to figure out what.
One separate thing i would say, though, is that the asterisk by that indicated (this was said at the beginning of the section) that it was not necessary for the proposition AI being an existential threat—it only helps the argument. This is true for many things on that list.
Yea—you’re totally right. They’re not independent propositions making it pretty complicated (I did briefly not the fact that they had to be independent and thought it was clear enough that they weren’t, but maybe not). I agree this is really difficult to estimate probabilities on the basis of this, and I recommend big error bars and less certainty!
Thanks for the helpful feedback, though!
The claim The current neural architecture paradigm can scale up to Artificial General Intelligence (AGI) (especially without great breakthroughs) was not claimed to be necessary for the proposition (as was indicated with the asterisk and statement towards the beginning of that section). This is gonna sound bad, but it is not meant to be attacking in any way: while I really appreciate the counterarguments, please read the section more carefully before countering it.
Tbh, I don’t think what I think is actually so important. The project was mainly to take arguments and compile them in a way that I thought was most convincing. I think these arguments have various degrees of validity in my mind, but i don’t know how much saying those actually matter.
Also, and this is definitely not your fault for not catching this, I write tell me why I’m wrong at the end of every blog post, so it was not a statement of endorsement. My previous blog post is entitled against utilitarianism, but I would largely consider myself to be a utilitarian (as I write there).
Also, I can think the best arguments for a given position are still pretty bad.
I much appreciate the constructive criticism, however.
Good point!
Good call. Thanks!
People: “Ah, yes. We should trust OpenAI with AGI.” OpenAI: https://www.nytimes.com/2024/07/04/technology/openai-hack.html “But the executives decided not to share the news publicly because no information about customers or partners had been stolen, the two people said. The executives did not consider the incident a threat to national security because they believed the hacker was a private individual with no known ties to a foreign government. The company did not inform the F.B.I. or anyone else in law enforcement.”
Nice post! Someone want to explain how to do problem 2? I’ve looked at it for about 3 mins and can’t figure it out and am now just curious lol
Hate to be that guy but was he cryopreserved?
I think this has the potential to be a great thing to make note of. I would consider adding weird rituals and communal synchronization into the mix (in form of song or dance). I don’t think the synchrony stuff has to be crazy (a mere songs in front a camp fire feels like a religious experience to many). Although it does sound very weird (and very well may be), I find it not merely coincidental that this a strong part of what makes religions’ communities survive and thrive. I do see potential downsides 1) in how weird it would be to the outside (making the rat culture even more insular) and 2) actually getting people to participate in those activities seriously without belief in something beyond mere instrumental utility.
Thank you for your help! I was already in the Discord and have been in contact (and am a part of) the EA group here.
Tyler Cowen often has really good takes (even some good stuff against AI as an x-risk!), but this was not one of them: https://marginalrevolution.com/marginalrevolution/2024/10/a-funny-feature-of-the-ai-doomster-argument.html
Title: A funny feature of the AI doomster argument
If you ask them whether they are short the market, many will say there is no way to short the apocalypse. But of course you can benefit from pending signs of deterioration in advance. At the very least, you can short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends.
Still, in a recent informal debate at the wonderful Roots of Progress conference in Berkeley, many of the doomsters insisted to me that “the end” will come as a complete surprise, given the (supposed) deceptive abilities of AGI.
But note what they are saying. If markets will not fall at least partially in advance, they are saying the passage of time, and the events along the way, will not persuade anyone. They are saying that further contemplation of their arguments will not persuade any marginal investors, whether directly or indirectly. They are predicting that their own ideas will not spread any further.
I take those as signs of a pretty weak argument. “It will never get more persuasive than it is right now!” “There’s only so much evidence for my argument, and never any more!” Of course, by now most intelligent North Americans with an interest in these issues have heard these arguments and they are most decidedly not persuaded.
There is also a funny epistemic angle here. If the next say twenty years of evidence and argumentation are not going to persuade anyone else at the margin, why should you be holding this view right now? What is it that you know, that is so resistant to spread and persuasion over the course of the next twenty years?
I would say that to ask such questions is to answer them.