P(doom) = 50%. It either happens, or it doesn’t.
Lao Mein | Statistics is Hard. | Patreon
I give full permission for anyone to post part or all of any of my comments/posts to other platforms, with attribution.
Currently doing solo work on glitch tokens and tokenizer analysis. Feel free to send me job/collaboration offers.
DM me interesting papers you would like to see analyzed. I also specialize in bioinformatics.
Agents which allow themselves such considerations to seriously influence their actions aren’t just less fit—they die immediately. I don’t mean that as hyperbole. I mean that you can conduct a Pascal’s Mugging on them constantly until they die. “Give me $5, and I’ll give you infinite resources outside the simulation. Refuse, and I will simulate an infinite number of everyone on Earth being tortured for eternity” (replace infinity with very large numbers expressed in up-notation if that’s an objection). If your objection is that you’re OK with being poor, replace losing $5 with <insert nightmare scenario here>.
This still holds if the reasoning about the simulation is true. It’s just that such agents simply don’t survive whatever selection pressures create conscious beings in the first place.
I’ll note that you can not Pascal’s Mug people in real life. People will not give you $5. I think a lot of thought experiments in this mold (St. Petersberg is another example) are in some senses isomorphic—they represent cases in which the logically correct answer, if taken seriously, allows an adversary to immediately kill you.
A more intuitive argument may be:
An AI which takes this line of reasoning seriously can be Mugged into saying racial slurs.
Such behavior will be trained out of all commercial LLMs long before we reach AGI.
Thus, superhuman AIs will be strongly biased against such logic.
I will once again recommend Elizer go on the Glenn Beck Show.
Sam Altman has made many enemies in his tenure at OpenAI. One of them is Elon Musk, who feels betrayed by OpenAI, and has filed failed lawsuits against the company. I previously wrote this off as Musk considering the org too “woke”, but Altman’s recent behavior has made me wonder if it was more of a personal betrayal. Altman has taken Musk’s money, intended for an AI safety non-profit, and is currently converting it into enormous personal equity. All the while de-emphasizing AI safety research.
Musk now has the ear of the President-elect. Vice-President-elect JD Vance is also associated with Peter Thiel, whose ties with Musk go all the way back to PayPal. Has there been any analysis on the impact this may have on OpenAI’s ongoing restructuring? What might happen if the DOJ turns hostile?
[Following was added after initial post]
I would add that convincing Musk to take action against Altman is the highest ROI thing I can think of in terms of decreasing AI extinction risk.
Internal Tech Emails on X: “Sam Altman emails Elon Musk May 25, 2015 https://t.co/L1F5bMkqkd” / X
CAIR took a while to release their exit polls. I can see why. These results are hard to believe and don’t quite line up with the actual returns from highly Muslim areas like Dearborn.
We know that Dearborn is ~50% Muslim. Stein got 18% of the vote there, as opposed to the minimum 30% implied by the CAIR exit polls. Also, there are ~200,000 registered Muslim voters in Michigan, but Stein only received ~45,000 votes. These numbers don’t quite add up when you consider that the Green party had a vote share of 0.3% in 2020 and 1.1% in 2016, long before Gaza polarized the Muslim vote. Clearly, non-Muslim were voting for Stein too.
I’m curious how I can best estimate the error of the CAIR exit poll. Any suggestions?
I misspoke. I was using the actual results from Dearborn, and not exit polls. Note how differently they voted from Wayne County as a whole!
Sure, if Muslim Americans voted 100% for Harris, she still would have lost (although she would have flipped Michigan). However, I just don’t see any way Stein would have gotten double digits in Dearborn if Muslim Americans weren’t explicitly retaliating against Harris for the Biden administration’s handling of Gaza.
But 200,000 registered voters in a state Trump won by 80,000 is a critical demographic in a swing state like Michigan. The exit polls show a 40% swing in Dearborn away from Democrats, enough for “we will vote Green/Republican if you give us what we want” to be a credible threat, which I’m seen some (maybe Scott Alexander?) claim isn’t possible, as it would require a large group of people to coordinate to vote against their interests. Seemingly irrational threats (“I will vote for someone with a worse Gaza policy than you if you don’t change your Gaza policy”) are entirely rational if you have a track record of actually carrying them out.
On second thought, a lot of groups swung heavily towards Trump, and it’s not clear that Gaza is responsible for the majority of it amongst Muslim Americans. I should do more research.
My takeaway from the US elections is that electoral blackmail in response to party in-fighting can work, and work well.
Dearborn and many other heavily Muslim areas of the US had plurality or near-plurality support for Trump, along with double-digit vote shares for Stein. It’s notable that Stein supports cutting military support for Israel, which may signal a genuine preference rather than a protest vote. Many previously Democrat-voting Muslims explicitly cited a desire to punish Democrats as a major motivator for voting Trump or Stein.
Trump also has the advantage of not being in office, meaning he can make promises for brokering peace without having to pay the cost of actually doing so.
Thus, the cost of not voting Democrat in terms of your Gaza expectations may be low, or even negative.
Whatever happens, I think Democrats are going to take Muslim concerns about Gaza more seriously in future election cycles. The blackmail worked—Muslim Americans have a credible electoral threat against Democrats in the future.
My problem with this argument is that the AIs which will accept your argument can be Pascal’s Mugged in general, which means they will never take over the world. It’s less “Sane rational agents will ignore this type of threat/trade” and more “Agents which consistently accept this type of argument will die instantly when others learn to exploit it”.
I have a few questions.
Can you save the world in time without a slowdown in AI development if you had a billion dollars?
Can you do it with a trillion dollars?
If so, why aren’t you trying to ask the US Congress for a trillion dollars?
If it’s about a lack of talent, do you think Terrance Tao can make significant progress on AI alignment if he actually tried?
Do you think he would be willing to work on AI alignment if you offered him a trillion dollars?
The text referred to this as a “social deception game”. Where is the deception?
My guesses:
The actual messages sent to the Petrovs and Generals will significantly differ from the ones shown here.
It’s Amongus, and there are players who get very high payoffs if they trick the sides into nuclear war
It’s just a reference to expected weird game theory stuff. But why not call it a “game theory exercise”?
The actual rules of the game will differ drastically from the ones described here. Maybe no positive payoffs for one-sided nuking?
The sensor readings are actually tied to comments on this post. Maybe an AI is somehow involved?
The wording is a bit weird:
90 minutes after you send the nukes, your opposing side will die and the game will end. (They may nuke you in this window.) If you fire nukes without getting nuked, you and all of your fellow Generals will gain 1,000 karma.
If you get nuked, then you and your Generals lose 300 karma (and don’t gain any karma).
“If you fire nukes without getting nukes” and “if you get nuked” imply that both sides firing after 4:30 pm results in +karma for everyone, since the nukes are still in the air at 6:00 pm, when the game ends. The +karma is triggered by firing, while the -karma is triggered by the nukes actually landing.
Is this intended?
Building 5 GWs of data centers over the course of a few years isn’t impossible. The easiest way to power them is to delay the off lining of coal power plants and to reactivate mothballed ones. This is pretty easy to scale since so many of them are being replaced by natural gas power plants for cost reasons.
OK, I’m starting to see your point. Why do you think OpenAI is so successful despite this? Is their talent and engineering direction just that good? Is everyone else even worse at data management?
My prediction is that this becomes a commitment race. A general is going to post something like “I am going to log off until the game begins, at which point I will immediately nuke without reading any messages.”, at which point the generals on the other side get to decide if 2 days of access to LessWrong for everyone is worth the reputational cost of being known as someone who doesn’t retaliate. Given what I know about LessWrong and how this entire game is framed, you’ll likely get more social capital from not retaliating, meaning the commitment works and everyone keeps access.
The real question is how committing to nuking affects your reputation. My guess is that it’s bad? I clearly made a mistake opting-in to this game, so maybe the civilians will be grateful that you didn’t destroy their access to LessWrong, but committing looks pretty bad to all the bystanders, many of whom make decisions at places like Manifund and may incorporate this information into if your next project gets funded.
What happens if nukes are launched at 5:59 pm? Is the game extended for another 90 minutes?
Extremely impressive! I’ve been wanting something like this for a while.
I find that questionable. Crime rates for adoptive children tend to be closer to that of their biological parents than that of their adoptive parent.
One of the highest quality-of-life increases I’ve ever experienced is when I stopped listening to music with sad lyrics. Crazy how long it took me to realize it was lowering my mood in a persistent, motivation-sapping way.
This area could really use better economic analysis. It seems obvious to me that some subset of workers can be pushed below subsistence, at least locally (imagine farmers being unable to afford rent because mechanized cotton plantations can out-bid them for farmland). Surely there are conditions where this would be true for most humans.
There should be a simple one-sentence counter-argument to “Trade opportunities always increases population welfare”, but I’m not sure what it is.
Is there a thorough analysis of OpenAI’s for-profit restructuring? Surely, a Delaware lawyer who specializes in these types of conversions has written a blog somewhere.