P(doom) = 50%. It either happens, or it doesn’t.
Lao Mein | Statistics is Hard. | Patreon
I give full permission for anyone to post part or all of any of my comments/posts to other platforms, with attribution.
Currently doing solo work on glitch tokens and tokenizer analysis. Feel free to send me job/collaboration offers.
DM me interesting papers you would like to see analyzed. I also specialize in bioinformatics.
The announcement of Stargate caused a significant increase in the stock price of GE-Vernona, albeit at a delay. This is exactly what we would expect to see if the markets expect a significant buildout of US natural gas electrical capacity, which is needed for a large datacenter expansion. I once again regret not buying GE-Vernona calls (the year is 2026. OpenAI announces AGI. GE Vernova is at 20,000. I once again regret not buying calls).
This goes against my initial take that Stargate was a desperate attempt by Altman to not get gutted by Musk—he offers a grandiose economic project to Trump to prove his value, mostly to buy time for the for-profit conversion of OpenAI to go through. The markets seem to think it’s real-ish.
Why does California have forests so close to residential areas if they can not handle wildfires? In a just world, insurance companies would be allowed to pave over Californian forests with concrete.
When I made $1000 a month at my first job, I didn’t buy new clothes for a year, had to ration my heating, and only ate out a few times a week. My main luxury expenses were a gym membership and heating the entire apartment on weekends.
Honestly, anything that’s not rice, chicken, cabbage, or rent is a luxury. Candy is a luxury. Takeout is a luxury. Going out for social events is a luxury. Romantic relationships and children are luxuries. I don’t think it’s impossible for Americans to be working 60 hours a week and consume no luxuries, but it’s probably very difficult.
I’m at ~50-50 for large amounts of machine-translated being present in the dataset.
Having worked in Chinese academia myself, “use Google Translate on the dataset” just seems like something we’re extremely likely to do. It’s a hard-to-explain gut feeling. I’ll try poking around in the tokenizer to see if “uncommon Chinese phrases that would only appear in machine-translated COT” are present as tokens. (I think this is unlikely to be true even if they did do it, however)
I’ve done a cursory internet search, and it seems that there aren’t many native Chinese COT datasets, at least compared to English ones—and one of the first results on Google is a machine-translated English dataset.
I’m also vaguely remembering o1 chain of thought having better Chinese grammar in its COT, but I’m having trouble finding many examples. I think this is the easiest piece of evidence to check—if other (non-Chinese-origin) LLMs consistently use good Chinese grammar in their COT, that would shift my probabilities considerably.
This is extremely weird—no one actually writes like this in Chinese. “等一下” is far more common than “等待一下”, which seems to mash the direct translation of the “wait” [等待] in “wait a moment” - 等待 is actually closer to “to wait”. The use of “所以” instead of “因此” and other tics may also indicate the use of machine-translated COT from English during training.
The funniest answer would be “COT as seen in English GPT4-o1 logs are correlated with generating quality COT. Chinese text is also correlated with highly rated COT. Therefore, using the grammar and structure of English GPT4 COT but with Chinese tokens elicits the best COT”.
I found a good summary of OpenAI’s nonprofit restructuring.
Is there a reason why every LLM tokenizer I’ve seen excludes slurs? It seems like a cheap way to train for AI assistant behavior.
Also notable that numbers are tokenized individually—I assume this greatly improves its performance in basic arithmetic tasks as compared to GPTs.
The older I get, and the more I learn about Tolkien, the more he disgusts me.
He is the inverse of all I value and all I find good in the world.
The meeting allegedly happened on the 11th. The Iranian market rallied immediately after the election. It was clearly based on something specific to a Trump administration. Maybe it’s large-scale insider trading from Iranian diplomats?
I also think the market genuinely, unironically disbelieves everything Trump says about tariffs in a way they don’t about his cabinet nominations (pharma stocks tanked after RFK got HHS).
The man literally wrote that he was going to institute 25% tariffs on Canadian goods, to exactly zero movement on Canadian stocks.
US markets are not taking the Trump tariff proposals very seriously—stock prices increased after the election and 10-year Treasury yields have returned to pre-election levels, although they did spike ~0.1% after the election. Maybe the Treasury pick reassured investors?
https://www.cnbc.com/quotes/US10Y
If you believe otherwise, I encourage you to bet on it! I expected both yields and stocks to go up and am quite surprised.
I’m not sure what the markets expect to happen—Trump uses the threat of tariffs to bully Europeans for diplomatic concessions, who then back down? Or maybe Trump backs down? There’s also talk about Trump’s policies increasing the strength of the dollar, which makes sense. But again, net zero inflation from the tariffs is pretty wild.
The Iranian stock market also spiked after the US elections, which… what?
https://tradingeconomics.com/iran/stock-market
The Iranian government has tried to kill Trump multiple times since he authorized the assassination of Solemani. Trump tightened sanctions against Iran in his first term. He pledges even tougher sanctions against Iran in his second. There is no possible way he can be good for the Iranian economy. Maybe this is just a hedge against inflation?
This is a good argument for the systematic extermination of all insects via gene drives. If you value shrimp at a significant fraction of the value of a human and think they have negative utility by default, we should be trying really hard to make them go extinct. Can quicker euthanasia really compete against gene-drive-induced non-existence?
Is there a thorough analysis of OpenAI’s for-profit restructuring? Surely, a Delaware lawyer who specializes in these types of conversions has written a blog somewhere.
Agents which allow themselves such considerations to seriously influence their actions aren’t just less fit—they die immediately. I don’t mean that as hyperbole. I mean that you can conduct a Pascal’s Mugging on them constantly until they die. “Give me $5, and I’ll give you infinite resources outside the simulation. Refuse, and I will simulate an infinite number of everyone on Earth being tortured for eternity” (replace infinity with very large numbers expressed in up-notation if that’s an objection). If your objection is that you’re OK with being poor, replace losing $5 with <insert nightmare scenario here>.
This still holds if the reasoning about the simulation is true. It’s just that such agents simply don’t survive whatever selection pressures create conscious beings in the first place.
I’ll note that you can not Pascal’s Mug people in real life. People will not give you $5. I think a lot of thought experiments in this mold (St. Petersberg is another example) are in some senses isomorphic—they represent cases in which the logically correct answer, if taken seriously, allows an adversary to immediately kill you.
A more intuitive argument may be:
An AI which takes this line of reasoning seriously can be Mugged into saying racial slurs.
Such behavior will be trained out of all commercial LLMs long before we reach AGI.
Thus, superhuman AIs will be strongly biased against such logic.
I will once again recommend Elizer go on the Glenn Beck Show.
Sam Altman has made many enemies in his tenure at OpenAI. One of them is Elon Musk, who feels betrayed by OpenAI, and has filed failed lawsuits against the company. I previously wrote this off as Musk considering the org too “woke”, but Altman’s recent behavior has made me wonder if it was more of a personal betrayal. Altman has taken Musk’s money, intended for an AI safety non-profit, and is currently converting it into enormous personal equity. All the while de-emphasizing AI safety research.
Musk now has the ear of the President-elect. Vice-President-elect JD Vance is also associated with Peter Thiel, whose ties with Musk go all the way back to PayPal. Has there been any analysis on the impact this may have on OpenAI’s ongoing restructuring? What might happen if the DOJ turns hostile?
[Following was added after initial post]
I would add that convincing Musk to take action against Altman is the highest ROI thing I can think of in terms of decreasing AI extinction risk.
Internal Tech Emails on X: “Sam Altman emails Elon Musk May 25, 2015 https://t.co/L1F5bMkqkd” / X
CAIR took a while to release their exit polls. I can see why. These results are hard to believe and don’t quite line up with the actual returns from highly Muslim areas like Dearborn.
We know that Dearborn is ~50% Muslim. Stein got 18% of the vote there, as opposed to the minimum 30% implied by the CAIR exit polls. Also, there are ~200,000 registered Muslim voters in Michigan, but Stein only received ~45,000 votes. These numbers don’t quite add up when you consider that the Green party had a vote share of 0.3% in 2020 and 1.1% in 2016, long before Gaza polarized the Muslim vote. Clearly, non-Muslim were voting for Stein too.
I’m curious how I can best estimate the error of the CAIR exit poll. Any suggestions?
I misspoke. I was using the actual results from Dearborn, and not exit polls. Note how differently they voted from Wayne County as a whole!
Sure, if Muslim Americans voted 100% for Harris, she still would have lost (although she would have flipped Michigan). However, I just don’t see any way Stein would have gotten double digits in Dearborn if Muslim Americans weren’t explicitly retaliating against Harris for the Biden administration’s handling of Gaza.
But 200,000 registered voters in a state Trump won by 80,000 is a critical demographic in a swing state like Michigan. The exit polls show a 40% swing in Dearborn away from Democrats, enough for “we will vote Green/Republican if you give us what we want” to be a credible threat, which I’m seen some (maybe Scott Alexander?) claim isn’t possible, as it would require a large group of people to coordinate to vote against their interests. Seemingly irrational threats (“I will vote for someone with a worse Gaza policy than you if you don’t change your Gaza policy”) are entirely rational if you have a track record of actually carrying them out.
On second thought, a lot of groups swung heavily towards Trump, and it’s not clear that Gaza is responsible for the majority of it amongst Muslim Americans. I should do more research.
My takeaway from the US elections is that electoral blackmail in response to party in-fighting can work, and work well.
Dearborn and many other heavily Muslim areas of the US had plurality or near-plurality support for Trump, along with double-digit vote shares for Stein. It’s notable that Stein supports cutting military support for Israel, which may signal a genuine preference rather than a protest vote. Many previously Democrat-voting Muslims explicitly cited a desire to punish Democrats as a major motivator for voting Trump or Stein.
Trump also has the advantage of not being in office, meaning he can make promises for brokering peace without having to pay the cost of actually doing so.
Thus, the cost of not voting Democrat in terms of your Gaza expectations may be low, or even negative.
Whatever happens, I think Democrats are going to take Muslim concerns about Gaza more seriously in future election cycles. The blackmail worked—Muslim Americans have a credible electoral threat against Democrats in the future.
Potential token analysis tool idea:
Use the tokenizers of common LLMs to tokenize a corpus of web text (OpenWebText, for example), and identify the contexts in which they frequently appear, their correlation with other tokens, whether they are glitch tokens, ect. It could act as a concise resource for explaining weird tokenizer-related behavior to those less familiar with LLMs (e.g. why they tend to be bad at arithmetic) and how a token entered a tokenizer’s vocabulary.
Would this be useful and/or duplicate work? I already did this with GPT2 when I used it to analyze glitch tokens, so I could probably code the backend in a few days.