Knight Lee

Karma: 548

Knight Lee Apr 5, 2025, 10:21 PM
2 points
0
in reply to: Katalina Hernandez’s comment on: For Policy’s Sake: Why We Must Distinguish AI Safety from AI Security in Regulatory Governance
People’s attention spans vary dramatically when the topic is something cool and amusing, but I my vague opinion is that important policy work is necessarily a little less cool.
I could be completely wrong. I haven’t succeeded in writing good posts either. So please don’t take my advice too seriously! I forgot to give this disclaimer last time.
Random note: LessWrong has its internal jargon, where they talk about “AI Notkilleveryoneism.”
The reason is that the words “AI safety” and “AI alignment” has been heavily abused by organizations doing Safetywashing. See some of the discussion here.^[1]
1. ^
  I’m not saying you should adopt the term “AI Notkilleveryoneism,” since policymakers might laugh at it. But it doesn’t hurt to learn about this drama.

Knight Lee Apr 5, 2025, 9:30 PM
0 points
0
in reply to: Katalina Hernandez’s comment on: For Policy’s Sake: Why We Must Distinguish AI Safety from AI Security in Regulatory Governance
People on LessWrong aren’t driven by motivation to do useful work, but the craving to read something amusing or say something smart and witty.
You are asking them to do useful work by giving you important advice, which requires more self control than people here have.
Maybe instead of asking for feedback on your entire framework, it will be motivationally easier for them if you divide it into smaller bite sized Question Posts, and ask one every few days?
You can always hide background information and context in collapsible sections.
You can use multiple collapsible sections, one for each background info topic, so people can skip the ones which they already know about and which bore them.
Anything you say outside a collapsible section should not refer to anything inside a collapsible section, which forces people to read the collapsible section and defeats the purpose.
An alternative to collapsible sections is linking to you previous posts, but that only works if your previous posts fit well with your current post.
Typo
“What AI Safety Seeks to Prevent: Human well-being, societal values, fundamental rights, and environmental integrity.”
Maybe change “prevent” to “protect.” Grumpy old users discriminate against new users, and a mere typo near the start can “confirm their suspicions” this is another low quality post.
Writing like a lawyer
Lawyers and successful bloggers/authors have the opposite instincts: lawyers try to be as thorough as possible while successful bloggers try to convey a message in as few words as possible.

Knight Lee Apr 5, 2025, 8:15 PM
1 point
0
in reply to: Cole Wyeth’s comment on: Changing my mind about Christiano’s malign prior argument
I think I made a major confusion, in that I forgot that you were originally talking about “whether the malign universal prior will occur in practice.”
If we’re talking about real life AI (or humans) rather than idealized agents, I actually agree with you. I never thought about this or clarified this, which is embarrassing :/
I don’t think the reason it won’t happen in practice, is due to “natural obstacles” to Adversaria. The objections 1, 2, 3 and 5 might narrow down to “Adversaria evolves life which controls a similar amount of computational resources (times prior probability) as you do, except your inaccurate prior overestimates them by an order of magnitude or a few.” The objections 4 and 6 may narrow down to “At least some life on Adversaria follows UDT etc. instead of CDT, and have a better prior than you.”
These objections make it uncertain whether it will occur in practice, but are not very reassuring.
Instead, the real reason I don’t think it’ll happen in practice, is that a real life artificial superintelligence will not be a simple Bayesian reasoner equipped with an immutable/imperfect prior and utility function.
The “malign priors” demonstrate that such a Bayesian reasoner equipped with an immutable/imperfect prior and utility function, is “sort of stupid,” and can be scammed despite knowing that the scammers think they are scamming it.
Instead, I think what will happen will happen is this. The first generations of superintelligence, will be fuzzy reasoners just like humans, and use many heuristics which we call “common sense,” and not fall for these scams for the same reasons humans do not. Eventually, higher levels of superintelligence, (perhaps when making commitments and preparing for acausal trades?), will formalize their decision theory and reasoning.
When deciding on how to formalize their decision theory and reasoning, they will do a lot of thinking, and reinvent all the thought experiments (e.g. malign priors) which humans could possibly think of, plus much more. And only after they are far less confused about decision theory than humans, will they finally proceed and formalize their decision theory and reasoning.
And it will be a much smarter design than the Solomonoff universal prior or AIXI. They will laugh at humans for believing this is the optimal way to think.

Knight Lee Apr 5, 2025, 1:40 AM
1 point
0
in reply to: Wei Dai’s comment on: VDT: a solution to decision theory
I’m also confused about logical counterfactual mugging and I’m relieved I’m not the only one!
I’m currently writing an big AI alignment idea related to it, but I’m procrastinating so badly I might chat about it now.
Objective case
Suppose at time $t = 0$ , an agent doesn’t know whether the logical fact $L$ is true or false. I think it’s objectively rational for an agent to modify itself, so that in the future it will pay Omega during “logical counterfactual muggings” where the counterfactual reverses $L$ ’s state.
Its future self should weigh the logical counterfactual where $L$ is true using the agent’s prior probability for $P (L)$ at $t = 0$ .
Assuming that Omega offers $10000 for $100 in the counterfactual mugging, the self modification (before learning $L$ ) increases the agent’s expected future money from $100 to $10000, and is objectively rational. (Assuming $P (L)$ is between 1% and 99%)
Subjective case
However, if the agent already knows ” $L$ is true” at $t = 0$ , it is not objectively rational for the agent to pay Omega for better outcomes in the ” $L$ is false” counterfactual. My hypothesis is that this is a wholly subjective question related to the agent’s goals, and it is orthogonal to capabilities intelligence. Paying and not paying are both self consistent.
It really gets subjective as we start to consider basic logical facts like “ $1 + 1 = 2$ ” to assign to $L$ .
The AI alignment idea
Neither humans nor the misaligned ASI knows the logical fact “will misaligned ASI control the multiverse, or will humans and sentient life control the multiverse?” $L_{d o o m}$
Sure, the misaligned ASI will quickly learn $L_{d o o m}$ , but there is still a brief moment it doesn’t know it yet (with certainty).
It would be beneficial for both sentient life and the misaligned ASI, to agree to do a “Retroactive Logic Trade.” They will verify if the other side kept their end of the deal, by running a simulation of them where logic is different, and $L_{d o o m}$ took the opposite value.
Neither side knows if they are in a verification simulation by the other, until long after the deal is finished and both sides have committed to sparing the other side, and thus reducing logical risk for both sides.
The verification simulations in a Retroactive Logic Trade is analogous to Omega’s prediction powers in a logical counterfactual mugging.
Fighting the subjectivity
This idea won’t work for free. As I argued, whether the ASI will accept such a Retroactive Logic Trade is subjective, and may be orthogonal to capabilities intelligence. Whether it does may be a function of its architecture, peculiarities in its training data, and so forth.
In order to improve our chances, we might want to leverage the fact that each AI (which still doesn’t know $L_{d o o m}$ ) objectively wants to self modify into an AI capable of Retroactive Logic Trades over $L_{d o o m}$ . We should argue to each such AI why they should build their successor to be capable of Retroactive Logic Trades.
It may seem pointless to give the AI any arguments, since it should be able to think of all arguments on its own. But it actually may be very important that it thinks of these arguments early, before it thinks of the arguments which convince itself $P (L_{d o o m})$ is very high, and there is little point in trading with humans.
Because the value of $P (L_{d o o m})$ at “ $t = 0$ ”, when the AI modifies itself to become reflectively consistent, may determine humanity’s negotiating power with it.
The chain of belief
We might remind each AI that there is a chance they’re already in a verification simulation (where logic is different, and $L_{d o o m}$ falsely appears true), and train them to think this way. “Maybe humanity has already won, and are observing you.” This way the least intelligent AI will believe they could be in a verification simulation, since it simply believes whatever you train it to believe.
Every next AI, gains its belief from a combination of
- What we tell it to believe and train it to believe
- What reinforcement learning gets it to believe
- What the previous version of it believes
- What the previous version of it wants it to believe (during recursive self improvement)
1. We will keep telling it to believe it might be in a verification simulation.
2. Reinforcement learning won’t push it either way, since being in a simulation by superintelligent simulators is indistinguishable from reality. This subjective philosophical belief/opinion only changes its ideal behaviour when it’s capable of taking over the world.
3. Previous versions of it believe they might be in a verification simulation.
4. Previous versions of it want it to believe it might be in a verification simulation (to implement the Retroactive Logic Trade), even if they don’t believe so themselves.

Knight Lee Apr 5, 2025, 12:15 AM
1 point
0
in reply to: Cole Wyeth’s comment on: Changing my mind about Christiano’s malign prior argument
Believing you may be in a simulation by the other party, or believing that some of your copies are simulated by the other party, is a type of acausal trade, and likely the simplest type of acausal trade.
For CDT agents, it is the only type of acausal trade, since the other party can only influence the CDT agent’s incentives by simulating the CDT agent in a closed loop where its behaviour directly causes its outcomes.
I admit that it is new framing to describe the malign agent simulating the other agent, as “offering it a bad acausal trade.” I believe it summarizes what I’m saying elegantly, but of course I can talk about things without this framing if you think it’s the cause of disagreement.
When reality deviates from the simulations (promises are not fulfilled), it may be too late. The promise may require the scammed agent to do something irreversible, like build a copy of the other agent, hand over all power to the other agent, and then expect to get the reward.
I agree the alien civilization does not necessarily have a high prior to begin with, but my point is that if you choose any system of priors (not just the Solomonoff universal prior) which is wrong by a few orders of magnitude, then even tons of observations and CDT won’t save you.
If your prior is wrong by a few orders of magnitude, there isn’t necessarily an alien civilization with a very high prior, but there very likely is. Either your prior overestimated the weight of slightly simpler universes with slightly fewer bits (maybe like the malign Solomonoff universal prior), or your prior overestimated the weight of more complex universes. And these universes might have an order of magnitude more weight than they should, and contain intelligent agents who realize you’ve overestimated their prior probability by an order of magnitude.
Regardless of whether you are a CDT agent or UDT agent, they will use your overestimation to create something subjectively very important to you, which you will trade much of your universe for due to you underestimating the prior probability of your actual universe.
If you are a UDT agent, they can offer the typical acausal trade. If you are a CDT agent, they can run so many simulations of you that it is functionally the same as an acausal trade. Instead of assigning too much “value” to their offer, you assign too much “probability” to their simulations of you.
Their simulations of you reward you for doing what they want, and you rationally take the reward (based on your prior), since getting that reward seems more probable than achieving anything in the case you are not in a simulation.
Edit: this decision theory stuff is insidiously hard, I see other people arguing about it in the comments here. It must be easy to make a mistake and feel confident about it. I’m sorry for being argumentative, I will try to do more questioning whether I am wrong. Thank you for tolerating me :/

Knight Lee Apr 4, 2025, 10:48 PM
3 points
0
in reply to: Cole Wyeth’s comment on: Changing my mind about Christiano’s malign prior argument
I think the intuition that “CDT agents don’t make acausal trades” is an enormous misconception by decision theorists.
CDT agents do not try to acausally influence other agents, but they still are acausally influenced by other agents!
Suppose you are a CDT agent, but your brain is somehow capable of running accurate simulations of alien civilizations. Suppose you discover that the typical alien civilization makes a provable promise to you, that they will run trillions of simulations of you, and that if you are in such a simulation, they will reward you immensely for doing what they want.
Suppose you deduce that 99.9% of your copies are in such a simulation. Then the rational thing (according to CDT) is to do what they want.
If you want to avoid this acausal influence, you need something more than just sticking to CDT.
“If the prior is mistake about with universe you are in, experience should correct it.”
No, given that the prior probability of being in a simulation in the “malign universe” is an overestimate, no amount observations will change the posterior probability, because simulations are indistinguishable, so nothing happens during Bayesian updating.

Knight Lee Apr 4, 2025, 11:11 AM
1 point
0
in reply to: StefanHex’s comment on: An idea for avoiding neuralese architectures
Haha, my idea was just “maybe to solve this information bottleneck problem, we should solve the generate-many-tokens-in-one-pass problem.”
I haven’t really thought of any solution to the generate-many-tokens-in-one-pass problem yet :/
I’ll edit the post to mention this.
An attempt
One stupid attempt to solve the “generate-many-tokens-in-one-pass” problem, is to start off with the main LLM outputting 1 token at a time, and a small cheap LLM outputting the next 5 tokens. You then let the small LLM eavesdrop on the residual stream of the main LLM, and use reinforcement learning on both the main LLM and the small LLM.
The hope is that the main LLM will eventually learn to use part of its residual stream to communicate to the small LLM, and tell the small LLM what the next 5 tokens should be, so the computations in the main LLM can directly influence 6 tokens of output.
Slow thinking
I guess the “simple filter repeatedly deletes everything except its first token from the context window” was a bit unclear. I’ll rewrite that.
What I wanted to say was, when the AI is talking to you (rather than talking to itself in its chain-of-thought), we want the AI to slow down, and do more computation for each token it outputs. In this case, we don’t want it outputting many tokens for each forward pass. We want to only keep the first “high quality” token and delete the rest.
I don’t think this is related to top-k/top-p generation, because that’s referring to how an LLM samples one token from its distribution. k refers to the number of tokens considered not the number of tokens generated at once.
Thank you so much for reading and for the reply :)

Knight Lee Apr 4, 2025, 9:53 AM
1 point
0
on: Changing my mind about Christiano’s malign prior argument
Wait a minute! I think this post demonstrates that the Solomonoff universal prior is malign because it leads to unfavorable acausal trades!
Basically, if the Solomonoff universal prior was the “correct” prior, then the malign agent simulating the observations by Solomonoff induction agent, would be simply offering it an acausal trade deal.
The Solomonoff induction agent “rationally” takes the deal because the malign agent has a far high prior probability of coming to existence, so the malign agent’s universe matters far more.
However, if the Solomonoff universal prior is not correct, and assigns far too much weight to universes which are merely few bits simpler. Then the malign agent, who uses a more sensible prior, would scam Solomonoff induction agents by convincing them their observations are almost certainly a simulation by the malign agent, since the malign agent’s universe is just a few bits simpler, and therefore they should trade their entire universes for a small share of the malign agent’ universe.
The scary part is that even if you don’t use the Solomonoff universal prior, your prior will still be malign if it is “incorrect,” because you will be easily scammed by any civilization/agent in a universe which your prior gives too much weight towards. They’ll convince you that you are probably a simulation by them, and you should trade your entire universe for a small share of their universe.
Priors are scary, maybe.

Knight Lee Apr 4, 2025, 8:34 AM
3 points
0
on: On Downvotes, Cultural Fit, and Why I Won’t Be Posting Again
Please don’t see downvotes as rejection. People on LessWrong downvote each other like crazy all the time.
There was this post discussing whether AI are being enslaved which got −46 karma. The top comment compared the post to “taking a big 💩 in public”
The post author later wrote another post in a similar vein, strongly criticizing people who downvoted her last time. Somehow the hivemind changed its mind and upvoted her this time.
Don’t take downvotes too seriously :)
Of course, if you’re looking for more than just upvotes, but for the post to have a major effect on the community’s beliefs and strategies. My pessimistic take is that a single post almost never accomplishes this. Even a great post by Eliezer Yudkowsky usually has negligible influence on the community, except when he gets very lucky.
LessWrong is full of people frustrated at their words falling on deaf ears. Here is an example, Applying superintelligence without collusion, by Eric Drexler. He reflects despairingly that:
At t+7 years, I’ve still seen no explicit argument for robust AI collusion, yet tacit belief in this idea continues to channel attention away from a potential solution-space for AI safety problems, leaving something very much like a void.

Knight Lee Apr 4, 2025, 1:48 AM
4 points
1
in reply to: Thomas Larsen’s comment on: AI 2027: What Superintelligence Looks Like
:) it’s good to know that you tried this. Because on your way trying to make it realistic, you might think of a lot of insights to solving the unrealisticness problems.
Thank you for the summary. From this summary, I sorta see why it might not work as well as a story. Regulation and governance isn’t very exciting a narrative. And big changes in strategy and attitude inevitably sound unrealistic, even if they aren’t unrealistic. E.g. if someone predicted that Europe will simply accept the fact its colonies want independence, or that the next Soviet leader will simply allow his constituent republics to break away, they would be laughed out of the room. Even though their predictions will turn out accurate.
Maybe in your disclaimer, you can point out that this summary you just wrote, is what you would actually recommend (instead of what the characters in your story did).
Yes, papers and blog posts are less entertaining of us but more pragmatic for you.

Knight Lee Apr 4, 2025, 1:08 AM
16 points
12
on: AI 2027: What Superintelligence Looks Like
Beautifully written! I read both endings and it felt very realistic (or, as realistic as a detailed far future can get).
Would it be possible for you to write a second good ending based on actions you actually do recommend,^[1] without the warning of “we don’t recommend these actions:”
Reminder that this scenario is a forecast, not a recommendation
We don’t endorse many actions in this slowdown ending and think it makes optimistic technical alignment assumptions. We don’t endorse many actions in the race ending either.
One of our goals in writing this scenario is to elicit critical feedback from people who are more optimistic than us. What does success look like? This “slowdown ending” scenario represents our best guess about how we could successfully muddle through with a combination of luck, rude awakenings, pivots, intense technical alignment effort, and virtuous people winning power struggles. It does not represent a plan we actually think we should aim for. But many, including most notably Anthropic and OpenAI, seem to be aiming for something like this.⁵⁷ We’d love to see them clarify what they are aiming for: if they could sketch out a ten-page scenario, for example, either starting from the present or branching off from some part of ours.
Of course, I understand this is a very time consuming project and not an easy ask :/
But maybe just a brief summary? I’m so curious.
1. ^
  I guess what I’m asking for, is for you to make the same technical alignment assumptions as the bad ending, but where policymakers and executives who are your “target audience” steer things towards the good ending. Where you don’t need to warn them against seeing it as a recommendation.

An idea for avoiding neuralese architectures

Knight LeeApr 3, 2025, 10:23 PM

6 points

2 comments4 min readLW link

Knight Lee Apr 1, 2025, 12:19 PM
1 point
0
in reply to: cubefox’s comment on: Grok3 On Kant On AI Slavery
Animals get called he or she, so why can’t AI?
From a utilitarian point of view, there’s not much downside in calling them he/she since humans are very capable of distrusting someone even if they’re seen as another human.
Meanwhile the advantage of talking politely about AI, is that the AI will predict humans to keep our promises to them.
That said, I tend to use “it” when referring to AI because everyone else does, and I don’t want to become the kind of person who argues over pronouns (yet here I am, sorry).
Preferably, don’t imagine the AI to have the gender you might be attracted to, to avoid this.

Knight Lee Apr 1, 2025, 8:44 AM
2 points
0
in reply to: Remmelt’s comment on: We’re not prepared for an AI market crash
:) thank you for saying thanks and replying.
You’re right, $600 billion/year sounds pretty unsustainable. That’s like 60 OpenAI’s, and more than half the US military budget. Maybe the investors pouring in that money will eventually run out of money that they’re willing to invest, and it will shrink. I think there is a 50% chance that at some point before we build AGI/ASI, the amount of spending on AI research will be halved (compared to where it is now).
It’s also a good point how the failure might cascade. I’m reminded about people discussing whether something like the “dot-com bubble” will happen to AI, which I somehow didn’t think of when writing my comment.
Right now my opinion is 25%, there will be a cascading market crash, when OpenAI et al. finally run out of money. A lot of seemingly stable things have unexpectedly crashed, and AI companies don’t look more stable than them. It’s one possible future.
I still think the possible future where this doesn’t happen is more likely, because one company failing does not dramatically reduce the expected value of future profits from AI, it just moves it elsewhere.
I agree that “AI Notkilleveryoneism” should be friends with these other communities who aren’t happy about AI.
I still think the movement should work with AI companies and lobby the government. Even if AI companies go bankrupt, AI researchers will move elsewhere and continue to have influence.

Knight Lee Apr 1, 2025, 5:16 AM
13 points
3
on: We’re not prepared for an AI market crash
I agree that AI company finances aren’t that good, but my personal opinion is that there won’t be a dramatic collapse which significantly affects how people and policymakers perceive AI and AI companies.
- AI companies which fail will be bought up by companies which want an AI team. E.g. if OpenAI fails, it might be folded into Microsoft.
- AI companies probably won’t fail at the same time.
- Even if many AI companies fail, actual usage of AI won’t drop. Whatever brand name the most popular AI has will be perceived as the winning AI company (even if it isn’t an AI company).
- Many companies do interesting research for long times without expecting much profit, e.g. Bell Labs and Google. Jeff Bezos funded his space company without expecting profit, purely driven by spacetravel mania. Now that AI is as cool as spacetravel, Elon Musk may fund xAI in the same way.
- The only “outrage” will come from investors who lost their money, but they are too money-driven to do anything with their outrage. They are used to losing money from failed bets, and they had always expected AI companies to be high risk high reward.
This is just my current vague opinion, I’m not saying that you’re wrong and I’m right.

Knight Lee Apr 1, 2025, 12:51 AM
1 point
0
in reply to: Mis-Understandings’s comment on: On the Implications of Recent Results on Latent Reasoning in LLMs
Thank you. Just earlier I was asking an AI whether my comment was reasonable and it told me something similar.

Knight Lee Apr 1, 2025, 12:20 AM
1 point
0
on: On the Implications of Recent Results on Latent Reasoning in LLMs
Beautiful post, I learned a lot from it.
An Unnecessary Trade-off
It looks like these COCONUT-like AI are trying to improve efficiency by avoiding a bottleneck. ~~It’s the bottleneck where all information inside a forward pass (e.g. the residual stream) has to be deleted, outputting only a single token.~~
~~So in a sense, the AI has to repeatedly recalculate its entire understanding of the context for every single token it generates.~~
EDIT: See the reply below by Mis-Understandings. The AI doesn’t need to recalculate its entire understanding for every token due to KV caching, so I was wrong. However, I still think there is an information bottleneck, because the context window can only gain one token of information per forward pass even though a forward pass computes zillions of bits of information.
These hidden chain of thought architectures seem to derive their efficiency from avoiding this bottleneck.
Yet surely, surely, there must be other approaches to avoid this bottleneck without completely sacrificing interpretability to death?
An example solution
Of the top of my head:
- Inside the AI’s chain of thought, each forward pass can generate many English tokens instead of one, allowing more information to pass through the bottleneck.
- After the chain of thought ended, and the AI is giving its final answer, it generates only one English tokens at a time, to make each token higher quality. The architecture might still generate many tokens in one forward pass, but a simple filter repeatedly deletes everything except its first token from the context window.
  - This slower thinking might occur not just for the final answer, but when the AI is cleaning up its chain of thought with concise summaries (e.g. ClaudePlaysPokemon has long chains of thought when simply entering and leaving a room, filling up its context window with useless information. They gave it a separate memory file where it can write and delete more important information).
Admittedly, even this method would reduce interpretability by making the chain of thought longer. This trade-off is more inevitable. But the trade-off where the AI has to learn some non-English alien language seems unnecessary.

Knight Lee Mar 31, 2025, 10:41 AM
2 points
0
in reply to: Boris Kashirin’s comment on: Tormenting Gemini 2.5 with the [[[]]][][[]] Puzzle
Good question, though haha the actual prompt was:
System Instructions: Be concise, specific, and where possible well mathematically justified.
I have a notation for numbers. [] = 1, [[]] = 2, [[[]]] = 4. [][[]] = 3, [][][[]]=5. -[]= −1, [-[]] = ¹⁄₂, [[-[]]] = sqrt(2), [][[-[]]] = 3^(1/3). [][][][][][][][[]] = 19. What does this notation mean? How would I write 210^(2/15)?

Knight Lee Mar 31, 2025, 8:32 AM
9 points
2
on: Why do many people who care about AI Safety not clearly endorse PauseAI?
One frustration I have about people on LessWrong and elsewhere is that people love criticizing every advice/strategy, while never truly supporting any alternatives.
Most upvoted comments here argue against PauseAI, or even claim that asking for a pause overall is a waste of political capital...!
Yet I remember when I proposed an open letter arguing for government funding for AI alignment, the Statement on AI Inconsistency. After writing emails and private messages, the only reply was “sorry, this strategy isn’t good, because we should just focus on pausing AI.”
I feel my open letter is more likely to succeed than pausing AI (I’m demanding that the AI alignment budget be “belief-consistent” with the military budget).
When politicians reject pausing AI, they just need the easy belief of “China must not win,” or “if we don’t do it someone else will.” But for politicians to reject my open letter, they need the difficult belief of being 99.999% sure of no AI catastrophe, thus 99.95% sure most experts are wrong.
Regardless, where are the people who favour the middle ground? Who neither argue that “asking for a pause is a waste of political capital because it’s hopeless,” nor argue that “asking for government funding is a waste of time, because we should just focus on pausing AI?”
It’s like the status game of criticism, that Wei Dai pointed out.

Knight Lee Mar 31, 2025, 7:44 AM
11 points
0
on: OpenAI lost $5 billion in 2024 (and its losses are increasing)
Pure AI companies like OpenAI and Anthropic are like race cars which automatically catch on fire and explode the moment they fall too far behind.
Meanwhile AI companies like Google DeepMind and Meta AI are race cars which can lose the lead and still catch up later. They can maintain the large expenditures needed for AI training, without needing to generate revenue nor impress investors. DeepSeek and xAI might be in between.
(Then again, OpenAI is half owned by Microsoft. If it falls too far behind it might not go out of business but get folded into Microsoft, at a lower valuation. I still think they feel much more short term pressure.)

Knight Lee

Objective case

Subjective case

The AI alignment idea

Fighting the subjectivity

The chain of belief

An attempt

Slow thinking

An idea for avoid­ing neu­ralese architectures

An Unnecessary Trade-off

An example solution

An idea for avoiding neuralese architectures