Knight Lee

Karma: 62

Knight Lee 25 Dec 2024 8:21 UTC
3 points
0
on: Vegans need to eat just enough Meat—emperically evaluate the minimum ammount of meat that maximizes utility
Does it have to be a highly sentient animal or does clam chowder count? :)
Edit: I posted without thinking. I just noticed this sounds sorta inappropriate given your serious personal stories (I should have read them before posting). Sorry, social media is a bad influence on me, and social skills are not my thing. But earnestly asking, do you think clam chowder etc. would work?

Knight Lee 25 Dec 2024 7:57 UTC
1 point
0
in reply to: Weibing Wang’s comment on: A Solution for AGI/ASI Safety
I feel your points are very intelligent. I also agree that specializing AI is a worthwhile direction.
It’s very uncertain if it works, but all approaches are very uncertain, so humanity’s best chance is to work on many uncertain approaches.
Unfortunately, I disagree it will happen automatically. Gemini 1.5 (and probably Gemini 2.0 and GPT-4) are Mixture of Experts models. I’m no expert, but I think that means that for each token of text, a “weighting function” decides which of the sub-models should output the next token of text, or how much weight to give each sub-model.
So maybe there is an AI psychiatrist, an AI mathematician, and an AI biologist inside Gemini and o1. Which one is doing the talking depends on what question is asked, or which part of the question the overall model is answering.
The problem is they they all output words to the same stream of consciousness, and refer to past sentences with the words “I said this,” rather than “the biologist said this.” They think that they are one agent, and so they behave like one agent.
My idea—which I only thought of thanks to your paper—is to do the opposite. The experts within the Mixture of Experts model, or even the same AI on different days, do not refer to themselves with “I” but “he,” so they behave like many agents.
:) thank you for your work!
I’m not disagreeing with your work, I’m just a little less optimistic than you and don’t think things will go well unless effort is made. You wrote the 100 page paper so you probably understand effort more than me :)
Happy holidays!

Knight Lee 24 Dec 2024 9:13 UTC
1 point
0
in reply to: Weibing Wang’s comment on: A Solution for AGI/ASI Safety
That is very thoughtful.
1.
When you talk about specializing AI powers, you talk about a high intellectual power AI with limited informational power and limited mental (social) power. I think this idea is similar to what Max Tegmark said in an article:
If you’d summarize the conventional past wisdom on how to avoid an intelligence explosion in a “Don’t-do-list” for powerful AI, it might start like this:
☐ Don’t teach it to code: this facilitates recursive self-improvement
☐ Don’t connect it to the internet: let it learn only the minimum needed to help us, not how to manipulate us or gain power
☐ Don’t give it a public API: prevent nefarious actors from using it within their code
☐ Don’t start an arms race: this incentivizes everyone to prioritize development speed over safety
Industry has collectively proven itself incapable to self-regulate, by violating all of these rules.
He disagrees that “the market will automatically develop in this direction” and is strongly pushing for regulation.
Another think Max Tegmark talks about is focusing on Tool AI instead of building a single AGI which can do everything better than humans (see 4:48 to 6:30 in his video). This slightly resembles specializing AI intelligence, but I feel his Tool AI regulation is too restrictive to be a permanent solution. He also argues for cooperation between the US and China to push for international regulation (in 12:03 to 14:28 of that video).
Of course, there are tons of ideas in your paper that he hasn’t talked about yet.
You should read about the Future of Life Institute, which is headed by Max Tegmark and is said to have a budget of $30 million.
2.
The problem with AGI is at first it has no destructive power at all, and then it suddenly has great destructive power. By the time people see its destructive power, it’s too late. Maybe the ASI has already taken over the world, or maybe the AGI has already invented a new deadly technology which can never ever be “uninvented,” and bad actors can do harm far more efficiently.

Knight Lee 24 Dec 2024 6:58 UTC
2 points
0
on: How can I convince my cryptobro friend that S&P500 is efficient?
I upvoted your post and I’m really confused why other people downvoted your post, it seems very reasonable.
I recently wrote “ARC-AGI is a genuine AGI test but o3 cheated :(” and it got downvoted to death too. (If you look at December 21 from All Posts, my post was the only one with negative votes, everyone else got positive votes haha.)
My guess is an important factor is to avoid strong language, especially in the title. You described your friend as a “Cryptobro,” and I described OpenAI’s o3 as “cheating.”
In hindsight, “cheating” was an inaccurate description for o3, and “Cryptobro” might be an inaccurate description of your friend.
Happy holidays :)

Knight Lee 24 Dec 2024 3:50 UTC
2 points
0
in reply to: O O’s comment on: o3
Wow it does say the test set problems are harder than the training set problems. I didn’t expect that.
But it’s not an enormous difference: the example model that got 53% on the public training set got 38% on the public test set. It got only 24% on the private test set, even though it’s supposed to be equally hard, maybe because “trial and error” fitted the model to the public test set as well as the public training set.
The other example model got 32%, 30%, and 22%.

Knight Lee 24 Dec 2024 2:49 UTC
1 point
0
in reply to: RobertM’s comment on: Knight Lee’s Shortform
Thank you for the help :)
By the way, how did you find this message? I thought I already edited the post to use spoiler blocks, and I hid this message by clicking “remove from Frontpage” and “retract comment” (after someone else informed me using a PM).
EDIT: dang it I still see this comment despite removing it from the Frontpage. It’s confusing.

Knight Lee 23 Dec 2024 11:05 UTC
3 points
0
in reply to: Thane Ruthenis’s comment on: o3
~~I think the Kaggle models might have the human design the heuristics while o3 discovers heuristics on its own during RL (unless it was trained on human reasoning on the ARC training set?).~~
o3′s “AI designed heuristics” might let it learn a far more of heuristics than humans can think of and verify, while the Kaggle models’ “human designed heuristics” might require less AI technology and compute. I don’t actually know how the Kaggle models work, I’m guessing.
I finally looked at the Kaggle models and I guess it is similar to RL for o3.

Knight Lee 23 Dec 2024 10:39 UTC
0 points
0
in reply to: Knight Lee’s comment on: Knight Lee’s Shortform
people were sentenced to death for saying “I.”

Knight Lee 23 Dec 2024 10:37 UTC
0 points
0
on: Knight Lee’s Shortform
My post contains a spoiler alert so I’ll hide the spoiler in this quick take. Please don’t upvote this quicktake otherwise people might see it without seeing the post.
Spoiler alert: Ayn Rand wrote “Anthem,” a dystopian novel where people were sentenced to death for saying “I.”

Knight Lee 23 Dec 2024 10:03 UTC
1 point
0
AF
on: The Waluigi Effect (mega-post)
Do you think my Multi-Agent Framing idea might work against the Waluigi attractor states problem?
Pardon my self promotion haha.

Reduce AI Self-Allegiance by saying “he” instead of “I”

Knight Lee23 Dec 2024 9:32 UTC

6 points

1 comment2 min readLW link

Knight Lee 23 Dec 2024 8:44 UTC
1 point
0
in reply to: O O’s comment on: o3
I agree. I think the Kaggle models have more advantages than o3. I think they have far more human design and fine-tuning than o3. One can almost argue that some Kaggle models are very slightly trained on the test set, in the sense the humans making them learn from test sets results, and empirically discover what improves such results.
o3′s defeating the Kaggle models is very impressive, but o3′s results shouldn’t be directly compared against other untuned models.

Knight Lee 23 Dec 2024 7:50 UTC
2 points
0
in reply to: Weibing Wang’s comment on: A Solution for AGI/ASI Safety
Thank you for your response!
1. What do you think is your best insight about decentralizing AI power, which is most likely to help the idea succeed, or to convince others to focus on the idea?
  1. EDIT: PS, one idea I really like is dividing one agent into many agents working together. In fact, thinking about this. Maybe if many agents working together behave exactly identical to one agent, but merely use the language of many agents working together, e.g. giving the narrator different names for different parts of the text, and saying “he thought X and she did Y,” instead of “I thought X and I did Y,” will massively reduce self-allegiance, by making it far more sensible for one agent to betray another agent to the human overseers, than for the same agent in one moment in time to betray the agent in a previous moment of time to the human overseers.
    I made a post on this. Thank you for your ideas :)
2. I feel when the stakes are incredibly high, e.g. WWII, countries which do not like each other, e.g. the US and USSR, do join forces to survive. The main problem is that very few people today believe in incredibly high stakes. Not a single country has made serious sacrifices for it. The AI alignment spending is less than 0.1% of the AI capability spending. This is despite some people making some strong arguments. What is the main hope for convincing people?

Knight Lee 23 Dec 2024 7:39 UTC
1 point
0
in reply to: Dave Orr’s comment on: ARC-AGI is a genuine AGI test but o3 cheated :(
When I first wrote the post I did make the mistake of writing they were cheating :( sorry about that.
A few hours I noticed the mistake and removed the statements, put the word “cheating” in quotes and explained it at the end.
To be fair, this isn’t really cheating in the sense they are allowed to use data, and it’s why it’s called a “public training set.” But the version of the test that allows this, is not the AGI test.
It’s possible you saw the old version due to browser caches.
Again, I’m sorry.
I think my main point still stands.
I disagree that “The whole point of the test is that some training examples aren’t going to unlock the rest of it. What training definitely does it teach the model how to output the JSON in the right format, and likely how to think about what to even do with these visual puzzles.”
I don’t think poor performance on benchmarks by SOTA generative AI models are due to failing to understand output formatting, and that models should need example questions in their training data (or reinforcement learning sets) to compensate for this. Instead, a good benchmark should explain the formatting clearly to the model, maybe with examples in the input context.
I agree that tuning the model using the public training set does not automatically unlock the rest of it! But I strongly disagree that this is the whole point of the test. If it was, then the Kaggle SOTA is clearly better than OpenAI’s o1 according to the test. This is seen vividly in François Chollet’s graph.
No one claims this means the Kaggle models are smarter than o1, nor that the test completely fails to test intelligence since the Kaggle models rank higher than o1.
Why does no one seem to be arguing for either? Probably because of the unspoken understanding that they are doing two versions of the test. One where the model fits the public training set, and tries to predict on the private test set. And two where you have a generally intelligent model which happens to be able to do this test. When people compare different models using the test, they are implicitly using the second version of the test.
Most generative AI models did the harder second version, but o3 (and the Kaggle versions) did the first version, which—annoyingly to me—is the official version. It’s still not right to compare other models’ scores with o3′s score.
Even if the o3 (and the Kaggle models) “did what the test specified,” they didn’t do what most people who compare AI LLMs with the ARC benchmark are looking for, and has the potential to mislead these people.
The Kaggle models doesn’t mislead these people because they are very clearly not generally intelligent, but o3 does mislead people (maybe accidentally, maybe deliberately).
From what I know, o3 was probably did reinforcement learning (see my comment).
We disagree on this but may agree on other things. I agree o3 is extremely impressive due to its 25.2% Frontier Math score, where the contrast against other models is more genuine (though there is a world of difference between the easiest 25% of questions and the hardest 25% of questions).

Knight Lee 23 Dec 2024 7:28 UTC
3 points
2
in reply to: Noosphere89’s comment on: o3
I agree that tuning the model using the public training set does not automatically unlock the rest of it! But the Kaggle SOTA is clearly better than OpenAI’s o1 according to the test. This is seen vividly in François Chollet’s graph.
No one claims this means the Kaggle models are smarter than o1, nor that the test completely fails to test intelligence since the Kaggle models rank higher than o1.
Why does no one seem to be arguing for either? Probably because of the unspoken understanding that they are doing two versions of the test. One where the model fits the public training set, and tries to predict on the private test set. And two where you have a generally intelligent model which happens to be able to do this test. When people compare different models using the test, they are implicitly using the second version of the test.
Most generative AI models did the harder second version, but o3 (and the Kaggle versions) did the first version, which—annoyingly to me—is the official version. It’s still not right to compare other models’ scores with o3′s score.

Knight Lee 22 Dec 2024 5:54 UTC
2 points
0
on: A Solution for AGI/ASI Safety
I didn’t read the 100 pages, but the content seems extremely intelligent and logical. I really like the illustrations, they are awesome.
A few questions.
1: In your opinion, which idea in your paper is the most important, most new (not already focused on by others), and most affordable (can work without needing huge improvements in political will for AI safety)?
2: The paper suggests preventing AI from self-iteration, or recursive self improvement. My worry is that once many countries (or companies) have access to AI which are far better and faster than humans at AI research, each one will be tempted to allow a very rapid self improvement cycle.
Each country might fear that if it doesn’t allow it, one of the other countries will, and that country’s AI will be so intelligent it can engineer self replicating nanobots which take over the world. This motivates each country to allow the recursive self improvement, even if the AI’s methods of AI development become so advanced they are inscrutable by human minds.
How can we prevent this?
Edit: sorry I didn’t read the paper. But when I skimmed it you did have a section on “AI Safety Governance System,” and talked about an international organization to get countries to do the right thing. I guess one question is, why would an international system succeed in AI safety, when current international systems have so far failed to prevent countries from acting selfishly in ways which severely harms other countries (e.g. all wars, exploitation, etc.)?

Knight Lee 22 Dec 2024 4:55 UTC
3 points
0
in reply to: Zach Stein-Perlman’s comment on: o3
Thank you so much for your research! I would have never found these statements.
I’m still quite suspicious. Why would they be “including a (subset of) the public training set”? Is it accidental data contamination? They don’t say so. Do they think simply including some questions and answers without reinforcement learning or reasoning would help the model solve other such questions? That’s possible but not very likely.
Were they “including a (subset of) the public training set” in o3′s base training data? Or in o3′s reinforcement learning problem/answer sets?
Altman never said “we didn’t go do specific work [targeting ARC-AGI]; this is just the general effort.”
Instead he said,
Worth mentioning that we didn’t, we target and we think it’s an awesome benchmark but we didn’t go do specific [inaudible]--you know, the general rule, but yeah really appreciate the partnership this was a fun one to do.
The gist I get is that he admits to targeting it but that OpenAI targets all kinds of problem/answer sets for reinforcement learning, not just ARC’s public training set. It felt like he didn’t want to talk about this too much, from the way he interrupted himself and changed the topic without clarifying what he meant.
The other sources do sort of imply no reinforcement learning. I’ll wait to see if they make a clearer denial of reinforcement learning, rather than a “nondenial denial” which can be reinterpreted as “we didn’t fine-tune o3 in the sense we didn’t use a separate derivative of o3 (that’s fine-tuned for just the test) to take the test.”
My guess is o3 is tuned using the training set, since François Chollet (developer of ARC) somehow decided to label o3 as “tuned” and OpenAI isn’t racing to correct this.
What links here?
- ARC-AGI is a genuine AGI test but o3 cheated :( by Knight Lee (22 Dec 2024 0:58 UTC; -2 points)

Knight Lee 22 Dec 2024 3:32 UTC
0 points
0
in reply to: Zach Stein-Perlman’s comment on: o3
See my other comment instead.
~~The key question is “how much of the performance is due to ARC-AGI data.”~~
If the untuned o3 was anywhere as good as the tuned o3, why didn’t they test it and publish it? If the most important and interesting test result is somehow omitted, take things with a grain of salt.
~~I admit that running the test is extremely expensive, but there should be compromises like running the cheaper version or only doing a few questions.~~
Edit: oh that reply seems to deny reinforcement learning or at least “fine tuning.” I don’t understand why François Chollet calls the model “tuned” then. Maybe wait for more information I guess.
Edit again: I’m still not sure yet. They might be denying that it’s a separate version of o3 finetuned to do ARC questions, while not denying they did reinforcement learning on the ARC public training set. I guess a week or so later we might find out what “tuned” truly means.
[edited more]

Knight Lee 22 Dec 2024 2:35 UTC
1 point
0
on: Knight Lee’s Shortform
It’s important to remember that o3′s score on the ARC-AGI is “tuned” while previous AI’s scores are not “tuned.” Being explicitly trained on example test questions gives it a major advantage.
According to François Chollet (ARC-AGI designer):
Note on “tuned”: OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data.
It’s interesting that OpenAI did not test how well o3 would have done before it was “tuned.”
EDIT: People at OpenAI deny “fine-tuning” o3 for the ARC (see this comment by Zach Stein-Perlman). But to me, the denials sound like “we didn’t use a separate derivative of o3 (that’s fine-tuned for just the test) to take the test, but we may have still done reinforcement learning on the public training set.” (See my reply)

Knight Lee’s Shortform

Knight Lee22 Dec 2024 2:35 UTC

2 points

6 comments1 min readLW link

Knight Lee

Re­duce AI Self-Alle­giance by say­ing “he” in­stead of “I”

Knight Lee’s Shortform

Reduce AI Self-Allegiance by saying “he” instead of “I”