mishka

Karma: 1,223

mishka 17 Jan 2025 5:17 UTC
4 points
−2
on: AI #99: Farewell to Biden

The alternative hypothesis does need to be said, especially after someone at a party outright claimed it was obviously true, and with the general consensus that the previous export controls were not all that tight. That alternative hypothesis is that DeepSeek is lying and actually used a lot more compute and chips it isn’t supposed to have. I can’t rule it out.

Re DeepSeek cost-efficiency, we are seeing more claims pointing in that direction.

In a similarly unverified claim, the founder of 01.ai (who is sufficiently known in the US according to https://en.wikipedia.org/wiki/Kai-Fu_Lee) seems to be claiming that the training cost of their Yi-Lightning model is only 3 million dollars or so. Yi-Lightning is a very strong model released in mid-Oct-2024 (when one compares it to DeepSeek-V3, one might want to check “math” and “coding” subcategories on https://lmarena.ai/?leaderboard; the sources for the cost claim are https://x.com/tsarnick/status/1856446610974355632 and https://www.tomshardware.com/tech-industry/artificial-intelligence/chinese-company-trained-gpt-4-rival-with-just-2-000-gpus-01-ai-spent-usd3m-compared-to-openais-usd80m-to-usd100m, and we probably should similarly take this with a grain of salt).

But all this does seem to be well within what’s possible. Here is the famous https://github.com/KellerJordan/modded-nanogpt ongoing competition, and it took people about 8 months to accelerate Andrej Karpathy’s PyTorch GPT-2 trainer from llm.c by 14x on a 124M parameter GPT-2 (what’s even more remarkable is that almost all that acceleration is due to better sample efficiency with the required training data dropping from 10 billion tokens to 0.73 billion tokens on the same training set with the fixed order of training tokens).

Some of the techniques used by the community pursuing this might not scale to really large models, but most of them probably would scale (as we see in their mid-Oct experiment demonstrating scaling of what has been 3-4x acceleration back then to the 1.5B version).

So when an org is claiming 10x-20x efficiency jump compared to what it presumably took a year or more ago, I am inclined to say, “why not, and probably the leaders are also in possession of similar techniques now, even if they are less pressed by compute shortage”.

The real question is how fast will these numbers continue to go down for the similar levels of performance… It’s has been very expensive to be the very first org achieving a given new level, but the cost seems to be dropping rapidly for the followers...

mishka 10 Jan 2025 4:19 UTC
1 point
0
on: Rebuttals for ~all criticisms of AIXI
However, I don’t view safe tiling as the primary obstacle to alignment. Constructing even a modestly superhuman agent which is aligned to human values would put us in a drastically stronger position and currently seems out of reach. If necessary, we might like that agent to recursively self-improve safely, but that is an additional and distinct obstacle. It is not clear that we need to deal with recursive self-improvement below human level.

I am not sure that treating recursive self-improvement via tiling frameworks is necessarily a good idea, but setting this aspect aside, one obvious weakness with this argument is that it mentions a superhuman case and a below human level case, but it does not mention the approximately human level case.

And it is precisely the approximately human level case where we have a lot to say about recursive self-improvement, and where it feels that avoiding this set of considerations would be rather difficult.
1. Humans often try to self-improve, and human-level software will have advantage over humans at that.
Humans are self-improving in the cognitive sense by shaping their learning experiences, and also by controlling their nutrition and various psychoactive factors modulating cognition. The desire to become smarter and to improve various thinking skills is very common.

Human-level software would have great advantage over humans at this, because it can hack at its own internals with great precision at the finest resolution and because it can do so in a reversible fashion (on a copy, or after making a backup), and so can do it in a relatively safe manner (whereas a human has difficulty hacking their own internals with required precision and is also taking huge personal risks if hacking is sufficiently radical).
1. Collective/multi-agent aspects are likely to be very important.
People are already talking about possibilities of “hiring human-level artificial software engineers” (and, by extension, human-level artificial AI researchers). The wisdom of having an agent form-factor here is highly questionable, but setting this aspect aside and focusing only on technical feasibility, we see the following.

One can hire multiple artificial software engineers with long-term persistence (of features, memory, state, and focus) into an existing team of human engineers. Some of those teams will work on making next generations of better artificial software engineers (and artificial AI researchers). So now we are talking about mixed teams with human and artificial members.

By definition, we can say that those artificial software engineers and artificial AI researchers have reached human level, if a team of those entities would be able to fruitfully work on the next generation of artificial software engineers and artificial AI researchers even in the absence of any human team members.

This multi-agent setup is even more important than individual self-improvement, because this is what the mainstream trend might actually be leaning towards, judging by some recent discussions. Here we are talking about a multi-agent setup, and about recursive self-improvement of the community of agents, rather than focusing on self-improvement of individual agents.
1. Current self-improvement attempts.
We actually do see a lot of experiments with various forms of recursive self-improvements even at the current below human level. We are just lucky that all those attempts have been saturating at the reasonable levels so far.

We currently don’t have good enough understanding to predict when they stop saturating, and what would the dynamics be when they stop saturating. But self-improvement by a community of approximately human-level artificial software engineers and artificial AI researchers competitive with top human software engineers and top human AI researcher seems unlikely to saturate (or, at least, we should seriously consider the possibility that it won’t saturate).
1. At the same time, the key difficulties of AI existential safety are tightly linked to recursive self-modifications.
The most intractable aspect of the whole thing is how to preserve any properties indefinitely through radical self-modifications. I think this is the central difficulty of AI existential safety. Things will change unpredictably. How can one shape this unpredictable evolution so that some desirable invariants do hold?

These invariants would be invariant properties of the whole ecosystem, not of individual agents; they would be the properties of a rapidly changing world, not of a particular single system (unless one is talking about a singleton which is very much in control of everything). This seems to be quite central to our overall difficulty with AI existential safety.

mishka 5 Jan 2025 14:41 UTC
28 points
10
on: Chinese Researchers Crack ChatGPT: Replicating OpenAI’s Advanced AI Model
I think this is a misleading clickbait title. It references a popular article with the same misleading clickbait title, and the only thing that popular article references is a youtube video with the misleading clickbait title, “Chinese Researchers Just CRACKED OpenAI’s AGI Secrets!”

However, the description of that youtube video does reference the paper in question and a twitter thread describing this paper:

Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective, https://arxiv.org/abs/2412.14135

https://x.com/rohanpaul_ai/status/1872713137407049962

Nothing is “cracked” here. It’s just a roadmap which might work or not, depending on luck and efforts. It might correspond to what’s under the hood of o1 models or not (never mind o3, the paper is published a couple of days before the o3 announcement).

The abstract of the paper ends with

“Existing open-source projects that attempt to reproduce o1 can be seem as a part or a variant of our roadmap. Collectively, these components underscore how learning and search drive o1′s advancement, making meaningful contributions to the development of LLM.”

The abstract also has distinct feeling of being written by an LLM. The whole paper is just a discussion of various things one could try if one wants to reproduce o1. It also references a number of open source and closed source implementations of reasoners over LLMs. There are no new technical advances in the paper.

mishka 1 Jan 2025 8:54 UTC
4 points
−2
in reply to: Daniel Kokotajlo’s comment on: o3, Oh My
Right. We should probably introduce a new name, something like narrow AGI, to denote a system which is AGI-level in coding and math.

This kind of system will be “AGI” as redefined by Tom Davidson in https://www.lesswrong.com/posts/Nsmabb9fhpLuLdtLE/takeoff-speeds-presentation-at-anthropic:

“AGI” (=AI that could fully automate AI R&D)

This is what matters for AI R&D speed and for almost all recursive self-improvement.

Zvi is not quite correct when he is saying

If o3 was as good on most tasks as it is at coding or math, then it would be AGI.

o3 is not that good in coding and math (e.g. it only gets 71.7% on SWE-bench verified), it is not a “narrow AGI” yet. But it is strong enough, it’s a giant step forward.

For example, if one takes Sakana’s “AI scientist”, upgrades it slightly, and uses o3 as a back-end, it is likely that one can generate NeurIPS/ICLR quality papers and as many of those as one wants.

So, another upgrade (or a couple of upgrades) beyond o3, and we will reach that coveted “narrow AGI” stage.

What OpenAI has demonstrated is that it is much easier to achieve “narrow AGI” than “full AGI”. This does suggest a road to ASI without going through anything remotely close to a “full AGI” stage, with missing capabilities to be filled afterwards.

mishka 24 Nov 2024 14:10 UTC
4 points
0
in reply to: Leon Lang’s comment on: mishka’s Shortform
Indeed

mishka 24 Nov 2024 7:25 UTC
38 points
3
on: mishka’s Shortform
METR releases a report, Evaluating frontier AI R&D capabilities of language model agents against human experts: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Daniel Kokotajlo and Eli Lifland both feel that one should update towards shorter timelines remaining until the start of rapid acceleration via AIs doing AI research based on this report:

https://x.com/DKokotajlo67142/status/1860079440497377641

https://x.com/eli_lifland/status/1860087262849171797

mishka 22 Nov 2024 21:23 UTC
2 points
0
in reply to: duck_master’s comment on: AI Safety Salon with Steve Omohundro
the meetup page says 7:30pm, but actually the building asks people to leave by 9pm

mishka 14 Nov 2024 8:05 UTC
29 points
2
on: mishka’s Shortform
Gwern was on Dwarkesh yesterday: https://www.dwarkeshpatel.com/p/gwern-branwen

We recorded this conversation in person. In order to protect Gwern’s anonymity, we created this avatar. This isn’t his voice. This isn’t his face. But these are his words.

mishka 17 Oct 2024 1:23 UTC
4 points
0
on: Bitter lessons about lucid dreaming
Thanks, that’s very useful.

If one decides to use galantamine, is it known if one should take it right before bedtime, or anytime during the preceding day, or in some other fashion?

mishka 16 Oct 2024 11:48 UTC
3 points
0
on: Distillation Of DeepSeek-Prover V1.5
I think it’s a good idea to include links to the originals:

https://arxiv.org/abs/2408.08152 - “DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search”

https://github.com/deepseek-ai/DeepSeek-Prover-V1.5

mishka 15 Oct 2024 4:30 UTC
5 points
−5
on: mishka’s Shortform
Scott Alexander wrote a very interesting post covering the details of the political fight around SB 1047 a few days ago: https://www.astralcodexten.com/p/sb-1047-our-side-of-the-story

I’ve learned a lot of things new to me reading it (which is remarkable given how much material related to SB 1047 I have seen before)

mishka 13 Oct 2024 5:34 UTC
3 points
0
in reply to: queelius’s comment on: How Should We Use Limited Time to Maximize Long-Term Impact?

the potential of focusing on chemotherapy treatment timing

More concretely (this is someone’s else old idea), what I think is still not done is the following. Chemo kills dividing cells, this is why the rapidly renewing tissues and cell populations are particularly vulnerable.

If one wants to spare one of those cell types (say, a particular population of immune cells), one should take the typical period of its renewal, and use that as a period of chemo sessions (time between chemo sessions, a “resonance” of sorts between that and the period of the cell population renewal for the selected cell type). Then one should expect to spare most of that population (and might potentially be able to use higher doses for better effect, if the spared population is the most critical one; this does need some precision, not a typical today’s “relaxed logistics” approach where a few days this or that way in the schedule is nothing to worry about).

I don’t know if that ever progressed beyond the initial idea...

(That’s just one example, of course, there is a lot of things which can be considered and, perhaps, tried.)

mishka 13 Oct 2024 3:11 UTC
3 points
0
on: How Should We Use Limited Time to Maximize Long-Term Impact?
This depends on many things (one’s skills, one’s circumstances, one’s preferences and inclinations (the efficiency of one’s contributions greatly depends on one’s preferences and inclinations)).

I have stage 4 cancer, so statistically, my time may be more limited than most. I’m a PhD student in Computer Science with a strong background in math (Masters).

In your case, there are several strong arguments for you to focus on research efforts which can improve your chances of curing it (or, at least, of being able to maintain the situation for a long time), and a couple of (medium strength?) arguments against this choice.

For:
- If you succeed, you’ll have more time to make impact (and so if your chance of success is not too small, this will contribute to your ability to maximize your overall impact, statistically speaking).
- Of course, any success here will imply a lot of publicly valuable impact (there are plenty of people in a similar position health-wise, and they badly need progress to occur ASAP).
- The rapid development of applied AI models (both general purpose models and biology-specific models) creates new opportunities to datamine and juxtapose a variety of potentially relevant information and to uncover new connections which might lead to effective solutions. Our tools progress so fast that people are slow to adapt their thinking and methods to that progress. So new people with fresh outlook have reasonable shots (of course, they should aim for collaborations). In this sense, your PhD CS studies and your strong math is very helpful (a lot of the relevant models are dynamic systems, timing of interventions is typically not managed correctly as far as I know (there are plenty of ways to be nice to particularly vulnerable tissues by timing the chemo right and thus being able to make it more effective, but this is not a part of the standard-of-care yet as far as I know), and so on).
- You are likely to be strongly motivated and to be able to maintain strong motivation. At the same time you’ll know that it is the result that counts here, not the effort, and so you will be likely to try your best to approach this in a smart way, not in a brute force effort way.
Possibly against:
- The psychological implications of working on your own life-and-death problem are non-trivial. One might choose to embrace them or to avoid them.
- Focusing on “one’s own problem” might be compatible or not very compatible with this viewpoint you once expressed: https://www.lesswrong.com/posts/KFWZg6EbCuisGcJAo/immortality-or-death-by-agi-1?commentId=QYDvovQZevDmGtfXY
(Of course, there are plenty of other interesting things one can do with this background (PhD CS studies and strong math). For example, one might decide to disregard the health situation and to dive into technical aspects of AI development and AI existential safety issues, especially if one’s estimate of AI timelines yields really short timelines.)

mishka 2 Oct 2024 17:29 UTC
2 points
1
in reply to: peterslattery’s comment on: My 10-year retrospective on trying SSRIs
Thanks for the references.

Yes, the first two of those do mention co-occurring anxiety in the title.

The third study suggests a possibility that it might just work as an effective anti-depressant as well. (I hope there will be further studies like that; yes, this might be a sufficient reason to try it for depression, even if one does not have anxiety. It might work, but it’s clearly not a common knowledge yet.)

mishka 2 Oct 2024 13:50 UTC
2 points
0
on: Three main arguments that AI will save humans and one meta-argument
Your consideration seems to assume that the AI is an individual, not a phenomenon of “distributed intelligence”:

The first argument is that AI thinks it may be in a testing simulation, and if it harms humans, it will be turned off.

etc. That is, indeed, the only case we are at least starting to understand well (unfortunately, our understanding of situations where AIs are not individuals seems to be extremely rudimentary).

If the AI is an individual, then one can consider a case of a “singleton” or a “multipolar case”.

In some sense, for a self-improving ecosystem of AIs, a complicated multipolar scenario seems more natural, as new AIs are getting created and tested quite often in realistic self-improvement scenarios. In any case, a “singleton” only looks “monolithic” from the outside; from the inside, it is still likely to be a “society of mind” of some sort.

If there are many such AI individuals with uncertain personal future (individuals who can’t predict their future trajectory and their future relative strength in the society and who care about their future and self-preservation), then AI individuals might be interested in a “world order based on individual rights”, and then rights of all individuals (including humans) might be covered in such a “world order”.

This consideration is my main reason for guarded optimism, although there are many uncertainties.

In some sense, my main reasons for guarded optimism are in hoping that the AI ecosystem will manage to act rationally and will manage to avoid chaotic destructive developments. As you say

It is not rational to destroy a potentially valuable thing.

And my main reasons for pessimism are in being afraid that the future will resemble uncontrolled super-fast chaotic accelerating “natural evolution” (in this kind of scenarios AIs seem to be likely to destroy everything including themselves, they do have an existential safety problem of their own as they can easily destroy the “fabric of reality” if they don’t exercise collaboration and self-control).

mishka 2 Oct 2024 13:18 UTC
6 points
0
on: Should we abstain from voting? (In nondeterministic elections)
One might consider that some people have strong preferences for the outcome of an election and some people have weak preferences, but that there is usually no way to express the strength of one’s preferences during a vote, and the probability that one would actually go ahead and vote in a race does correlate with the strength of one’s preferences.

So, perhaps, this is indeed working as intended. People who have stronger preferences are more likely to vote, and so their preferences are more likely to be taken into account in a statistical sense.

It seems that the strength of one’s preferences is (automatically, but imperfectly) taken into account via this statistical mechanism.

mishka 2 Oct 2024 0:11 UTC
29 points
26
on: Newsom Vetoes SB 1047
Thanks for the great post!

Also it’s California, so there’s some chance this happens, seriously please don’t do it, nothing is so bad that you have to resort to a ballot proposition, choose life

Why are you saying this? In what sense “nothing is so bad”?

The reason why people who have libertarian sensibilities, distrust for government track record in general and specifically for its track record in tech regulation are making exception in this case is the future AI strong potential for catastrophic and existential risks.

So, why people who generally dislike the mechanism and track record of California ballot propositions should not make an exception here as well?

The whole point of all this effort around SB 1047 is that “nothing is so bad” is an incorrect statement.

And especially given that you are correctly saying:

Thus I reiterate the warning: SB 1047 was probably the most well-written, most well-considered and most light touch bill that we were ever going to get. Those who opposed it, and are now embracing the use-case regulatory path as an alternative thinking it will be better for industry and innovation, are going to regret that. If we don’t get back on the compute and frontier model based path, it’s going to get ugly.

There is still time to steer things back in a good direction. In theory, we might even be able to come back with a superior version of the model-based approach, if we all can work together to solve this problem before something far worse fills the void.

But we’ll need to work together, and we’ll need to move fast.

Sure, there is still a bit of time for a normal legislative effort (this time with a close coordination with Newsom, otherwise he will just veto it again), but if you really think that if a normal route fails, the ballot route is still counter-productive, you need to make a much stronger case for that.

Especially given that the ballot measure will probably pass with large margin and flying colors...

mishka 23 Sep 2024 19:06 UTC
5 points
0
in reply to: peterslattery’s comment on: My 10-year retrospective on trying SSRIs

Silexan

For anxiety treatment only, if I understand it correctly.

There is no claim that it works as an antidepressant, as far as I know.

mishka 21 Sep 2024 21:23 UTC
3 points
0
in reply to: Roko’s comment on: A Nonconstructive Existence Proof of Aligned Superintelligence
No, not microscopic.

Coherent light produced by lasers is not microscopic, we see its traces in the air. And we see the consequences (old fashioned holography and the ability to cut things with focused light, even at large distances). Room temperature is fine for that.

Superconductors used in the industry are not microscopic (and the temperatures are high enough to enable industrial use of them in rather common devices such as MRI scanners).

mishka 21 Sep 2024 21:19 UTC
2 points
0
in reply to: Roko’s comment on: A Nonconstructive Existence Proof of Aligned Superintelligence
It’s just… having a proof is supposed to boost our confidence that the conclusion is correct...

if the proof relies on assumptions which are already quite far from the majority opinion about our actual reality (and are probably going to deviate further, as AIs will be better physicists and engineers than us and will leverage the strangeness of our physics much further than we do), then what’s the point of that “proof”?

how does having this kind of “proof” increase our confidence in what seems informally correct for a single branch reality (and rather uncertain in a presumed multiverse, but we don’t even know if we are in a multiverse, so bringing a multiverse in might, indeed, be one of the possible objections to the statement, but I don’t know if one wants to pursue this line of discourse, because it is much more complicated than what we are doing here so far)?

(as an intellectual exercise, a proof like that is still of interest, even under the unrealistic assumption that we live in a computable reality, I would not argue with that; it’s still interesting)