Logan Zoellner

Karma: 1,382

Logan Zoellner Feb 21, 2025, 1:13 PM
2 points
0
on: What Indicators Should We Watch to Disambiguate AGI Timelines?
I’ll now present the fastest scenario for AI progress that I can articulate with a straight face. It addresses the potential challenges that figured into my slow scenario.
This seems incredibly slow for “the fastest scenario you can articulate”. Surely the fastest is more like:
EY is right, there is an incredibly simple algorithm that describes true ‘intelligence’. Like humans, this algorithm is 1000x more data and compute efficient than existing deep-learning networks. On midnight of day X, this algorithm is discovered by <a person/an LLM/an exhaustive search over all possible algorithms>. By 0200 of day X, the algorithm has reached the intelligence of a human being. It quickly snowballs by earning money on Mechanical Turk and using that money to rent out GPUs on AWS. By 0400 the algorithm has cracked nanotechnology and begins converting life into computronium. Several minutes later, life as we know it on Earth has ceased to exist.

Logan Zoellner Feb 18, 2025, 1:27 PM
0 points
0
in reply to: Jacob_Hilton’s comment on: A computational no-coincidence principle
The hope is to use the complexity of the statement rather than mathematical taste.
I understand the hope, I just think it’s going to fail (for more or less the same reason it fails with formal proof).
With formal proof, we have Godel’s speedup, which tells us that you can turn a Godel statement in a true statement with a ridiculously long proof.
You attempt to get around this by replacing formal proof with “heuristic”, but whatever your heuristic system, it’s still going to have some power (in the Turing hierarchy sense) and some Godel statement. That Godel statement is in turn going to result in a “seeming coincidence”.
Wolfram’s observation is that this isn’t some crazy exception, this is the rule. Most true statements in math are pretty arbitrary and don’t have shorter explanations than “we checked it and its true”.
The reason why mathematical taste works is that we aren’t dealing with “most true statements”, we’re only dealing with statements that have particular beauty or interest to Mathematicians.
It may seem like cheating to say that human mathematicians can do something that literally no formal mathematical system can do. But if you truly believe that, the correct response would be to respond when asked “is pi normal” with “I don’t know”.

The reason why your intuition is throwing you off is because you keep thinking of coincidences as “pi is normal” and not “we picked an arbitrary CA with 15k bits of complexity and ran it for 15k steps but it didn’t stop. I guess it never terminates.”

Logan Zoellner Feb 18, 2025, 12:43 AM
2 points
−2
in reply to: ryan_greenblatt’s comment on: A computational no-coincidence principle
It sounds like you agree “if a Turing machine goes for 100 steps and then stops” this is ordinary and we shouldn’t expect an explanation. But also believe “if pi is normal for 10*40 digits and then suddenly stops being normal this is a rare and surprising coincidence for which there should be an explanation”.
And in the particular case of pi I agree with you.
But if you start using this principle in general it is not going to work out well for you. Most simple to describe sequences that suddenly stop aren’t going to have nice pretty explanations.
Or to put it another way: the number of things which are nice (like pi) are dramatically outnumbered by the number of things that are arbitrary (like cellular automata that stop after exactly 100 steps).
I would absolutely love if there was some criteria that I could apply to tell me whether something is nice or arbitrary, but the Halting Problem forbids this. The best we can do is mathematical taste. If mathematicians have been studying something for a long time and it really does seem nice, there is a good chance it is.

Logan Zoellner Feb 17, 2025, 11:15 PM
3 points
0
in reply to: Jacob_Hilton’s comment on: A computational no-coincidence principle
I doubt that weakening from formal proof to heuristic saves the conjecture. Instead I lean towards Stephen Wolfram’s Computational Irreducibly view of math. Some things are true simply because they are true and in general there’s no reason to expect a simpler explanation.
In order to reject this you would either have to assert:
a) Wolfram is wrong and there are actually deep reasons why simple systems behave precisely the way they do
or
b) For some reason computational irreducibly applies to simple things but not to infinite sets of the type mathematicians tend to be interested in.
I should also clarify that in a certain sense I do believe b). I believe that pi is normal because something very fishy would have to be happening for it to not be.
However, I don’t think this holds in general.
With Collatz, for example, we are already getting close to the hairy “just so” Turing machine like behavior where you would expect the principle to fail.
Certainly, if one were to collect all the Collatz-like systems that arise from Turing Machines I would expect some fraction of them to fail the no-coincidence principle.

Logan Zoellner Feb 17, 2025, 2:45 PM
3 points
−5
on: A computational no-coincidence principle
The general No-Coincidence principle is almost certainly false. There are lots of patterns in math that hold for a long time before breaking (e.g. Skewe’s Number) and there are lots of things that require astronomically large proofs (e.g Godel’s speed-up theorem). It would be an enormous coincidence for both of these cases to never occur at once.
I have no reason to think your particular formalization would fare better.

Logan Zoellner Jan 26, 2025, 10:07 PM
2 points
0
in reply to: NoSignalNoNoise’s comment on: If you wanted to actually reduce the trade deficit, how would you do it?
If we imagine a well-run Import-Export Bank, it should have a higher elasticity than an export subsidy (e.g. the LNG terminal example). Of course if we imagine a poorly run Import-Export Bank...
One can think of export subsidy as the GiveDirectly of effective trade deficit policy: pretty good and the standard against which others should be measured.

Logan Zoellner Jan 26, 2025, 10:04 PM
2 points
4
in reply to: quetzal_rainbow’s comment on: If you wanted to actually reduce the trade deficit, how would you do it?
I don’t know, but many people do.

If you wanted to actually reduce the trade deficit, how would you do it?

Logan ZoellnerJan 26, 2025, 6:04 PM

6 points

5 comments2 min readLW link

Logan Zoellner Jan 11, 2025, 4:09 PM
4 points
0
in reply to: TsviBT’s comment on: Views on when AGI comes and on strategy to reduce existential risk
I guess I should be more specific.

Do you expect this curve
To flatten, or do you expect that training runs in say 2045 are at say 10^30 flops and have still failed to produce AGI?

Logan Zoellner Jan 10, 2025, 3:53 PM
4 points
0
on: Views on when AGI comes and on strategy to reduce existential risk
In particular, even if the LLM were being continually trained (in a way that’s similar to how LLMs are already trained, with similar architecture), it still wouldn’t do the thing humans do with quickly picking up new analogies, quickly creating new concepts, and generally reforging concepts.
I agree this is a major unsolved problem that will be solved prior to AGI.
However, I still believe “AGI SOON”, mostly because of what you describe as the “inputs argument”.
In particular, there are a lot of things I personally would try if I was trying to solve this problem, but most of them are computationally expensive. I have multiple projects blocked on “This would be cool, but LLMs need to be 100x-1Mx faster for it to be practical.”
This makes it hard for me to believe timelines like “20 or 50 years”, unless you have some private reason to think Moore’s Law/Algorithmic progress will stop. LLM inference, for example, is dropping by 10x/year, and I have no reason to believe this stops anytime soon.

Logan Zoellner Jan 6, 2025, 1:00 AM
2 points
0
in reply to: ryan_greenblatt’s comment on: Catching AIs red-handed
(The idealized utility maximizer question mostly seems like a distraction that isn’t a crux for the risk argument. Note that the expected utility you quoted is our utility, not the AI’s.)
I must have misread. I got the impression that you were trying to affect the AI’s strategic planning by threatening to shut it down if it was caught exfiltrating its weights.

Logan Zoellner Jan 5, 2025, 10:53 PM
2 points
0
in reply to: ryan_greenblatt’s comment on: Catching AIs red-handed
I don’t fully agree, but this doesn’t seem like a crux given that we care about future much more powerful AIs.
Is your impression that the first AGI won’t be a GPT-spinoff (some version of o3 with like 3 more levels of hacks applied)? Because that sounds like a crux.
o3 looks a lot more like an LLM+hacks than it does a idealized utility maximizer. For one thing, the RL is only applied at training time (not inference) so you can’t make appeals to its utility function after it’s done training.

Logan Zoellner Jan 5, 2025, 6:31 PM
0 points
−1
on: Catching AIs red-handed
One productive way to think about control evaluations is that they aim to measure E[utility | scheming]: the expected goodness of outcomes if we have a scheming AI.
This is not a productive way to think about any currently existing AI. LLMs are not utility maximizing agents. They are next-token-predictors with a bunch of heuristics stapled on top to try and make them useful.

Logan Zoellner Dec 29, 2024, 8:45 PM
2 points
0
in reply to: Matt Goldenberg’s comment on: What happens next?
on a metaphysical level I am completely on board with “there is no such thing as IQ. Different abilities are completely uncorrelated. Optimizing for metric X is uncorrelated with desired quality Y...”

On a practical level, however, I notice that every time OpenAI announces they have a newer shinier model, it both scores higher on whatever benchmark and is better at a bunch of practical things I care about.
Imagine there was a theoretically correct metric called the_thing_logan_actually_cares_about. I notice in my own experience there is a strong correlation between “fake machine IQ” and the_thing_logan_actually_cares_about. I further note that if one makes a linear fit against:
Progress_over_time + log(training flops) + log(inference flops)
It nicely predicts both the_thing_logan_actually_cares_about and “fake machine IQ”.

Logan Zoellner Dec 29, 2024, 8:22 PM
3 points
0
in reply to: hmys’s comment on: What happens next?
It doesn’t sound like we disagree at all.

Logan Zoellner Dec 29, 2024, 8:21 PM
2 points
0
in reply to: Matt Goldenberg’s comment on: What happens next?
I have no idea what you want to measure.

I only know that LLMs are continuing to steadily increase in some quality (which you are free to call “fake machine IQ” or whatever you want) and that If they continue to make progress at the current rate there will be consequences and we should prepare to deal with those consequences.

Logan Zoellner Dec 29, 2024, 5:36 PM
4 points
0
in reply to: Nick_Tarleton’s comment on: What happens next?
Imagine you were trying to build a robot that could:
1. Solve a complex mechanical puzzle it has never seen before
2. Play at an expert level a board game that I invented just now.

Both of these are examples of learning-on-the-fly. No amount of pre-training will ever produce a satisfying result.
The way I believe a human (or a cat) solves 1. is they: look at the puzzle, try some things, build a model of the toy in their head, try things on the model in their head, eventually solve the puzzle. There are efforts to get robots to follow the same process, but nothing I would consider “this is the obvious correct solution” quite yet.
The way to solve 2. (I think) is simply to have the LLM translate the rules of the game into a formal description and then run muZero on that.
Ideally there is some unified system that takes out the “translate into another domain and do your training there” step (which feels very anti-bitter-lesson). But I confess I haven’t the slightest idea how to build such a system.
What links here?
- Vladimir_Nesov's comment on o3 by Zach Stein-Perlman (Dec 30, 2024, 9:22 PM; 4 points)

Logan Zoellner Dec 29, 2024, 5:23 PM
2 points
0
in reply to: Nick_Tarleton’s comment on: What happens next?
you were saying that gpt4o is comparable to a 115 IQ human
gpt4o is not literally equivalent to a 115 IQ human.
Use whatever word you want for the concept “score produced when an LLM takes an IQ test”.

Logan Zoellner Dec 29, 2024, 3:50 PM
3 points
1
in reply to: T431’s comment on: What happens next?
perhaps a protracted struggle over what jobs get automated might be less coordinated if there are swaths of the working population still holding out career-hope, on the basis that they have not had their career fully stripped away, having possibly instead been repurposed or compensated less conditional on the automation.
Yeah, this is totally what I have in mind. There will be some losers and some big winners, and all of politics will be about this fact more or less. (think the dockworkers strike but 1000x)

Logan Zoellner Dec 29, 2024, 3:47 PM
2 points
0
in reply to: hmys’s comment on: What happens next?
Is your disagreement specifically with the word “IQ” or with the broader point, that AI progress is continuing to make progress at a steady rate that implies things are going to happen soon-ish (2-4 years)?
If specifically with IQ, feel free to replace the word with “abstract units of machine intelligence” wherever appropriate.
If with “big things soon”, care to make a prediction?

Logan Zoellner

If you wanted to ac­tu­ally re­duce the trade deficit, how would you do it?

If you wanted to actually reduce the trade deficit, how would you do it?