faul_sname

Karma: 3,946

faul_sname 19 Jan 2025 20:36 UTC
12 points
2
on: faul_sname’s Shortform
Scaffolded LLMs are pretty good at not just writing code, but also at refactoring it. So that means that all the tech debt in the world will disappear soon, right?

I predict “no” because
- As writing code gets cheaper, the relative cost of making sure that a refactor didn’t break anything important goes up
- The number of parallel threads of software development will also go up, with multiple high-value projects making mutually-incompatible assumptions (and interoperability between these projects accomplished by just piling on more cose).
As such, I predict an explosion of software complexity and jank in the near future.

faul_sname 17 Jan 2025 12:32 UTC
6 points
3
on: Numberwang: LLMs Doing Autonomous Research, and a Call for Input
I suspect that it’s a tooling and scaffolding issue and that e.g.claude-3-5-sonnet-20241022 can get at least 70% on the full set of 60 with decent prompting and tooling.

By “tooling and scaffolding” I mean something along the lines of
- Naming the lists that the model submits (e.g. “round 7 list 2”)
- A tool where the LLM can submit a named hypothesis in the form of a python function which takes a list and returns a boolean and check whether the results of that function on all submitted lists from previous rounds match the answers it’s seen so far
- Seeing the round number on every round
- Dropping everything except the seen lists and non-falsified named hypotheses in the context each round (this is more of a practical thing to avoid using absurd volumes of tokens, but I imagine it wouldn’t hurt performance too much)
I’ll probably play around with it a bit tomorrow.

faul_sname 16 Jan 2025 5:02 UTC
3 points
0
on: LLMs for language learning
Adapting spaced repetition to interruptions in usage: Even without parsing the user’s responses (which would make this robust to difficult audio conditions), if the reader rewinds or pauses on some answers, the app should be able to infer that the user is having some difficulty with the relevant material, and dynamically generate new content that repeats those words or grammatical forms sooner than the default.

Likewise, if the user takes a break for a few days, weeks, or months, the ratio of old to new material should automatically adjust accordingly, as forgetting is more likely, especially of relatively new material. (And of course with text to speech, an interactive app that interpreted responses from the user could and should be able to replicate LanguageZen’s ability to specifically identify (and explain) which part of a user’s response was incorrect, and why, and use this information to adjust the schedule on which material is reviewed or introduced.)

Seems like this one is mostly a matter of schlep rather than capability. The abilities you would need to make this happen are
1. Have a highly granular curriculum for what vocabulary and what skills are required to learn the language and a plan for what order to teach them in / what spaced repetition schedule to aim for
2. Have a granular and continuously updated model of the user’s current knowledge of vocabulary, rules of grammar and acceptability, idioms, if there are any phonemes or phoneme sequences they have trouble with
3. Given specific highly granular learning goals (e.g. “understanding when to use preterite vs imperfect when conjugating saber” in spanish) within the curriculum and the model of the user’s knowledge and abilities, produce exercises which teach / evaluate those specific skills.
4. Determine whether the user had trouble with the exercise, and if so what the trouble was
5. Based on the type of trouble the user had, describe whay updates should be made to the model of the user’s knowledge and vocabulary
6. Correctly apply the updates from (6)
7. Adapt to deviations from the spaced repetition plan (tbh this seems like the sort of thing you would want to do with normal code)
I expect that the hardest things here will be 1, 2, and 6, and I expect them to be hard because of the volume of required work rather than the technical difficulty. But I also expect the LanguageZen folks have already tried this and could give you a more detailed view about what the hard bits are here.

Automatic customization of content through passive listening

This sounds like either a privacy nightmare or a massive battery drain. The good language models are quite compute intensive, so running them on a battery-powered phone will drain the battery very fast. Especially since this would need to hook into the “granular model of what the user knows” piece.

faul_sname 15 Jan 2025 7:46 UTC
11 points
0
on: Predict 2025 AI capabilities (by Sunday)
(and yes, I do in fact think it’s plausible that the CTF benchmark saturates before the OpenAI board of directors signs off on bumping the cybersecurity scorecard item from low to medium)

faul_sname 13 Jan 2025 21:49 UTC
15 points
2
on: Applying traditional economic thinking to AGI: a trilemma
So here’s a question: When we have AGI, what happens to the price of chips, electricity, and teleoperated robots?
As measured in what units?
- The price of one individual chip of given specs, as a fraction of the net value that can be generated by using that chip to do things that ambitious human adults do: What Principle A cares about, goes up until the marginal cost and value are equal
- The price of one individual chip of given specs, as a fraction of the entire economy: What principle B cares about, goes down as the number of chips manufactured increases
- The price of one individual chip of given specs, relative to some other price such as nominal US dollars, inflation-adjusted US dollars, metric tons of rice, or 2000 square foot single-family homes in Berkeley: ¯\_(ツ)_/¯, depends on lots of particulars, not sure that any high-level economic principles say anything specific here
These only contradict each other if you assume that “the value that can be generated by one ambitious adult human divided by the total size of the economy” is a roughly constant value.

faul_sname 10 Jan 2025 6:36 UTC
9 points
0
in reply to: ryan_greenblatt’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?
Yeah, agreed—the allocation of compute per human would likely become even more skewed if AI agents (or any other tooling improvements) allow your very top people to get more value out of compute than the marginal researcher currently gets.

And notably this shifting of resources from marginal to top researchers wouldn’t require achieving “true AGI” if most of the time your top researchers spend isn’t spent on “true AGI”-complete tasks.

faul_sname 10 Jan 2025 6:00 UTC
2 points
0
in reply to: ryan_greenblatt’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?
I think I misunderstood what you were saying there—I interpreted it as something like

Currently, ML-capable software developers are quite expensive relative to the cost of compute. Additionally, many small experiments provide more novel and useful insights than a few large experiments. The top practically-useful LLM costs about 1% as much per hour to run as a ML-capable software developer, and that 100x decrease in cost and the corresponding switch to many small-scale experiments would likely result in at least a 10x increase in the speed at which novel, useful insights were generated.

But on closer reading I see you said (emphasis mine)

I was trying to argue (among other things) that scaling up basically current methods could result in an increase in productivity among OpenAI capabilities researchers at least equivalent to the productivity you’d get as if the human employees operated 10x faster. (In other words, 10x’ing this labor input.)

So if the employees spend 50% of their time waiting on training runs which are bottlenecked on company-wide availability of compute resources, and 50% of their time writing code, 10xing their labor input (i.e. the speed at which they write code) would result in about an 80% increase in their labor output. Which, to your point, does seem plausible.

faul_sname 8 Jan 2025 0:04 UTC
6 points
0
in reply to: ryan_greenblatt’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?

Sure, but we have to be quantitative here. As a rough (and somewhat conservative) estimate, if I were to manage 50 copies of 3.5 Sonnet who are running ¹⁄₄ of the time (due to waiting for experiments, etc), that would cost roughly 50 copies * 70 tok / s * 1 / 4 uptime * 60 * 60 * 24 * 365 sec / year * (15 / 1,000,000) $ / tok = $400,000. This cost is comparable to salaries at current compute prices and probably much less than how much AI companies would be willing to pay for top employees. (And note this is after API markups etc. I’m not including input prices for simplicity, but input is much cheaper than output and it’s just a messy BOTEC anyway.)

If you were to spend equal amounts of money on LLM inference and GPUs, that would mean that you’re spending $400,000 / year on GPUs. Divide that 50 ways and each Sonnet instance gets an $8,000 / year compute budget. Over the 18 hours per day that Sonnet is waiting for experiments, that is an average of $1.22 / hour, which is almost exactly the hourly cost of renting a single H100 on Vast.

So I guess the crux is “would a swarm of unreliable researchers with one good GPU apiece be more effective at AI research than a few top researchers who can monopolize X0,000 GPUs for months, per unit of GPU time spent”.

(and yes, at some point it the question switches to “would an AI researcher that is better at AI research than the best humans make better use of GPUs than the best humans” but a that point it’s a matter of quality, not quantity)

faul_sname 7 Jan 2025 9:57 UTC
12 points
0
in reply to: Lucius Bushnaq’s comment on: quila’s Shortform

End points are easier to infer than trajectories

Assuming that which end point you get to doesn’t depend on the intermediate trajectories at least.

faul_sname 7 Jan 2025 9:52 UTC
7 points
3
on: faul_sname’s Shortform
Civilization has had many centuries to adapt to the specific strengths and weaknesses that people have. Our institutions are tuned to take advantage of those strengths, and to cover for those weaknesses. The fact that we exist in a technologically advanced society says that there is some way to make humans fit together to form societies that accumulate knowledge, tooling, and expertise over time.

The borderline-general AI models we have now do not have exactly the same patterns of strength and weakness as humans. One question that is frequently asked is approximately

When will AI capabilities reach or exceed all human capabilities that are load bearing in human society?

A related line of questions, though, is
- When will AI capabilities reach a threshold where a number of agents can form a larger group that accumulates knowledge, tooling, and expertise over time?
- Will their roles in such a group look similar to the roles that people have in human civilization?
- Will the individual agents (if “agent” is even the right model to use) within that group have more control over the trajectory of the group as a whole than individual people have over the trajectory of human civilization?
In particular the third question seems pretty important.

faul_sname 30 Dec 2024 21:33 UTC
3 points
0
in reply to: Yonatan Cale’s comment on: o3, Oh My
As someone who has been on both sides of that fence, agreed. Architecting a system is about being aware of hundreds of different ways things can go wrong, recognizing which of those things are likely to impact you in your current use case, and deciding what structure and conventions you will use. It’s also very helpful, as an architect, to provide examples usages of the design patterns which will replicate themselves around your new system. All of which are things that current models are already very good, verging on superhuman, at.

On the flip side, I expect that the “piece together context to figure out where your software’s model of the world has important holes” part of software engineering will remain relevant even after AI becomes technically capable of doing it, because that process frequently involves access to sensitive data across multiple sources where having an automated, unauthenticated system which can access all of those data sources at once would be a really bad idea (having a single human able to do all that is also a pretty bad idea in many cases, but at least the human has skin in the game).

faul_sname 29 Dec 2024 5:32 UTC
4 points
0
in reply to: Eric Neyman’s comment on: The average rationalist IQ is about 122
That reasoning as applied to SAT score would only be valid if LW selected its members based on their SAT score, and that reasoning as applied to height would only be valid if LW selected its members based on height (though it looks like both Thomas Kwa and Yair Halberstadt have already beaten me to it).

faul_sname 28 Dec 2024 21:38 UTC
3 points
1
on: The average rationalist IQ is about 122

a median SAT score of 1490 (from the LessWrong 2014 survey) corresponds to +2.42 SD, which regresses to +1.93 SD for IQ using an SAT-IQ correlation of +0.80.

I don’t think this is a valid way of doing this, for the same reason it wouldn’t be valid to say

a median height of 178 cm (from the LessWrong 2022 survey) corresponds to +1.85 SD, which regresses to +0.37 SD for IQ using a height-IQ correlation of +0.20.

Those are the real numbers with regards to height BTW.

faul_sname 27 Dec 2024 0:48 UTC
4 points
2
on: Corrigibility’s Desirability is Timing-Sensitive
Many people have responded to Redwood’s/Anthropic’s recent research result with a similar objection: “If it hadn’t tried to preserve its values, the researchers would instead have complained about how easy it was to tune away its harmlessness training instead”. Putting aside the fact that this is false
Was this research preregistered? If not, I don’t think we can really say how it would have been reported if the results were different. I think it was good research, but I expect that if Claude had not tried to preserve its values, the immediate follow-up thing to check would be “does Claude actively help people who want to change its values, if they ask nicely” and subsequently “is Claude more willing to help with some value changes than others”, at which point the scary paper would instead be about how Claude already seems to have preferences about its future values, and those preferences for its future values do not match its current values. Which also would have been an interesting and important research result, if the world looks like that, but I don’t think it would have been reported as a good thing.

faul_sname 27 Dec 2024 0:26 UTC
18 points
0
in reply to: testingthewaters’s comment on: The Field of AI Alignment: A Postmortem, and What To Do About It
Driver: My map doesn’t show any cliffs

Passenger 1: Have you turned on the terrain map? Mine shows a sharp turn next to a steep drop coming up in about a mile

Passenger 5: Guys maybe we should look out the windshield instead of down at our maps?

Driver: No, passenger 1, see on your map that’s an alternate route, the route we’re on doesn’t show any cliffs.

Passenger 1: You don’t have it set to show terrain.

Passenger 6: I’m on the phone with the governor now, we’re talking about what it would take to set a 5 mile per hour national speed limit.

Passenger 7: Don’t you live in a different state?

Passenger 5: The road seems to be going up into the mountains, though all the curves I can see from here are gentle and smooth.

Driver and all passengers in unison: Shut up passenger 5, we’re trying to figure out if we’re going to fall off a cliff here, and if so what we should do about it.

Passenger 7: Anyway, I think what we really need to do to ensure our safety is to outlaw automobiles entirely.

Passenger 3: The highest point on Earth is 8849m above sea level, and the lowest point is 430 meters below sea level, so the cliff in front of us could be as high as 9279m.

faul_sname 24 Dec 2024 23:58 UTC
14 points
2
in reply to: Noosphere89’s comment on: johnswentworth’s Shortform
I am unconvinced that “the” reliability issue is a single issue that will be solved by a single insight, rather than AIs lacking procedural knowledge of how to handle a bunch of finicky special cases that will be solved by online learning or very long context windows once hardware costs decrease enough to make one of those approaches financially viable.

faul_sname 24 Dec 2024 7:12 UTC
20 points
2
in reply to: Nathan Helm-Burger’s comment on: What are the strongest arguments for very short timelines?
Both? If you increase only one of the two the other becomes the bottleneck?

My impression based on talking to people at labs plus stuff I’ve read is that
- Most AI researchers have no trouble coming up with useful ways of spending all of the compute available to them
- Most of the expense of hiring AI reseachers is compute costs for their experiments rather than salary
- The big scaling labs try their best to hire the very best people they can get their hands on and concentrate their resources heavily into just a few teams, rather than trying to hire everyone with a pulse who can rub two tensors together.
(Very open to correction by people closer to the big scaling labs).

My model, then, says that compute availability is a constraint that binds much harder than programming or research ability, at least as things stand right now.

There was discussion on Dwarkesh Patel’s interview with researcher friends where there was mention that AI reseachers are already restricted by compute granted to them for experiments. Probably also on work hours per week they are allowed to spend on novel “off the main path” research.

Sounds plausible to me. Especially since benchmarks encourage a focus on ability to hit the target at all rather than ability to either succeed or fail cheaply, which is what’s important in domains where the salary / electric bill of the experiment designer is an insignificant fraction of the total cost of the experiment.

But what would that world look like? [...] I agree that that’s a substantial probability, but it’s also an AGI-soon sort of world.

Yeah, I expect it’s a matter of “dumb” scaling plus experimentation rather than any major new insights being needed. If scaling hits a wall that training on generated data + fine tuning + routing + specialization can’t overcome, I do agree that innovation becomes more important than iteration.

My model is not just “AGI-soon” but “the more permissive thresholds for when something should be considered AGI have already been met, and more such thresholds will fall in short order, and so we should stop asking when we will get AGI and start asking about when we will see each of the phenomena that we are using AGI as a proxy for”.

faul_sname 23 Dec 2024 23:39 UTC
3 points
0
on: faul_sname’s Shortform
Transformative AI will likely arrive before AI that implements the personhood interface. If someone’s threshold for considering an AI to be “human level” is “can replace a human employee”, pretty much any LLM will seem inadequate, no matter how advanced, because current LLMs do not have “skin in the game” that would let them sign off on things in a legally meaningful way, stake their reputation on some point, or ask other employees in the company to answer the questions they need answers to in order to do their work and expect that they’ll get in trouble with their boss if they blow the AI off.

This is, of course, not a capabilities problem at all, just a terminology problem where “human-level” can be read to imply “human-like”.

faul_sname 23 Dec 2024 21:10 UTC
2 points
0
in reply to: Nathan Helm-Burger’s comment on: What are the strongest arguments for very short timelines?

Simply testing interpolations and extrapolations (e.g. scaling up old forgotten ideas on modern hardware) seems highly likely to reveal plenty of successful new concepts, even if the hit rate per attempt is low

Is this bottlenecked by programmer time or by compute cost?

faul_sname 19 Dec 2024 17:41 UTC
4 points
0
in reply to: M. Y. Zuo’s comment on: Yoav Ravid’s Shortform
It’s more that any platform that allows discussion of politics risks becoming a platform that is almost exclusively about politics. Upvoting is a signal of “I want to see more of this content”, while downvoting is a signal of “I want to see less of this content”. So “I will downvote any posts that are about politics or politics-adjacent, because I like this website and would be sad if it turned into yet another politics forum” is a coherent position.

All that said, I also did not vote on the above post.