faul_sname

Karma: 4,116

faul_sname Apr 19, 2025, 7:44 AM
3 points
2
on: faul_sname’s Shortform
Prediction:
1. We will soon see the first high-profile example of “misaligned” model behavior where a model does something neither the user nor the developer want it to do, but which instead appears to be due to scheming.
2. On examination, the AI’s actions will not actually be a good way to accomplish that goal. Other instances of the same model will be capable of recognizing this.
3. The AI’s actions will make a lot of sense as an extrapolated of some contextually-activated behavior which led to better average performance on some benchmark.
That is to say, the traditional story is
1. We use RL to train AI
2. AI learns to predict reward
3. AI decides that its goal is to maximize reward
4. AI reasons about what behavior will lead to maximal reward
5. AI does something which neither its creators nor the user want it to do, but that thing serves the AI’s long term goals, or at least it thinks that’s the case
6. We all die when the AI releases a bioweapon (or equivalent) to ensure no future competition
7. The AI takes to the stars, but without us
My prediction here is
1. We use RL to train AI
2. AI learns to recognize what the likely loss/reward signal is for its current task
3. AI learns a heuristic like “if the current task seems to have a gameable reward and success seems unlikely by normal means, try to game the reward”
4. AI ends up in some real-world situation which it decides resembles an unwinnable task (it knows it’s not being evaluated, but that doesn’t matter)
5. AI decides that some random thing it just thought of looks like success criterion
6. AI thinks of some plan which has an outside chance of “working” by that success criterion it just came up with
7. AI does some random pants-on-head stupid thing which its creators don’t want, the user doesn’t want, and which doesn’t serve any plausible long-term goal.
8. We all die when the AI releases some dangerous bioweapon because doing so pattern-matches to some behavior that helped in training, but not actually in a way that kills everyone and not only after it can take over the roles humans had

faul_sname Apr 18, 2025, 5:54 PM
2 points
0
in reply to: jefftk’s comment on: Host Keys and SSHing to EC2
If you were to edit ~/.ssh/known_hosts to add an entry for each EC2 host you use, but put them all under the alias ec2, that would work.

So your ~/.ssh/known_hosts would look like
```
ec2 ssh-ed25519 AAAA...w7lG
ec2 ssh-ed25519 AAAA...CxL+
ec2 ssh-ed25519 AAAA...M5fX
```
That would mean that host key checking only works to say “is this any one of my ec2 instances” though.

Edit: You could also combine the two approaches, e.g. have
```
ec2 ssh-ed25519 AAAA...w7lG
ec2_01 ssh-ed25519 AAAA...w7lG
ec2 ssh-ed25519 AAAA...CxL+
ec2_02 ssh-ed25519 AAAA...CxL+
ec2 ssh-ed25519 AAAA...M5fX
ec2_nf ssh-ed25519 AAAA...M5fX
```
and leave ssh_ec2nf as doing ssh -o "StrictHostKeyChecking=yes" -o "HostKeyAlias=ec2nf" "$ADDR" while still having git, scp, etc work with $ADDR. If “I want to connect to these instances in an ad-hoc manner not already covered by my shell scripts” is a problem you ever run into. I kind of doubt it is, I was mainly responding to the “I don’t see how” part of your comment rather than claiming that doing so would be useful.

faul_sname Apr 18, 2025, 4:42 PM
2 points
0
in reply to: jefftk’s comment on: Host Keys and SSHing to EC2
I think Dagon is saying that any time you’re doing ssh -o "OptionKey=OptionValue" you can instead add OptionKey OptionValue under that host in your .ssh/config, which in this case might look like
```
Host ec2-*.compute-1.amazonaws.com
    HostKeyAlias aws-ec2-compute
    StrictHostKeyChecking yes
```
i.e. you would still need step 1 but not step 2 in the above post.

faul_sname Apr 17, 2025, 11:23 PM
4 points
0
on: faul_sname’s Shortform
Semi-crackpot hypothesis: we already know how to make LLM-based agents with procedural and episodic memory, just via having agents explicitly decide to start continuously tracking things and construct patterns of observation-triggered behavior.

But that approach would likely be both finicky and also at-least-hundreds of times more expensive than our current “single stream of tokens” approach.

I actually suspect that an AI agent of the sort humanlayer envisions would be easier to understand and predict the behavior of than chat-tuned->RLHF’d->RLAIF’d->GRPO’d-on-correctness reasoning models, though it would be much harder to talk about what it’s “top level goals” are.

faul_sname Apr 15, 2025, 9:07 PM
4 points
0
in reply to: samuelshadrach’s comment on: xpostah’s Shortform
Did you mean to imply something similar to the pizza index?

The Pizza Index refers to the sudden, trackable increase of takeout food orders (not necessarily of pizza) made from government offices, particularly the Pentagon and the White House in the United States, before major international events unfold.

Government officials order food from nearby restaurants when they stay late at the office to monitor developing situations such as the possibility of war or coup, thereby signaling that they are expecting something big to happen. This index can be monitored through open resources such as Google Maps, which show when a business location is abnormally busy.

If so, I think it’s a decent idea, but your phrasing may have been a bit unfortunate—I originally read it as a proposal to stalk AI lab employees.

faul_sname Apr 11, 2025, 12:28 AM
8 points
0
in reply to: Oxidize’s comment on: How familiar is the Lesswrong community as a whole with the concept of Reward-modelling?
Can you give one extremely concrete example of a scenario which involves reward modeling, and point to the part of the scenario that you call “reward modeling”?

faul_sname Apr 11, 2025, 12:13 AM
7 points
0
on: New Paper: Infra-Bayesian Decision-Estimation Theory
Alright, as I’ve mentioned before I’m terrible at abstract thinking, so I went through the post and came up with a concrete example. Does this seem about right?
We are running a quantitative trading firm, we care about the closing prices of the S&P 500 stocks. We have a forecasting system which is designed in a robust, “prediction market” style. Instead of having each model output a single precise forecast, each model $M^{j} (for j = 1, \dots, k)$
outputs a set of probability distributions over tomorrow’s closing prices. That is, for each action $a$ (for example, “buy” or “sell” a particular stock, or more generally “execute a trade decision”), our model returns a set
$M j (a) \subseteq Δ (F u t u r e P r i c e s), M^{j} (a) \subseteq Δ (Future Prices),$
which means that $M^{j} (a)$ is a nonempty, convex set of probability distributions over outcomes.
This definition allows “nature” (the market) to choose the distribution that actually occurs from within the set provided by our model. In our robust framework, we assume that the market might select the worst-case distribution in $M^{j} (a)$ . In other words, our multivalued model is a function
$M : A \to □ (O), M : A \to □ (O),$
with $□ (O)$ representing the collection of sets of probability distributions over outcomes. Instead of pinning down exactly what tomorrow’s price will be if we take a particular action, each model provides us a “menu” of plausible distributions over possible closing prices.
We do this because there is some hidden-to-us world state that we cannot measure directly, and this state might even be controlled adversarially by market forces. Instead of trying to infer or estimate the exact hidden state, we posit that there are a finite number of plausible candidates for what this hidden state might be. For each candidate hidden state, we associate one probability distribution over the closing prices. Thus, when we look at the model for an action, rather than outputting a single forecast, the model gives us a “menu” of distributions, each corresponding to one possible hidden state scenario. In this way, we deliberately refrain from committing to a single prediction about the hidden state, thereby preparing for a worst-case (or adversarial) realization.
Given a bettor $B$ , MDP $M$ and policy $π$ we define ${bet}_{M, π}^{B} : ([0, 1] \times (states \times acts \times [0, 1])^{[H]}) \to R$ , the aggregate betting function, as
${bet}_{M, π}^{B} (r_{0}, s_{1}, a_{1}, . . . r_{H}) := 1 + \sum_{h = 0}^{H} ({lbet}_{M, π}^{B, h, s_{h}, a_{h}} (r_{h}, s_{h + 1}, a_{h + 1}) - 1)$
Where $t r_{t}$ and $π_{t}$ are the trajectory in episode $t$ and policy in episode $t$ , respectively.
Next, our trading system aggregates these imprecise forecasts via a prediction-market-like mechanism. Inside our algorithm we maintain a collection of “bettors”. Each “bettor” (besides the pessimistic and uniform bettors, which do the obvious thing their names imply) corresponds to one of our underlying models (or to aspects of a model). Each bettor $B$ is associated with its own preferred prediction (derived from its hypothesis set) and a current “wealth” (i.e. credibility). Instead of simply choosing one model, every bettor places a bet based on (?)how well our market prediction aligns with its own view(?),
$Robust Universal Estimator (RUE)$
$Parameters: Hypothesis class H, rounds T, reward function r, prior ζ_{1}$
$ϵ \leftarrow min (\frac{1}{2}, \sqrt{\frac{ln (2)}{T}})$
$Function estimate(ζ, a):$
$r e t u r n argmin μ \in Δ O E B \sim ζ [if B \neq ∙, 2 D_{H}^{2} (μ \to M_{B} (a)), else ε \cdot E o \sim μ [r (a, o)]]$
$Function update(ζ, ¯ ¯¯¯¯ ¯ M, a, o):$
$f o r B \in H \cup {u} :$
$μ_{B} \leftarrow {argmin}_{μ \in M_{B} (a)} D_{H}^{2} (¯ ¯¯¯¯ ¯ M (a), μ)$
$ξ (B) \leftarrow ζ (B) \cdot (\sqrt{\frac{μ_{B} (o)}{¯ ¯¯¯ ¯ M (a) (o)}} + D_{H}^{2} (¯ ¯¯¯¯ ¯ M (a) \to M_{B} (a)))$
$ξ (∙) \leftarrow ζ (∙) \cdot (1 + ε (E_{o^{'} \sim ¯ ¯¯¯ ¯ M (a)} [r (a, o^{'})] - r (a, o)))$
$return ξ$
$for 1 \leq t \leq T :$
${ˆ M}_{t} \leftarrow λ a . estimate (ζ_{t}, a)$
$return {ˆ M}_{t}$
$Receive a_{t}, o_{t} from the environment$
$ζ_{t + 1} \leftarrow update (ζ_{t}, {ˆ M}_{t}, a_{t}, o_{t})$
$end$
To calculate our market prediction $ˆ M$ , we solve a convex minimization problem that balances the differing opinions of all our bettors (weighted by their current wealth $ζ$ ) in such a way that it (?)minimizes their expected value on update(?).
The key thing here is that we separate the predictable / non-adversarial parts of our environment from the possibly-adversarial ones, and so our market prediction $ˆ M$ reflects our best estimate of the outcomes of our actions if the parts of the universe we don’t observer are out to get us.

Is this a reasonable interpretation? If so, I’m pretty interested to see where you go with this.

faul_sname Apr 4, 2025, 9:53 PM
6 points
0
in reply to: Daniel Kokotajlo’s comment on: AI 2027: What Superintelligence Looks Like
In that case, “2027-level AGI agents are not yet data efficient but are capable of designing successors that solve the data efficiency bottleneck despite that limitation” seems pretty cruxy.

I probably want to bet against that. I will spend some time this weekend contemplating how that could be operationalized, and particularly trying to think of something where we could get evidence before 2027.

faul_sname Apr 4, 2025, 4:47 PM
7 points
1
in reply to: Steven Byrnes’s comment on: AI 2027: What Superintelligence Looks Like
Nope, I just misread. Over on ACX I saw that Scott had left a comment

Our scenario’s changes are partly due to change in intelligence, but also partly to change in agency/time horizon/planning, and partly serial speed. Data efficiency comes later, downstream of the intelligence explosion.

I hadn’t remembered reading that in the post Still “things get crazy before models get data-efficient” does sound like the sort of thing which could plausibly fit with the world model in the post (but would be understated if so). Then I re-skimmed the post, and in the October 2027 section I saw

The gap between human and AI learning efficiency is rapidly decreasing.

Agent-3, having excellent knowledge of both the human brain and modern AI algorithms, as well as many thousands of copies doing research, ends up making substantial algorithmic strides, narrowing the gap to an agent that’s only around 4,000x less compute-efficient than the human brain

and when I read that my brain silently did a s/compute-efficient/data-efficient.

Though now I am curious about the authors’ views on how data efficiency will advance over the next 5 years, because that seems very world-model-relevant.

faul_sname Apr 4, 2025, 10:33 AM
9 points
0
on: AI 2027: What Superintelligence Looks Like
Agent-3, having excellent knowledge of both the human brain and modern AI algorithms, as well as many thousands of copies doing research, ends up making substantial algorithmic strides, narrowing the gap to an agent that’s only around 4,000x less compute-efficient than the human brain
I recognize that this is not the main point of this document, but am I interpreting correctly that you anticipate that rapid recursive improvement in AI research / AI capabilities is cracked before sample efficiency is cracked (e.g. via active learning)?
If so, that does seem like a continuation of current trends, but the implications seem pretty wild. e.g.
- Most meme-worthy: We’ll get the discount sci-fi future where humanoid robots become commonplace, not because the human form is optimal, but because it lets AI systems piggyback off human imitation for physical tasks even when that form is wildly suboptimal for the job
- Human labor will likely become more valuable relative to raw materials, not less (as long as most humans are more sample efficient than the best AI). In a world where all repetitive, structured tasks can be automated, humans will be prized specifically for handling novel one-off tasks that remain abundant in the physical world
- Repair technicians and debuggers of physical and software systems become worth their weight in gold. The ability to say “This situation reminds me of something I encountered two years ago in Minneapolis” becomes humanity’s core value proposition
- Large portions of the built environment begin resembling Amazon warehouses—robot restricted areas and corridors specifically designed to minimize surprising scenarios, with humans stationed around the perimeter for exception handling
- We accelerate toward living in a panopticon, not primarily for surveillance, but because ubiquitous observation provides the massive datasets needed for AI training pipelines
Still, I feel like I have to be misinterpreting what you mean by “4,000x less sample efficient” here, because passages like the following don’t make sense under that interpretation

> The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. For many of their research ideas, the AIs immediately respond with a report explaining that their idea was tested in-depth 3 weeks ago and found unpromising.

faul_sname Apr 1, 2025, 10:09 PM
1 point
0
in reply to: lc’s comment on: LessWrong has been acquired by EA
As a newly-minted +1 strong upvote, I disagree, though I feel that this change reflects the level of care and attention to detail that I expect out of EA.

faul_sname Mar 21, 2025, 6:11 AM
2 points
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
I am not one of them—I was wondering the same thing, and was hoping you had a good answer.

If I was trying to answer this question, I would probably try to figure out what fraction of all economically-valuable labor each year was cognitive, the breakdown of which tasks comprise that labor, and the year-on-year productivity increases on those task, then use that to compute the percentage of economically-valuable labor that is being automated that year.

Concretely, to get a number for the US in 1900 I might use a weighted average of productivity increases across cognitive tasks in 1900, in an approach similar to how CPI is computed
- Look at the occupations listed in the 1900 census records
- Figure out which ones are common, and then sample some common ones and make wild guesses about what those jobs looked like in 1900
- Classify those tasks as cognitive or non-cognitive
- Come to estimate that record-keeping tasks are around a quarter to a half of all cognitive labor
- Notice that typewriters were starting to become more popular - about 100,000 typewriters sold per year
- Note that those 100k typewriters were going to the people who would save the most time by using them
- As such, estimate 1-2% productivity growth in record-keeping tasks in 1900
- Multiply the productivity growth for record-keeping tasks by the fraction of time (technically actually 1-1/productivity increase but when productivity increase is small it’s not a major factor)
- Estimate that 0.5% of cognitive labor was automated by specifically typewriters in 1900
- Figure that’s about half of all cognitive labor automation in 1900
and thus I would estimate ~1% of all cognitive labor was automated in 1900. By the same methodology I would probably estimate closer to 5% for 2024.

Again, though, I am not associated with Open Phil and am not sure if they think about cognitive task automation in the same way.

faul_sname Mar 20, 2025, 8:53 PM
3 points
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform

What fraction of economically-valuable cognitive labor is already being automated today?

Did e.g. a telephone operator in 1910 perform cognitive labor, by the definition we want to use here?

faul_sname Mar 18, 2025, 1:22 AM

2 points

in reply to: kman’s comment on: kman’s Shortform

Oh, indeed I was getting confused between those. So as a concrete example of your proof we could consider the following degenerate example case

def f(N: int) -> int:
    if N == 0x855bdad365f9331421ab4b13737917cf97b5e8d26246a14c9af1adb060f9724a:
        return 1
    else:
        return 0

def check(x: int, y: float) -> bool:
    return f(x) >= y

def argsat(y: float, max_search: int = 2**64) -> int or None:
    # We postulate that we have this function because P=NP
    if y > 1:
        return None
    elif y <= 0:
        return 0
    else:
        return 0x855bdad365f9331421ab4b13737917cf97b5e8d26246a14c9af1adb060f9724a

but we could also replace our degenerate f with e.g. sha256.

Is that the gist of your proof sketch?

faul_sname Mar 18, 2025, 1:05 AM
2 points
0
in reply to: kman’s comment on: kman’s Shortform
Finding the input x such that f(x) == argmax(f(x)) is left as an exercise for the reader though.

faul_sname Mar 18, 2025, 12:03 AM
14 points
15
in reply to: DAL’s comment on: DAL’s Shortform
Is Amodei forecasting that, in 3 to 6 months, AI will produce 90% of the value derived from written code, or just that AI will produce 90% of code, by volume? It would not surprise me if 90% of new “art” (defined as non-photographic, non-graph images) by volume is currently AI-generated, and I would not be surprised to see the same thing happen with code.

And in the same way that “AI produces 90% of art-like images” is not the same thing as “AI has solved art”, I expect “AI produces 90% of new lines of code” is not the same thing as “AI has solved software”.

faul_sname Mar 13, 2025, 8:33 PM
10 points
0
in reply to: Bogdan Ionut Cirstea’s comment on: Bogdan Ionut Cirstea’s Shortform
I’m skeptical.
Did the Sakana team publish the code that their scientist agent used to write the compositional regularization paper? The post says
For our choice of workshop, we believe the ICBINB workshop is a highly relevant choice for the purpose of our experiment. As we wrote in the main text, we selected this workshop because of its broader scope, challenging researchers (and our AI Scientist) to tackle diverse research topics that address practical limitations of deep learning, unlike most workshops with a narrow focus on one topic.
This workshop focuses particularly on understanding limitations of deep learning methods applied to real world problems, and encourages participants to study negative experimental outcomes. Some may criticize our choice of a workshop that encourages discussion of “negative results” (implying that papers discussing negative results are failed scientific discoveries), but we disagree, and we believe this is an important topic.
and while it is true that “negative results” are important to report, “we report a negative result because our AI agent put forward a reasonable and interesting hypothesis, competently tested the hypothesis, and found that the hypothesis was false” looks a lot like “our AI agent put forward a reasonable and interesting hypothesis, flailed around trying to implement it, had major implementation problems, and wrote a plausible-sounding paper describing its failure as a fact about the world rather than a fact about its skill level”.
The paper has a few places with giant red flags where it seems that the reviewer assumes that there were solid results that the author of the paper was simply not reporting skillfully, for example in section B2

,
I favor an alternative hypothesis: the Sakana agent determines where a graph belongs, what would be on the X and Y axis of that graph, what it expects that the graph would look like, and how to generate that graph. It then generates the graph and inserts the caption the graph would show if its hypothesis was correct. The agent has no particular ability to notice that its description doesn’t work with the graph.

faul_sname Mar 10, 2025, 4:20 PM
5 points
1
in reply to: Mateusz Bagiński’s comment on: johnswentworth’s Shortform
Plausibly going off into the woods decreases the median output while increasing the variance.

faul_sname Mar 8, 2025, 9:40 AM
2 points
0
on: faul_sname’s Shortform
Has anyone trained a model to, given a prompt-response pair and an alternate response, generate an alternate prompt which is close to the original and causes the alternate response to be generated with high probability?

I ask this because
1. It strikes me that many of the goals of interpretability research boil down to “figure out why models say the things they do, and under what circumstances they’d say different things instead”. If we could reliably ask the model and get an intelligible and accurate response back, that would almost trivialize this sort of research.
2. This task seems like it has almost ideal characteristics for training on—unlimited synthetic data, granular loss metric, easy for a human to see if the model is doing some weird reward hacky thing by spot checking outputs
A quick search found some vaguely adjacent research, but nothing I’d rate as a super close match.
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (2020) Automatically creates prompts by searching for words that make language models produce specific outputs. Related to the response-guided prompt modification task, but mainly focused on extracting factual knowledge rather than generating prompts for custom responses.
- RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022) Uses reinforcement learning to find the best text prompts by rewarding the model when it produces desired outputs. Similar to the response-guided prompt modification task since it tries to find prompts that lead to specific outputs, but doesn’t start with existing prompt-response pairs.
- GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models Makes simple edits to instructions to improve how well language models perform on tasks. Relevant because it changes prompts to get better results, but mainly focuses on improving existing instructions rather than creating new prompts for specific alternative responses.
- Large Language Models are Human-Level Prompt Engineers (2022) Uses language models themselves to generate and test many possible prompts to find the best ones for different tasks. Most similar to the response-guided prompt modification task as it creates new instructions to achieve better performance, though not specifically designed to match alternative responses.
If this research really doesn’t exist I’d find that really surprising, since it’s a pretty obvious thing to do and there are O(100,000) ML researchers in the world. And it is entirely possible that it does exist and I just failed to find it with a cursory lit review.

Anyone familiar with similar research / deep enough in the weeds to know that it doesn’t exist?

faul_sname Mar 8, 2025, 6:35 AM
6 points
0
in reply to: ztzuliios’s comment on: Lots of brief thoughts on Software Engineering
I think the ability to “just look up this code” is a demonstration of fluency—if your way of figuring out “what happens when I invoke this library function” is “read the source code”, that indicates that you are able to fluently read code.

That said, fluently reading code and fluently writing code are somewhat different skills, and the very best developers relative to their toolchain can do both with that toolchain.