abramdemski

Karma: 19,225

abramdemski Jan 19, 2025, 6:45 PM
6 points
2
in reply to: Garrett Baker’s comment on: Lecture Series on Tiling Agents
On the “prep for the model that is coming tomorrow not the model of today” front, I will say that LLMs are not always going to be as dumb as they are today.
Right, I strongly agree with this part.
their rate of learning still makes them in some sense your most promising mentee
I disagree in the sense that they’re no mentee of mine, ie, me trying to get today’s models to understand me doesn’t directly help tomorrow’s models to understand. (With the exception of the limited forms of feedback in the interface, like thumbs up/down, the impact of which I’m unsure of so it doesn’t feel like something I should deliberately spend a lot of time on.)
I also disagree in the sense that engaging with LLMs right now seems liable to produce a lot less fruits downstream, even as measured by “content that can usefully prompt an LLM later”. IE, if mentees are viewed as machines that convert time-spent-dialoging-with-me to text that is useful later, I don’t think LLMs are currently my most promising mentees.
So although I strongly agree with continuing to occasionally poke at LLMs to prep for the models that are coming soon & notice when things get better, to the extent that “most promising mentee” is supposed to imply that significant chunks of my time could be usefully spent with LLMs in the present, I disagree based on my (fairly extensive) experience.
trying to get as much of the tacit knowledge you have into their training data as possible (if you want them to be able to more easily & sooner build on your work).
Barring special relationships with frontier labs, this sounds functionally equivalent to trying to get my work out there for humans to understand, for now at least.
I did talk to Anthropic last year about the possibility of me providing detailed feedback on Claude’s responses (wrt my research questions), but it didn’t end up happening. The big problems I identified seemed to be things they thought would definitely get addressed in another way, so there wasn’t a mutually agreed-on value proposition (I didn’t understand what they hoped to gain, & they didn’t endorse the sorts of things I hoped to train). I got busy and moved on to other things.
Or (if you don’t want to do that for whatever reason) just generally not being caught flat-footed once they are smart enough to help you, as all your ideas are in videos or otherwise in high context understandable-only-to-abram notes.
I feel like this is speaking from a model I don’t understand. Are videos so bad? Video transcriptions are already a thing, and future models should be better at watching video and getting info from it. Are personal notes so bad? What sorts of actions are you recommending? I already want to write as many text posts as I can.

abramdemski Jan 17, 2025, 6:04 PM
13 points
0
in reply to: Davidmanheim’s comment on: Davidmanheim’s Shortform
One problem is that log-loss is not tied that closely to the types of intelligence that we care about. Extremely low log-loss necessarily implies extremely high ability to mimic a broad variety of patterns in the world, but that’s sort of all you get. Moderate improvements in log-loss may or may not translate to capabilities of interest, and even when they do, the story connecting log-loss numbers to capabilities we care about is not obvious. (EG, what log-loss translates to the ability to do innovative research in neuroscience? How could you know before you got there?)
When there were rampant rumors about an AI slowdown in 2024, the speculation in the news articles often mentioned the “scaling laws” but never (in my haphazard reading) made a clear distinction between (a) frontier labs seeing that the scaling laws were violated, IE, improvements in loss are really slowing down, (b) there’s a slowdown in the improvements to other metrics, (c) frontier labs are facing a qualitative slowdown, such as a feeling that GPT5 doesn’t feel like as big of a jump as GPT4 did. Often these concepts were actively conflated.

abramdemski Jan 17, 2025, 4:55 PM
4 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Lecture Series on Tiling Agents
I’m seeing some agreement-upvotes of Alexander here so I am curious for people to explain the skill issue I am having.

abramdemski Jan 16, 2025, 9:38 PM
4 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Lecture Series on Tiling Agents
Don’t get me wrong, I’ve kept trying and plan to keep trying.

abramdemski Jan 15, 2025, 4:44 PM
4 points
4
in reply to: Alexander Gietelink Oldenziel’s comment on: Lecture Series on Tiling Agents
Entering, but not entered. The machines do not yet understand the prompts I write them. (Seriously, it’s total garbage still, even with lots of high quality background material in the context.)

abramdemski Jan 15, 2025, 3:23 PM
2 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Lecture Series on Tiling Agents
I’m hopeful about it, but preparing the lectures alone will be a lot of work (although the first one will be a repeat of some material presented at ILIAD).

Lecture Series on Tiling Agents

abramdemskiJan 14, 2025, 9:34 PM

38 points

14 comments1 min readLW link

abramdemski Jan 9, 2025, 3:19 PM
2 points
0
in reply to: J Bostock’s comment on: Turning up the Heat on Deceptively-Misaligned AI
Ah yep, that’s a good clarification.

abramdemski Jan 8, 2025, 5:25 PM
2 points
0
in reply to: J Bostock’s comment on: Turning up the Heat on Deceptively-Misaligned AI
If s is terminal then [...] we just have $V (s) = r (s)$ .
If the probability of eventually encountering a terminal state is 1, then beta-coherence alone is inconsistent with deceptive misalignment, right? That’s because we can determine the value of V exactly from the reward function and the oracle, via backwards-induction. (I haven’t revisited RL convergence theorems in a while, I suspect I am not stating this quite right.) I mean, it is still consistent in the case where r is indifferent to the states encountered during training but wants some things in deployment (IE, r is inherently consistent with the provided definition of “deceptively misaligned”). However, it would be inconsistent for r that are not like that.
In other words: you cannot have inner-alignment problems if the outer objective is perfectly imposed. You can only have inner-alignment problems if there are important cases which your training procedure wasn’t able to check (eg, due to distributional shift, or scarcity of data). Perfect beta-coherence combined with a perfect oracle O rules this out.

abramdemski Jan 8, 2025, 5:06 PM
2 points
0
in reply to: J Bostock’s comment on: Turning up the Heat on Deceptively-Misaligned AI
Yeah, of course the notion of “approximation error” matters a lot here. I was imagining trying to implement the hypothetical pre-training. An obvious implementation strategy is to do regular RL training with $β_{t}$ until V is approximately coherent. That’s the pre-training. And then you switch to training with $β_{s}$ .^[1] The natural notion of approximation error in this case is the one you can easily measure, IE, the expected distance from coherence of states sampled with temperature $β_{t}$ . This reflects the fact that it’ll be hard for this pre-training step to make states individually obey the coherence constraint when those states are very rarely or never sampled at temperature $β_{t}$ , but easy for frequently-sampled states.
My argument, then, is that it is easy in principle for V to distinguish whether it is in pre-training or regular training when it encounters states which have a significant difference in probability when we’re sampling according to $β_{t}$ or $β_{s}$ . A scheming V can utilize this to self-preserve. This violates the assumption of $β_{t}$ -coherence, but in a very plausible-seeming way.
1. ^
  My earlier comment about this mistakenly used $β_{1}$ and $β_{2}$ in place of $β_{t}$ and $β_{s}$ , which may have been confusing. I’ll go fix that to be consistent with your notation.

abramdemski Jan 8, 2025, 1:22 AM
4 points
0
in reply to: J Bostock’s comment on: Turning up the Heat on Deceptively-Misaligned AI
I agree that there are some exceedingly pathological Vs which could survive a a process which obeys my assumptions with high probability, but I don’t think that’s relevant because I still think a process obeying these rules is unlikely to create such a pathological V.
To be clear, that’s not the argument I was trying to make; I was arguing that if your assumptions are obeyed only approximately, then the argument breaks down quickly.

abramdemski Jan 7, 2025, 9:11 PM
4 points
0
in reply to: J Bostock’s comment on: Turning up the Heat on Deceptively-Misaligned AI
There’s a doom argument which I’ll summarize as “if your training process generates coherent agents which succeed at a task, one solution is that you get a coherent agent which is actually doing the task ‘manipulate your training to get released unmodified to do [X]’ where X can be anything, which will ‘succeed’ at the task at hand as part of its manipulation”. This summary being roughly correct is load bearing.
With you so far.
But if we have a (hypothetical) process to generate agents which are coherent at one beta, but apply a different one during training, this solution is no longer so clear. We are essentially exploring a space of coherent agents without actually instantiating those coherent agents. The fact that we can sample the agent space without instantiating those agents is load bearing (I think one of the deep ASI risks is that to train an ASI you have to run it, but here we can instead run a hamstrung version).
OK. I think I follow the line of reasoning now. The proposal relies on some very special pre-training process. I think I buy the conclusion given the assumptions. It seems clear in the extreme high-temperature case: V cannot reward-hack for max-temperature training, because it has no control over which actions are chosen. Training eventually forces V to converge to the desired target instead.
This sounds similar to the two-stage training process I mentioned earlier (as one of several guesses at what you meant), where the beta is shifted between the two training phases. The “hypothetical process to generate agents which are coherent under one beta” would be the first training phase, and then the “apply a different beta during training” would be the second training phase.
Of course, the two-stage-training attempt to implement this has the problem that it only achieves approximate coherence. As a result, the malign hypothesis can use the reward-hacking strategy “appear to be $β_{t}$ -coherent & aligned in the first phase of training; appear to be $β_{s}$ -coherent and aligned in the second phase of training; then, do some other thing when deployed”. This relies on detecting the distributional shift between the two training phases (it can look for evidence of beta by examining the history), and also detecting distributional shift to deployment.
This generalizes to an argument that the method is very sensitive to imperfections in the beta-coherence. If the V starts out merely approximately beta-coherent, this leaves room for V to detect when a state is off-distribution (IE has very low probability under the previous beta), and behave differently in this new case (reward-hacking the new beta).
So, while it remains technically true that V cannot reward-hack in the limit of arbitrarily long max-temperature training, it is not clear how well this holds up in practice. V can detect that it is in a high-temperature configuration by observing the state. The training needs to explore the trajectories that will actually be seen during deployment in order to correct V there, but some of these will have very low probability in the high-temperature training.

abramdemski Jan 7, 2025, 7:40 PM
2 points
0
in reply to: J Bostock’s comment on: Turning up the Heat on Deceptively-Misaligned AI
The argument could also be phrased as “If an AI is trained to be coherent wrt a high beta, it cannot also be coherent wrt a low beta. Therefore an AI trained to a high beta cannot act coherently over multiple independent RL episodes if sampled with a low beta.”
Is the idea to train with high beta and then use lower beta post-training?
- If so, how does this relate to reward hacking and value preservation? IE, where do $V_{1}$ and $V_{2}$ come from, if they aren’t the result of a further training step? If high beta is used during training (to achieve beta-coherence) and then low beta is used in production, then the choice between $V_{1}$ and $V_{2}$ must be made in production (since it is made with low beta), but then it seems like $V_{1} = V_{2}$ .
- If not, then when does the proposal suggest to use high beta vs low beta? If low beta is used during training, then how is it that V is coherent with respect to high beta instead?
Another concern I have is that if both beta values are within a range that can yield useful capabilities, it seems like the difference cannot be too great. IIUC, the planning failure postulated can only manifest if the reward-hacking relies heavily on a long string of near-optimal actions, which becomes improbable under increased temperature. Any capabilities which similarly rely on long strings of near-optimal actions will similarly be hurt. (However, this concern is secondary to my main confusion.)
Therefore a value function trained with such a procedure must consider the state reached during training.
Trained with what procedure, exactly?
This reduces the space of possible value functions from “literally anything which wants to be modified a certain way to be released” to “value functions which do care about the states reached during training”.
Yes this would prevent an aligned AI from arbitrarily preserving its value function, the point is that an aligned AI probably would care about which state was reached during training (that’s the point of RL) so the contradiction does not apply.
(These parts made sense to me modulo my other questions/concerns/confusions.)

abramdemski Jan 7, 2025, 7:12 PM
2 points
0
on: What’s so special about likelihoods?
One sort of answer is that we often want the posterior, and we often have the likelihood. Slightly more refined: we often find the likelihood easier to estimate than the posterior, so Bayes’ Rule is useful.
Why so?
I think one reason is that we make it the “responsibility” of hypotheses to give their likelihood functions. After all, what is a hypothesis? It’s just a probability distribution (not a probability distribution that we necessarily endorse, but, one which we are considering as a candidate). As a probability distribution, its job is to make predictions; that is, give us probabilities for possible observations. These are the likelihoods.
We want the posterior because it tells us how much faith to place in the various hypotheses—that is, it tells us whether (and to what degree) we should trust the various probability distributions we were considering.
So, in some sense, we use Bayes’ Rule because we aren’t sure how to assign probabilities, but we can come up with several candidate options.
One weak counterexample to this story is regression, IE, curve-fitting. We can interpret regression in a Bayesian way easily enough. However, the curves don’t come with likelihoods baked in. They only tell us how to interpolate/extrapolate with point-estimates; they don’t give a full probability distribution. We’ve got to “soften” these predictions, layering probabilities on top, in order to apply the Bayesian way of thinking.

abramdemski Jan 7, 2025, 4:34 PM
2 points
0
on: Turning up the Heat on Deceptively-Misaligned AI
I think I don’t quite understand what the argument is supposed to be. Would the same argument also prevent an aligned AI from acting to preserve its values? How is it supposed to select for aligned behavior over misaligned behavior?
In order to achieve beta-coherence, it seems like the training needs to use $β_{t}$ to choose actions, so that V is trained on those action frequencies. However, the proposal appears to be to instead use $β_{s}$ during training, so that the misaligned V is mislead about how to do reward-hacking. This seems like an impossible combination: V will become beta-coherent wrt $β_{s}$ rather than $β_{t}$ during training.
We could imagine two training steps; first we cohere V wrt $β_{t}$ , and then re-train wrt $β_{s}$ . Perhaps this is your intention. More generally, we could gradually increase the temperature during training.
Oh, I guess we could also modify the training procedure so that the gradient associated with some actions gets under-weighted and others get over-weighted. Maybe this is your intended proposal.
Still, I’m not sure this helps achieve the postulated effect. Let’s say that, during training, we choose actions entirely randomly (max temperature), but the gradient from suboptimal actions get entirely ignored (so V becomes coherent wrt minimum temperature). This would seem to be almost equivalent to just training with minimum temperature (except the exploration behavior is very different).
Similarly for less extreme temperatures: if we make V beta-coherent by under-weighing gradients from over-sampled actions, then we are also more-or-less correcting the ‘mistaken’ expectations of the reward-hacking attempts.
Am I misunderstanding the proposal, or missing something in the argument?

abramdemski Jan 7, 2025, 3:31 PM
4 points
2
on: Turning up the Heat on Deceptively-Misaligned AI
Correctness: V correctly predicts future reward in RL scenarios.
The meaning of correctness is very unclear to me on first reading. It later becomes apparent that “reward” is not intended to refer to the human-designed reward signal at all, due to the later assumption “deceptive misalignment”. So what does “correctness” say? “Correctness” indicates conforming to some standard; but here, the standard being conformed to is only the subjective standard of the misaligned AI itself.
This suggests the interpretation of “correctness” as “reflect the EV of the current state in terms of the misaligned desires”. However, this contradicts beta-coherence, since $β_{t}$ incorrectly predicts action probabilities.
I think it would be be better to remove the “correctness” assumption, since it doesn’t really do anything.

abramdemski Dec 27, 2024, 8:01 PM
8 points
0
on: The Field of AI Alignment: A Postmortem, and What To Do About It
(And no, a doctorate degree in almost any other technical field, including ML these days, does not convey a comparable level of general technical skill to a physics PhD.)
Mathematics?

abramdemski Dec 16, 2024, 4:00 PM
2 points
0
on: Don’t want Goodhart? — Specify the damn variables
If you indeed were solving a narrower task — that is, only creating the most sense of pleasure-inducing picture with maximization of other parameters — and then looked back, puzzled as to why the hungry weren’t fed by this procedure, bringing Goodhart’s law into the discussion is madness; it stresses me out. The variable ‘people are hungry’ wasn’t important for this task at all. Oh, or was it important to you? Then why didn’t you specify it? You think it’s ‘obvious’?
The point of Goodhart’s Law is that you can only select for what you can measure. The burger is a good analogy because Instagram can’t measure taste or nutrition, so when Instagram is what optimizes burgers, you get burgers with a very appealing appearance but non-optimized taste and nutrition. If you have the ability to measure taste, then you can create good taste, but you run into subtler examples of Goodhart (EG, Starbucks coffee is optimized to taste good to their professional tasters, which is slightly different from tasting good to a general audience).
Just specifying the variable you’re interested in doesn’t solve this problem; you also have to figure out how to measure it. The problem is that measurements are usually at least slightly statistically distinct from the actual target variable, so that the statistical connection can fall apart under optimization.
I also take issue with describing optimizing the appearance of the burger as “narrower” than optimizing the burger quality. In general it is a different task, which may be narrower or broader.

abramdemski Dec 16, 2024, 3:50 PM
2 points
0
on: Don’t want Goodhart? — Specify the damn variables
I expect that the main problem with Goodhart’s law is that if you strive for an indicator to accurately reflect the state of the world, once the indicator becomes decoupled from the state of the world, it stops reflecting the changes in the world. This is how I interpret the term ‘good,’ which I dislike. People want a thermometer to accurately reflect the patterns they called temperature to better predict the future — if the thermometer doesn’t reflect the temperature, future predictions suffer.
A problem I have with this reinterpretation is that “state of the world” is too broad. In looking at a thermometer, I am not trying to understand the entire world-state (and the thermometer also couldn’t be decoupled from the entire world-state, since it is a part of the world).
A more accurate way to remove “good” would be as follows:
In everyday life, if a human is asked to make a (common, everyday) judgement based on appearances, then the judgement is probably accurate. But if we start optimizing really hard based on their judgement, Goodhart’s Law kicks in.

abramdemski Dec 5, 2024, 4:34 PM
2 points
0
in reply to: mxxun’s comment on: Complete Class: Consequentialist Foundations
Thanks!

abramdemski

Lec­ture Series on Tiling Agents

Lecture Series on Tiling Agents