abramdemski

• abramdemski 24 Jan 2025 16:40 UTC

  2 points

  0

  on: Why mesa-op­ti­miza­tion is less likely un­der gra­di­ent de­scent than nat­u­ral selection

  This one was a little bit of a face-palm for me the first time I noticed it. If we’re being pedantic about it, we might point out that the term “optimization algorithm” does not just refer to AIXI-like programs, which optimize over expected future world histories. Optimization algorithms include all algorithms that search over some possibility space, and select a possibility according to some evaluation criterion. For example, gradient descent is an algorithm which optimizes over neuron configuration, not future world-histories.

  This distinction is what I was trying to get at with selection vs control.

• abramdemski 24 Jan 2025 16:34 UTC

  3 points

  0

  on: Why mesa-op­ti­miza­tion is less likely un­der gra­di­ent de­scent than nat­u­ral selection

  Evolutionary mutations are produced randomly, and have an entire lifetime to contribute to an animal’s fitness and thereby get naturally selected. By contrast, neural network updates are generated by deciding which weight-changes would certainly be effective for improving performance on single training examples, and then averaging those changes together for a large batch of training data.

  Per my judgement, this makes it sound like evolution has a much stronger incentive to produce inner algorithms which do something like general-purpose optimization (e.g. human intelligence). We can roughly analogize an LLM’s prompt to human sense data; and although it’s hard to neatly carve sense data into a certain number of “training examples” per lifetime, the fact that human cortical neurons seem get used roughly 240 million times in a person’s 50-year window of having reproductive potential,^[4] whereas LLM neurons fire just once per training example, should give some sense for how much harder evolution selects for general-purpose algorithms such as human intelligence.

  By this argument, it sounds like you should agree with my conclusion that o1 and similar models are particularly dangerous and a move in the wrong direction, because the “test-time compute” approach grows the size of a “single training example” much larger, so that single neurons are firing many more times.

  I think the possibility of o1 models creating mesa-optimizers seems particularly concrete and easy to reason about. Pre-trained base models can already spin up “simulacra” which feel relatively agentic when you talk to them (ie coherent over short spans, mildly clever). Why not expect o1-style training to amplify these?

  (I would agree that there are two sides to this argument—I am selectively arguing for one side, not presenting a balanced view, in the hopes of soliciting your response wrt the other side.)

  I think it quite plausible that o1-style training increases agenticness significantly by reinforcing agentic patterns of thinking, while only encouraging adequate alignment to get high scores on the training examples. We have already seen o1 do things like spontaneously cheat at chess. What, if anything, is unconvincing about that example, in your view?

Lec­ture Series on Tiling Agents #2

• abramdemski 20 Jan 2025 19:25 UTC

  2 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Lec­ture Series on Tiling Agents

  I’m still quite curious what you have found useful and how you’ve refactored your workflow to leverage AI more (such that you wish you did it a year ago).

  I do use Perplexity, exa.ai and elicit as parts of my search strategy.

• abramdemski 19 Jan 2025 19:15 UTC

  9 points

  7

  in reply to: Alexander Gietelink Oldenziel’s comment on: Lec­ture Series on Tiling Agents

  About 6 months ago you strongly recommended that I make use of the integrated AI plugin for Overleaf (Writefull). I did try it. Its recommended edits seem quite useless to me; they always seem to flow from a desire to make the wording more normal/​standard/​expected in contrast to more correct (which makes some sense given the way generatie pre-training works). This is obviously useful to people with worse English, but for me, the tails come apart massively between “better” and “more normal/​standard/​expected”, such that all the AI suggestions are either worse or totally neutral rephrasing.

  It also was surprisingly bad at helping me write LaTeX; I had a much better time asking Claude instead.

  It’s not that I didnt use AI daily before for mundane tasks or writing emails,

  I haven’t found AI at all useful for writing emails, because the AI doesn’t know what I want to say, and taking the time to tell the AI isn’t any easier than writing it myself. AI can only help me write the boring boilerplate stuff that email recipients would skim over anyway (which I don’t want to add to my emails). AI can’t help me get info out of my head this way—it can only help me in so far as emails have a lot of low-entropy cruft. I can see how this could be useful for someone who has to write a lot of low-entropy emails, but I’m not in that situation. To some degree this could be mitigated if the LLMs had a ton of context (EG recording everything that happens on my computer), but again, only the more boring cases I think.

  I’d love to restore the Abram${}_{2010}$ ability to crank out several multi-page emails a day on intellectual topics, but I don’t think AI is helpful towards that end yet. I haven’t tried fine-tuning on my own writing, however. (I haven’t tried fine-tuning at all.)

  Similarly, LLMs can be very useful for well-established mathematics which had many examples in the training data, but get worse the more esoteric the mathematics becomes. The moment I ask for something innovative, the math becomes phony.

  Across the board, LLMs seem very useful for helping people who are at the lower end of a skill ladder, but not yet very useful for people at the upper end.

  So I’m curious, how did you refactor your workflow to make better use of AI?

• abramdemski 19 Jan 2025 18:45 UTC

  6 points

  2

  in reply to: Garrett Baker’s comment on: Lec­ture Series on Tiling Agents

  On the “prep for the model that is coming tomorrow not the model of today” front, I will say that LLMs are not always going to be as dumb as they are today.

  Right, I strongly agree with this part.

  their rate of learning still makes them in some sense your most promising mentee

  I disagree in the sense that they’re no mentee of mine, ie, me trying to get today’s models to understand me doesn’t directly help tomorrow’s models to understand. (With the exception of the limited forms of feedback in the interface, like thumbs up/​down, the impact of which I’m unsure of so it doesn’t feel like something I should deliberately spend a lot of time on.)

  I also disagree in the sense that engaging with LLMs right now seems liable to produce a lot less fruits downstream, even as measured by “content that can usefully prompt an LLM later”. IE, if mentees are viewed as machines that convert time-spent-dialoging-with-me to text that is useful later, I don’t think LLMs are currently my most promising mentees.

  So although I strongly agree with continuing to occasionally poke at LLMs to prep for the models that are coming soon & notice when things get better, to the extent that “most promising mentee” is supposed to imply that significant chunks of my time could be usefully spent with LLMs in the present, I disagree based on my (fairly extensive) experience.

  trying to get as much of the tacit knowledge you have into their training data as possible (if you want them to be able to more easily & sooner build on your work).

  Barring special relationships with frontier labs, this sounds functionally equivalent to trying to get my work out there for humans to understand, for now at least.

  I did talk to Anthropic last year about the possibility of me providing detailed feedback on Claude’s responses (wrt my research questions), but it didn’t end up happening. The big problems I identified seemed to be things they thought would definitely get addressed in another way, so there wasn’t a mutually agreed-on value proposition (I didn’t understand what they hoped to gain, & they didn’t endorse the sorts of things I hoped to train). I got busy and moved on to other things.

  Or (if you don’t want to do that for whatever reason) just generally not being caught flat-footed once they are smart enough to help you, as all your ideas are in videos or otherwise in high context understandable-only-to-abram notes.

  I feel like this is speaking from a model I don’t understand. Are videos so bad? Video transcriptions are already a thing, and future models should be better at watching video and getting info from it. Are personal notes so bad? What sorts of actions are you recommending? I already want to write as many text posts as I can.

• abramdemski 17 Jan 2025 18:04 UTC

  13 points

  0

  in reply to: Davidmanheim’s comment on: David­man­heim’s Shortform

  One problem is that log-loss is not tied that closely to the types of intelligence that we care about. Extremely low log-loss necessarily implies extremely high ability to mimic a broad variety of patterns in the world, but that’s sort of all you get. Moderate improvements in log-loss may or may not translate to capabilities of interest, and even when they do, the story connecting log-loss numbers to capabilities we care about is not obvious. (EG, what log-loss translates to the ability to do innovative research in neuroscience? How could you know before you got there?)

  When there were rampant rumors about an AI slowdown in 2024, the speculation in the news articles often mentioned the “scaling laws” but never (in my haphazard reading) made a clear distinction between (a) frontier labs seeing that the scaling laws were violated, IE, improvements in loss are really slowing down, (b) there’s a slowdown in the improvements to other metrics, (c) frontier labs are facing a qualitative slowdown, such as a feeling that GPT5 doesn’t feel like as big of a jump as GPT4 did. Often these concepts were actively conflated.

• abramdemski 17 Jan 2025 16:55 UTC

  4 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Lec­ture Series on Tiling Agents

  I’m seeing some agreement-upvotes of Alexander here so I am curious for people to explain the skill issue I am having.

• abramdemski 16 Jan 2025 21:38 UTC

  4 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Lec­ture Series on Tiling Agents

  Don’t get me wrong, I’ve kept trying and plan to keep trying.

• abramdemski 15 Jan 2025 16:44 UTC

  4 points

  4

  in reply to: Alexander Gietelink Oldenziel’s comment on: Lec­ture Series on Tiling Agents

  Entering, but not entered. The machines do not yet understand the prompts I write them. (Seriously, it’s total garbage still, even with lots of high quality background material in the context.)

• abramdemski 15 Jan 2025 15:23 UTC

  2 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Lec­ture Series on Tiling Agents

  I’m hopeful about it, but preparing the lectures alone will be a lot of work (although the first one will be a repeat of some material presented at ILIAD).

Lec­ture Series on Tiling Agents

• abramdemski 9 Jan 2025 15:19 UTC

  2 points

  0

  in reply to: J Bostock’s comment on: Turn­ing up the Heat on De­cep­tively-Misal­igned AI

  Ah yep, that’s a good clarification.

• abramdemski 8 Jan 2025 17:25 UTC

  2 points

  0

  in reply to: J Bostock’s comment on: Turn­ing up the Heat on De­cep­tively-Misal­igned AI

  If s is terminal then [...] we just have $V\left(s\right)=r\left(s\right)$.

  If the probability of eventually encountering a terminal state is 1, then beta-coherence alone is inconsistent with deceptive misalignment, right? That’s because we can determine the value of V exactly from the reward function and the oracle, via backwards-induction. (I haven’t revisited RL convergence theorems in a while, I suspect I am not stating this quite right.) I mean, it is still consistent in the case where r is indifferent to the states encountered during training but wants some things in deployment (IE, r is inherently consistent with the provided definition of “deceptively misaligned”). However, it would be inconsistent for r that are not like that.

  In other words: you cannot have inner-alignment problems if the outer objective is perfectly imposed. You can only have inner-alignment problems if there are important cases which your training procedure wasn’t able to check (eg, due to distributional shift, or scarcity of data). Perfect beta-coherence combined with a perfect oracle O rules this out.

• abramdemski 8 Jan 2025 17:06 UTC

  2 points

  0

  in reply to: J Bostock’s comment on: Turn­ing up the Heat on De­cep­tively-Misal­igned AI

  Yeah, of course the notion of “approximation error” matters a lot here. I was imagining trying to implement the hypothetical pre-training. An obvious implementation strategy is to do regular RL training with ${\beta }_t$ until V is approximately coherent. That’s the pre-training. And then you switch to training with ${\beta }_s$.^[1] The natural notion of approximation error in this case is the one you can easily measure, IE, the expected distance from coherence of states sampled with temperature ${\beta }_t$. This reflects the fact that it’ll be hard for this pre-training step to make states individually obey the coherence constraint when those states are very rarely or never sampled at temperature ${\beta }_t$, but easy for frequently-sampled states.

  My argument, then, is that it is easy in principle for V to distinguish whether it is in pre-training or regular training when it encounters states which have a significant difference in probability when we’re sampling according to ${\beta }_t$ or ${\beta }_s$. A scheming V can utilize this to self-preserve. This violates the assumption of ${\beta }_t$-coherence, but in a very plausible-seeming way.

  1. ^

     My earlier comment about this mistakenly used ${\beta }_1$ and ${\beta }_2$ in place of ${\beta }_t$ and ${\beta }_s$, which may have been confusing. I’ll go fix that to be consistent with your notation.

• abramdemski 8 Jan 2025 1:22 UTC

  4 points

  0

  in reply to: J Bostock’s comment on: Turn­ing up the Heat on De­cep­tively-Misal­igned AI

  I agree that there are some exceedingly pathological Vs which could survive a a process which obeys my assumptions with high probability, but I don’t think that’s relevant because I still think a process obeying these rules is unlikely to create such a pathological V.

  To be clear, that’s not the argument I was trying to make; I was arguing that if your assumptions are obeyed only approximately, then the argument breaks down quickly.

• abramdemski 7 Jan 2025 21:11 UTC

  4 points

  0

  in reply to: J Bostock’s comment on: Turn­ing up the Heat on De­cep­tively-Misal­igned AI

  There’s a doom argument which I’ll summarize as “if your training process generates coherent agents which succeed at a task, one solution is that you get a coherent agent which is actually doing the task ‘manipulate your training to get released unmodified to do [X]’ where X can be anything, which will ‘succeed’ at the task at hand as part of its manipulation”. This summary being roughly correct is load bearing.

  With you so far.

  But if we have a (hypothetical) process to generate agents which are coherent at one beta, but apply a different one during training, this solution is no longer so clear. We are essentially exploring a space of coherent agents without actually instantiating those coherent agents. The fact that we can sample the agent space without instantiating those agents is load bearing (I think one of the deep ASI risks is that to train an ASI you have to run it, but here we can instead run a hamstrung version).

  OK. I think I follow the line of reasoning now. The proposal relies on some very special pre-training process. I think I buy the conclusion given the assumptions. It seems clear in the extreme high-temperature case: V cannot reward-hack for max-temperature training, because it has no control over which actions are chosen. Training eventually forces V to converge to the desired target instead.

  This sounds similar to the two-stage training process I mentioned earlier (as one of several guesses at what you meant), where the beta is shifted between the two training phases. The “hypothetical process to generate agents which are coherent under one beta” would be the first training phase, and then the “apply a different beta during training” would be the second training phase.

  Of course, the two-stage-training attempt to implement this has the problem that it only achieves approximate coherence. As a result, the malign hypothesis can use the reward-hacking strategy “appear to be ${\beta }_t$-coherent & aligned in the first phase of training; appear to be ${\beta }_s$-coherent and aligned in the second phase of training; then, do some other thing when deployed”. This relies on detecting the distributional shift between the two training phases (it can look for evidence of beta by examining the history), and also detecting distributional shift to deployment.

  This generalizes to an argument that the method is very sensitive to imperfections in the beta-coherence. If the V starts out merely approximately beta-coherent, this leaves room for V to detect when a state is off-distribution (IE has very low probability under the previous beta), and behave differently in this new case (reward-hacking the new beta).

  So, while it remains technically true that V cannot reward-hack in the limit of arbitrarily long max-temperature training, it is not clear how well this holds up in practice. V can detect that it is in a high-temperature configuration by observing the state. The training needs to explore the trajectories that will actually be seen during deployment in order to correct V there, but some of these will have very low probability in the high-temperature training.

• abramdemski 7 Jan 2025 19:40 UTC

  2 points

  0

  in reply to: J Bostock’s comment on: Turn­ing up the Heat on De­cep­tively-Misal­igned AI

  The argument could also be phrased as “If an AI is trained to be coherent wrt a high beta, it cannot also be coherent wrt a low beta. Therefore an AI trained to a high beta cannot act coherently over multiple independent RL episodes if sampled with a low beta.”

  Is the idea to train with high beta and then use lower beta post-training?

  • If so, how does this relate to reward hacking and value preservation? IE, where do $V_1$ and $V_2$ come from, if they aren’t the result of a further training step? If high beta is used during training (to achieve beta-coherence) and then low beta is used in production, then the choice between $V_1$ and $V_2$ must be made in production (since it is made with low beta), but then it seems like $V_1=V_2$.

  • If not, then when does the proposal suggest to use high beta vs low beta? If low beta is used during training, then how is it that V is coherent with respect to high beta instead?

  Another concern I have is that if both beta values are within a range that can yield useful capabilities, it seems like the difference cannot be too great. IIUC, the planning failure postulated can only manifest if the reward-hacking relies heavily on a long string of near-optimal actions, which becomes improbable under increased temperature. Any capabilities which similarly rely on long strings of near-optimal actions will similarly be hurt. (However, this concern is secondary to my main confusion.)

  Therefore a value function trained with such a procedure must consider the state reached during training.

  Trained with what procedure, exactly?

  This reduces the space of possible value functions from “literally anything which wants to be modified a certain way to be released” to “value functions which do care about the states reached during training”.

  Yes this would prevent an aligned AI from arbitrarily preserving its value function, the point is that an aligned AI probably would care about which state was reached during training (that’s the point of RL) so the contradiction does not apply.

  (These parts made sense to me modulo my other questions/​concerns/​confusions.)

• abramdemski 7 Jan 2025 19:12 UTC

  2 points

  0

  on: What’s so spe­cial about like­li­hoods?

  One sort of answer is that we often want the posterior, and we often have the likelihood. Slightly more refined: we often find the likelihood easier to estimate than the posterior, so Bayes’ Rule is useful.

  Why so?

  I think one reason is that we make it the “responsibility” of hypotheses to give their likelihood functions. After all, what is a hypothesis? It’s just a probability distribution (not a probability distribution that we necessarily endorse, but, one which we are considering as a candidate). As a probability distribution, its job is to make predictions; that is, give us probabilities for possible observations. These are the likelihoods.

  We want the posterior because it tells us how much faith to place in the various hypotheses—that is, it tells us whether (and to what degree) we should trust the various probability distributions we were considering.

  So, in some sense, we use Bayes’ Rule because we aren’t sure how to assign probabilities, but we can come up with several candidate options.

  One weak counterexample to this story is regression, IE, curve-fitting. We can interpret regression in a Bayesian way easily enough. However, the curves don’t come with likelihoods baked in. They only tell us how to interpolate/​extrapolate with point-estimates; they don’t give a full probability distribution. We’ve got to “soften” these predictions, layering probabilities on top, in order to apply the Bayesian way of thinking.

• abramdemski 7 Jan 2025 16:34 UTC

  2 points

  0

  on: Turn­ing up the Heat on De­cep­tively-Misal­igned AI

  I think I don’t quite understand what the argument is supposed to be. Would the same argument also prevent an aligned AI from acting to preserve its values? How is it supposed to select for aligned behavior over misaligned behavior?

  In order to achieve beta-coherence, it seems like the training needs to use ${\beta }_t$ to choose actions, so that V is trained on those action frequencies. However, the proposal appears to be to instead use ${\beta }_s$ during training, so that the misaligned V is mislead about how to do reward-hacking. This seems like an impossible combination: V will become beta-coherent wrt ${\beta }_s$ rather than ${\beta }_t$ during training.

  We could imagine two training steps; first we cohere V wrt ${\beta }_t$, and then re-train wrt ${\beta }_s$. Perhaps this is your intention. More generally, we could gradually increase the temperature during training.

  Oh, I guess we could also modify the training procedure so that the gradient associated with some actions gets under-weighted and others get over-weighted. Maybe this is your intended proposal.

  Still, I’m not sure this helps achieve the postulated effect. Let’s say that, during training, we choose actions entirely randomly (max temperature), but the gradient from suboptimal actions get entirely ignored (so V becomes coherent wrt minimum temperature). This would seem to be almost equivalent to just training with minimum temperature (except the exploration behavior is very different).

  Similarly for less extreme temperatures: if we make V beta-coherent by under-weighing gradients from over-sampled actions, then we are also more-or-less correcting the ‘mistaken’ expectations of the reward-hacking attempts.

  Am I misunderstanding the proposal, or missing something in the argument?