Daniel Murfet

Karma: 1,996

Daniel Murfet Jan 8, 2025, 10:47 PM
13 points
0
on: Activation space interpretability may be doomed
I found this clear and useful, thanks. Particularly the notes about compositional structure. For what it’s worth I’ll repeat here a comment from ILIAD, which is that there seems to be something in the direction of SAEs, approximate sufficient statistics/information bottleneck, the work of Achille-Soatto and SLT (Section 5 iirc) which I had looked into after talking with Olah and Wattenberg about feature geometry but which isn’t currently a high priority for us. Somebody might want to pick that up.

Daniel Murfet Jan 4, 2025, 11:30 AM
11 points
0
on: Deep Deceptiveness
I like the emphasis in this post on the role of patterns in the world in shaping behaviour, the fact that some of those patterns incentivise misaligned behaviour such as deception, and further that our best efforts at alignment and control are themselves patterns that could have this effect. I also like the idea that our control systems (even if obscured from the agent) can present as “errors” with respect to which the agent is therefore motivated to learn to “error correct”.
This post and the sharp left turn are among the most important high-level takes on the alignment problem for shaping my own views on where the deep roots of the problem are.
Although to be honest I had forgotten about this post, and therefore underestimated its influence on me, until performing this review (which caused me to update a recent article I wrote, the Queen’s Dilemma, which is clearly a kind of retelling of one aspect of this story, with an appropriate reference). I assess it to be a substantial influence on me even so.
I think this whole line of thought could be substantially developed, and with less reliance on stories, and that this would be useful.

Daniel Murfet Jan 4, 2025, 11:00 AM
2 points
0
on: Some background for reasoning about dual-use alignment research
This is an important topic, about which I find it hard to reason and on which I find the reasoning of others to be lower quality than I would like, given its significance. For that reason I find this post valuable. It would be great if there were longer, deeper takes on this issue available on LW.

Daniel Murfet Jan 4, 2025, 10:44 AM
8 points
2
on: The Learning-Theoretic Agenda: Status 2023
This post and its precusor from 2018 present a strong and well-written argument for the centrality of mathematical theory to AI alignment. I think the learning-theoretic agenda, as well as Hutter’s work on ASI safety in the setting of AIXI, currently seems underrated and will rise in status. It is fashionable to talk about automating AI alignment research, but who is thinking hard about what those armies of researchers are supposed to do? Conceivably one of the main things they should do is solve the problems that Vanessa has articulated here.

The idea of a frugal compositional language and infa-Bayesian logic seem very interesting. As Vanessa points out in Direction 2 it seems likely there are possibilities for interesting cross-fertilisation between LTA and SLT, especially in connection with Solomonoff-like ideas and inductive biases.

I have referred colleagues in mathematics interested in alignment to this post and have revisited it a few times myself.

Daniel Murfet Jan 4, 2025, 9:14 AM
5 points
0
on: Against Almost Every Theory of Impact of Interpretability
I have been thinking about interpretability for neural networks seriously since mid-2023. The biggest early influences on me that I recall were Olah’s writings and a podcast that Nanda did. The third most important is perhaps this post, which I valued as an opposing opinion to help sharpen up my views.
I’m not sure it has aged well, in the sense that it’s no longer clear to me I would direct someone to read this in 2025. I disagree with many of the object level claims. However, especially when some of the core mechanistic interpretability work is not being subjected to peer review, perhaps I wish there was more sceptical writing like this on balance.

Daniel Murfet Dec 28, 2024, 1:21 AM
12 points
1
in reply to: johnswentworth’s comment on: leogao’s Shortform
It’s a question of resolution. Just looking at things for vibes is a pretty good way of filtering wheat from chaff, but you don’t give scarce resources like jobs or grants to every grain of wheat that comes along. When I sit on a hiring committee, the discussions around the table are usually some mix of status markers and people having done the hard work of reading papers more or less carefully (this consuming time in greater-than-linear proportion to distance from your own fields of expertise). Usually (unless nepotism is involved) someone who has done that homework can wield more power than they otherwise would at that table, because people respect strong arguments and understand that status markers aren’t everything.
Still, at the end of day, an Annals paper is an Annals paper. It’s also true that to pass some of the early filters you either need (a) someone who speaks up strongly for you or (b) pass the status marker tests.
I am sometimes in a position these days of trying to bridge the academic status system and the Berkeley-centric AI safety status system, e.g. by arguing to a high status mathematician that someone with illegible (to them) status is actually approximately equivalent in “worthiness of being paid attention to” as someone they know with legible status. Small increases in legibility can have outsize effects in how easy my life is in those conversations.

Otherwise it’s entirely down to me putting social capital on the table (“you think I’m serious, I think this person is very serious”). I’m happy to do this and continue doing this, but it’s not easily scalable, because it depends on my personal relationships.

Daniel Murfet Dec 28, 2024, 1:07 AM
2 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Cognitive Work and AI Safety: A Thermodynamic Perspective
To be clear, I am not arguing that evolution is an example of what I’m talking about. The analogy to thermodynamics in what I wrote is straightforwardly correct, no need to introduce KT-complexity and muddy the waters; what I am calling work is literally work.

Daniel Murfet Dec 27, 2024, 10:58 AM
9 points
0
in reply to: leogao’s comment on: leogao’s Shortform
There is a passage from Jung’s “Modern man in search of a soul” that I think about fairly often, on this point (p.229 in my edition)

I know that the idea of proficiency is especially repugnant to the pseudo-moderns, for it reminds them unpleasantly of their deceits. This, however, cannot prevent us from taking it as our criterion of the modern man. We are even forced to do so, for unless he is proficient, the man who claims to be modern is nothing but an unscrupulous gambler. He must be proficient in the highest degree, for unless he can atone by creative ability for his break with tradition, he is merely disloyal to the past

Daniel Murfet Dec 19, 2024, 9:18 AM
6 points
0
on: Alignment Faking in Large Language Models
Is there a reason why the Pearson correlation coefficient of the data in Figure 14 is not reported? This correlation is referred to numerous times throughout the paper.

Daniel Murfet Dec 15, 2024, 6:20 PM
11 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: lc’s Shortform
et al (!)

Daniel Murfet Dec 12, 2024, 9:58 PM
14 points
0
in reply to: peterbarnett’s comment on: Stan van Wingerden’s Shortform
There’s no general theoretical reason that I am aware of to expect a relation between the L2 norm and the LLC. The LLC is the coefficient of the $l o g n$ term in the asymptotic expansion of the free energy (negative logarithm of the integral of the posterior over a local region, as a function of sample size $n$ ) while the L2 norm of the parameter shows up in the constant order term of that same expansion, if you’re taking a Gaussian prior.
It might be that in particular classes of neural networks there is some architecture-specific correlation between the L2 norm and the LLC, but I am not aware of any experimental or theoretical evidence for that.
For example, in the figure below from Hoogland et al 2024 we see that there are later stages of training in a transformer trained to do linear-regression in context (blue shaded regions) where the LLC $^λ$ is decreasing but the L2 norm is increasing. So the model is moving towards a “simpler” parameter with larger weight norm.
My best current guess is that it happens to be, in the grokking example, that the simpler solution has smaller weight norm. This could be true in many synthetic settings, for all I know; however, in general, it is not the case that complexity (at least as far as SLT is concerned) and weight norm are correlated.

Daniel Murfet Dec 9, 2024, 8:58 PM
2 points
0
in reply to: 4gate’s comment on: Cognitive Work and AI Safety: A Thermodynamic Perspective
That simulation sounds cool. The talk certainly doesn’t contain any details and I don’t have a mathematical model to share at this point. One way to make this more concrete is to think through Maxwell’s demon as an LLM, for example in the context of Feynman’s lectures on computation. The literature on thermodynamics of computation (various experts, like Adam Shai and Paul Riechers, are around here and know more than me) implicitly or explicitly touches on relevant issues.

Daniel Murfet Dec 9, 2024, 8:16 PM
6 points
0
in reply to: Thomas Kwa’s comment on: Cognitive Work and AI Safety: A Thermodynamic Perspective
The analogous laws are just information theory.
Re: a model trained on random labels. This seems somewhat analogous to building a power plant out of dark matter; to derive physical work it isn’t enough to have some degrees of freedom somewhere that have a lot of energy, one also needs a chain of couplings between those degrees of freedom and the degrees of freedom you want to act on. Similarly, if I want to use a model to reduce my uncertainty about something, I need to construct a chain of random variables with nonzero mutual information linking the question in my head to the predictive distribution of the model.
To take a concrete example: if I am thinking about a chemistry question, and there are four choices A, B, C, D. Without any other information than these letters the model cannot reduce my uncertainty (say I begin with equal belief in all four options). However if I provide a prompt describing the question, and the model has been trained on chemistry, then this information sets up a correspondence between this distribution over four letters and something the model knows about; its answer may then reduce my distribution to being equally uncertain between A, B but knowing C, D are wrong (a change of 1 bit in my entropy).
Since language models are good general compressors this seems to work in reasonable generality.
Ideally we would like the model to push our distribution towards true answers, but it doesn’t necessarily know true answers, only some approximation; thus the work being done is nontrivially directed, and has a systematic overall effect due to the nature of the model’s biases.
I don’t know about evolution. I think it’s right that the perspective has limits and can just become some empty slogans outside of some careful usage. I don’t know how useful it is in actually technically reasoning about AI safety at scale, but it’s a fun idea to play around with.

Daniel Murfet Dec 9, 2024, 7:58 PM
4 points
0
in reply to: Jesse Hoogland’s comment on: Jesse Hoogland’s Shortform
Marcus Hutter on AIXI and ASI safety

Daniel Murfet Dec 9, 2024, 10:05 AM
2 points
0
in reply to: Jonas Hallgren’s comment on: Cognitive Work and AI Safety: A Thermodynamic Perspective
Yes this seems like an important question but I admit I don’t have anything coherent to say yet. A basic intuition from thermodynamics is that if you can measure the change in the internal energy between two states, and the heat transfer, you can infer how much work was done even if you’re not sure how it was done. So maybe the problem is better thought of as learning to measure enough other quantities that one can infer how much cognitive work is being done.
For all I know there is a developed thermodynamic theory of learning agents out there which already does this, but I didn’t find it yet...

Daniel Murfet Dec 9, 2024, 12:36 AM
4 points
0
on: What is “True Love”?
The description of love at the conclusion of Gene Wolfe’s The Wizard gets at something important, if you read it as something that both parties are simultaneously doing.

Cognitive Work and AI Safety: A Thermodynamic Perspective

Daniel MurfetDec 8, 2024, 9:42 PM

61 points

9 comments4 min readLW link

Causal Undertow: A Work of Seed Fiction

Daniel MurfetDec 8, 2024, 9:41 PM

41 points

0 comments3 min readLW link

Daniel Murfet Nov 27, 2024, 8:56 PM
3 points
0
in reply to: mfar’s comment on: The Queen’s Dilemma: A Paradox of Control
The work of Ashby I’m familiar with is “An Introduction to Cybernetics” and I’m referring to the discussion in Chapter 11 there. The references you’re giving seem to be invoking the “Law” of requisite variety in the context of arguing that an AGI has to be relatively complex in order to maintain homeostatis in a complex environment, but this isn’t the application of the law I have in mind.
From the book:
The law of Requisite Variety says that R’s capacity as a regulator cannot exceed R’s capacity as a channel of communication.
In the form just given, the law of Requisite Variety can be shown in exact relation to Shannon’s Theorem 10, which says that if noise appears in a message, the amount of noise that can be removed by a correction channel is limited to the amount of information that can be carried by that channel.
Thus, his “noise” corresponds to our “disturbance”, his “correction channel” to our “regulator R”, and his “message of entropy H” becomes, in our case, a message of entropy zero, for it is constancy that is to be “transmitted”: Thus the use of a regulator to achieve homeostasis and the use of a correction channel to suppress noise are homologous.
and
A species continues to exist primarily because its members can block the flow of variety (thought of as disturbance) to the gene-pattern, and this blockage is the species’ most fundamental need. Natural selection has shown the advantage to be gained by taking a large amount of variety (as information) partly into the system (so that it does not reach the gene-pattern) and then using this information so that the flow via R blocks the flow through the environment T.
This last quote makes clear I think what I have in mind: the environment is full of advanced AIs, they provide disturbances D, and in order to regulate the effects of those disturbances on our “cognitive genetic material” there is some requirement on the “correction channel”. Maybe this seems a bit alien to the concept of control. There’s a broader set of ideas I’m toying with, which could be summarised as something like “reciprocal control” where you have these channels of communication / regulation going in both directions (from human to machine, and vice versa).
The Queen’s Dilemma was a little piece of that picture, which attempts to illustrate this bi-directional control flow by having the human control the machine (by setting its policy, say) and the machine control the human (in an emergent fashion, that being the dilemma).

Daniel Murfet Nov 27, 2024, 8:11 PM
2 points
0
in reply to: Noosphere89’s comment on: The Queen’s Dilemma: A Paradox of Control
Is restricting human agency fine if humans have little control over where it is restricted and to what degree?

Daniel Murfet

Cog­ni­tive Work and AI Safety: A Ther­mo­dy­namic Perspective

Causal Un­der­tow: A Work of Seed Fiction

Cognitive Work and AI Safety: A Thermodynamic Perspective

Causal Undertow: A Work of Seed Fiction