Daniel Murfet

• Daniel Murfet 15 Dec 2024 18:20 UTC

  11 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: lc’s Shortform

  et al (!)

• Daniel Murfet 12 Dec 2024 21:58 UTC

  12 points

  0

  in reply to: peterbarnett’s comment on: Stan van Winger­den’s Shortform

  There’s no general theoretical reason that I am aware of to expect a relation between the L2 norm and the LLC. The LLC is the coefficient of the $logn$ term in the asymptotic expansion of the free energy (negative logarithm of the integral of the posterior over a local region, as a function of sample size $n$) while the L2 norm of the parameter shows up in the constant order term of that same expansion, if you’re taking a Gaussian prior.

  It might be that in particular classes of neural networks there is some architecture-specific correlation between the L2 norm and the LLC, but I am not aware of any experimental or theoretical evidence for that.

  For example, in the figure below from Hoogland et al 2024 we see that there are later stages of training in a transformer trained to do linear-regression in context (blue shaded regions) where the LLC $\hat{\lambda }$ is decreasing but the L2 norm is increasing. So the model is moving towards a “simpler” parameter with larger weight norm.

  My best current guess is that it happens to be, in the grokking example, that the simpler solution has smaller weight norm. This could be true in many synthetic settings, for all I know; however, in general, it is not the case that complexity (at least as far as SLT is concerned) and weight norm are correlated.

  

• Daniel Murfet 9 Dec 2024 20:58 UTC

  2 points

  0

  in reply to: 4gate’s comment on: Cog­ni­tive Work and AI Safety: A Ther­mo­dy­namic Perspective

  That simulation sounds cool. The talk certainly doesn’t contain any details and I don’t have a mathematical model to share at this point. One way to make this more concrete is to think through Maxwell’s demon as an LLM, for example in the context of Feynman’s lectures on computation. The literature on thermodynamics of computation (various experts, like Adam Shai and Paul Riechers, are around here and know more than me) implicitly or explicitly touches on relevant issues.

• Daniel Murfet 9 Dec 2024 20:16 UTC

  4 points

  2

  in reply to: Thomas Kwa’s comment on: Cog­ni­tive Work and AI Safety: A Ther­mo­dy­namic Perspective

  The analogous laws are just information theory.

  Re: a model trained on random labels. This seems somewhat analogous to building a power plant out of dark matter; to derive physical work it isn’t enough to have some degrees of freedom somewhere that have a lot of energy, one also needs a chain of couplings between those degrees of freedom and the degrees of freedom you want to act on. Similarly, if I want to use a model to reduce my uncertainty about something, I need to construct a chain of random variables with nonzero mutual information linking the question in my head to the predictive distribution of the model.

  To take a concrete example: if I am thinking about a chemistry question, and there are four choices A, B, C, D. Without any other information than these letters the model cannot reduce my uncertainty (say I begin with equal belief in all four options). However if I provide a prompt describing the question, and the model has been trained on chemistry, then this information sets up a correspondence between this distribution over four letters and something the model knows about; its answer may then reduce my distribution to being equally uncertain between A, B but knowing C, D are wrong (a change of 1 bit in my entropy).

  Since language models are good general compressors this seems to work in reasonable generality.

  Ideally we would like the model to push our distribution towards true answers, but it doesn’t necessarily know true answers, only some approximation; thus the work being done is nontrivially directed, and has a systematic overall effect due to the nature of the model’s biases.

  I don’t know about evolution. I think it’s right that the perspective has limits and can just become some empty slogans outside of some careful usage. I don’t know how useful it is in actually technically reasoning about AI safety at scale, but it’s a fun idea to play around with.

• Daniel Murfet 9 Dec 2024 19:58 UTC

  4 points

  0

  in reply to: Jesse Hoogland’s comment on: Jesse Hoogland’s Shortform

  Marcus Hutter on AIXI and ASI safety

• Daniel Murfet 9 Dec 2024 10:05 UTC

  2 points

  0

  in reply to: Jonas Hallgren’s comment on: Cog­ni­tive Work and AI Safety: A Ther­mo­dy­namic Perspective

  Yes this seems like an important question but I admit I don’t have anything coherent to say yet. A basic intuition from thermodynamics is that if you can measure the change in the internal energy between two states, and the heat transfer, you can infer how much work was done even if you’re not sure how it was done. So maybe the problem is better thought of as learning to measure enough other quantities that one can infer how much cognitive work is being done.

  For all I know there is a developed thermodynamic theory of learning agents out there which already does this, but I didn’t find it yet...

• Daniel Murfet 9 Dec 2024 0:36 UTC

  4 points

  0

  on: What is “True Love”?

  The description of love at the conclusion of Gene Wolfe’s The Wizard gets at something important, if you read it as something that both parties are simultaneously doing.

Cog­ni­tive Work and AI Safety: A Ther­mo­dy­namic Perspective

Causal Un­der­tow: A Work of Seed Fiction

• Daniel Murfet 27 Nov 2024 20:56 UTC

  3 points

  0

  in reply to: mfar’s comment on: The Queen’s Dilemma: A Para­dox of Control

  The work of Ashby I’m familiar with is “An Introduction to Cybernetics” and I’m referring to the discussion in Chapter 11 there. The references you’re giving seem to be invoking the “Law” of requisite variety in the context of arguing that an AGI has to be relatively complex in order to maintain homeostatis in a complex environment, but this isn’t the application of the law I have in mind.

  From the book:

  The law of Requisite Variety says that R’s capacity as a regulator cannot exceed R’s capacity as a channel of communication.

  In the form just given, the law of Requisite Variety can be shown in exact relation to Shannon’s Theorem 10, which says that if noise appears in a message, the amount of noise that can be removed by a correction channel is limited to the amount of information that can be carried by that channel.

  Thus, his “noise” corresponds to our “disturbance”, his “correction channel” to our “regulator R”, and his “message of entropy H” becomes, in our case, a message of entropy zero, for it is constancy that is to be “transmitted”: Thus the use of a regulator to achieve homeostasis and the use of a correction channel to suppress noise are homologous.

  and

  A species continues to exist primarily because its members can block the flow of variety (thought of as disturbance) to the gene-pattern, and this blockage is the species’ most fundamental need. Natural selection has shown the advantage to be gained by taking a large amount of variety (as information) partly into the system (so that it does not reach the gene-pattern) and then using this information so that the flow via R blocks the flow through the environment T.

  This last quote makes clear I think what I have in mind: the environment is full of advanced AIs, they provide disturbances D, and in order to regulate the effects of those disturbances on our “cognitive genetic material” there is some requirement on the “correction channel”. Maybe this seems a bit alien to the concept of control. There’s a broader set of ideas I’m toying with, which could be summarised as something like “reciprocal control” where you have these channels of communication /​ regulation going in both directions (from human to machine, and vice versa).

  The Queen’s Dilemma was a little piece of that picture, which attempts to illustrate this bi-directional control flow by having the human control the machine (by setting its policy, say) and the machine control the human (in an emergent fashion, that being the dilemma).

• Daniel Murfet 27 Nov 2024 20:11 UTC

  2 points

  0

  in reply to: Noosphere89’s comment on: The Queen’s Dilemma: A Para­dox of Control

  Is restricting human agency fine if humans have little control over where it is restricted and to what degree?

• Daniel Murfet 27 Nov 2024 20:09 UTC

  3 points

  1

  in reply to: mfar’s comment on: The Queen’s Dilemma: A Para­dox of Control

  Re: your first point. I think I’m still a bit confused here and that’s partly why I wanted to write this down and have people poke at it. Following Sen (but maybe I’m misunderstanding him) I’m not completely convinced I know how to factor human agency into “winning”. One part of me wants to say that whatever notion of agency I have, in some sense it’s a property of world states and in principle I could extract it with enough monitoring of my brain or whatever, and then any prescribed tradeoff between “measured sense of agency” and “score” is something I could give to the machine as a goal.

  So then I end up with the machine giving me the precise amount of leeway that lets me screw up the game just right for my preferences.

  I don’t see a fundamental problem with that, but it’s also not the part of the metaphor that seems most interesting to me. What I’m more interested in is human inferiority as a pattern, and the way that pattern pervades the overall system and translates into computational structure, perhaps in surprising and indirect ways.

• Daniel Murfet 27 Nov 2024 20:03 UTC

  3 points

  0

  in reply to: mfar’s comment on: The Queen’s Dilemma: A Para­dox of Control

  I’ll reply in a few branches. Re: stochastic chess. I think there’s a difference between a metaphor and a toy model; this is a metaphor, and the ingredients are chosen to illustrate in a microcosm some features I think are relevant about the full picture. The speed differential, and some degree of stochasticity, seem like aspects of human intervention in AI systems that seem meaningful to me.

  I do agree that if one wanted to isolate the core phenomena here mathematically and study it, chess might not be the right toy model.

• Daniel Murfet 27 Nov 2024 18:48 UTC

  2 points

  0

  in reply to: Rauno Arike’s comment on: The Queen’s Dilemma: A Para­dox of Control

  The metaphor is a simplification, in practice I think it is probably impossible to know whether you have achieved complete alignment. The question is then: how significant is the gap? If there is an emergent pressure across the vast majority of learning machines that dominate your environment to push you from de facto to de jure control, not due to malign intent but just as a kind of thermodynamic fact, then the alignment gap (no matter how small) seems to loom larger.

The Queen’s Dilemma: A Para­dox of Control

• Daniel Murfet 17 Nov 2024 20:23 UTC

  17 points

  −1

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  Re: the SLT dogma.
  
  For those interested, a continuous version of the padding argument is used in Theorem 4.1 of Clift-Murfet-Wallbridge to show that the learning coefficient is a lower bound on the Kolmogorov complexity (in a sense) in the setting of noisy Turing machines. Just take the synthesis problem to be given by a TM’s input-output map in that theorem. The result is treated in a more detailed way in Waring’s thesis (Proposition 4.19). Noisy TMs are of course not neural networks, but they are a place where the link between the learning coefficient in SLT and algorithmic information theory has already been made precise.
  
  For what it’s worth, as explained in simple versus short, I don’t actually think the local learning coefficient is algorithmic complexity (in the sense of program length) in neural networks, only that it is a lower bound. So I don’t really see the LLC as a useful “approximation” of the algorithmic complexity.

  For those wanting to read more about the padding argument in the classical setting, Hutter-Catt-Quarel “An Introduction to Universal Artificial Intelligence” has a nice detailed treatment.

  What links here?

  • Deep Learn­ing is cheap Solomonoff in­duc­tion? by Lucius Bushnaq (7 Dec 2024 11:00 UTC; 31 points)

• Daniel Murfet 17 Nov 2024 2:05 UTC

  2 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  Typo, I think you meant singularity theory :p

• Daniel Murfet 17 Nov 2024 2:04 UTC

  11 points

  5

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  Modern mathematics is less about solving problems within established frameworks and more about designing entirely new games with their own rules. While school mathematics teaches us to be skilled players of pre-existing mathematical games, research mathematics requires us to be game designers, crafting rule systems that lead to interesting and profound consequences

  I don’t think so. This probably describes the kind of mathematics you aspire to do, but still the bulk of modern research in mathematics is in fact about solving problems within established frameworks and usually such research doesn’t require us to “be game designers”. Some of us are of course drawn to the kinds of frontiers where such work is necessary, and that’s great, but I think this description undervalues the within-paradigm work that is the bulk of what is going on.

Aus­tralian AI Safety Fo­rum 2024

• Daniel Murfet 22 Sep 2024 2:49 UTC

  4 points

  0

  in reply to: yanni kyriacos’s comment on: yanni’s Shortform

  It might be worth knowing that some countries are participating in the “network” without having formal AI safety institutes