jacob_drori

• jacob_drori 21 Oct 2024 5:47 UTC

  1 point

  0

  on: Ex­plor­ing the Pla­tonic Rep­re­sen­ta­tion Hy­poth­e­sis Beyond In-Distri­bu­tion Data

  This seems very interesting, but I think your post could do with a lot more detail. How were the correlations computed? How strongly do they support PRH? How was the OOD data generated? I’m sure the answers could be pieced together from the notebook, but most people won’t click through and read the code.

• jacob_drori 14 Oct 2024 20:45 UTC

  3 points

  2

  in reply to: Lucius Bushnaq’s comment on: Cir­cuits in Su­per­po­si­tion: Com­press­ing many small neu­ral net­works into one

  Ah, I think I understand. Let me write it out to double-check, and in case it helps others.
  
  Say $\delta =0$, for simplicity. Then $A^l={\sum }_t{E}_t{a}_t^l$ . This sum has $k$ nonzero terms.

  
  In your construction, $W^{l,in}={\sum }_t{V}_t^l{W}_t^{l,in}{E}_t^T$. Focussing on a single neuron, labelled by $i$, we have $(W^{l,in}{)}_i={\sum }_t(V_t^l{)}_i{W}_t^{l,in}{E}_t^T$. This sum has $\sim pT$ nonzero terms.

  
  So the preactivation of an MLP hidden neuron in the big network is $p_i^l={\sum }_{t,t^{\prime }}(V_t^l{)}_i{W}_t^{l,in}{E}_t^T{E}_{t^{\prime }}{a}_{t^{\prime }}^l$ . This sum has $\sim kpT$ nonzero terms.

  We only “want” the terms where $t=t^{\prime }$; the rest (i.e. the majority) are noise. Each noise term in the sum is a random vector, so each of the $\sim kpT$ different noise terms are roughly orthogonal, and so the norm of the noise is $O\left(\sqrt{kpT}\right)$ (times some other factors, but this captures the $T$-dependence, which is what I was confused about).

• jacob_drori 14 Oct 2024 17:25 UTC

  1 point

  0

  on: Cir­cuits in Su­per­po­si­tion: Com­press­ing many small neu­ral net­works into one

  I’m confused by the read-in bound:

  ${ϵ}_t^{l,in}=O\left(wa\sqrt{kT\frac{md}{MD}\log M}\right)$

  Sure, each neuron reads from $T\frac{n\log M}{M}$of the random subspaces. But in all but $k$ of those subspaces, the big network’s activations are smaller than $\delta$, right? So I was expecting a tighter bound—something like:

  ${ϵ}_t^{l,in}=O\left(wa\sqrt{(k+T\delta )\frac{md}{MD}\log M}\right)$

There is a globe in your LLM

Do­main-spe­cific SAEs

• jacob_drori 24 Sep 2024 20:02 UTC

  1 point

  0

  in reply to: tailcalled’s comment on: tail­called’s Shortform

  Ah, so I think you’re saying “You’ve explained to me the precise reason why energy and momentum (i.e. time and space) are different at the fundamental level, but why does this lead to the differences we observe between energy and momentum (time and space) at the macro-level?

  This is a great question, and as with any question of the form “why does this property emerge from these basic rules”, there’s unlikely to be a short answer. E.g. if you said “given our understanding of the standard model, explain how a cell works”, I’d have to reply “uhh, get out a pen and paper and get ready to churn through equations for several decades”.

  In this case, one might be able to point to a few key points that tell the rough story. You’d want to look at properties of solutions PDEs on manifolds with metric of signature (1,3) (which means “one direction on the manifold is different to the other three, in that it carries a minus sign in the metric compared to the others in the metric”). I imagine that, generically, these solutions behave differently with respect to the “1″ direction and the “3” directions. These differences will lead to the rest of the emergent differences between space and time. Sorry I can’t be more specific!

• jacob_drori 24 Sep 2024 18:41 UTC

  1 point

  0

  in reply to: tailcalled’s comment on: tail­called’s Shortform

  > could one replace the energy-first formulations of quantum mechanics with momentum-first formulations?
  
  Momentum is to space what energy is to time. Precisely, energy generates (in the Lie group sense) time-translations, whereas momentum generates spatial translations. So any question about ways in which energy and momentum differ is really a question about how time and space differ.
  
  In ordinary quantum mechanics, time and space are treated very differently: $t$ is a coordinate whereas $x$ is a dynamical variable (which happens to be operator-valued). The equations of QM tell us how $x$ evolves as a function of $t$.
  
  But ordinary QM was long-ago replaced by quantum field theory, in which time and space are on a much more even footing: they are both coordinates, and the equations of QFT tell us how a third thing (the field $\varphi (x,t)$) evolves as a function of $x$ and $t$. Now, the only difference between time and space is that there is only one dimension of the former but three of the latter (there may be some other very subtle differences I’m glossing over here, but I wouldn’t be surprised if they ultimately stem from this one).
  
  All of this is to say: our best theory of how nature works (QFT), is neither formulated as “energy-first” nor as “momentum-first”. Instead, energy and momentum are on fairly equal footing.

• jacob_drori 21 Sep 2024 1:20 UTC

  8 points

  0

  in reply to: tailcalled’s comment on: tail­called’s Shortform

  Sure, there are plenty of quantities that are globally conserved at the fundamental (QFT) level. But most most of.these quantities aren’t transferred between objects at the everyday, macro level we humans are used to.

  E.g. 1: most everyday objects have neutral electrical charge (because there exist positive and negative charges, which tend to attract and roughly cancel out) so conservation of charge isn’t very useful in day-to-day life.

  E.g. 2: conservation of color charge doesn’t really say anything useful about everyday processes, since it’s only changed by subatomic processes (this is again basically due to the screening effect of particles with negative color charge, though the story here is much more subtle, since the main screening effect is due to virtual particles rather than real ones).

  The only other fundamental conserved quantity I can think of that is nontrivially exchanged between objects at the macro level is momentum. And… momentum seems roughly as important as energy?

  I guess there is a question about why energy, rather than momentum, appears in thermodynamics. If you’re interested, I can answer in a separate comment.

• jacob_drori 30 Aug 2024 19:21 UTC

  3 points

  0

  on: Does a time-re­versible phys­i­cal law/​Cel­lu­lar Au­toma­ton always im­ply the First Law of Ther­mo­dy­nam­ics?

  I’ll just answer the physics question, since I don’t know anything about cellular automata.

  When you say time-reversal symmetry, do you mean that t → T-t is a symmetry for any T?

  If so, the composition of two such transformations is a time-translation, so we automatically get time-translation symmetry, which implies the 1st law.

  If not, then the 1st law needn’t hold. E.g. take any time-dependent hamiltonian satisfying H(t) = H(-t). This has time-reversal symmetry about t=0, but H is not conserved.

Open Source Au­to­mated In­ter­pretabil­ity for Sparse Au­toen­coder Features

• jacob_drori 12 Jun 2024 20:10 UTC

  1 point

  0

  on: DSLT 1. The RLCT Mea­sures the Effec­tive Di­men­sion of Neu­ral Networks

  The theorem guarantees the existence of a $d$-dimensional analytic manifold $M$ and a real analytic map

  $$g:M\ni u\mapsto w\in W$$

  such that for each coordinate $M_{\alpha }$ of $M$ one can write

  $$\begin{array}{cc}K\left(g\right(u\left)\right)& =u_1^{2k_1}\dots u_d^{2k_d}\\ ...& \end{array}$$

  I’m a bit confused here. First, I take it that $\alpha$ labels coordinate patches? Second, consider the very simple case with $d=2$ and $K\left(w\right)=w_1^2+w_2^2$. What $g$ would put $K$ into the stated form?

• jacob_drori 26 Apr 2024 0:00 UTC

  LW: 0 AF: -1

  0

  AF

  on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  Nice work! I’m not sure I fully understand what the “gated-ness” is adding, i.e. what the role the Heaviside step function is playing. What would happen if we did away with it? Namely, consider this setup:

  Let $f$ and $\hat{x}$ be the encoder and decoder functions, as in your paper, and let $x$ be the model activation that is fed into the SAE.

  The usual SAE reconstruction is $\hat{x}\left(f\right(x\left)\right)$, which suffers from the shrinkage problem.

  Now, introduce a new learned parameter $t\in R^{n_{features}}$, and define an “expanded” reconstruction $y_{expanded}=\hat{x}(t\odot f(x\left)\right)$, where $\odot$ denotes elementwise multiplication.

  Finally, take the loss to be:

  $L=||{\hat{x}}_{copy}\left(f\right(x\left)\right)-x|{|}_2^2+||y_{expanded}-x|{|}_2^2+\lambda ||f\left(x\right)|{|}_1$.

  where ${\hat{x}}_{copy}$ ensures the decoder gets no gradients from the first term. As I understand it, this is exactly the loss appearing in your paper. The only difference in the setup is the lack of the Heaviside step function.

  Did you try this setup? Or does it fail for an obvious reason I missed?

• jacob_drori 5 Mar 2024 16:22 UTC

  2 points

  0

  in reply to: Bart Bussmann’s comment on: Do sparse au­toen­coders find “true fea­tures”?

  The peaks at 0.05 and 0.3 are strange. What regulariser did you use? Also, could you check whether all features whose nearest neighbour has cosine similarity 0.3 have the same nearest neighbour (and likewise for 0.05)?

• jacob_drori 4 Mar 2024 18:57 UTC

  LW: 7 AF: 3

  0

  AF

  on: Some costs of superposition

  The typical noise on feature $f_1$ caused by 1 unit of activation from feature $f_2$, for any pair of features $f_1$, $f_2$, is (derived from Johnson–Lindenstrauss lemma)

  $ϵ=\sqrt{\frac{8\ln \left(m\right)}{n}}$ ^[1]

  1. … This is a worst case scenario. I have not calculated the typical case, but I expect it to be somewhat less, but still same order of magnitude

  Perhaps I’m misunderstanding your claim here, but the “typical” (i.e. RMS) inner product between two independently random unit vectors in $R^n$ is $n^{-1/2}$. So I think the $\sqrt{8\ln m}$ shouldn’t be there, and the rest of your estimates are incorrect.

  This means that we can have at most $l<n/\left(8\ln \right(m\left)\right)$ simultaneously active features

  This conclusion gets changed to $l<n$.

• jacob_drori 28 Feb 2024 18:10 UTC

  1 point

  0

  on: At­ten­tion SAEs Scale to GPT-2 Small

  Paging hijohnnylin—it’d be awesome to have neuronpedia dashboards for these features. Between these, OpenAI’s MLP features, and Joseph Bloom’s resid_pre features, we’d have covered pretty much the whole model!

• jacob_drori 27 Feb 2024 20:24 UTC

  3 points

  0

  on: Do sparse au­toen­coders find “true fea­tures”?

  For each SAE feature (i.e. each column of W_dec), we can look for a distinct feature with the maximum cosine similarity to the first. Here is a histogram of these max cos sims, for Joseph Bloom’s SAE trained at resid_pre, layer 10 in gpt2-small. The corresponding plot for random features is shown for comparison:

  

  
  

  The SAE features are much less orthogonal than the random ones. This effect persists if, instead of the maximum cosine similarity, we look at the 10th largest, or the 100th largest:

  

  

  I think it’s a good idea to include a loss term to incentivise feature orthogonality.

• jacob_drori 26 Feb 2024 19:18 UTC

  1 point

  0

  in reply to: Logan Riggs’s comment on: Do sparse au­toen­coders find “true fea­tures”?

  Nice, this is exactly what I was asking for. Thanks!

• jacob_drori 23 Feb 2024 18:13 UTC

  1 point

  0

  on: Do sparse au­toen­coders find “true fea­tures”?

  I’m confused about your three-dimensional example and would appreciate more mathematical detail.

  Call the feature directions f1, f2, f3.

  Suppose SAE hidden neurons 1,2 and 3 read off the components along f1, f2, and f1+f2, respectively. You claim that in some cases this may achieve lower L1 loss than reading off the f1, f2, f3 components.

  [note: the component of a vector X along f1+f2 here refers to ¹⁄₂ * (f1+f2) \cdot X]

  Can you write down the encoder biases that would achieve this loss reduction? Note that e.g. when the input is f1, there is a component of ¹⁄₂ along f1+f2, so you need a bias < −1/​2 on neuron 3 to avoid screwing up the reconstruction.

• jacob_drori 16 Feb 2024 17:47 UTC

  3 points

  0

  on: Map­ping the se­man­tic void II: Above, be­low and be­tween to­ken em­bed­dings

  Nice post. I was surprised that the model provides the same nonsense definition regardless of the token when the embedding is rescaled to be large, and moreover that this nonsense definition is very similar to the one given when the embedding is rescaled to be small. Here’s an explanation I find vaguely plausible. Suppose the model completes the task as follows:

  • The model sees the prompt 'A typical definition of <token> would be '.

  • At some attention head A1, the <token> position attends back to 'definition' and gains a component in the residual stream direction that represents the I am the token being defined feature.

  • At some later attention head A2, the final position of the prompt attends back to positions with the I am the token being defined feature, and moves whatever information from that position is needed for defining the corresponding token.

  Now, suppose we rescale the <token> embedding to be very large. The size of the I am the token being defined component moved to the <token> position by A1 stays roughly the same as before (since no matter how much we scale query vectors, attention probabilities can never exceed 1). So, as a fraction of the total norm of the residual stream at that position, we’ve made the I am the token being defined component a lot smaller.

  Then, when the residual stream is fed into the layernorm preceding A2, the I am the token being defined component gets squashed down to almost zero: it has been “squeezed out” by the very large token embedding. Hence, when the QK matrix of A2 looks for positions with the I am the token being defined feature, it finds nothing, and all the model can do is give some generic nonsense definition. Unsurprisingly, this nonsense definition ends up being pretty similar to the one given when the token embedding is sent to zero, since in both cases the model is essentially trying to define a token that isn’t there.

  The details of this explanation may be totally wrong, and I haven’t checked any of this. But my guess is that something roughly along these lines is correct.

• jacob_drori 28 Dec 2023 22:51 UTC

  1 point

  0

  in reply to: Seth Herd’s comment on: In­ter­nal in­de­pen­dent re­view for lan­guage model agent alignment

  I hope that type of learning isn’t used

  I share your hope, but I’m pessimistic. Using RL to continuously train the outer loop of an LLM agent seems like a no-brainer from a capabilities standpoint.
  
  The alternative would be to pretrain the outer loop, and freeze the weights upon deployment. Then, I guess your plan would be to only use the independent reviewer after deployment, so that the reviewer’s decision never influences the outer-loop weights. Correct me if I’m wrong here.

  I’m glad you plan to address this in a future post, and I look forward to reading it.