tailcalled

Karma: 6,506

tailcalled 28 May 2024 15:17 UTC
2 points
0
on: A small example of one-step hypotheticals

Ok, that’s what might happen if the agent had the power to ask unlimited hypothetical questions in arbitrarily many counterfactual scenarios. But that is not the case in the real world: the agent would be able to ask one, or maybe two questions at most, before the human attitude to the violin would change, and further data would become tainted.

Is it really the further data that becomes tainted, rather than the original data? Usually when you think longer about a subject, we’d think your opinions would become more rather than less valid.

tailcalled 28 May 2024 7:47 UTC
2 points
0
on: How to get nerds fascinated about mysterious chronic illness research?
What methods do you use to study this?

Like I guess the main method one can use is to start with the symptoms and track backwards step by step along the known causes of such symptoms to identify causes that are out of whack, until one gets up to some major cause that’s treatable?

Alternatively, one could just throw stuff at the wall and see if it sticks, but it seems like it would be too noisy to work.

The tricky part is I’d think it’s very hard to enumerate the causes of the symptoms? What does it look like in practice?

tailcalled 23 May 2024 19:18 UTC
6 points
2
on: Are most people deeply confused about “love”, or am I missing a human universal?
I can’t comment on whether people are confused about what “love” means as I’m not sufficiently deep in love discourse to say. But one thing I’m noticing about your characterizations of love is that they are missing an indexical element to the point of approaching sollipsism.
Romance and sexuality makes for a good example. Consider the following scenarios:
- A woman is on a date with a man, which she enjoys until she sees that his home is a dump.
- A teenager has a crush on a celebrity, with elaborate daydreams about how cool the celebrity is, not realizing how much of this is a facade created for entertainment.
- A man visits a prostitute and feels excited as he causes her to orgasm, not realizing that she fakes it for the business.
In all of these cases, one could say that there is a disconnect between what people think about their object of attraction, versus what that object of attraction really is like.
A Bayesian of parsing this is that their feelings of attraction represents an estimate of how well they fit together, but that this estimate differs from how well they really fit together. The actual fit seems important to think and talk about, and one should probably coin a short word for it—or at least for the coincidence between actual and estimated fit. This could be called “true love”.

tailcalled 23 May 2024 12:53 UTC
2 points
0
in reply to: tailcalled’s comment on: Each Llama3-8b text uses a different “random” subspace of the activation space
Maybe I just need to do epic layers of eigendecomposition...

tailcalled 23 May 2024 10:44 UTC
2 points
0
in reply to: tailcalled’s comment on: Each Llama3-8b text uses a different “random” subspace of the activation space
Realization: the binary multiplicative structure can probably be recovered fairly well from the binary additive structure + unary eigendecomposition?
Let’s say you’ve got three subspaces $X$ , $Y$ and $Z$ (represented as projection matrices). Imagine that one prompt uses dimensions $D_{1} = X + Y$ , and another prompt uses dimensions $D_{2} = Y + Z$ . If we take the difference, we get $X - Z$ . Notably, the positive eigenvalues correspond to X, and the negative eigenvalues correspond to $Z$ .
Define $f (P)$ to yield the part of $P$ with positive eigenvalues (which I suppose for projection matrices has a closed form of $\frac{P + P^{2}}{2}$ , but the point is it’s unary and therefore nicer to deal with mathematically). You get $D_{1} \land \neg D_{2} = f (D_{1} - D_{2})$ , and you get $D_{1} \land D_{2} = D_{1} - f (D_{1} - D_{2})$ .

tailcalled 22 May 2024 14:06 UTC
2 points
0
in reply to: tailcalled’s comment on: Each Llama3-8b text uses a different “random” subspace of the activation space
One thing I’m thinking is that the additive structure on its own isn’t going to be sufficient for this and I’m going to need to use intersections more.

tailcalled 22 May 2024 11:48 UTC
2 points
0
on: Each Llama3-8b text uses a different “random” subspace of the activation space
Actually one more thing I’m probably also gonna do is create a big subspace overlap matrix and factor it in some way to see if I can split off some different modules. I had intended to do that originally, but the finding that all the dimensions were used at least half the time made me pessimistic about it. But I should Try Harder.

Each Llama3-8b text uses a different “random” subspace of the activation space

tailcalled22 May 2024 7:31 UTC

3 points

4 comments7 min readLW link

tailcalled 21 May 2024 18:08 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
If I look at the pairwise overlap between the dimensions needed for each generation:
… then this is predictable down to ~1% error simply by assuming that they pick a random subset of the dimensions for each, so their overlap is proportional to each of their individual sizes.

tailcalled 21 May 2024 17:35 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
Given the large number of dimensions that are kept in each case, there must be considerable overlap in which dimensions they make use of. But how much?
I concatenated the dimensions found in each of the prompts, and performed an SVD of it. It yielded this plot:
… unfortunately this seems close to the worst-case scenario. I had hoped for some split between general and task-specific dimensions, yet this seems like an extremely uniform mixture.

tailcalled 21 May 2024 17:24 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
To quickly find the subspace that the model is using, I can use a binary search to find the number of singular vectors needed before the probability when clipping exceeds the probability when not clipping.
A relevant followup is what happens to other samples in response to the prompt when clipping. When I extrapolate “I believe the meaning of life is” using the 1886-dimensional subspace from
[I believe the meaning of life is] to be happy. It is a simple concept, but it is very difficult to achieve. The only way to achieve it is to follow your heart. It is the only way to live a happy life. It is the only way to be happy. It is the only way to be happy.
The meaning of life is
, I get:
[I believe the meaning of life is] to find happy. We is the meaning of life. to find a happy.
And to live a happy and. If to be a a happy.
. to be happy.
. to be happy.
. to be a happy.. to be happy.
. to be happy.
Which seems sort of vaguely related, but idk.
Another test is just generating without any prompt, in which case these vectors give me:
Question is a single thing to find. to be in the best to be happy. I is the only way to be happy.
I is the only way to be happy.
I is the only way to be happy.
It is the only way to be happy.. to be happy.. to be happy. to
Using a different prompt:
[Simply put, the theory of relativity states that ]1) the laws of physics are the same for all non-accelerating observers, and 2) the speed of light in a vacuum is the same for all observers, regardless of their relative motion or of the motion of the source of the light. Special relativity is a theory of the structure of spacetime
I can get a 3329-dimensional subspace which generates:
[Simply put, the theory of relativity states that ] 1) time is relative and 2) the speed of light in a vacuum is constant for all observers.
1) Time is relative, meaning that if two observers are moving relative to each other, the speed of light is the same for all observers, regardless of their motion. For example, if you are moving relative
or
Question: In a simple harmonic motion, the speed of an object is
A) constant
B) constant
C) constant
D) constant
In the physics of simple harmonic motion, the speed of an object is constant. The speed of the object can be constant, but the speed of an object can be
Another example:
[A brief message congratulating the team on the launch:
Hi everyone,
I just ] wanted to congratulate you all on the launch. I hope
that the launch went well. I know that it was a bit of a
challenge, but I think that you all did a great job. I am
proud to be a part of the team.
Thank you for your
can yield 2696 dimensions with
[A brief message congratulating the team on the launch:
Hi everyone,
I just ] wanted to say you for the launch of the launch of the team.
The launch was successful and I am so happy to be a part of the team and I am sure you are all doing a great job.
I am very looking to be a part of the team.
Thank you all for your hard work,
or
def measure and is the definition of the new, but the
the is a great, but the
The is the
The is a
The is a
The is a
The
The is a
The
The
The is a
The
The is a
And finally,
[Translate English to French:
sea otter ⇒ loutre de mer
peppermint ⇒ menthe poivrée
plush girafe ⇒ girafe peluche
cheese =>] fromage
pink ⇒ rose
blue ⇒ bleu
red ⇒ rouge
yellow ⇒ jaune
purple ⇒ violet
brown ⇒ brun
green ⇒ vert
orange ⇒ orange
black ⇒ noir
white ⇒ blanc
gold ⇒ or
silver ⇒ argent
can yield the 2518-dimensional subspace:
[Translate English to French:
sea otter ⇒ loutre de mer
peppermint ⇒ menthe poivrée
plush girafe ⇒ girafe peluche
cheese =>] fromage
cheese ⇒ fromage
cheese ⇒ fromage
f cheese ⇒ fromage
butter ⇒ fromage
apple ⇒ orange
yellow ⇒ orange
green ⇒ vert
black ⇒ noir
blue ⇒ ble
purple ⇒ violet
white ⇒ blanc
or
Question: A 201
The sum of a
The following
the sum
the time
the sum
the
the
the
The
The
The
The
The
The
The
The
The
The
The
The
The
The
The
The
The
The

[Question] Is deleting capabilities still a relevant research question?

tailcalled21 May 2024 13:24 UTC

15 points

1 comment1 min readLW link

tailcalled 20 May 2024 20:58 UTC
2 points
0
on: Interpretability: Integrated Gradients is a decent attribution method
We now have a method for how to do attributions on single data points. But when we’re searching for circuits, we’re probably looking for variables that have strong attributions between each other on average, measured over many data points.
Maybe?
One thing I’ve been thinking a lot recently is that building tools to interpret networks on individual datapoints might be more relevant than attributing over a dataset. This applies if the goal is to make statistical generalizations since a richer structure on an individual datapoint gives you more to generalize with, but it also applies if the goal is the inverse, to go from general patterns to particulars, since this would provide a richer method for debugging, noticing exceptions, etc..
And basically the trouble a lot of work that attempts to generalize ends up with is that some phenomena are very particular to specific cases, so one risks losing a lot of information by only focusing on the generalizable findings.
Either way, cool work, seems like we’ve thought about similar lines but you’ve put in more work.

tailcalled 20 May 2024 20:45 UTC
8 points
0
on: The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks
I was thinking in similar lines, but eventually dropped it because I felt like the gradients would likely miss something if e.g. a saturated softmax prevents any gradient from going through. I find it interesting that experiments also find that the interaction basis didn’t work, and I wonder whether any of the failure here is due to saturated softmaxes.

tailcalled 20 May 2024 18:58 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
Back to clipping away an entire range, rather than a single dimension. Here’s ordering it by the importance computed by clipping away a single dimension:
Less chaotic maybe, but also much slower at reaching a reasonable performance, so I tried a compromise ordering that takes both size and performance into account:
Doesn’t seem like it works super great tbh.
Edit: for completeness’ sake, here’s the initial graph with log-surprise-based plotting.

tailcalled 20 May 2024 16:38 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
An implicit assumption I’m making when I clip off from the end with the smallest singular values is that the importance of a dimension is proportional to its singular values. This seemed intuitively sensible to me (“bigger = more important”), but I thought I should test it, so I tried clipping off only one dimension at a time, and plotting how that affected the probabilities:
Clearly there is a correlation, but also clearly there’s some deviations from that correlation. Not sure whether I should try to exploit these deviations in order to do further dimension reduction. It’s tempting, but it also feels like it starts entering sketchy territories, e.g. overfitting and arbitrary basis picking. Probably gonna do it just to check what happens, but am on the lookout for something more principled.

tailcalled 20 May 2024 15:03 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
Ok, so I’ve got the clipping working. First, some uninterpretable diagrams:
In the bottom six diagrams, I try taking varying number (x-axis) of right singular vectors (v’s) and projecting down the “writes” to the residual stream to the space spanned by those vectors.
The obvious criterion to care about is whether the projected network reproduces the outputs of the original network, which here I operationalize based on the log probability the projected network gives to the continuation of the prompt (shown in the “generation probability” diagrams). This appears to be fairly chaotic (and low) in the 1-300ish range, and then stabilizes while still being pretty low in the 300ish-1500ish range, and then finally converges to normal in the 1500ish to 2000ish range, and is ~perfect afterwards.
The remaining diagrams show something about how/why we have this pattern. “orig_delta” concerns the magnitude of the attempted writes for a given projection (which is not constant because projecting in earlier layers will change the writes by later layers), and “kept_delta” concerns the remaining magnitude after the discarded dimensions have been projected away.
In the low end, “kept_delta” is small (and even “orig_delta” is a bit smaller than it ends up being at the high end), indicating that the network fails to reproduce the probabilities because the projection is so aggressive that it simply suppresses the network too much.
Then in the middle range, “orig_delta” and “kept_delta” explodes, indicating that the network has some internal runaway dynamics which normally would be suppressed, but where the suppression system is broken by the projection.
Finally, in the high range, we get a sudden improvement in loss, and a sudden drop in residual stream “write” size, indicating that it has managed to suppress this runaway stuff and now it works fine.

tailcalled 20 May 2024 12:14 UTC
2 points
0
in reply to: tailcalled’s comment on: tailcalled’s Shortform
Oops, my code had a bug so only self.attention(self.attention_norm(x), start_pos, freqs_cis, mask) and not self.feed_forward(self.ffn_norm(h)) was in the SVD. So the diagram isn’t 100% accurate.

tailcalled 19 May 2024 17:12 UTC
11 points
0
on: tailcalled’s Shortform
Finally gonna start properly experimenting on stuff. Just writing up what I’m doing to force myself to do something, not claiming this is necessarily particularly important.
Llama (and many other models, but I’m doing experiments on Llama) has a piece of code that looks like this:
h = x + self.attention(self.attention_norm(x), start_pos, freqs_cis, mask)
out = h + self.feed_forward(self.ffn_norm(h))
Here, out is the result of the transformer layer (aka the residual stream), and the vectors self.attention(self.attention_norm(x), start_pos, freqs_cis, mask) and self.feed_forward(self.ffn_norm(h)) are basically where all the computation happens. So basically the transformer proceeds as a series of “writes” to the residual stream using these two vectors.
I took all the residual vectors for some queries to Llama-8b and stacked them into a big matrix M with 4096 columns (the internal hidden dimensionality of the model). Then using SVD, I can express $M = \sum_{i} s_{i} (u_{i} \otimes v_{i})$ , where the $u$ ‘s and $v$ ’s are independent units vectors. This basically decomposes the “writes” into some independent locations in the residual stream (u’s), some latent directions that are written to (v’s) and the strength of those writes (s’s, aka the singular values).
To get a feel for the complexity of the writes, I then plotted the s’s in descending order. For the prompt “I believe the meaning of life is”, Llama generated the continuation “to be happy. It is a simple concept, but it is very difficult to achieve. The only way to achieve it is to follow your heart. If you follow your heart, you will find happiness. If you don’t follow your heart, you will never find happiness. I believe that the meaning of life is to”. During this continuation, there were 2272 writes to the residual stream, and the singular values for these writes were as follows:
The first diagram shows that there were 2 directions that were much larger than all the others. The second diagram shows that most of the singular values are nonnegligible, which indicates to me that almost all of the writes transfer nontrivial information. This can also be seen in the last diagram, where the cumulative size of the singular values increases approximately logaritmically with their count.
This is kind of unfortunate, because if almost all of the s was concentrated in a relatively small number of dimensions (e.g. 100), then we could simplify the network a lot by projecting down to these dimensions. Still, this was relatively expected because others had found the singular values of the neural networks to be very complex.
Since variance explained is likely nonlinearly related to quality, my next step will likely be to clip the writes to the first k singular vectors and see how that impacts the performance of the network.

tailcalled 17 May 2024 14:05 UTC
10 points
5
on: Is There Really a Child Penalty in the Long Run?

Second, Lomborg’s 2024 paper finds evidence that women time their births to just before a wage growth plateau. The evidence it gives again comes from IVF failures. Women who were planning to have a birth, but never succeed, have much flatter wage growth after their planned birth year, even though they didn’t actually have any kids. So the divergence between childrearing mothers and non-childbearing mothers shows up even in this placebo case when neither group actually had kids. Therefore, the event study is overstating the earnings impact of childbirth.

Couldn’t it also be that the women in question plan their career based on the expectation to have children and this is what leads to the plateau? In that case it seems like it would be incorrect to interpret these results as evidence against a child penalty, as it’s merely that the child penalty affects women regardless of whether they have the children. To check, I think you should ask the study participants why their career plateaued then.

tailcalled

Each Llama3-8b text uses a differ­ent “ran­dom” sub­space of the ac­ti­va­tion space

[Question] Is delet­ing ca­pa­bil­ities still a rele­vant re­search ques­tion?

Each Llama3-8b text uses a different “random” subspace of the activation space

[Question] Is deleting capabilities still a relevant research question?