Jett Janiak(Jett Mayzner)

• Jett Janiak 26 Sep 2024 13:49 UTC

  2 points

  0

  on: Char­ac­ter­iz­ing sta­ble re­gions in the resi­d­ual stream of LLMs

  I believe there are two phenomena happening during training

  1. Predictions corresponding to the same stable region become more similar, i.e. stable regions become more stable. We can observe this in the animations.

  2. Existing regions split, resulting in more regions.

  3. 

  I hypothesize that

  1. could be some kind of error correction. Models learn to rectify errors coming from superposition interference or another kind of noise.

  2. could be interpreted as more capable models picking up on subtler differences between the prompts and adjusting their predictions.

Char­ac­ter­iz­ing sta­ble re­gions in the resi­d­ual stream of LLMs

Eval­u­at­ing Syn­thetic Ac­ti­va­tions com­posed of SAE La­tents in GPT-2

• Jett Janiak 30 Aug 2024 17:55 UTC

  2 points

  0

  on: AIS ter­minol­ogy pro­posal: stan­dard­ize terms for prob­a­bil­ity ranges

  Scott In Continued Defense Of Non-Frequentist Probabilities

• Jett Janiak 17 May 2024 10:08 UTC

  15 points

  0

  on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  This is such a cool result! I tried to reproduce it in this notebook
  

  

• Jett Janiak 17 May 2024 9:47 UTC

  1 point

  0

  in reply to: Adam Shai’s comment on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  For the two sets of mess3 parameters I checked the stationary distribution was uniform.

AISC pro­ject: TinyEvals

Poly­se­man­tic At­ten­tion Head in a 4-Layer Transformer

• Jett Janiak 9 Oct 2023 8:41 UTC

  1 point

  0

  on: A Com­pre­hen­sive Mechanis­tic In­ter­pretabil­ity Ex­plainer & Glossary

  The activation patching, causal tracing and resample ablation terms seem to be out of date, compared to how you define them in your post on attribution patching.

An ad­ver­sar­ial ex­am­ple for Direct Logit At­tri­bu­tion: mem­ory man­age­ment in gelu-4l

A cir­cuit for Python doc­strings in a 4-layer at­ten­tion-only transformer