Adam Pearce

• Adam PearceAug 16, 2023, 2:50 AM

  1 point

  0

  in reply to: salman.mohammadi’s comment on: An in­ter­ac­tive in­tro­duc­tion to grokking and mechanis­tic interpretability

  Lots of custom d3 https://​​github.com/​​PAIR-code/​​ai-explorables/​​tree/​​master/​​source/​​grokking

• Adam PearceAug 7, 2023, 9:28 PM

  7 points

  2

  on: Grow­ing Bon­sai Net­works with RNNs

  • The optimization section of Learning Transformer Programs might work with your task/​model

  • You’ve probably seen David Ha’s work, but something like https://​​es-clip.github.io/​​ could be a good starting point for dropping backprop.

  • The exotic activation function almost feels like cheating? Like I want the model the model to discover these useful structures, then try to understand them. But trying to do everything at once may be too hard.

  • Incredibility minor, but changing from onchange to oninput and dropping the animation will make the slider feel much slicker.

  What links here?

  • gwern's comment on Grow­ing Bon­sai Net­works with RNNs by ameo (Aug 7, 2023, 10:57 PM; 8 points)

An in­ter­ac­tive in­tro­duc­tion to grokking and mechanis­tic interpretability