Really cool work!
Would it be accurate to say that MoE models are an extremely coarse form of parameter decomposition? They check the box for faithfulness, and they’re an extreme example of optimizing minimality (each input x only uses one component of the model if you define each expert as a component) while completely disregarding simplicity.
I’ve recently come to see the world through a lens of H.D.R.E.A.M. (Hegelian dialectics rule everything around me). Thank you for this writeup which very cogently articulates a perfect example.
Thesis: status quo society, filled with many people who, as you put it, “don’t seem to be on the same page at all” with high agency, avoidance of busy work, the drive to do something by doing something different than everyone else.
Antithesis: repetitive calls to be contrarian that ironically have become somewhat conformist/mainstream
Synthesis:
I used to be really irritated with a constant drumbeat of messages about “agency” and “ambition” and a rejection of “conformism”, because it seemed to be preaching an extreme ideal that I couldn’t live up to.
But the thing is, I was already on board with “you should try to do cool things” and “you should think for yourself” and “some things worth doing are hard.” Those are pretty normal baseline assumptions! I just didn’t like the guilt-trippy, shaming, aggressively superior tone that can come across in popular online writing.
And then I realized…oh wait. Some people are really non-agentic, unambitious, and obsessed with social approval, and don’t share my baseline assumptions. I don’t like the extreme on this axis, any more than I like the extreme “nobody and nothing in this world is good enough for me, the Grumpy Ubermensch” types.
Very cool — one of the best explanations of Shapley values I’ve seen! Thinking abstractly — is it fair to say that the Great Man Theory of history is essentially an argument about the magnitude of the Shapley values of prominent historical figures (where the payoff is some measure of societal impact)?
“Living willpower” is willpower that is consciously understood as a bet on unknowns: “I don’t know whether this project will pay off, but I am betting my finite credibility on it anyhow.”
This feels related to the idea of Slack that Scott Alexander writes about here in SSC.
He gives this example towards the end:
7. Ideas. These are in constant evolutionary competition – this is the insight behind memetics. The memetic equivalent of slack is inferential range, aka “willingness to entertain and explore ideas before deciding that they are wrong”.
Willpower and money are both ways to create slack for yourself and others so that you can explore ideas/projects with an uncertain payoff.
Training on CoT traces seems like a particular instance of a general class of “self-defeating strategies.” Other examples include antibiotics/bacterial resistance (treating bacterial infections creates selective pressure that promotes resistant bacterial populations, gradually rendering the antibiotics ineffective for future use) and the dilemma in The Imitation Game after Turing and his team have cracked Enigma (acting upon the deciphered messages would tip off the Nazis and remove the Allies’ informational advantage).