It’s not a Schelling point if you communicate about it!
utilistrutil
Karma: 364
MATS Winter 2023-24 Retrospective
How LLMs Work, in the Style of The Economist
GDP is an absolute quantity. If GDP doubles, then that means something. So readers should be thinking about the distance between the curve and the x-axis.
But 1980 is arbitrary. When comparing 2020 to 2000, all that matters is that they’re 20 years apart. No one cares that “2020 is twice as far from 1980 as 2000” because time did not start in 1980.
This is the difference between a ratio scale and a cardinal scale. In a cardinal scale, the distance between points is meaningful, e.g., “The gap between 1 and 2 is twice as big as the gap between 2 and 4.” In a ratio scale, there is a well-defined zero point, which means the ratios of points are also meaningful, e.g., “4 is twice as large as 2.”
I just came across this word from John Koenig’s Dictionary of Obscure Sorrows, that nicely capture the thesis of All Debates Are Bravery Debates.
redesis n. a feeling of queasiness while offering someone advice, knowing they might well face a totally different set of constraints and capabilities, any of which might propel them to a wildly different outcome—which makes you wonder if all of your hard-earned wisdom’s fundamentally nonstraferable, like handing someone a gift card in your name that probably expired years ago.
Analogy Bank for AI Safety
(and perhaps also reversing some past value-drift due to the structure of civilization and so on)
Can you say more about why this would be desirable?
A lot of this piece is unique to high school debate formats. In the college context, every judge is themself a current or previous debater, so some of these tricks don’t work. (There are of course still times when optimizing for competitive success distracts from truth-seeking.)
Here are some responses to Rawls from my debate files:
A2 RawlsAhistorical
Violates property rights
Does not account for past injustices eg slavery, just asks what kind of society would you design from scratch. Thus not a useful guide for action in our fucked world.
Acontextual
Veil of ignorance removes contextual understanding, which makes it impossible to assess different states of the world. Eg from the original position, Rawls prohibits me from using my gender to inform my understanding of gender in different states of the world
Identity is not arbitrary! It is always contingent, yes, but morality is concerned with the interactions of real people, who have capacities, attitudes, and preferences. There are reasons for these things that are located in individual experiences and contexts, so they are not arbitrary.
But even if they were the result of pure chance, it’s unclear that these coincidences are the legitimate subject of moral scrutiny. I *am* a white man—I can’t change that. They need to explain why morality should be pretend otherwise. Only after conditioning on our particular context can we begin to reason morally.
The one place Rawls is interested in context is bad: he says the principle should only be applied within a society: but this precludes action on global poverty.
Rejects economic growth: the current generation is the one that is worst-off; saving now for future growth necessarily comes at the cost of foregone consumption, which hurts the current generation.
1. It’s pretty much a complete guide to action? Maybe there are decisions where it is silent, but that’s true of like every ethical theory like this (“but util doesn’t care about X!”). I don’t think the burden is on him to incorporate all the other concepts that we typically associate with justice. At very least not a problem for “justifying the kind of society he supports”
2. Like the two responses to this are either “Rawls tells you the true conception of the good, ignore the other ones” or “just allow for other-regarding preferences and proceed as usual” and either seems workable
3. Sure
4. Agree in general that Rawls does not account for different risk preferences but infinite risk aversion isn’t necessary for most practical decisions
5. Agree Rawls doesn’t usually account for future. But you could just use veil of ignorance over all future and current people, which collapses this argument into a specific case of “maximin is stupid because it doesn’t let us make the worst-off people epsilon worse-off in exchange for arbitrary benefits to others”
I think (B) is getting at a fundamental problem
What % evals/demos and what % mech interp would you expect to see if there wasn’t Goodharting? 1⁄3 and 1⁄5 doesn’t seem that high to me, given the value of these agendas and the advantages of touching reality that Ryan named.
these are the two main ways I would expect MATS to have impact: research output during the program and future research output/career trajectories of scholars.
We expect to achieve more impact through (2) than (1); per the theory of change section above, MATS’ mission is to expand the talent pipeline. Of course, we hope that scholars produce great research through the program, but often we are excited about scholars doing research (1) as a means for scholars to become better researchers (2). Other times, these goals are in tension. For example, some mentors explicitly de-emphasize research outputs, encouraging scholars to focus instead on pivoting frequently to develop better taste.
If you have your own reason to think (1) is comparably important to (2), I wonder if you also think providing mentees to researchers whose agendas you support is a similarly important source of MATS’ impact? From the Theory of Change section:
By mentoring MATS scholars, these senior researchers also benefit from research assistance
Thanks, Aaron! That’s helpful to hear. I think “forgetting” is a good candidate explanation because scholars answered that question right after competing Alignment 201, which is designed for breadth. Especially given the expedited pace of the course, I wouldn’t be surprised if people forgot a decent chunk of A201 material over the next couple months. Maybe for those two scholars, forgetting some A201 content outweighed the other sources of breadth they were afforded, like seminars, networking, etc.
MATS Summer 2023 Retrospective
Today I am thankful that Bayes’ Rule is unintuitive.
Much ink has been spilled complaining that Bayes’ Rule can yield surprising results. As anyone who has taken an introductory statistics class knows, it is difficult to solve a problem that requires an application of Bayes’ Rule without plugging values into the formula, at least for a beginner. Eventually, the student of Bayes may gain an intuition for the Rule (perhaps in odds form), but at that point they can be trusted to wield their intuition responsibly because it was won through disciplined practice.
This unintuitiveness is a feature, not a bug because it discourages motivated reasoning. If Bayes’ Rule were more intuitive, it would be simple to back out what P(A), P(B), and P(B|A) must be to justify your preferred posterior belief, and then argue for these quantities. It would also be simple to work backwards to select your prediction A from a favorable hypothesis space. Because Bayes’ Rule is unintuitive, these are challenging moves, and formally updating your beliefs is less vulnerable to motivated reasoning.
Happy Thanksgiving!
utilistrutil’s Shortform
Update: We have finalized our selection of mentors.
Apply for MATS Winter 2023-24!
I have a friend Balaam who has a very hard time saying no. If I ask him, “Would it bother you if I eat the last slice of pizza?” he will say “Of course that’s fine!” even if it would be somewhat upsetting or costly to him.
I think this is a reference to Guess/Ask/Tell Culture, so I’m linking that post for anyone interested :)
This happens in chess all the time!
Sign of the effect of open source on hype? Or of hype on timelines? I’m not sure why either would be negative.
Open source --> more capabilities R&D --> more profitable applications --> more profit/investment --> shorter timelines
The example I’ve heard cited is Stable Diffusion leading to LORA.
There’s a countervailing effect of democratizing safety research, which one might think outweighs because it’s so much more neglected than capabilities, more low-hanging fruit.