Was on Vivek Hebbar’s team at MIRI, now working with Adrià Garriga-Alonso onvarious empirical alignment projects.
I’m looking for projects in interpretability, activation engineering, and control/oversight; DM me if you’re interested in working with me.
I have signed no contracts or agreements whose existence I cannot mention.
Someone I know who works at Anthropic, not on alignment, has thought pretty hard about this and concluded it was better than alternatives. Some factors include
by working on capabilities, you free up others for alignment work who were previously doing capabilities but would prefer alignment
more competition on product decreases aggregate profits of scaling labs
At one point some kind of post was planned but I’m not sure if this is still happening.
I also think there are significant upskilling benefits to working on capabilities, though I believe this less than I did the other day.
I do think they disagree based on my experience working with Nate and Vivek. Eliezer has said he has only shared 40% of his models with even Nate for infosec reasons [1] (which surprised me!), so it isn’t surprising to me that they would have different views. Though I don’t know Eliezer well, I think he does believe in the basic point of Deep Deceptiveness (because it’s pretty basic) but also believes in coherence/utility functions more than Nate does. I can maybe say more privately but if it’s important asking one of them is better.
[1] This was a while ago so he might have actually said that Nate only has 40% of his models. But either way my conclusion is valid.
I think that it’s mostly Eliezer who believes so strongly in utility functions. Nate Soares’ post Deep Deceptiveness, which I claim is a central part of the MIRI threat model insofar as there is one, doesn’t require an agent coherent enough to satisfy VNM over world-states. In fact it can depart from coherence in several ways and still be capable and dangerous:
It can flinch away from dangerous thoughts;
Its goals can drift over time;
Its preferences can be incomplete;
Maybe every 100 seconds it randomly gets distracted for 10 seconds
The important property it has is a goal about the real world and applies general problem-solving skills to achieve it, and has no stable desire to use its full intelligence to be helpful/good for humans. No one has formalized this and so no one has proved interesting things about such an agent model.
Such a world could even be more dangerous. LLMs are steerable and relatively weak at consequentialist planning. There is AFAICT no fundamental reason why the next paradigm couldn’t be even less interpretable, less steerable, and more capable of dangerous optimization at a given level of economic utility.
VTI is basically the entire US market, and companies are added to the index as they go public. You will gain unless the value goes to private companies, but then you couldn’t invest in them anyway. There are other indices for international equities.
“want to pick a war with America” is really strange wording because China’s strategic goals are not “win a war against nuclear-armed America”, but things like “be able to control its claims in the South China Sea including invading Taiwan without American interference”. Likewise Russia doesn’t want to “pick a war with the EU” but rather annex Ukraine; if they were stupid enough to want the former they would have just bombed Paris. I don’t know whether national security people relate to the phrasing the same way but they do understand this.
I don’t know how to say this without sounding elitist, but my guess is that people who prolifically write LW comments and whose karma:(comments+posts) ratio is less than around 1.5:1 or maybe even 2:1 should be more selective in what they say. Around this range, it would be unwarranted for mods to rate-limit you, but perhaps the bottom 30% of content you produce is net negative considering the opportunity cost to readers.
Of course, one should not Goodhart for karma and Eliezer is not especially virtuous by having a 16:1 ratio, but 1.5:1 is a quite low ratio. You get 1.5 karma if 0.75 high-karma users or 1.5 low-karma users weakly upvote your comment. Considering that it takes barely a second to upvote and comments get tens of views, this is not a high bar for the average comment.
Caveats:
If your absolute number of comments + posts is low, the risk of making new users participate less outweighs the risk of comments clogging up the frontpage.
In some contexts votes might underestimate quality, like if your comment is on some esoteric topic that few people engage with.
Application seems better than other kinds of capabilities. I was thinking of capabilities as everything other than alignment work, so inclusive of application.
This is really exciting to me. If this work generalizes to future models capable of sophisticated planning in the real world, we will be able to forecast future actions that internally justify an AI’s current actions and thus tell whether they’re planning to coup or not, whether or not an explicit general-purpose representation of the objective exists.
Since many people seem to disagree, I’m going to share some reasons why I believe this:
AGI that poses serious existential risks seems at least 6 years away, and safety work seems much more valuable at crunch time, such that I think more than half of most peoples’ impact will be more than 5 years away. So skilling up quickly should be the primary concern, until your timelines shorten.
Mentorship for safety is still limited. If you can get an industry safety job or get into MATS, this seems better than some random AI job, but most people can’t.
I think there are also many policy roles that are not crowded and potentially time-critical, which people should also take if available.
Funding is also limited in the current environment. I think most people cannot get funding to work on alignment if they tried? This is fairly cruxy and I’m not sure of it, so someone should correct me if I’m wrong.
The relative impact of working on capabilities is smaller than working on alignment—there are still 10x as many people doing capabilities as alignment, so unless returns don’t diminish or you are doing something unusually harmful, you can work for 1 year on capabilities and 1 year on alignment and gain 10x.
However, it seems bad to work at places that have particularly bad safety cultures and are explicitly trying to create AGI, possibly including OpenAI.
Safety could get even more crowded, which would make upskilling to work on safety net negative. This should be a significant concern, but I think most people can skill up faster than this.
Skills useful in capabilities are useful for alignment, and if you’re careful about what job you take there isn’t much more skill penalty in transferring them than, say, switching from vision model research to language model research.
Capabilities often has better feedback loops than alignment because you can see whether the thing works or not. Many prosaic alignment directions also have this property. Interpretability is getting there, but not quite. Other areas, especially in agent foundations, are significantly worse.
I suspect that working on capabilities (edit: preferably applications rather than building AGI) in some non-maximally-harmful position is actually the best choice for most junior x-risk concerned people who want to do something technical. Safety is just too crowded and still not very tractable.
We considered that “catastrophic” might have that connotation, but we couldn’t think of a better name and I still feel okay about it. Our intention with “catastrophic” was to echo the standard ML term of “catastrophic forgetting”, not a global catastrophe. In catastrophic forgetting the model completely forgets how to do task A after it is trained on task B, it doesn’t do A much worse than random. So we think that “catastrophic Goodhart” gives the correct idea to people who come from ML.
The natural question is then: why didn’t we study circumstances in which optimizing for a proxy gives you utility in the limit? Because it isn’t true under the assumptions we are making. We wanted to study regressional Goodhart, and this naturally led us to the independence assumption. Previous work like Zhuang et al and Skalse et al has already formalized the extremal Goodhart / “use the atoms for something else” argument that optimizing for one goal would be bad for another goal, and we thought the more interesting part was showing that bad outcomes are possible even when error and utility are independent. Under the independence assumption, it isn’t possible to get less than 0 utility.
To get utility in the frame where proxy = error + utility, you would need to assume something about the dependence between error and utility, and we couldn’t think of a simple assumption to make that didn’t have too many moving parts. I think extremal Goodhart is overall more important, but it’s not what we were trying to model.
Lastly, I think you’re imagining “average” outcome as a random policy, which is an agent incapable of doing significant harm. The utility of the universe is still positive because you can go about your life. But in a different frame, random is really bad. Right now we pretrain models and then apply RLHF (and hopefully soon, better alignment techniques). If our alignment techniques produce no more utility than the prior, this means the model is no more aligned than the base model, which is a bad outcome for OpenAI. Superintelligent models might be arbitrarily capable of doing things, so the prior might be better thought of as irreversibly putting the world in a random state, which is a global catastrophe.
Disagree. Epistemics is a group project and impatiently interrupting people can make both you and your interlocutor less likely to combine your information into correct conclusions. It is also evidence that you’re incurious internally which makes you worse at reasoning, though I don’t want to speculate on Eliezer’s internal experience in particular.
agent that maximizes some combination of the entropy of its actions, and their expected utility. i.e. the probability of taking an action a is proportional to exp(βE[U|a]) up to a normalization factor.
Note that if the distribution of utility under the prior is heavy-tailed, you can get infinite utility even with arbitrarily low relative entropy, so the optimal policy is undefined. In the case of goal misspecification, optimization with a KL penalty may be unsafe or get no better utility than the prior.
I believe Nanosystems is mostly valid physics (though I am still unsure about this) and in the far future, after GDP has doubled ten or twenty times, we will think of it like current rocket scientists think of Tsiolkovsky’s writing: speculative science that gave a glimpse surprisingly far into the future through an understanding of the timeless basic principles at play, though misses many implementation details. And just like the sense of perspective by knowing in 1914 it’s theoretically possible to send people to Mars on a ship with airlocks, fueled by hydrogen-oxygen engines and steered by cold gas thrusters, I think we gain an enormously valuable perspective on the universe by knowing that it is (probably) theoretically possible to perform most chemical reactions and many molecular assembly tasks with 99% efficiency using machines that precisely place atoms, self-replicate once an hour, and require only ultrapure gases, various trace metals, and electricity as input.
The title of this dialogue promised a lot, but I’m honestly a bit disappointed by the content. It feels like the authors are discussing exactly how to run particular mentorship programs and structure grants and how research works in full generality, while no one is actually looking at the technical problems. All field-building efforts must depend on the importance and tractability of technical problems, and this is just as true when the field is still developing a paradigm. I think a paradigm is established only when researchers with many viewpoints build a sense of which problems are important, then try many approaches until one successfully solves many such problems, thus proving the value of said approach. Wanting to find new researchers to have totally new takes and start totally new illegible research agendas is a level of helplessness that I think is unwarranted—how can one be interested in AF without some view on what problems are interesting?
I would be excited about a dialogue that goes like this, though the format need not be rigid:
What are the most important [1] problems in agent foundations, with as much specificity as possible?
Responses could include things like:
A sound notion of “goals with limited scope”: can’t nail down precise desiderata now, but humans have these all the time, we don’t know what they are, and they could be useful in corrigibility or impact measures.
Finding a mathematical model for agents that satisfies properties of logical inductors but also various other desiderata
Further study of corrigibility and capability of agents with incomplete preferences
Participants discuss how much each problem scratches their itch of curiosity about what agents are.
What techniques have shown promise in solving these and other important problems?
Does [infra-Bayes, Demski’s frames on embedded agents, some informal ‘shard theory’ thing, …] have a good success to complexity ratio?
probably none of them do?
What problems would benefit the most from people with [ML, neuroscience, category theory, …] expertise?
[1]: (in the Hamming sense that includes tractability)
Does MIRI have a statement on recent OpenAI events? I’m pretty excited about frank reflections on current events as helping people to orient.
Points 1-3 and the idea that superintelligences will be able to understand our values (which I think everyone believes). But the conclusion needs a bunch of additional assumptions.
I suggest people read both that and Deep Deceptiveness (which is not about deceptiveness in particular) and think about how both could be valid, because I think they both are.
Jane at FakeLab has a background in interpretability but is currently wrangling data / writing internal tooling / doing some product thing because the company needs her to, because otherwise FakeLab would have no product and be unable to continue operating including its safety research. Steve has comparative advantage at Jane’s current job.
It seems net bad because the good effect of slowing down OpenAI is smaller than the bad effect of GM racing? But OpenAI is probably slowed down—they were already trying to build AGI and they have less money and possibly less talent. Thinking about the net effect is complicated and I don’t have time to do it here. The situation with joining a lab rather than founding one may also be different.