Jacob_Hilton(Jacob Hilton)

A bird’s eye view of ARC’s research

• Jacob_Hilton 11 Sep 2024 3:50 UTC

  8 points

  0

  in reply to: Raemon’s comment on: Rae­mon’s Short­form Feed

  The LLM output looks correct to me.

Back­doors as an anal­ogy for de­cep­tive alignment

• Jacob_Hilton 4 Jul 2024 20:19 UTC

  LW: 5 AF: 3

  0

  AF

  in reply to: John Schulman’s comment on: For­mal ver­ifi­ca­tion, heuris­tic ex­pla­na­tions and sur­prise accounting

  Yes, I think the most natural way to estimate total surprise in practice would be to use sampling like you suggest. You could try to find the best explanation for “the model does $bad_thing with probability less than 1 in a million” (which you believe based on sampling) and then see how unlikely $bad_thing is according to the resulting explanation. In the Boolean circuit worked example, the final 23-bit explanation is likely still the best explanation for why the model outputs TRUE on at least 99% of inputs, and we can use this explanation to see that the model actually outputs TRUE on all inputs.

  Another possible approach is analogous to fine-tuning. You could start by using surprise accounting to find the best explanation for “the loss of the model is L” (where L is estimated during training), which should incentivize rich explanations of the model’s behavior in general. Then to estimate the probability that model does some rare $bad_thing, you could “fine-tune” your explanation using an objective that encourages it to focus on the relevant tails of the distribution. We have more ideas about estimating the probability of events that are too rare to estimate via sampling, and have been considering objectives other than surprise accounting for this. We plan to share these ideas soon.

• Jacob_Hilton 28 Jun 2024 20:39 UTC

  5 points

  0

  in reply to: Kaarel’s comment on: For­mal ver­ifi­ca­tion, heuris­tic ex­pla­na­tions and sur­prise accounting

  Yes, that’s a clearer way of putting it in the case of the circuit in the worked example. The reason I said “for no apparent reason” is that there could be some redundancy in the explanation. For example, if you already had an explanation for the output of some subcircuit, you shouldn’t pay additional surprise if you then check the output of that subcircuit in some particular case. But perhaps this was a distracting technicality.

• Jacob_Hilton 27 Jun 2024 16:29 UTC

  5 points

  2

  in reply to: Alexander Gietelink Oldenziel’s comment on: For­mal ver­ifi­ca­tion, heuris­tic ex­pla­na­tions and sur­prise accounting

  I would say that they are motivated by the same basic idea, but are applied to different problems. The MDL (or the closely-related BIC) is a method for model selection given a dataset, whereas surprise accounting is a method for evaluating heuristic explanations, which don’t necessarily involve model selection.

  Take the Boolean circuit worked example: what is the relevant dataset? Perhaps it is the 256 (input, TRUE) pairs. But the MDL would select a much simpler model, namely the circuit that ignores the input and outputs TRUE (or “x_1 OR (NOT x_1)” if it has to consist of AND, OR and NOT gates). On the other hand, a heuristic explanation is not interested choosing a simpler model, but is instead interested in explaining why the model we have been given behaves in the way it does.

  The heuristic explanations in the post do use a single prior or over the set of circuits, which we also call a “reference class”. But we wish to allow explanations that use other reference classes, as well as explanations that combine multiple reference classes, and perhaps even explanations that use “subjective” reference classes that do not seem to correspond to any precise prior. These are the sorts of issues explored in the upcoming paper. Ultimately, though, a lot of our heuristic arguments and the surprise accounting for them remain somewhat ambiguous or informal.

• Jacob_Hilton 26 Jun 2024 21:25 UTC

  6 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: For­mal ver­ifi­ca­tion, heuris­tic ex­pla­na­tions and sur­prise accounting

  Yes, the cost of 1 bit for the OR gate was based on the somewhat arbitrary choice to consider only OR and AND gates. A bit more formally, the heuristic explanations in the post implicitly use a “reference class” of circuits where each binary gate was randomly chosen to be either an OR or an AND, and each input wire to a binary gate was randomly chosen to have a NOT or not. The arbitrariness of this choice of reference class is one obstruction to formalizing heuristic explanations and surprise accounting. We are currently preparing a paper that explores this and related topics, but unfortunately the core issue remains unresolved.

For­mal ver­ifi­ca­tion, heuris­tic ex­pla­na­tions and sur­prise accounting

• Jacob_Hilton 31 May 2024 21:52 UTC

  11 points

  −2

  in reply to: Adam Scholl’s comment on: Non-Dis­par­age­ment Ca­naries for OpenAI

  See the statement from OpenAI in this article:

  We’re removing nondisparagement clauses from our standard departure paperwork, and we’re releasing former employees from existing nondisparagement obligations unless the nondisparagement provision was mutual. We’ll communicate this message to former employees.

  They have communicated this to me and I believe I was in the same category as most former employees.

  I think the main reasons so few people have mentioned this are:

  • As I mentioned, there is still some legal ambiguity and additional avenues for retaliation

  • Some people are taking their time over what they want to say

  • Most people don’t want to publicly associate themselves with a controversial situation

  • Most people aren’t inclined to disparage their former employer anyway, and so they may not think of their own situation as that big of a deal

• Jacob_Hilton 31 May 2024 14:47 UTC

  13 points

  −19

  in reply to: Amalthea’s comment on: Non-Dis­par­age­ment Ca­naries for OpenAI

  Yeah I agree with this, and my original comment comes across too strongly upon re-reading. I wanted to point out some counter-considerations, but the comment ended up unbalanced. My overall view is:

  • It was highly inappropriate for the company to have been issuing these agreements so widely, especially using such aggressive tactics and without allowing disclosure of the agreement, given the technology that it is developing.

  • The more high-profile and credible a person is, the more damaging it is for this person to have been subject to the agreement.

  • Nevertheless, it is a mistake to think of potential “disparagement” as part of the job duties of most of the people mentioned, and the post appears to wildly misinterpret the meaning of this term as “taking any actions which might make the company less valuable”.

  • Ultimately, it would have looked extremely bad for the company to enforce one of these agreements, so the primary effect of the contract comes down to how individuals felt that it constrained their behavior. We don’t have great visibility into this. It’s possible that some of these people felt quite constrained, and it’s also possible that some of these people weren’t even aware of the non-disparagement clause because they didn’t notice it when they signed.

  • Thankfully, most of this is now moot as the company has retracted the contract. I should emphasize that there may remain some legal ambiguity and additional avenues for retaliation, but I am optimistic that these will be cleaned up in the near future. There will still be non-disparagement agreements in place in cases where “the non-disparagement provision was mutual” (in the words of the company), but my strong guess is that this refers only to the original Anthropic departures and perhaps a handful of other individuals who were high up at the company.

  • It remains important for people to disclose their financial interest in the company when appropriate, or in some cases give up this interest to avoid a conflict of interest.

  Note: I have a financial interest in the company and was subject to one of these agreements until recently.

• Jacob_Hilton 31 May 2024 6:50 UTC

  2 points

  −50

  on: Non-Dis­par­age­ment Ca­naries for OpenAI

  We were especially alarmed to notice that the list contains at least 12 former employees currently working on AI policy, and 6 working on safety evaluations. This includes some in leadership positions, for example:

  I don’t really follow this reasoning. If anything, playing a leadership role in AI policy or safety evaluations will usually give you an additional reason not to publicly disparage AI companies, to avoid being seen as partisan, making being subject to such an agreement less of an issue. I would be pretty surprised if such people subject to these agreements felt particularly constrained in what they could say as part of their official duties, although if I am wrong about this then it does seem like quite a concerning thing to have happened. The obvious exception to this is if the role involves unofficial public commentary about labs, but it’s not obvious to me that this has been a big part of the role of any of the people on your list, and even then, they may not have felt especially constrained, depending on the individual. It’s also worth noting that several of these roles require the holder to give up or donate lab equity to avoid any conflict of interest, regardless of any non-disparagement agreements.

  I suspect the crux may be our differing interpretations of the agreement. I’m not sure where your interpretation that it prohibits “taking any actions which might make the company less valuable” comes from, maybe you could highlight the part of the agreement you are basing that on.

• Jacob_Hilton 24 May 2024 8:10 UTC

  5 points

  0

  in reply to: Celer’s comment on: Com­mon mis­con­cep­tions about OpenAI

  If the question is whether I think they were true at time given the information I have now, I think all of the individual points hold up except for the first and third “opinions”. I am now less sure about what OpenAI leadership believed or cared about. The last of the “opinions” also seems potentially overstated. Consequently, the overall thrust now seems off, but I still think it was good to share my views at the time, to start a discussion.

  If the question is about the state of the organization now, I know less about that because I haven’t worked there in over a year. But the organization has certainly changed a lot since this post was written over 18 months ago.

• Jacob_Hilton 16 Dec 2023 5:39 UTC

  LW: 11 AF: 8

  0

  AF

  on: Com­mon mis­con­cep­tions about OpenAI

  Since this post was written, OpenAI has done much more to communicate its overall approach to safety, making this post somewhat obsolete. At the time, I think it conveyed some useful information, although it was perceived as more defensive than I intended.

  My main regret is bringing up the Anthropic split, since I was not able to do justice to the topic. I was trying to communicate that OpenAI maintained its alignment research capacity, but should have made that point without mentioning Anthropic.

  Ultimately I think the post was mostly useful for sparking some interesting discussion in the comments.

• Jacob_Hilton 20 Jun 2023 7:22 UTC

  LW: 9 AF: 7

  0

  AF

  on: Mode col­lapse in RL may be fueled by the up­date equation

  I think KL/​entropy regularization is usually used to prevent mode collapse partly because it has nice theoretical properties. In particular, it is easy to reason about the optimal policy for the regularized objective—see for example the analysis in the paper Equivalence Between Policy Gradients and Soft Q-Learning.

  Nevertheless, action-dependent baselines do appear in the literature, although the story is a bit confusing. This is my understanding of it from some old notes:

  • The idea was explored in Q-Prop. But unlike you, their intention was not to change the optimal policy, but rather to reduce the variance of the policy gradient. Therefore they also incorporated an additional term to cancel out the bias introduced by the action-dependent baseline. (Incidentally, perhaps this analysis is also relevant to understanding ACTDE.)

  • Later, The Mirage of Action-Dependent Baselines showed that in fact the variance reduction due the action-dependent baseline was negligible, and the entire benefit of Q-Prop was essentially due to a bug! The implementation normalized advantage estimates, but failed to apply the same adjustment to the bias-correction term, which turned out to be independently helpful because it’s essentially the DDPG training objective.

• Jacob_Hilton 19 Jun 2023 16:33 UTC

  5 points

  1

  in reply to: RGRGRG’s comment on: ARC is hiring the­o­ret­i­cal researchers

  We will do our best to fairly consider all applications, but realistically there is probably a small advantage to applying earlier. This is simply because there is a limit to how quickly we can grow the organization, so if hiring goes better than expected then it will be longer before we can take on even more people. With that being said, we do not have a fixed number of positions that we are hiring for; rather, we plan to vary the number of hires we make based on the strength of the applications we receive. Moreover, if we were unable to hire someone due to capacity constraints, we would very likely be interested in hiring them at a later date. For these reasons, I think the advantage to applying earlier is a fairly small consideration overall, and it sounds like it would make more sense for you to apply whenever you are comfortable.

• Jacob_Hilton 14 Jun 2023 19:19 UTC

  LW: 6 AF: 3

  0

  AF

  in reply to: Gghhuutt’s comment on: ARC is hiring the­o­ret­i­cal researchers

  The questions on the take-home test vary in difficulty but are generally easier than olympiad problems, and should be accessible to graduates with relevant background. However, it is important to note that we are ultimately interested in research ability rather than the ability to solve self-contained problems under timed conditions. So although the take-home test forms part of our assessment, we also look at other signals such as research track-record (while recognizing that assessing research ability is unfortunately very hard).

  (Note: I am talking about the current version of the test, it’s possible that the difficulty will change as we refine our interview process.)

• Jacob_Hilton 12 Jun 2023 21:03 UTC

  17 points

  8

  in reply to: Shmi’s comment on: ARC is hiring the­o­ret­i­cal researchers

  I think the kind of mathematical problem solving we’re engaged in is common across theoretical physics (although this is just my impression as a non-physicist). I’ve noticed that some specific topics that have come up (such as Gaussian integrals and permanents) also crop up in quantum field theory, but I don’t think that’s a strong reason to prefer that background particularly. Broad areas that often come up include probability theory, computational complexity and discrete math, but it’s not necessary to have a lot of experience in those areas, only to be able to pick things up from them as needed.

ARC is hiring the­o­ret­i­cal researchers

• Jacob_Hilton 4 May 2023 17:33 UTC

  5 points

  2

  in reply to: paulfchristiano’s comment on: Prizes for ma­trix com­ple­tion problems

  It’s not quite this simple, the same issue arises if every PSD completion of the known-diagonal minor has zero determinant (e.g. ((?, 1, 2), (1, 1, 1), (2, 1, 1))). But I think in that case making the remaining diagonal entries large enough still makes the eigenvalues at least −ε, which is good enough.

• Jacob_Hilton 21 Feb 2023 17:28 UTC

  35 points

  33

  on: AI al­ign­ment re­searchers don’t (seem to) stack

  I think the examples you give are valid, but there are several reasons why I think the situation is somewhat contingent or otherwise less bleak than you do:

  1. Counterexamples: I think there are research agendas that are less pre-paradigmatic than the ones you’re focused on that are significantly more (albeit not entirely) parallelizable. For example, mechanistic interpretability and scalable oversight both have multiple groups focused on them and have grown substantially over the last couple of years. I’m aware that we disagree about how valuable these directions are.

  2. Survival of the fittest: Unfortunately I think in cases where an individual has been pursuing a research direction for many years and has tried but failed to get anyone else on board with it, there is some explanatory power to the hypothesis that the direction is not that productive. Note that I’m not claiming to have a strong view on any particular agenda, and there are of course other possibilities in any given case. On the flip side, I expect promising directions to gain momentum over time, even if only gradually, and I consider the counterexamples from point 1 to be instances of this effect.

  3. Fixable coordination/​deference failures: I think it would be a mistake for absolutely everyone to go off and try to develop their own alignment strategy from scratch, and it’s plausible that the group you’re focused on is erring too far in this direction. My own strategy has been to do my best to develop my own inside view (which I think is important for research prioritization and motivation as well from a group epistemics perspective), use this to narrow down my set of options to agendas I consider plausible, but be considerably more willing to defer when it comes to making a final call about which agenda to pursue.

  4. Clarity from AI advances: If the risk from AI is real, then I expect the picture of it to become clearer over time as AI improves. As a consequence, it should become clearer to people which directions are worth pursuing, and theoretical approaches should evolve into practical ones than can be iterated on empirically. This should both cause the field to grow and lead to more parallelizable work. I think this is already happening, and even the public at large is picking up on the spookiness of current alignment failures (even though the discourse is unsurprisingly very muddled).