Rubi J. Hudson

• Rubi J. Hudson 6 May 2024 3:32 UTC

  LW: 2 AF: 1

  1

  AF

  in reply to: silentbob’s comment on: Search­ing for Search­ing for Search

  I’d take an agnostic view on whether LLMs are doing search internally. Crucially, though, I think the relevant output to be searching over is distributions of tokens, rather than the actual token that gets chosen. Search is not required to generate a single distribution over next tokens.

  I agree that external search via scaffolding can also be done, and would be much easier to identify, but without understanding the internals it’s hard to know how powerful the search process will be.

• Rubi J. Hudson 4 Apr 2024 16:05 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: ryan_greenblatt’s comment on: The Case for Pre­dic­tive Models

  Thanks for taking the time to write out your response. I think the last point you made gets at the heart of our difference in perspectives.

  • You could hope for substantial coordination to wait for bigger models that you only use via CPM, but I think bigger models are much riskier than well elicited small models so this seems to just make the situation worse putting aside coordination feasibility.

  If we’re looking at current LLMs and asking whether conditioning provides an advantage in safely eliciting useful information, then for the most part I agree with your critiques. I also agree that bigger models are much riskier, but I have the expectation that we’re going to get them anyway. With those more powerful models come new potential issues, like predicting manipulated observations and performative prediction, that we don’t see in current systems. Strategies like RLHF also become riskier, as deceptive alignment becomes more of a live possibility with greater capabilities.

  My motivation for this approach is in raising awareness and addressing the risks that seem likely to arise in future predictive models, regardless of the ends to which they’re used. Then, success in avoiding the dangers from powerful predictive models would open the possibility of using them to reduce all-cause existential risk.

• Rubi J. Hudson 3 Apr 2024 20:35 UTC

  LW: 5 AF: 4

  0

  AF

  in reply to: ryan_greenblatt’s comment on: The Case for Pre­dic­tive Models

  I’d be very interested in hearing the reasons why you’re skeptical of the approach, even a bare-bones outline if that’s all you have time for.

The Case for Pre­dic­tive Models

Search­ing for Search­ing for Search

• Rubi J. Hudson 8 Feb 2024 0:08 UTC

  1 point

  0

  in reply to: Jacob Pfau’s comment on: Abram Dem­ski’s ELK thoughts and pro­posal—distillation

  Ah, ok, I see what you’re saying now. I don’t see any reason why restricting to input space counterfactuals wouldn’t work, beyond the issues described with predictor-state counterfactuals. Possibly a performance hit from needing to make larger changes. In the worst case, a larger minimum change size might hurt with specifying the direct reporter.

• Rubi J. Hudson 7 Feb 2024 19:48 UTC

  LW: 1 AF: 1

  0

  AF

  in reply to: Jacob Pfau’s comment on: Abram Dem­ski’s ELK thoughts and pro­posal—distillation

  Sorry, I’m not quite clear what you mean by this, so I might be answering the wrong question.

  I believe counterfactuals on the input space are a subset of counterfactuals on the predictor’s state, because the input space’s influence is through the predictor’s state, but modifying the predictor’s state can also reach states that don’t correspond to any input. As such, I don’t think counterfactuals on the input space add any power to the proposal.

• Rubi J. Hudson 14 Dec 2023 10:56 UTC

  1 point

  0

  on: AI Views Snapshots

  I find one consistent crux I have with people not concerned about AI risk is that they believe massively more resources will be invested into technical safety before AGI is developed.

  In the context of these statements, I would put it as something like “The number of people working full-time on technical AI Safety will increase by an order of magnitude by 2030”.

• Rubi J. Hudson 6 Dec 2023 6:40 UTC

  LW: 1 AF: 1

  0

  AF

  on: List of strate­gies for miti­gat­ing de­cep­tive alignment

  Long-term planning is another capability that is likely necessary for deceptive alignment that could. Obviously a large alignment tax, but there are potentially ways to mitigate that. It seems at least as promising as some other approaches you listed.

• Rubi J. Hudson 18 Nov 2023 2:02 UTC

  LW: 1 AF: 1

  0

  AF

  in reply to: Steven Byrnes’s comment on: New re­port: “Schem­ing AIs: Will AIs fake al­ign­ment dur­ing train­ing in or­der to get power?”

  I don’t find goal misgeneralization vs schemers to be as much as a dichotomy as this comment is making it out to be. While they may be largely distinct for the first period of training, the current rollout method for state of the art seems to be “give a model situational awareness and deploy it to the real world, use this to identify alignment failures, retrain the model, repeat steps 2 and 3”. If you consider this all part of the training process (and I think that’s a fair characterization), model that starts with goal misgeneralization quickly becomes a schemer too.

• Rubi J. Hudson 16 Nov 2023 2:52 UTC

  LW: 2 AF: 1

  0

  AF

  on: Game The­ory with­out Argmax [Part 1]

  I think this part uses an unfair comparison:

  Supposes that $X=\{x_1,\dots ,x_n\}$ and $W^{+}=\{w_1,\dots ,w_m\}$ are small finite sets. A task $f:X\to W^{+}$ can be implemented as dictionary whose keys lie in $X$ and whose values lie in $W^{+}$, which uses $n\log m$ bits. The functional $\psi :J^P(X,W^{+})$ can be implemented as a program which receives input of type $Dict[X,W]$ and returns output of type $List\left[X\right]$. Easy!

  In the subjective account, by contrast, the task $f:X\to R$ requires infinite bits to specify, and the functional $\psi :J^P(X,R)$ must somehow accept a representation of an arbitrary function $f:X\to R$. Oh no! This is especially troubling for embedded agency, where the agent’s decision theory must run on a physical substrate.

  If X and W+ are small finite sets, then any behavior can be described with a utility function requiring only a finite number of bits to specify. You only need to use R as the domain when W+ is infinite, such as when outcomes are continuous, in which case the dictionaries require infinite bits to specify too.

  I think this is representative of an unease I have with the framing of this sequence. It seems to be saying that the more general formulation allows for agents that behave in ways that utility maximizers cannot, but most of these behaviors exist for maximizers of certain utility functions. I’m still waiting for the punchline of what AI safety relevant aspect requires higher order game theory rather than just maximizing agents, particularly if you allow for informational constraints.

• Rubi J. Hudson 1 Jun 2023 20:34 UTC

  LW: 6 AF: 4

  5

  AF

  in reply to: Caspar Oesterheld’s comment on: Con­di­tional Pre­dic­tion with Zero-Sum Train­ing Solves Self-Fulfilling Prophecies

  I think, from an alignment perspective, having a human choose their action while being aware of the distribution over outcomes it induces is much safer than having it effectively chosen for them by their specification of a utility function. This is especially true because probability distributions are large objects. A human choosing between them isn’t pushing in any particular direction that can make it likely to overlook negative outcomes, while choosing based on the utility function they specify leads to exactly that. This is all modulo ELK, of course.

  I’m not sure I understand the variant you proposed. How is that different than the Othman and Sandholm MAX rule?

• Rubi J. Hudson 29 May 2023 11:23 UTC

  LW: 2 AF: 2

  0

  AF

  in reply to: CBiddulph’s comment on: Con­di­tional Pre­dic­tion with Zero-Sum Train­ing Solves Self-Fulfilling Prophecies

  Thanks for the comment. I agree that, ideally, we would find a way not to have two wholly separate models and instead somehow train a model against itself. I think a potential issue with your proposal is that small perturbations could have discontinuous effects, the anticipation of which distorts predictions. However, it would be interesting to think about further to see if there’s some way to avoid that issue.

• Rubi J. Hudson 29 May 2023 11:10 UTC

  LW: 7 AF: 5

  0

  AF

  in reply to: Caspar Oesterheld’s comment on: Con­di­tional Pre­dic­tion with Zero-Sum Train­ing Solves Self-Fulfilling Prophecies

  Thanks Caspar, your comments here and on earlier drafts are appreciated. We’ll expand more on the positioning within the related literature as we develop this into a paper.

  As for your work on Decision Scoring Rules and the proposal in your comment, the biggest distinction is that this post’s proposal does not require specifying the decision maker’s utility function in order to reward one of the predictors and shape their behavior into maximizing it. That seems very useful to me, as if we were able to properly specify the desired utility function, we could skip using predictive models and just train an AI to maximize that instead (modulo inner alignment).

• Rubi J. Hudson 29 May 2023 10:46 UTC

  LW: 3 AF: 3

  1

  AF

  in reply to: ryan_greenblatt’s comment on: Con­di­tional Pre­dic­tion with Zero-Sum Train­ing Solves Self-Fulfilling Prophecies

  For the first point, I agree that the SGD pushes towards closing any gaps. My concern is that at the moment, we don’t know how small the gaps need to be to get the desired behavior (and this is what we are working on modelling now). On top of that, depending on how the models are initialized, the starting gap may be quite large, so the dynamics of how gaps close throughout the training process seems important to study further.

  For the second point, I think we are also in agreement. If the training process leads the AI to learning “If I predict that this action will destroy the world, the humans won’t choose it”, which then leads to dishonest predictions. However, I also find the training process converging to a mesa-optimizer for the training objective (or something sufficiently close) to be somewhat more plausible.

Con­di­tional Pre­dic­tion with Zero-Sum Train­ing Solves Self-Fulfilling Prophecies

• Rubi J. Hudson 20 Feb 2023 3:22 UTC

  1 point

  0

  in reply to: Gurkenglas’s comment on: Con­di­tion­ing Pre­dic­tive Models: Outer al­ign­ment via care­ful conditioning

  In the first part of this sequence, we clarify that we are focusing on the case where the model is a predictive model of the world. The fourth part, on making inner alignment as easy as possible, outlines some reasons why we think this kind of predictive model is possible (even likely) outcome of the training process. Of course, it is also possible that the model is not precisely a predictive model, but is still close enough to one that the content of “Conditioning Predictive Models” is still relevant.

• Rubi J. Hudson 20 Feb 2023 2:07 UTC

  3 points

  0

  in reply to: Charlie Steiner’s comment on: Con­di­tion­ing Pre­dic­tive Models: In­ter­ac­tions with other approaches

  Yes, you are correct that RL with KL penalties only approximates a Bayesian update in the limit, after enough steps to converge. Determining the speed of this convergence, especially for LLMs, remains an area for future work.

Con­di­tion­ing Pre­dic­tive Models: Open prob­lems, Con­clu­sion, and Appendix

• Rubi J. Hudson 10 Feb 2023 19:03 UTC

  1 point

  0

  in reply to: Charlie Steiner’s comment on: Mechanism De­sign for AI Safety—Agenda Creation Retreat

  I’m not sure, but if anyone knows how to contact them, they could be a great fit.