David Scott Krueger (formerly: capybaralet)

Karma: 1,952

I’m more active on Twitter than LW/AF these days: https://twitter.com/DavidSKrueger

Bio from https://www.davidscottkrueger.com/:
I am an Assistant Professor at the University of Cambridge and a member of Cambridge’s Computational and Biological Learning lab (CBL). My research group focuses on Deep Learning, AI Alignment, and AI safety. I’m broadly interested in work (including in areas outside of Machine Learning, e.g. AI governance) that could reduce the risk of human extinction (“x-risk”) resulting from out-of-control AI systems. Particular interests include:

Reward modeling and reward gaming
Aligning foundation models
Understanding learning and generalization in deep learning and foundation models, especially via “empirical theory” approaches
Preventing the development and deployment of socially harmful AI systems
Elaborating and evaluating speculative concerns about more advanced future AI systems

David Scott Krueger (formerly: capybaralet) 14 Sep 2024 22:59 UTC
LW: 4 AF: 3
0
AF
in reply to: Nick Corvino’s comment on: Survey: How Do Elite Chinese Students Feel About the Risks of AI?
OK, so it’s not really just your results? You are aggregating across these studies (and presumably ones of “Westerners” as well)? I do wonder how directly comparable things are… Did you make an effort to translate a study or questions from studies, or are the questions just independently conceived and formulated?

David Scott Krueger (formerly: capybaralet) 14 Sep 2024 22:57 UTC
LW: 4 AF: 3
0
AF
in reply to: Daniel Kokotajlo’s comment on: Is there any rigorous work on using anthropic uncertainty to prevent situational awareness / deception?
No, I was only responding to the the first part.

David Scott Krueger (formerly: capybaralet) 4 Sep 2024 15:55 UTC
LW: 4 AF: 1
1
AF
in reply to: Daniel Kokotajlo’s comment on: Is there any rigorous work on using anthropic uncertainty to prevent situational awareness / deception?
Not necessarily fooling it, just keeping it ignorant. I think such schemes can plausibly scale to very high levels of capabilities, perhaps indefinitely, since intelligence doesn’t give one the ability to create information from thin air...

David Scott Krueger (formerly: capybaralet) 4 Sep 2024 13:18 UTC
LW: 4 AF: 2
0
AF
on: Consent across power differentials
This is a super interesting and important problem, IMO. I believe it already has significant real world practical consequences, e.g. powerful people find it difficult to avoid being surrounded by sychophants: even if they really don’t want to be, that’s just an extra constraint for the sychophants to satisfy (“don’t come across as sychophantic”)! I am inclined to agree that avoiding power differentials is the only way to really avoid these perverse outcomes in practice, and I think this is a good argument in favor of doing so.

--------------------------------------
This is also quite related to an (old, unpublished) work I did with Jonathan Binas on “bounded empowerment”. I’ve invited you to the Overleaf (it needs to clean-up, but I’ve also asked Jonathan about putting it on arXiv).

To summarize: Let’s consider this in the case of a superhuman AI, R, and a human H. The basic idea of that work is that R should try and “empower” H, and that (unlike in previous works on empowerment), there are two ways of doing this:
1) change the state of the world (as in previous works)
2) inform H so they know how to make use of the options available to them to achieve various ends (novel!)

If R has a perfect model of H and the world, then you can just compute how to effectively do these things (it’s wildly intractable, ofc). I think this would still often look “patronizing” in practice, and/or maybe just lead to totally wild behaviors (hard to predict this sort of stuff...), but it might be a useful conceptual “lead”.

Random thought OTMH: Something which might make it less “patronizing” is if H were to have well-defined “meta-preferences” about how such interactions should work that R could aim to respect.

David Scott Krueger (formerly: capybaralet) 4 Sep 2024 12:45 UTC
LW: 3 AF: 1
0
AF
on: Survey: How Do Elite Chinese Students Feel About the Risks of AI?
What makes you say this: “However, our results suggest that students are broadly less concerned about the risks of AI than people in the United States and Europe”?

[Question] Is there any rigorous work on using anthropic uncertainty to prevent situational awareness / deception?

David Scott Krueger (formerly: capybaralet)4 Sep 2024 12:40 UTC

17 points

7 comments1 min readLW link

David Scott Krueger (formerly: capybaralet) 9 Aug 2024 0:32 UTC
LW: 2 AF: 1
0
AF
on: ProLU: A Nonlinearity for Sparse Autoencoders
This activation function was introduced in one of my papers from 10 years ago ;)

See Figure 2 of https://arxiv.org/abs/1402.3337

An ML paper on data stealing provides a construction for “gradient hacking”

David Scott Krueger (formerly: capybaralet)30 Jul 2024 21:44 UTC

21 points

1 comment1 min readLW link

(arxiv.org)

David Scott Krueger (formerly: capybaralet) 7 Jun 2024 1:39 UTC
LW: 4 AF: 2
0
AF
in reply to: Joe Collman’s comment on: [Link Post] “Foundational Challenges in Assuring Alignment and Safety of Large Language Models”
Really interesting point!

I introduced this term in my slides that included “paperweight” as an example of an “AI system” that maximizes safety.

I sort of still think it’s an OK term, but I’m sure I will keep thinking about this going forward and hope we can arrive at an even better term.

[Link Post] “Foundational Challenges in Assuring Alignment and Safety of Large Language Models”

David Scott Krueger (formerly: capybaralet)6 Jun 2024 18:55 UTC

69 points

2 comments6 min readLW link

(llm-safety-challenges.github.io)

David Scott Krueger (formerly: capybaralet) 15 Mar 2024 12:09 UTC
3 points
0
in reply to: eggsyntax’s comment on: Testing for consequence-blindness in LLMs using the HI-ADS unit test.
You could try to do tests on data that is far enough from the training distribution that it won’t generalize in a simple immitative way there, and you could do tests to try and confirm that you are far enough off distribution. For instance, perhaps using a carefully chosen invented language would work.

David Scott Krueger (formerly: capybaralet) 15 Mar 2024 12:07 UTC
LW: 2 AF: 1
0
AF
in reply to: Ediz Ucar’s comment on: Quick thoughts on “scalable oversight” / “super-human feedback” research
I don’t disagree… in this case you don’t get agents for a long time; someone else does though.

David Scott Krueger (formerly: capybaralet) 6 Mar 2024 21:08 UTC
LW: 3 AF: 1
0
AF
in reply to: Ediz Ucar’s comment on: Quick thoughts on “scalable oversight” / “super-human feedback” research
I meant “other training schemes” to encompass things like scaffolding that deliberately engineers agents using LLMs as components, although I acknowledge they are not literally “training” and more like “engineering”.

Testing for consequence-blindness in LLMs using the HI-ADS unit test.

David Scott Krueger (formerly: capybaralet)24 Nov 2023 23:35 UTC

25 points

2 comments2 min readLW link

David Scott Krueger (formerly: capybaralet) 22 Nov 2023 20:38 UTC
LW: 2 AF: 1
AF
on: Reading the ethicists 2: Hunting for AI alignment papers
I would look at the main FATE conferences as well, which I view as being: FAccT, AIES, EEAMO.

David Scott Krueger (formerly: capybaralet) 14 Aug 2023 18:42 UTC
3 points
0
on: Ways I Expect AI Regulation To Increase Extinction Risk
I found this thought provoking, but I didn’t find the arguments very strong.

(a) Misdirected Regulations Reduce Effective Safety Effort; Regulations Will Almost Certainly Be Misdirected
(b) Regulations Generally Favor The Legible-To-The-State
(c) Heavy Regulations Can Simply Disempower the Regulator
(d) Regulations Are Likely To Maximize The Power of Companies Pushing Forward Capabilities the Most

Briefly responding:
a) The issue in this story seems to be that the company doesn’t care about x-safety, not that they are legally obligated to care about face-blindness.
b) If governments don’t have bandwidth to effectively vet small AI projects, it seems prudent to err on the side of forbidding projects that might pose x-risk.
c) I do think we need effective international cooperation around regulation. But even buying 1-4 years time seems good in expectation.
d) I don’t see the x-risk aspect of this story.

David Scott Krueger (formerly: capybaralet) 26 Jul 2023 23:15 UTC
LW: 5 AF: 3
0
AF
on: How LLMs are and are not myopic
This means that the model can and will implicitly sacrifice next-token prediction accuracy for long horizon prediction accuracy.
Are you claiming this would happen even given infinite capacity?
If so, can you perhaps provide a simple+intuitive+concrete example?

David Scott Krueger (formerly: capybaralet) 19 Jul 2023 23:03 UTC
LW: 2 AF: 1
0
AF
on: What Discovering Latent Knowledge Did and Did Not Find
What do you mean by “random linear probe”?

David Scott Krueger (formerly: capybaralet) 11 Jul 2023 15:25 UTC
4 points
0
on: Deceptive AI vs. shifting instrumental incentives
I skimmed this. A few quick comments:
- I think you characterized deceptive alignment pretty well.
- I think it only covers a narrow part of how deceptive behavior can arise.
- CICERO likely already did some of what you describe.

David Scott Krueger (formerly: capybaralet) 29 Jun 2023 10:26 UTC
LW: 3 AF: 2
1
AF
on: Instrumental Convergence? [Draft]
So let us specify a probability distribution over the space of all possible desires. If we accept the orthogonality thesis, we should not want this probability distribution to build in any bias towards certain kinds of desires over others. So let’s spread our probabilities in such a way that we meet the following three conditions. Firstly, we don’t expect Sia’s desires to be better satisfied in any one world than they are in any other world. Formally, our expectation of the degree to which Sia’s desires are satisfied at $W$ is equal to our expectation of the degree to which Sia’s desires are satisfied at $W^{*}$ , for any $W, W^{*}$ . Call that common expected value $μ$ ″. Secondly, our probabilities are symmetric around $μ$ . That is, our probability that $W$ satisfies Sia’s desires to at least degree $μ + x$ is equal to our probability that it satisfies her desires to at most degree $μ - x$ . And thirdly, learning how well satisfied Sia’s desires are at some worlds won’t tell us how well satisfied her desires are at other worlds. That is, the degree to which her desires are satisfied at some worlds is independent of how well satisfied they are at any other worlds. (See the appendix for a more careful formulation of these assumptions.) If our probability distribution satisfies these constraints, then I’ll say that Sia’s desires are ‘sampled randomly’ from the space of all possible desires.

This is a characterization, and it remains to show that there exist distributions that fit it (I suspect there are not, assuming the sets of possible desires and worlds are unbounded).

I also find the 3rd criteria counterintuitive. If worlds share features, I would expect these to not be independent.

David Scott Krueger (formerly: capybaralet)

[Question] Is there any rigor­ous work on us­ing an­thropic un­cer­tainty to pre­vent situ­a­tional aware­ness /​ de­cep­tion?

An ML pa­per on data steal­ing pro­vides a con­struc­tion for “gra­di­ent hack­ing”

[Link Post] “Foun­da­tional Challenges in As­sur­ing Align­ment and Safety of Large Lan­guage Models”

Test­ing for con­se­quence-blind­ness in LLMs us­ing the HI-ADS unit test.

[Question] Is there any rigorous work on using anthropic uncertainty to prevent situational awareness / deception?

An ML paper on data stealing provides a construction for “gradient hacking”

[Link Post] “Foundational Challenges in Assuring Alignment and Safety of Large Language Models”

Testing for consequence-blindness in LLMs using the HI-ADS unit test.