Nora Belrose

Karma: 700

Nora Belrose Mar 5, 2024, 7:10 AM
4 points
1
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
It’s not clear to me what an “algorithm” is supposed to be here, and I suspect that this might be cruxy. In particular I suspect (40-50% confidence) that:
- You think there are objective and determinate facts about what “algorithm” a neural net is implementing, where
- Algorithms are supposed to be something like a Boolean circuit or a Turing machine rather than a neural network, and
- We can run counting arguments over these objective algorithms, which are distinct both from the neural net itself and the function it expresses.
I reject all three of these premises, but I would consider it progress if I got confirmation that you in fact believe in them.

Nora Belrose Mar 5, 2024, 7:00 AM
1 point
−9
in reply to: TurnTrout’s comment on: Counting arguments provide no evidence for AI doom
I’m sorry to hear that you think the argumentation is weaker now.
the reader has to do the work to realize that indifference over functions is inappropriate
I don’t think that indifference over functions in particular is inappropriate. I think indifference reasoning in general is inappropriate.
I’m very happy with running counting arguments over the actual neural network parameter space
I wouldn’t call the correct version of this a counting argument. The correct version uses the actual distribution used to initialize the parameters as a measure, and not e.g. the Lebesgue measure. This isn’t appealing to the indifference principle at all, and so in my book it’s not a counting argument. But this could be terminological.

Nora Belrose Mar 5, 2024, 3:13 AM
1 point
−4
in reply to: ryan_greenblatt’s comment on: Counting arguments provide no evidence for AI doom
Fair enough if you never read any of these comments.
Yeah, I never saw any of those comments. I think it’s obvious that the most natural reading of the counting argument is that it’s an argument over function space (specifically, over equivalence classes of functions which correspond to “goals.”) And I also think counting arguments for scheming over parameter space, or over Turing machines, or circuits, or whatever, are all much weaker. So from my perspective I’m attacking a steelman rather than a strawman.

Nora Belrose Mar 5, 2024, 3:07 AM
0 points
−1
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
I’ve read every word of all of your comments.
I know that you think your criticism isn’t dependent on Solomonoff induction in particular, because you also claim that a counting argument goes through under circuit prior. It still seems like you view the Solomonoff case as the central one, because you keep talking about “bitstrings.” And I’ve repeatedly said that I don’t think the circuit prior works either, and why I think that.
At no point in this discussion have you provided any reason for thinking that in fact, the Solomonoff prior and/or circuit prior do provide non-negligible evidence about neural network inductive biases, despite the very obvious mechanistic disanalogies.
Yes—that’s exactly the sort of counting argument that I like!
Then make an NNGP counting argument! I have not seen such an argument anywhere. You seem to be alluding to unpublished, or at least little-known, arguments that did not make their way into Joe’s scheming report.

Nora Belrose Mar 5, 2024, 2:47 AM
LW: -14 AF: -8
−18
AF
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
So today we’ve learned that:
1. The real counting argument that Evan believes in is just a repackaging of Paul’s argument for the malignity of the Solomonoff prior, and not anything novel.
2. Evan admits that Solomonoff is a very poor guide to neural network inductive biases.
At this point, I’m not sure why you’re privileging the hypothesis of scheming at all.
you want to substitute it out for whatever the prior is that you think is closest to deep learning that you can still reason about theoretically.
I mean, the neural network Gaussian process is literally this, and you can make it more realistic by using the neural tangent kernel to simulate training dynamics, perhaps with some finite width corrections. There is real literature on this.

Nora Belrose Mar 5, 2024, 2:30 AM
LW: 3 AF: 2
0
AF
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
What makes you think that’s intended to be a counting argument over function space? I usually think of this as a counting argument over infinite bitstrings
I definitely thought you were making a counting argument over function space, and AFAICT Joe also thought this in his report.
The bitstring version of the argument, to the extent I can understand it, just seems even worse to me. You’re making an argument about one type of learning procedure, Solomonoff induction, which is physically unrealizable and AFAICT has not even inspired any serious real-world approximations, and then assuming that somehow the conclusions will transfer over to a mechanistically very different learning procedure, gradient descent. The same goes for the circuit prior thing (although FWIW I think you’re very likely wrong that minimal circuits can be deceptive).

Nora Belrose Mar 4, 2024, 7:11 PM
3 points
−3
in reply to: TurnTrout’s comment on: Counting arguments provide no evidence for AI doom
FWIW I object to 2, 3, and 4, and maybe also 1.

Nora Belrose Mar 4, 2024, 11:29 AM
−3 points
−5
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom

It is trivially easy to modify the formalism to search only over fixed-size algorithms, and in fact that’s usually what I do when I run this sort of analysis.

What? Which formalism? I don’t see how this is true at all. Please elaborate or send an example of “modifying” Solomonoff so that all the programs have fixed length, or “modifying” the circuit prior so all circuits are the same size.

No, I’m pretty familiar with your writing. I still don’t think you’re focusing on mainstream ML literature enough because you’re still putting nonzero weight on these other irrelevant formalisms. Taking that literature seriously would mean ceasing to take the Solomonoff or circuit prior literature seriously.

Nora Belrose Mar 4, 2024, 4:15 AM
LW: -5 AF: -6
−13
AF
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
Right, and I’ve explained why I don’t think any of those analyses are relevant to neural networks. Deep learning simply does not search over Turing machines or circuits of varying lengths. It searches over parameters of an arithmetic circuit of fixed structure, size, and runtime. So Solomonoff induction, speed priors, and circuit priors are all inapplicable. There has been a lot of work in the mainstream science of deep learning literature on the generalization behavior of actual neural nets, and I’m pretty baffled at why you don’t pay more attention to that stuff.

Nora Belrose Mar 4, 2024, 2:12 AM
−1 points
0
AF
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom

Then show me how! If you think there are errors in the math, please point them out.

I’m not aware of any actual math behind the counting argument for scheming. I’ve only ever seen handwavy informal arguments about the number of Christs vs Martin Luthers vs Blaise Pascals. There certainly was no formal argument presented in Joe’s extensive scheming report, which I assumed would be sufficient context for writing this essay.

Nora Belrose Mar 4, 2024, 1:06 AM
2 points
0
in reply to: Signer’s comment on: Counting arguments provide no evidence for AI doom
I’m saying <0.1% chance on “world is ended by spontaneous scheming.” I’m not saying no AI will ever do anything that might be well-described as scheming, for any reason.

Nora Belrose Mar 4, 2024, 1:02 AM
LW: 1 AF: 1
−3
AF
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
I obviously don’t think the counting argument for overfitting is actually sound, that’s the whole point. But I think the counting argument for scheming is just as obviously invalid, and misuses formalisms just as egregiously, if not moreso.

I deny that your Kolmogorov framework is anything like “the proper formalism” for neural networks. I also deny that the counting argument for overfitting is appropriately characterized as a “finite bitstring” argument, because that suggests I’m talking about Turing machine programs of finite length, which I’m not- I’m directly enumerating functions over a subset of the natural numbers. Are you saying the set of functions over 1...10,000 is not a well defined mathematical object?

Nora Belrose Mar 4, 2024, 12:43 AM
LW: 1 AF: 1
0
AF
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
I never used any kind of bitstring analysis.

Nora Belrose Feb 29, 2024, 4:44 PM
LW: 5 AF: 4
−3
AF
in reply to: Daniel Kokotajlo’s comment on: Counting arguments provide no evidence for AI doom
I think the infinite bitstring case has zero relevance to deep learning.
There does exist a concept you might call “simplicity” which is relevant to deep learning. The neural network Gaussian process describes the prior distribution over functions which is induced by the initialization distribution over neural net parameters. Under weak assumptions about the activation function and initialization variance, the NNGP is biased toward lower frequency functions. I think this cuts against scheming, and we plan to write up a post on this in the next month or two.

Nora Belrose Feb 29, 2024, 3:46 AM
28 points
10
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
I’m well aware of how it’s derived. I still don’t think it makes sense to call that an indifference prior, precisely because enforcing an uncomputable halting requirement induces an exponentially strong bias toward short programs. But this could become a terminological point.

I think relying on an obviously incorrect formalism is much worse than relying on no formalism at all. I also don’t think I’m relying on zero formalism. The literature on the frequency/spectral bias is quite rigorous, and is grounded in actual facts about how neural network architectures work.

Nora Belrose Feb 29, 2024, 3:31 AM
LW: 33 AF: 17
3
AF
in reply to: evhub’s comment on: Counting arguments provide no evidence for AI doom
Thanks for the reply. A couple remarks:
- “indifference over infinite bitstrings” is a misnomer in an important sense, because it’s literally impossible to construct a normalized probability measure over infinite bitstrings that assigns equal probability to each one. What you’re talking about is the length weighted measure that assigns exponentially more probability mass to shorter programs. That’s definitely not an indifference principle, it’s baking in substantive assumptions about what’s more likely.
- I don’t see why we should expect any of this reasoning about Turing machines to transfer over to neural networks at all, which is why I didn’t cast the counting argument in terms of Turing machines in the post. In the past I’ve seen you try to run counting or simplicity arguments in terms of parameters. I don’t think any of that works, but I at least take it more seriously than the Turing machine stuff.
- If we’re really going to assume the Solomonoff prior here, then I may just agree with you that it’s malign in Christiano’s sense and could lead to scheming, but I take this to be a reductio of the idea that we can use Solomonoff as any kind of model for real world machine learning. Deep learning does not approximate Solomonoff in any meaningful sense.
- Terminological point: it seems like you are using the term “simple” as if it has a unique and objective referent, namely Kolmogorov-simplicity. That’s definitely not how I use the term; for me it’s always relative to some subjective prior. Just wanted to make sure this doesn’t cause confusion.

Nora Belrose Feb 28, 2024, 9:37 PM
3 points
4
in reply to: peterbarnett’s comment on: Counting arguments provide no evidence for AI doom
I’m not actually sure the scheming problems are “compatible” with good performance on these metrics, and even if they are, that doesn’t mean they’re likely or plausible given good performance on our metrics.
Human brains are way more similar to other natural brains
So I disagree with this, but likely because we are using different conceptions of similarity. In order to continue this conversation we’re going to need to figure out what “similar” means, because the term is almost meaningless in controversial cases— you can fill in whatever similarity metric you want. I used the term earlier as a shorthand for a more detailed story about randomly initialized singular statistical models learned with iterative, local update rules. I think both artificial and biological NNs fit that description, and this is an important notion of similarity.

Nora Belrose Feb 28, 2024, 4:16 PM
3 points
−3
in reply to: Max H’s comment on: Counting arguments provide no evidence for AI doom
This is just an equivocation, though. Of course you could train an AI to “scheme” against people in the sense of selling a fake blood testing service. That doesn’t mean that by default you should expect AIs to spontaneously start scheming against you, and in ways you can’t easily notice.

Nora Belrose Feb 28, 2024, 4:11 PM
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Counting arguments provide no evidence for AI doom
Could you be more specific? In what way will there be non-mild distribution shifts in the future?

Nora Belrose Feb 28, 2024, 4:09 PM
LW: 3 AF: 1
0
AF
in reply to: Daniel Kokotajlo’s comment on: Counting arguments provide no evidence for AI doom
With respect to which measure though? You have to define a measure, there are going to be infinitely many possible measures you could define on this space. And then we’ll have to debate if your measure is a good one.