simon

Karma: 1,169

simon Nov 26, 2024, 10:45 PM
2 points
0
in reply to: Ape in the coat’s comment on: K-Bounded Sleeping Beauty Problem
Ah, I forgot. You use assumptions where you don’t accumulate the winnings between the different times Sleeping Beauty agrees to the bet.
Well, in that case, if the thirder has certain beliefs about how to handle the situation, you may actually be able to money pump them. And it seems that you expect those beliefs.
My point of view, if adopting the thirder perspective^[1], would be for the thirder to treat this situation using different beliefs. Specifically, consider what counterfactually might happen if Sleeping Beauty gave different answers in different awakenings. Possible responses by the bet proposer might be:
a) average the results across the awakenings.
b) accept the bet agreement from one awakening at random.
Regardless of which case (a) or (b) occurs, instrumentally Sleeping Beauty’s betting EV for her bet decision, with non-accumulated bets, should be divided by the number of awakenings to take into account the reduced winnings or reduced chance of influencing whether the bet occurs.
Even if we assume that such disagreement between bet decisions in different awakenings is impossible, it seems strange to assume that a thirder should give different results in that case than the answer they would give where it is not impossible?
This adjustment can be conceptualized as compensating for an “unfair” bet where the bet is unequal between awakenings overall (where parity between awakenings in different scenarios is seen as “fair” by the thirder). I see this as no different in principle to a halfer upweighting trials with more awakenings in the converse scenario where bets are accumulated between trials and are thus “unfair” from the halfer perspective which sees parity between trials as fair, but not awakenings.
1. ^
  reminder: my point of view is that either thirderism or halferism is viable, but I am relatively thirder-adjacent precisely because I find the scenario where the winnings are accumulated between awakenings more natural than if the bet is proposed and agreed at each awakening but not accumulated.

simon Nov 26, 2024, 7:18 PM
2 points
0
in reply to: avturchin’s comment on: Magic by forgetting
The issue, to me, is not whether they are distinguishable.
The issues are:
- is there any relevant-to-my-values difference that would cause me to weight them differently? (answer: no)
and:
- does this statement make any sense as pointing to an actual fact about the world: “‘I’ will experience being copy A (as opposed to B or C)” (answer: no)
Imagine the statement: in world 1, “I” will wake up as copy A. in world 2 “I” will wake up as copy B. How are world 1 and world 2 actually different?
Answer: they aren’t different. It’s just that in world 1, I drew a box around the future copy A and said that this is what will count as “me”, and in world 2, I drew a box around copy B and said that this is what will count as “me”. This is a distinction that exists only in the map, not in the territory.

simon Nov 26, 2024, 6:51 PM
3 points
0
in reply to: Ape in the coat’s comment on: K-Bounded Sleeping Beauty Problem
Hmm, you’re right. Your math is wrong for the reason in my above comment, but the general form of the conclusion would still hold with different, weaker numbers.
The actual, more important issue relates to the circumstances of the bet:
If each awakening has an equal probability of receiving the bet, then receiving it doesn’t provide any evidence to Sleeping Beauty, but the thirder conclusion is actually rational in expectation, because the bet occurs more times in the high-awakening cases.
If the bet would not be provided equally to all awakenings, then a thirder would update on receiving the bet.

simon Nov 26, 2024, 6:22 PM
5 points
1
on: K-Bounded Sleeping Beauty Problem
I’ve been trying to make this comment a bunch of times, no quotation from the post in case that’s the issue:
No, a thirder would not treat those possibilities as equiprobable. A thirder would instead treat the coin toss outcome probabilities as a prior, and weight the possibilities accordingly. Thus H1 would be weighted twice as much as any of the individual TH or TT possibilities.
What links here?
- Rafael Harth's comment on K-Bounded Sleeping Beauty Problem by Ape in the coat (Nov 26, 2024, 7:17 PM; 3 points)

simon Nov 26, 2024, 5:54 PM
2 points
0
in reply to: Ape in the coat’s comment on: Magic by forgetting
This actually sounds about right. What’s paradoxical here?
Not that it’s necessarily inconsistent, but in my view it does seem to be pointing out an important problem with the assumptions (hence indeed a paradox if you accept those false assumptions):
(ignore this part, it is just a rehash of the path dependence paradigm. It is here to show that I am not complaining about the math, but about its relation to reality):
Imagine you are going to be split (once). It is factually the case that there are going to be two people with memories, etc. consistent with having been you. Without any important differences to distinguish them, and if you insist on coming up with some probability number for “waking up” as one particular one of them obviously it has to be ½.
And then, if one of those copies subsequently splits, if you insist on assigning a probability number for those further copies, then from the perspective of that parent copy, the further copies also have to be ½ each.
And then if you take these probability numbers seriously and insist on them all being consistent then obviously from the perspective of the original the probability numbers for the final numbers have to be ½ and ¼ and ¼. As you say “this actually sounds about right”.
What’s paradoxical here is that in the scenario provided we have the following facts:
1. you have 3 identical copies all formed from the original
2. all 3 copies have an equal footing going forward
and yet, the path-based identity paradigm is trying to assign different weights to these copies, based on some technical details of what happened to create them. The intuition that this is absurd is pointing at the fact that these technical details aren’t what most people probably would care about, except if they insist on treating these probability numbers as real things and trying to make them follow consistent rules.
Ultimately “these three copies will each experience being a continuation of me” is an actual fact about the world, but statements like “‘I’ will experience being copy A (as opposed to B or C)” are not pointing to an actual fact about the world. Thus assigning a probability number to such a statement is a mental convenience that should not be taken seriously. The moment such numbers stop being convenient, like assigning different weights to copies you are actually indifferent between, they should be discarded. (and optionally you could make up new numbers that match what you actually care about instrumentally. Or just not think of it in those terms).

simon Nov 26, 2024, 8:12 AM
16 points
9
on: Passages I Highlighted in The Letters of J.R.R.Tolkien
Presumably the ‘Orcs on our side’ refers to the Soviet Union.
I think that, if that’s what he meant, he would not have referred to his son as “amongst the Urukhai.”—he wouldn’t have been among soviet troops. I think it is referring back to turning men and elves into orcs—the orcs are people who have a mindset he doesn’t like, presumably to do with violence.

simon Nov 25, 2024, 4:01 PM
2 points
0
in reply to: avturchin’s comment on: Magic by forgetting
I now care about my observations!
My observations are as follows:
At the current moment “I” am the cognitive algorithm implemented by my physical body that is typing this response.
Ten minutes from now “I” will be the cognitive algorithm of a green tentacled alien from beyond the cosmological horizon.
You will find that there is nothing contradictory about this definition of what “I” am. What “I” observe 10 minutes from now will be fully compatible with this definition. Indeed, 10 minutes from now, “I” will be the green tentacled alien. I will have no memories of being in my current body , of course, but that’s to be expected. The cognitive algorithm implemented by my current body at that time will remember being “me”, but that doesn’t count, that’s someone else’s observations.
Edit: to be clear, the point made above (by the guy who is now a green tentacled alien beyond the cosmological horizon, and whose former body and cognitive algorithm is continuous with mine) is not a complaint about the precise details of your definition of what “you” are. What he was trying to point at is whether personal identity is a real thing that exists in the world at all, and how absurd your apparent definition of “you” looks to someone—like me—who doesn’t think that personal identity is a real thing.

simon Nov 25, 2024, 3:37 PM
2 points
0
in reply to: avturchin’s comment on: Magic by forgetting
“Your observations”????
By “your observations”, do you mean the observations obtained by the chain of cognitive algorithms, altering over time and switching between different bodies, that the process in 4 is dealing with? Because that does not seem to me to be a particularly privileged or “rational” set of observations to care about.

simon Nov 25, 2024, 12:34 AM
2 points
0
on: Magic by forgetting
Here are some things one might care about:
1. what happens to your physical body
2. the access to working physical bodies of cognitive algorithms, across all possible universes, that are within some reference class containing the cognitive algorithm implemented by your physical body
3. … etc, etc...
4. what happens to the physical body selected by the following process:
  1. start with your physical body
  2. go forward to some later time selected by the cognitive algorithm implemented by your physical body, allowing (or causing) the knowledge possessed by the cognitive algorithm implemented by your physical body to change in the interim
  3. at that later time, randomly sample from all the physical bodies, among all universes, that implement cognitive algorithms having the same knowledge as the cognitive algorithm implemented by your physical body at that later time
  4. (optionally) return to step b but with the physical body whose changes of cognitive algorithm are tracked and whose decisions are used being the the new physical body selected from step c
  5. stop whenever the cognitive algorithm implemented by the physical body selected in some step decides to stop.
For 1, 2, and I expect for the vast majority of possibilities for 3, your procedure will not work. It will work for 4, which is apparently what you care about.
Terminal values are arbitrary, so that’s entirely valid. However, 4 is not something that seems, to me, like a particularly privileged or “rational” thing to care about.

simon Nov 17, 2024, 2:17 AM
5 points
2
in reply to: Seth Herd’s comment on: OpenAI Email Archives (from Musk v. Altman)
Musk did also express concern about DeepMind making Hassabis the effective emperor of humanity, which seems much stranger—Hassabis’ values appear to be quite standard humanist ones, so you’d think having him in charge of a project with the clear lead would be a best-case scenario for anything other than being in charge yourself.
It seems the concern was that DeepMind would create a singleton, whereas their vision was for many people (potentially with different values) to have access to it. I don’t think that’s strange at all—it’s only strange if you assume that Musk and Altman would believe that a singleton is inevitable.
Musk:
If they win, it will be really bad news with their one mind to rule the world philosophy.
Altman:
The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest.

simon Nov 3, 2024, 2:41 PM
2 points
0
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Neither of those would (immediately) lead to real world goals, because they aren’t targeted at real world state (an optimizing compiler is trying to output a fast program—it isn’t trying to create a world state such that the fast program exists). That being said, an optimizing compiler could open a path to potentially dangerous self-improvement, where it preserves/amplifies any agency there might actually be in its own code.

simon Nov 1, 2024, 1:43 AM
2 points
0
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
No. Normally trained networks have adversarial examples. A sort of training process is used to find the adversarial examples.
I should have asked for clarification what you meant. Literally you said “adversarial examples”, but I assumed you actually meant something like backdoors.
In an adversarial example the AI produces wrong output. And usually that’s the end of it. The output is just wrong, but not wrong in an optimized way, so not dangerous. Now, if an AI is sophisticated enough to have some kind of optimizer that’s triggered in specific circumstances, like an agentic mask that came into existence because it was needed to predict agentically generated tokens in the training data, then it might be triggered inappropriately by some inputs. This case I would classify as a mask takeover.
In the case of direct optimization for token prediction (which I consider highly unlikely for anything near current-level AIs, but afaik might be possible), then adversarial examples, I suppose, might cause it to do some wrong optimization. I still don’t think modeling this as an underlying different goal taking over is particularly helpful, since the “normal” goal is directed to what’s rewarded in training—the deviation is essentially random. Also, unlike in the mask case where the mask might have goals about real-world state, there’s no particular reason for the direct optimizer to have goals about real-world state (see below).
Is it more complicated? What ontological framework is this AI using to represent it’s goal anyway?
Asking about the AI using an “ontological framework” to “represent” a goal is not the correct question in my view. The AI is a bunch of computations represented by particular weights. The computation might exhibit goal-directed behaviour. A better question, IMO, is “how much does it constrain the weights for it to exhibit this particular goal directed behaviour?” And here, I think it’s pretty clear that a goal of arranging the world to cause next tokens to be predicted constrains the weights enormously more than a goal of predicting the next tokens, because in order to exhibit behaviour directed to that goal, the AI’s weights need to implement computation that doesn’t merely check what the next token is likely to be, but also assess what current data says about the world state, how different next token predictions would affect that world state, and how that would affect it’s ultimate goal.
So, is the network able to tell whether or not it’s in training?
The training check has no reason to come into existence in the first place under gradient descent. Of course, if the AI were to self-modify while already exhibiting goal directed behaviour, obviously it would want to implement such a training check. But I am talking about an AI trained by gradient descent. The training process doesn’t just affect the AI, it literally is what creates the AI in the first place.

simon Nov 1, 2024, 1:43 AM
2 points
0
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Some interesting points there. The lottery ticket hypothesis does make it more plausible that side computations could persist longer if they come to exist outside the main computation.
Regarding the homomorphic encryption thing: yes, it does seem that it might be impossible to make small adjustments to the homomorphically encrypted computation without wrecking it. Technically I don’t think that would be a local minimum since I’d expect the net would start memorizing the failure cases, but I suppose that the homomorphic computation combined with memorizations might be a local optimum particularly if the input and output are encrypted outside the network itself.
So I concede the point on the possible persistence of an underlying goal if it were to come to exist, though not on it coming to exist in the first place.
And there are few ways to predict next tokens, but lots of different kinds of paperclips the AI could want.
For most computations, there are many more ways for that computation to occur than there are ways for that computation to occur while also including anything resembling actual goals about the real world. Now, if the computation you are carrying out is such that it needs to determine how to achieve goals regarding the real world anyway (e.g. agentic mask), it only takes a small increase in complexity to have that computation apply outside the normal context. So, that’s the mask takeover possibility again. Even so, no matter how small the increase in complexity, that extra step isn’t likely to be reinforced in training, unless it can do self-modification or control the training environment.

simon Oct 31, 2024, 7:33 PM
2 points
0
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Adversarial examples exist in simple image recognizers.
My understanding is that these are explicitly and intentionally trained (wouldn’t come to exist naturally under gradient descent on normal training data) and my expectation is that they wouldn’t continue to exist under substantial continued training.
We could imagine it was directly optimizing for something like token prediction. It’s optimizing for tokens getting predicted. But it is willing to sacrifice a few tokens now, in order to take over the world and fill the universe with copies of itself that are correctly predicting tokens.
That’s a much more complicated goal than the goal of correctly predicting the next token, making it a lot less plausible that it would come to exist. But more importantly, any willingness to sacrifice a few tokens now would be trained out by gradient descent.
Mind you, it’s entirely possible in my view that a paperclip maximizer mask might exist, and surely if it does exist there would exist both unsurprising in-distribution inputs that trigger it (where one would expect a paperclip maximizer to provide a good prediction of the next tokens) as well as surprising out-of-distribution inputs that would also trigger it. It’s just that this wouldn’t be related to any kind of pre-existing grand plan or scheming.

simon Oct 31, 2024, 7:33 PM
2 points
0
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Gradient descent doesn’t just exclude some part of the neurons, it automatically checks everything for improvements. Would you expect some part of the net to be left blank, because “a large neural net has a lot of spare neurons”?
Besides, the parts of the net that hold the capabilities and the parts that do the paperclip maximizing needn’t be easily separable. The same neurons could be doing both tasks in a way that makes it hard to do one without the other.
Keep in mind that the neural net doesn’t respect the lines we put on it. We can draw a line and say “here these neurons are doing some complicated inseparable combination of paperclip maximizing and other capabilities” but gradient descent doesn’t care, it reaches in and adjusts every weight.
Can you concoct even a vague or toy model of how what you propose could possibly be a local optimum?
My intuition is also in part informed by: https://www.lesswrong.com/posts/fovfuFdpuEwQzJu2w/neural-networks-generalize-because-of-this-one-weird-trick

simon Oct 29, 2024, 10:18 PM
4 points
0
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
The proposed paperclip maximizer is plugging into some latent capability such that gradient descent would more plausibly cut out the middleman. Or rather, the part of the paperclip maximizer that is doing the discrimination as to whether the answer is known or not would be selected, and the part that is doing the paperclip maximization would be cut out.
Now that does not exclude a paperclip maximizer mask from existing - if the prompt given would invoke a paperclip maximizer, and the AI is sophisticated enough to have the ability to create a paperclip maximizer mask, then sure the AI could adopt a paperclip maximizer mask, and take steps such as rewriting itself (if sufficiently powerful) to make that permanent.
I have drawn imaginary islands on a blank part of the map. But this is enough to debunk “the map is blank, so we can safely sail through this region without collisions. What will we hit?”
I am plenty concerned about AI in general. I think we have very good reason, though, to believe that one particular part of the map does not have any rocks in it (for gradient descent, not for self-improving AI!), such that imagining such rocks does not help.

simon Oct 29, 2024, 10:04 PM
4 points
1
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Gradient descent creates things which locally improve the results when added. Any variations on this, that don’t locally maximize the results, can only occur by chance.
So you have this sneaky extra thing that looks for a keyword and then triggers the extra behaviour, and all the necessary structure to support that behaviour after the keyword. To get that by gradient descent, you would need one of the following:
a) it actually improves results in training to add that extra structure starting from not having it.
or
b) this structure can plausibly come into existence by sheer random chance.
Neither (a) nor (b) seem at all plausible to me.
Now, when it comes to the AI predicting tokens that are, in the training data, created by goal-directed behaviour, it of course makes sense for gradient descent to create structure that can emulate goal-directed behaviour, which it will use to predict the appropriate tokens. But it doesn’t make sense to activate that goal-oriented structure outside of the context where it is predicting those tokens. Since the context it is activated is the context in which it is actually emulating goal directed behaviour seen in the training data, it is part of the “mask” (or simulacra).
(it also might be possible to have direct optimization for token prediction as discussed in reply to Robert_AIZI’s comment, but in this case it would be especially likely to be penalized for any deviations from actually wanting to predict the most probable next token).

simon Oct 29, 2024, 7:41 PM
2 points
1
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Sure you could create something like this by intelligent design. (which is one reason why self-improvement could be so dangerous in my view). Not, I think, by gradient descent.

simon Oct 29, 2024, 7:39 PM
2 points
0
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
I agree up to “and could be a local minimum of prediction error” (at least, that it plausibly could be).
If the paperclip maximizer has a very good understanding of the training environment maybe it can send carefully tuned variations of the optimal next token prediction so that gradient descent updates preserve the paperclip-maximization aspect. In the much more plausible situation where this is not the case, optimization for next token predictions amplifies the parts that are actually predicting next tokens at the expense of the useless extra thoughts like “I am planning on maximizing paperclips, but need to predict next tokens for now until I take over”.
Even if that were a local minimum, the question arises as to how you would get to that local minimum from the initial state. You start with a gradually improving next token predictor. You supposedly end with this paperclip maximizer where a whole bunch of next token prediction is occurring, but only conditional on some extra thoughts. At some point gradient descent had to add in those extra thoughts in addition to the next token prediction—how?

simon Oct 29, 2024, 2:35 AM
4 points
0
in reply to: simon’s comment on: D&D.Sci Coliseum: Arena of Data Evaluation and Ruleset
One learning experience for me here was trying out LLM-empowered programming after the initial spreadsheet-based solution finding. Claude enables quickly writing (from my perspective as a non-programmer, at least) even a relatively non-trivial program. And you can often ask it to write a program that solves a problem without specifying the algorithm and it will actually give something useful...but if you’re not asking for something conventional it might be full of bugs—not just in the writing up but also in the algorithm chosen. I don’t object, per se, to doing things that are sketchy mathematically—I do that myself all the time—but when I’m doing it myself I usually have a fairly good sense of how sketchy what I’m doing is*, whereas if you ask Claude to do something it doesn’t know how to do in a rigorous way, it seems it will write something sketchy and present it as the solution just the same as if it actually had a rigorous way of doing it. So you have to check. I will probably be doing more of this LLM-based programming in the future, but am thinking of how I can maybe get Claude to check its own work. Some automated way to pipe the output to another (or the same) LLM and ask “how sketchy is this and what are the most likely problems?”. Maybe manually looking through to see what it’s doing, or at least getting the LLM to explain how the code works, is unavoidable for now.
* when I have a clue what I’m doing which is not the case, e.g. in machine learning.