justinpombrio

Karma: 718

justinpombrio Aug 28, 2024, 8:06 PM
1 point
0
in reply to: justinpombrio’s comment on: Am I confused about the “malign universal prior” argument?
Very curious what part of this people think is wrong.

justinpombrio Aug 28, 2024, 3:01 PM
4 points
−4
on: Am I confused about the “malign universal prior” argument?
Here’s a simple argument that simulating universes based on Turing machine number can give manipulated results.

Say we lived in a universe much like this one, except that:
- The universe is deterministic
- It’s simulated by a very short Turing machine
- It has a center, and
- That center is actually nearby! We can send a rocket to it.
So we send a rocket to the center of the universe and leave a plaque saying “the answer to all your questions is Spongebob”. Now any aliens in other universes that simulate our universe and ask “what’s in the center of that universe at time step 10^1000?” will see the plaque, search elsewhere in our universe for the reference, and watch Spongebob. We’ve managed to get aliens outside our universe to watch Spongebob.

I feel like it would be helpful to speak precisely about the universal prior. Here’s my understanding.

It’s a partial probability distribution over bit strings. It gives a non-zero probability to every bit string, but these probabilities add up to strictly less than 1. It’s defined as follows:

$P (bitstring) = Σ_{code} (TM(code) halts with bitstring) / 2^{len(code)}$

That is, describe Turing machines by a binary code, and assign each one a probability based on the length of its code, such that those probabilities add up to exactly 1. Then magically run all Turing machines “to completion”. For those that halt leaving a bitstring on their tape, attribute the probability of that Turing machine to that bitstring. Now we have a probability distribution over bitstrings, though the probabilities add up to less than one because not all of the Turing machines halted.

You cannot compute this probability distribution, but you can compute lower bounds on the probabilities of its bitstrings. (The Nth lower bound is the probability distribution you get from running the first N TMs for N steps.)

Call a TM that halts poisoned if its output is determined as follows:
- The TM simulates a complex universe full of intelligent life, then selects a tiny portion of that universe to output, erasing the rest.
- That intelligent life realizes this might happen, and writes messages in many places that could plausibly be selected.
- It works, and the TM’s output is determined by what the intelligent life it simulated chose to leave behind.
If we approximate the universal prior, the probability contribution of poisoned TMs will be precisely zero, because we don’t have nearly enough compute to simulate a poisoned TM until it halts. However, if there’s an outer universe with dramatically more compute available, and it’s approximating the universal prior using enough computational power to actually run the poisoned TMs, they’ll effect the probability distribution of the bitstrings, making bitstrings with the messages they choose to leave behind more likely.

So I think Paul’s right, actually (not what I expected when I started writing this). If you approximate the UP well enough, the distribution you see will have been manipulated.

justinpombrio Jul 26, 2024, 1:54 PM
0 points
0
in reply to: cubefox’s comment on: “AI achieves silver-medal standard solving International Mathematical Olympiad problems”
The feedback is from Lean, which can validate attempted formal proofs.

justinpombrio Jun 7, 2024, 3:05 PM
1 point
0
in reply to: VisionaryHera’s comment on: Is Wittgenstein’s Language Game used when helping Ai understand language?

This is one of the bigger reasons why I really don’t like RLHF—because inevitably you’re going to have to use a whole bunch of Humans who know less-than-ideal amounts about philosophy, pertaining to Ai Alignment.

What would these humans do differently, if they knew about philosophy? Concretely, could you give a few examples of “Here’s a completion that should be positively reinforced because it demonstrates correct understanding of language, and here’s a completion of the same text that should be negatively reinforced because it demonstrates incorrect understanding of language”? (Bear in mind that the prompts shouldn’t be about language, as that would probably just teach the model what to say when it’s discussing language in particular.)

It’s impossible for the Utility function of the Ai to be amenable to humans if it doesn’t use language the same way

What makes you think that humans all use language the same way, if there’s more than one plausible option? People are extremely diverse in their perspectives.

justinpombrio Jun 6, 2024, 12:15 PM
1 point
0
in reply to: VisionaryHera’s comment on: Is Wittgenstein’s Language Game used when helping Ai understand language?
As you’re probably aware, the fine tuning is done by humans rating the output of the LLM. I believe this was done by paid workers, who were probably given a list of criteria like that it should be helpful and friendly and definitely not use slurs, and who had probably not heard of Wittgenstein. How do you think they would rate LLM outputs that demonstrated “incorrect understanding of language”?

I have (tried to) read Wittgenstein, but don’t know what outputs would or would not constitute an “incorrect understanding of language”. Could you give some examples? The question is whether the tuners would rate those examples positively or negatively, and whether examples like those would arise during five tuning.

justinpombrio Jun 4, 2024, 9:52 PM
2 points
0
on: Is Wittgenstein’s Language Game used when helping Ai understand language?
You say “AI”, though I’m assuming you’re specifically asking about LLMs (large language models) like GPT, Llama, Claude, etc.

LLMs aren’t programmed, they’re trained. None of the code written by the developers of LLMs has anything to do with concepts, sentences, dictionary definitions, or different languages (e.g. English vs. Spanish). The code only deals with general machine learning, and streams of tokens (which are roughly letters, but encoded a bit differently).

The LLM is trained on huge corpuses of text. The LLM learns concepts, and what a sentence is, and the difference between English and Spanish, purely from the text. None of that is explicitly programmed into it; the programmers have no say in the matter.

As far as how it comes to understands language, and how that related to Wittgenstein’s thoughts on language, we don’t know much at all. You can ask it. And we’ve done some experiments like that recent one with the LLM that was made to think it was the Golden Gate Bridge, which you probably heard about. But that’s about it; we don’t really know how LLMs “think” internally. (We know what’s going on at a low-level, but not at a high-level.)

justinpombrio Apr 30, 2024, 5:02 PM
3 points
0
in reply to: avturchin’s comment on: Magic by forgetting
Exactly.

justinpombrio Apr 29, 2024, 3:33 PM
1 point
0
in reply to: avturchin’s comment on: Magic by forgetting

However, If I already know that I have the disease, and I am not altruistic to my copies, playing such game is a wining move to me?

Correct. But if you don’t have the disease, you’re probably also not altruistic to your copies, so you would choose not to participate. Leaving the copies of you with the disease isolated and unable to “trade”.

justinpombrio Apr 28, 2024, 1:41 PM
5 points
4
in reply to: avturchin’s comment on: Magic by forgetting
Not “almost no gain”. My point is that it can be quantified, and it is exactly zero expected gain under all circumstances. You can verify this by drawing out any finite set of worlds containing “mediators”, and computing the expected number of disease losses minus disease gains as:

num(people with disease)*P(person with disease meditates)*P(person with disease who meditates loses the disease) - num(people without disease)*P(person without disease meditates)*P(person without disease who meditates gains the disease)

My point is that this number is always exactly zero. If you doubt this, you should try to construct a counterexample with a finite number of worlds.

justinpombrio Apr 28, 2024, 3:54 AM
1 point
0
in reply to: avturchin’s comment on: Magic by forgetting
My point still stands. Try drawing out a specific finite set of worlds and computing the probabilities. (I don’t think anything changes when the set of worlds becomes infinite, but the math becomes much harder to get right.)

justinpombrio Apr 24, 2024, 9:34 PM
17 points
15
on: Magic by forgetting

There is a 0.001 chance that someone who did not have the disease will get it. But he can repeat the procedure.

No, that doesn’t work. It invalidates the implicit assumption you’re making that the probability that a person chooses to “forget” is independent of whether they have the disease. Ultimately, you’re “mixing” the various people who “forgot”, and a “mixing” procedure can’t change the proportion of people who have the disease.

When you take this into account, the conclusion becomes rather mundane. Some copies of you can gain the disease, while a proportional number of copies can lose it. (You might think you could get some respite by repeatedly trading off “who” has the disease, but the forgetting procedure ensures that no copy ever feels respite, as that would require remembering having the disease.)

justinpombrio Apr 3, 2024, 8:46 PM
3 points
0
in reply to: zhukeepa’s comment on: The Cognitive-Theoretic Model of the Universe: A Partial Summary and Review

I think formalizing it in full will be a pretty nontrivial undertaking, but formalizing isolated components feels tractable, and is in fact where I’m currently directing a lot of my time and funding.

Great. Yes, I think that’s the thing to do. Start small! I (and presumably others) would update a lot from a new piece of actual formal mathematics from Chris’s work. Even if that work was, by itself, not very impressive.

(I would also want to check that that math had something to do with his earlier writings.)

My current understanding is that he believes that his current written work should be sufficient for modern mathematicians and scientists to understand his core ideas

Uh oh. The “formal grammar” that I checked used formal language, but was not even close to giving a precise definition. So Chris either (i) doesn’t realize that you need to be precise to communicate with mathematicians, or (ii) doesn’t understand how to be precise.

Please be prepared for the possibility that Chris is very smart and creative, and that he’s had some interesting ideas (e.g. Syndiffeonesis), but that his framework is more of a interlocked collection of ideas than anything mathematical (despite using terms from mathematics). Litany of Tarsky and all that.

justinpombrio Apr 3, 2024, 2:48 PM
1 point
1
in reply to: zhukeepa’s comment on: The Cognitive-Theoretic Model of the Universe: A Partial Summary and Review
“gesture at something formal”—not in the way of the “grammar” it isn’t. I’ve seen rough mathematics and proof sketches, especially around formal grammars. This isn’t that, and it isn’t trying to be. There isn’t even an attempt at a rough definition for which things the grammar derives.

I think Chris’s work is most valuable to engage with for people who have independently explored philosophical directions similar to the ones Chris has explored

A big part of Chris’s preliminary setup is around how to sidestep the issues around making the sets well-ordered.

Nonsense! If Chris has an alternative to well-ordering, that’s of general mathematical interest! He would make a splash simply writing that up formally on its own, without dragging the rest of his framework along with it.

Except, I can already predict you’re going to say that no piece of his framework can be understood without the whole. Not even by making a different smaller framework that exists just to showcase the well-ordering alternative. It’s a little suspicious.

because someone else I’d funded to review Chris’s work

If you’re going to fund someone to do something, it should be to formalize Chris’s work. That would not only serve as a BS check, it would make it vastly more approachable.

I’m confused why you’re asking about specific insights people have gotten when Jessica has included a number of insights she’s gotten in her post

I was hoping people other than Jessica would share some specific curated insights they got. Syndiffeonesis is in fact a good insight.
What links here?
- CTMU insight: maybe consciousness *can* affect quantum outcomes? by zhukeepa (Apr 19, 2024, 3:23 PM; 13 points)

justinpombrio Mar 29, 2024, 2:46 PM
25 points
4
on: The Cognitive-Theoretic Model of the Universe: A Partial Summary and Review
tldr; a spot check calls bullshit on this.

I know a bunch about formal languages (PhD in programming languages), so I did a spot check on the “grammar” described on page 45. It’s described as a “generative grammar”, though instead of words (sequences of symbols) it produces “L_O spacial relationships”. Since he uses these phrases to describe his “grammar”, and they have their standard meaning because he listed their standard definition earlier in the section, he is pretty clearly claiming to be making something akin to a formal grammar.

My spot check is then: is the thing defined here more-or-less a grammar, in the following sense?
1. There’s a clearly defined thing called a grammar, and there can be more than one of them.
2. Each grammar can be used to generate something (apparently an L_O) according to clearly defined derivation rules that depend only on the grammar itself.
If you don’t have a thing plus a way to derive stuff from that thing, you don’t have anything resembling a grammar.

My spot check says:
1. There’s certainly a thing called a grammar. It’s a four-tuple, whose parts closely mimic that of a standard grammar, but using his constructs for all the basic parts.
2. There’s no definition of how to derive an “L_O spacial relationship” given a grammar. Just some vague references to using “telic recursion”.
I’d categorize this section as “not even wrong”; it isn’t doing anything formal enough to have a mistake in it.

Another fishy aspect of this section is how he makes a point of various things coinciding, and how that’s very different from the standard definitions. But it’s compatible with the standard definitions! E.g. the alphabet of a language is typically a finite set of symbols that have no additional structure, but there’s no reason you couldn’t define a language whose symbols were e.g. grammars over that very language. The definition of a language just says that its symbols form a set. (Perhaps you’d run into issues with making the sets well-ordered, but if so he’s running headlong into the same issues.)

I’m really not seeing any value in this guy’s writing. Could someone who got something out of it share a couple specific insights that got from it?

justinpombrio Mar 2, 2024, 5:47 PM
5 points
0
in reply to: Adam Zerner’s comment on: The Parable Of The Fallen Pendulum—Part 1
How did you find me? How do they always find me? No matter...

Have you tried applying your models to predict the day’s weather, or what your teacher will be wearing that day? I bet not: they wouldn’t work very well. Models have domains in which they’re meant to be applied. More precise models tend to have more specific domains.

Making real predictions about something, like what the result of a classroom experiment will be even if the pendulum falls over, is usually outside the domain of any precise model. That’s why your successful models are compound models, using Newtonian mechanics as a sub-model, and that’s why they’re so unsatisfyingly vague and cobbled together.

There is a skill to assembling models that make good predictions in messy domains, and it is a valuable skill. But it’s not the goal of your physics class. That class is trying to teach you about precise models like Newtonian mechanics. Figuring out exactly how to apply Newtonian mechanics to a real physical experiment is often harder than solving the Newtonian math! But surely you’ve noticed by now that, in the domains where Newtonian mechanics seems to actually apply, it applies very accurately?

This civilization we live in tends to have two modes of thinking. The first is ‘precise’ thinking, where people use precise models but don’t think about the mismatch between their domain and reality. The model’s domain is irrelevant in the real world, so people will either inappropriately apply the model outside its domain or carefully only make statements within the model’s domain and hope that others will make that incorrect leap on their own. The other mode of thinking is ‘imprecise’ thinking, where people ignore all models and rely on their gut feelings. We are extremely bad, at the moment, of the missing middle path of making and recognizing models for messy domains.

justinpombrio Mar 1, 2024, 3:49 PM
28 points
13
on: The Parable Of The Fallen Pendulum—Part 1
“There’s no such thing as ‘a Bayesian update against the Newtonian mechanics model’!” says a hooded figure from the back of the room. “Updates are relative: if one model loses, it must be because others have won. If all your models lose, it may hint that there’s another model you haven’t thought of that does better than all of them, or it may simply be that predicting things is hard.”

“Try adding a couple more models to compare against. Here’s one: pendulums never swing. And here’s another: Newtonian mechanics is correct but experiments are hard to perform correctly, so there’s a 80% probability that Newtonian mechanics gives the right answer and 20% probability spread over all possibilities including 5% on ‘the pendulum fails to swing’. Continue to compare these models during your course, and see which one wins. I think you can predict it already, despite your feigned ignorance.”

The hooded figure opens a window in the back of the room and awkwardly climbs through and walks off.

justinpombrio Feb 23, 2024, 3:48 PM
1 point
0
in reply to: numpyNaN’s comment on: The Pareto Best and the Curse of Doom
Are we assuming things are fair or something?

I would have modeled this as von Neumann getting 300 points and putting 260 of them into the maths and sciences and the remaining 40 into living life and being well adjusted.

justinpombrio Nov 22, 2023, 7:26 PM
1 point
0
in reply to: dyokomizo’s comment on: Implementing Decision Theory
Oh, excellent!

It’s a little hard to tell from the lack of docs, but you’re modelling dilemmas with Bayesian networks? I considered that, but wasn’t sure how to express Sleeping Beauty nicely, whereas it’s easy to express (and gives the right answers) in my tree-shaped dilemmas. Have you tried to express Sleeping Beauty?

And have you tried to express a dilemma like smoking lesion where the action that an agent takes is not the action their decision theory tells them to take? My guess is that this would be as easy as having a chain of two probabilistic events, where the first one is what the decision theory says to do and the second one is what the agent actually does, but I don’t see any of this kind of dilemma in your test cases.

justinpombrio Nov 16, 2023, 6:54 PM
1 point
in reply to: Waller-Hefkó János’s comment on: My attitude towards death
I have a healthy fear of death; it’s just that none of it stems from an “unobserved endless void”. Some of the specific things I fear are:
- Being stabbed is painful and scary (it’s scary even if you know you’re going to live)
- Most forms of dying are painful, and often very slow
- The people I love mourning my loss
- My partner not having my support
- Future life experiences, not happening
- All of the things I want to accomplish, not happening
The point I was making in this thread was that “unobserved endless void” is not on this list, I don’t know how to picture it, and I’m surprised that other people think it’s a big deal.

Who knows, maybe if I come close to dying some time I’ll suddenly gain a new ontological category of thing to be scared of.

justinpombrio Nov 12, 2023, 12:22 AM
2 points
0
in reply to: Nicolas Macé’s comment on: Implementing Decision Theory
What’s the utility function of the predictor? Is there necessarily a utility function for the predictor such that the predictor’s behavior (which is arbitrary) corresponds to maximizing its own utility? (Perhaps this is mentioned in the paper, which I’ll look at.)

EDIT: do you mean to reduce a 2-player game to a single-agent decision problem, instead of vice-versa?