mattmacdermott

Karma: 850

mattmacdermott 16 Dec 2024 0:32 UTC
1 point
0
on: OpenAI Email Archives (from Musk v. Altman)

Elon Musk to: Ilya Sutskever, Greg Brockman, Sam Altman—Feb 19, 2016 12:05 AM

Frankly, what surprises me is that the AI community is taking this long to figure out concepts. It doesn’t sound super hard. High-level linking of a large number of deep nets sounds like the right approach or at least a key part of the right approach

…

Ilya Sutskever to: Elon Musk, (cc: Greg Brockman, Sam Altman) - Feb 19, 2016 10:28 AM

Several points:

It is not the case that once we solve “concepts,” we get AI. Other problems that will have to be solved include unsupervised learning, transfer learning, and lifetime learning. We’re also doing pretty badly with language right now.

Seems like there was a misunderstanding here where Musk meant “the AI community is taking a long time to figure stuff out” but Ilya thought he meant “figure out how to make AI that can think in terms of concepts”.

mattmacdermott 14 Dec 2024 19:04 UTC
3 points
2
on: OpenAI Email Archives (from Musk v. Altman)

Compute is used in two ways: it is used to run a big experiment quickly, and it is used to run many experiments in parallel.

95% of progress comes from the ability to run big experiments quickly. The utility of running many experiments is much less useful. In the old days, a large cluster could help you run more experiments, but it could not help with running a single large experiment quickly.

For this reason, an academic lab could compete with Google, because Google’s only advantage was the ability to run many experiments. This is not a great advantage.

Recently, it has become possible to combine 100s of GPUs and 100s of CPUs to run an experiment that’s 100x bigger than what is possible on a single machine while requiring comparable time. This has become possible due to the work of many different groups. As a result, the minimum necessary cluster for being competitive is now 10–100x larger than it was before.

I think Ilya said almost exactly this in his NeurIPS talk yesterday. Sounds like “doing the biggest experiments fast is more important than doing many experiments in parallel” is a core piece of wisdom to him.

mattmacdermott 13 Dec 2024 20:26 UTC
1 point
0
in reply to: Raemon’s comment on: The “Think It Faster” Exercise
This might be the source of Sarah Constantin’s incredulity?

mattmacdermott 13 Dec 2024 20:13 UTC
1 point
0
on: The “Think It Faster” Exercise
Is Eliezer claiming to (train himself to perform only those steps) over thirty seconds, or train himself to (perform only those steps over thirty seconds)?

mattmacdermott 5 Dec 2024 10:02 UTC
3 points
0
on: mattmacdermott’s Shortform
I’m confused by this argument from Redwood’s AI control paper.

5.2 WHY WE DIDN’T CONSIDER STRATEGIES THAT INVOLVE TRAINING THE MODEL [...] Any protocol that involves training the model can be transformed into a validation-style protocol that merely validates the model’s performance on the dataset that was used as a train set. In any case where the validation-style protocol fails, we know that the red-team model is not distinguished from an aligned model by the measurements done by the protocol. Given that none of the training examples allow the reward-generating process to determine whether the policy it is training is undesirable, we don’t see why training would transform the model from an undesirable policy to a desirable policy. It is conceivable that we could find a training scheme that systematically converts undesirable policies to desirable ones, despite the lack of a reward differential causing it to do so. But we don’t know how.

If I understand correctly, the argument is:
1. In order for training on dataset D to convince you a model is desirable, good performance on D needs to be a sufficient condition for being desirable.
2. If good performance on D is a sufficient condition for being desirable, you can just test performance on D to decide whether a model is desirable.
But 2 seems wrong—if good performance on D is sufficient but not neccesary for being desirable, then you could have two models, one desirable and one undesirable, which are indistinguishable on D because they both perform badly, and then turn them both into desirable models by training on D.

As an extreme example, suppose your desirable model outputs correct code and your undesirable model backdoored code. Using D you train them both to output machine checkable proofs of the correctness of the code they write. Now both models are desirable because you can just check correctness before deploying the code.

mattmacdermott 3 Dec 2024 8:37 UTC
4 points
0
on: (The) Lightcone is nothing without its people: LW + Lighthaven’s big fundraiser
I gave $290. Partly because of the personal value I get out of LW, partly because I think it’s a solidly cost-effective donation.

mattmacdermott 3 Dec 2024 8:28 UTC
1 point
1
in reply to: elifland’s comment on: (The) Lightcone is nothing without its people: LW + Lighthaven’s big fundraiser
Do we have any data on p(doom) in the LW/rationalist community? I would guess the median is lower than 35-55%.

It’s not exactly clear where to draw the line, but I would guess this to be the case for, say, the 10% most active LessWrong users.

mattmacdermott 3 Dec 2024 8:16 UTC
1 point
0
in reply to: abramdemski’s comment on: Ayn Rand’s model of “living money”; and an upside of burnout
Not sure. I guess you also have to exclude policy gradient methods that make use of learned value estimates. “Learned evaluation vs sampled evaluation” is one way you could say it.

Model-based vs model-free does feel quite appropriate, shame it’s used for a narrower kind of model in RL. Not sure if it’s used in your sense in other contexts.

mattmacdermott 2 Dec 2024 18:48 UTC
5 points
0
in reply to: abramdemski’s comment on: Ayn Rand’s model of “living money”; and an upside of burnout
Under that definition you end up saying that what are usually called ‘model-free’ RL algorithms like Q-learning are model-based. E.g. in Connect 4, once you’ve learned that getting 3 in a row has a high value, you get credit for taking actions that lead to 3 in a row, even if you ultimately lose the game.

I think it is kinda reasonable to call Q-learning model-based, to be fair, since you can back out a lot of information about the world from the Q-values with little effort.

mattmacdermott 18 Nov 2024 18:11 UTC
3 points
4
on: “It’s a 10% chance which I did 10 times, so it should be 100%”
Nitpick: “odds of 63%” sounds to me like it means “odds of 63:100” i.e. “probability of around 39%”. Took me a while to realise this wasn’t what you meant.

Context-dependent consequentialism

Jeremy Gillen and mattmacdermott

4 Nov 2024 9:29 UTC

31 points

6 comments27 min readLW link

mattmacdermott 9 Oct 2024 8:14 UTC
3 points
0
in reply to: mattmacdermott’s comment on: mattmacdermott’s Shortform
I think the way to go, philosophically, might be to distinguish kindness-towards-conscious-minds and kindness-towards-agents. The former comes from our values, while the second may be decision theoretic.

mattmacdermott 9 Oct 2024 8:10 UTC
1 point
−1
on: mattmacdermott’s Shortform
The revealed preference orthogonality thesis

People sometimes say it seems generally kind to help agents achieve their goals. But it’s possible there need be no relationship between a system’s subjective preferences (i.e. the world states it experiences as good) and its revealed preferences (i.e. the world states it works towards).

For example, you can imagine an agent architecture consisting of three parts:
- a reward signal, experienced by a mind as pleasure or pain
- a reinforcement learning algorithm
- a wrapper which flips the reward signal before passing it to the RL algorithm.
This system might seek out hot stoves to touch while internally screaming. It would not be very kind to turn up the heat.

mattmacdermott 1 Oct 2024 11:11 UTC
1 point
−3
in reply to: David Matolcsi’s comment on: You can, in fact, bamboozle an unaligned AI into sparing your life
Even if you think a life’s work can’t make a difference but many can, you can still think it’s worthwhile to work on alignment for whatever reasons make you think it’s worthwhile to do things like voting.

(E.g. a non-CDT decision theory)

mattmacdermott 30 Sep 2024 17:25 UTC
69 points
37
on: the case for CoT unfaithfulness is overstated
Since o1 I’ve been thinking that faithful chain-of-thought is waaaay underinvested in as a research direction.

If we get models such that a forward pass is kinda dumb, CoT is superhuman, and CoT is faithful and legible, then we can all go home, right? Loss of control is not gonna be a problem.

And it feels plausibly tractable.

I might go so far as to say it Pareto dominates most people’s agendas on importance and tractability. While being pretty neglected.

mattmacdermott 29 Sep 2024 8:42 UTC
1 point
37
in reply to: Raemon’s comment on: “Slow” takeoff is a terrible term for “maybe even faster takeoff, actually”
Gradual/Sudden

mattmacdermott 14 Sep 2024 21:50 UTC
11 points
0
in reply to: Stephen McAleese’s comment on: Stephen McAleese’s Shortform
Do we know that the test set isn’t in the training data?

mattmacdermott 13 Sep 2024 14:08 UTC
7 points
2
in reply to: p.b.’s comment on: OpenAI o1
You can read examples of the hidden reasoning traces here.

mattmacdermott 12 Sep 2024 22:31 UTC
3 points
0
in reply to: eggsyntax’s comment on: eggsyntax’s Shortform

But it’s not clear to me that in practice it would say naughty things, since it’s easier for the model to learn one consistent set of guidelines for what to say or not say than it is to learn two.

If I think about asking the model a question about a politically sensitive or taboo subject, I can imagine it being useful for the model to say taboo or insensitive things in its hidden CoT in the course of composing its diplomatic answer. The way they trained it may or may not incentivise using the CoT to think about the naughtiness of its final answer.

But yeah, I guess an inappropriate content filter could handle that, letting us see the full CoT for maths questions and hiding it for sensitive political ones. I think that does update me more towards thinking they’re hiding it for other reasons.

mattmacdermott 12 Sep 2024 20:32 UTC
11 points
5
in reply to: eggsyntax’s comment on: eggsyntax’s Shortform
CoT optimised to be useful in producing the correct answer is a very different object to CoT optimised to look good to a human, and a priori I expect the former to be much more likely to be faithful. Especially when thousands of tokens are spent searching for the key idea that solves a task.
For example, I have a hard time imagining how the examples in the blog post could be not faithful (not that I think faithfulness is guaranteed in general).

If they’re avoiding doing RL based on the CoT contents,

Note they didn’t say this. They said the CoT is not optimised for ‘policy compliance or user preferences’. Pretty sure what they mean is the didn’t train the model not to say naughty things in the CoT.

‘We also do not want to make an unaligned chain of thought directly visible to users.’ Why?

I think you might be overthinking this. The CoT has not been optimised not to say naughty things. OpenAI avoid putting out models that haven’t been optimised not to say naughty things. The choice was between doing the optimising, or hiding the CoT.

Edit: not wanting other people to finetune on the CoT traces is also a good explanation.

mattmacdermott

Con­text-de­pen­dent consequentialism

The revealed preference orthogonality thesis

Context-dependent consequentialism