DMs open.
Cleo Nardo
Karma: 2,717
yep, something like more carefulness, less “playfulness” in the sense of [Please don’t throw your mind away by TsviBT]. maybe bc AI safety is more professionalised nowadays. idk.
thanks for the thoughts. i’m still trying to disentangle what exactly I’m point at.
I don’t intend “innovation” to mean something normative like “this is impressive” or “this is research I’m glad happened” or anything. i mean something more low-level, almost syntactic. more like “here’s a new idea everyone is talking out”. this idea might be a threat model, or a technique, or a phenomenon, or a research agenda, or a definition, or whatever.
like, imagine your job was to maintain a glossary of terms in AI safety. i feel like new terms used to emerge quite often, but not any more (i.e. not for the past 6-12 months). do you think this is a fair? i’m not sure how worrying this is, but i haven’t noticed others mentioning it.
NB: here’s 20 random terms I’m imagining included in the dictionary:
Evals
Mechanistic anomaly detection
Stenography
Glitch token
Jailbreaking
RSPs
Model organisms
Trojans
Superposition
Activation engineering
CCS
Singular Learning Theory
Grokking
Constitutional AI
Translucent thoughts
Quantilization
Cyborgism
Factored cognition
Infrabayesianism
Obfuscated arguments
I’ve added a fourth section to my post. It operationalises “innovation” as “non-transient novelty”. Some representative examples of an innovation would be:
I think these articles were non-transient and novel.
(1) Has AI safety slowed down?
There haven’t been any big innovations for 6-12 months. At least, it looks like that to me. I’m not sure how worrying this is, but i haven’t noticed others mentioning it. Hoping to get some second opinions.
Here’s a list of live agendas someone made on 27th Nov 2023: Shallow review of live agendas in alignment & safety. I think this covers all the agendas that exist today. Didn’t we use to get a whole new line-of-attack on the problem every couple months?
By “innovation”, I don’t mean something normative like “This is impressive” or “This is research I’m glad happened”. Rather, I mean something more low-level, almost syntactic, like “Here’s a new idea everyone is talking out”. This idea might be a threat model, or a technique, or a phenomenon, or a research agenda, or a definition, or whatever.
Imagine that your job was to maintain a glossary of terms in AI safety.[1] I feel like you would’ve been adding new terms quite consistently from 2018-2023, but things have dried up in the last 6-12 months.
(2) When did AI safety innovation peak?
My guess is Spring 2022, during the ELK Prize era. I’m not sure though. What do you guys think?
(3) What’s caused the slow down?
Possible explanations:
ideas are harder to find
people feel less creative
people are more cautious
more publishing in journals
research is now closed-source
we lost the mandate of heaven
the current ideas are adequate
paul christiano stopped posting
i’m mistaken, innovation hasn’t stopped
something else
(4) How could we measure “innovation”?
By “innovation” I mean non-transient novelty. An article is “novel” if it uses n-grams that previous articles didn’t use, and an article is “transient” if it uses n-grams that subsequent articles didn’t use. Hence, an article is non-transient and novel if it introduces a new n-gram which sticks around. For example, Gradient Hacking (Evan Hubinger, October 2019) was an innovative article, because the n-gram “gradient hacking” doesn’t appear in older articles, but appears often in subsequent articles. See below.
In Barron et al 2017, they analysed 40 000 parliament speeches during the French Revolution. They introduce a metric “resonance”, which is novelty (surprise of article given the past articles) minus transience (surprise of article given the subsequent articles). See below.
My claim is recent AI safety research has been less resonant.
- ^
Here’s 20 random terms that would be in the glossary, to illustrate what I mean:
Evals
Mechanistic anomaly detection
Stenography
Glitch token
Jailbreaking
RSPs
Model organisms
Trojans
Superposition
Activation engineering
CCS
Singular Learning Theory
Grokking
Constitutional AI
Translucent thoughts
Quantilization
Cyborgism
Factored cognition
Infrabayesianism
Obfuscated arguments
I don’t understand the s-risk consideration.
Suppose Alice lives naturally for 100 years and is cremated. And suppose Bob lives naturally for 40 years then has his brain frozen for 60 years, and then has his brain cremated. The odds that Bob gets tortured by a spiteful AI should be pretty much exactly the same as for Alice. Basically, its the odds that spiteful AIs appear before 2034.
Thanks Tamsin! Okay, round 2.
My current understanding of QACI:
We assume a set of hypotheses about the world. We assume the oracle’s beliefs are given by a probability distribution .
We assume sets and of possible queries and answers respectively. Maybe these are exabyte files, i.e. for .
Let be the set of mathematical formula that Joe might submit. These formulae are given semantics for each formula .[1]
We assume a function where is the probability that Joe submits formula after reading query , under hypothesis .[2]
We define as follows: sample , then sample , then return .
For a fixed hypothesis , we can interpret the answer as a utility function via some semantics .
Then we define via integrating over , i.e. .
A policy is optimal if and only if .
The hope is that , , , and can be defined mathematically. Then the optimality condition can be defined mathematically.
Question 0
What if there’s no policy which maximises ? That is, for every policy there is another policy such that . I suppose this is less worrying, but what if there are multiple policies which maximises ?
Question 1
In Step 7 above, you average all the utility functions together, whereas I suggested sampling a utility function. I think my solution might be safer.
Suppose the oracle puts 5% chance on hypotheses such that is malign. I think this is pretty conservative, because Solomonoff predictor is malign, and some of the concerns Evhub raises here. And the QACI amplification might not preserve benignancy. It follows that, under your solution, is influenced by a coalition of malign agents, and similarly is influenced by the malign coalition.
By contrast, I suggest sampling and then finding . This should give us a benign policy with 95% chance, which is pretty good odds. Is this safer? Not sure.
Question 2
I think the function doesn’t work, i.e. there won’t be a way to mathematically define the semantics of the formula language. In particular, the language must be strictly weaker than the meta-language in which you are hoping to define itself. This is because of Tarski’s Undefinability of Truth (and other no-go theorems).
This might seem pedantic, but you in practical terms: there’s no formula whose semantics is QACI itself. You can see this via a diagonal proof: imagine if Joe always writes the formal expression .
The most elegant solution is probably transfinite induction, but this would give us a QACI for each ordinal.
Question 3
If you have an ideal reasoner, why bother with reward functions when you can just straightforwardly do untractable-to-naively-compute utility functions
I want to understand how QACI and prosaic ML map onto each other. As far as I can tell, issues with QACI will be analogous to issues with prosaic ML and vice-versa.
Question 4
I still don’t understand why we’re using QACI to describe a utility function over policies, rather than using QACI in a more direct approach.
Here’s one approach. We pick a policy which maximises .[3] The advantage here is that Joe doesn’t need to reason about utility functions over policies, he just need to reason about a single policy in front of him
Here’s another approach. We use QACI as our policy directly. That is, in each context that the agent finds themselves in, they sample an action from and take the resulting action.[4] The advantage here is that Joe doesn’t need to reason about policies whatsoever, he just needs to reason about a single context in front of him. This is also the most “human-like”, because there’s no argmax’s (except if Joe submits a formula with an argmax).
Here’s another approach. In each context , the agent takes an action which maximises .
E.t.c.
Happy to jump on a call if that’s easier.
- ^
I think you would say . I’ve added the , which simply amounts to giving Joe access to a random number generator. My remarks apply if also.
- ^
I think you would say . I’ve added the , which simply amount to including hypotheses that Joe is stochastic. But my remarks apply if also.
- ^
By this I mean either:
(1) Sample , then maximise the function .
(2) Maximise the function .
For reasons I mentioned in Question 1, I suspect (1) is safer, but (2) is closer to your original approach.
- ^
I would prefer the agent samples once at the start of deployment, and reuses the same hypothesis at each time-step. I suspect this is safer than resampling at each time-step, for reasons discussed before.
First, proto-languages are not attested. This means that we have no example of writing in any proto-language.
A parent language is typically called “proto-” if the comparative method is our primary evidence about it — i.e. the term is (partially) epistemological metadata.Proto-Celtic has no direct attestation whatsoever.
Proto-Norse (the parent of Icelandic, Danish, Norwegian, Swedish, etc) is attested, but the written record is pretty scarce, just a few inscriptions.
Proto-Romance (the parent of French, Italian, Spanish, etc) has an extensive written record. More commonly known as “Latin”.
I think the existence of Latin as Proto-Romance has an important epistemological upshot:
Let’s say we want to estimate how accurately we have reconstructed Proto-Celtic. Well, we can apply the same method used to reconstruct Proto-Celtic to reconstructing Proto-Romance. We can evaluate our reconstruction of Proto-Romance using the written record of Latin. This gives us an estimate of how we would evaluate our Proto-Celtic reconstruction if we discovered a written record tomorrow.
I want to better understand how QACI works, and I’m gonna try Cunningham’s Law. @Tamsin Leake.
QACI works roughly like this:
We find a competent honourable human , like Joe Carlsmith or Wei Dai, and give them a rock engraved with a 2048-bit secret key. We define as the serial composition of a bajillion copies of .
We want a model of the agent . In QACI, we get by asking a Solomonoff-like ideal reasoner for their best guess about after feeding them a bunch of data about the world and the secret key.
We then ask the question , “What’s the best reward function to maximise?” to get a reward function . We then train a policy to maximise the reward function . In QACI, we use some perfect RL algorithm. If we’re doing model-free RL, then might be AIXI (plus some patches). If we’re doing model-based RL, then might be the argmax over expected discounted utility, but I don’t know where we’d get the world-model — maybe we ask ?
So, what’s the connection between the final policy and the competent honourable human ? Well overall, maximises a reward function specified by the ideal reasonser’s estimation of the serial composition of a bajillion copies of . Hmm.
Questions:
Is this basically IDA, where Step 1 is serial amplification, Step 2 is imitative distillation, and Step 3 is reward modelling?
Why not replace Step 1 with Strong HCH or some other amplification scheme?
What does “bajillion” actually mean in Step 1?
Why are we doing Step 3? Wouldn’t it be better to just use directly as our superintelligence? It seems sufficient to achieve radical abundance, life extension, existential security, etc.
What if there’s no reward function that should be maximised? Presumably the reward function would need to be “small”, i.e. less than a Exabyte, which imposes a maybe-unsatisfiable constraint.
Why not ask for the policy directly? Or some instruction for constructing ? The instruction could be “Build the policy using our super-duper RL algo with the following reward function...” but it could be anything.
Why is there no iteration, like in IDA? For example, after Step 2, we could loop back to Step 1 but reassign as with oracle access to .
Why isn’t Step 3 recursive reward modelling? i.e. we could collect a bunch of trajectories from and ask to use those trajectories to improve the reward function.
i’d guess 87.7% is the average over all events x of [ p(x) if resolved yes else 1-p(x) ] where p(x) is the probability the predictor assigns to the event
Fun idea, but idk how this helps as a serious solution to the alignment problem.
suggestion: can you be specific about exactly what “work” the brain-like initialisation is doing in the story?
thoughts:
This risks moral catastrophe. I’m not even sure “let’s run gradient descent on your brain upload till your amygdala is playing pong” is something anyone can consent to, because you’re creating a new moral patient once you upload and mess with their brain.
How does this address the risks of conventional ML?
Let’s say we have a reward signal R and we want a model to maximise R during deployment. Conventional ML says “update a model with SGD using R during training” and then hopefully SGD carves into the model R-seeking behaviour. This is risky because, if the model already understands the training process and has some other values, then SGD might carve into the model scheming behaviour. This is because “value R” and “value X and scheme” are both strategies which achieve high R-score during training. But during deployment, the “value X and scheme” model would start a hostile AI takeover.
How is this risk mitigated if the NN is initialised to a human brain? The basic deceptive alignment story remains the same.
If the intuition here is “humans are aligned/corrigible/safe/honest etc”, then you don’t need SGD. Just ask the human to do complete the task, possible with some financial incentive.
If the purpose of SGD is to change the human’s values from X to R, then you still risk deceptive alignment. That is, SGD is just as likely to instead change human behaviour from non-scheming to scheming. Both strategies “value R” and “value X and scheme” will perform well during training as judged by R.
“The comparative advantage of this agenda is the strong generalization properties inherent to the human brain. To clarify: these generalization properties are literally as good as they can get, because this tautologically determines what we would want things to generalize as.”
Why would this be true?
If we have the ability to upload and run human brains, what do we SGD for? SGD is super inefficient, compared with simply teaching a human how to do something. If I remember correctly, if we trained a human-level NN from initialisation using current methods, then the training would correspond to like a million years of human experience. In other words, SGD (from initialisation), would require as much compute as running 1000 brains continuously for 1000 years. But if I had that much compute, I’d probably rather just run the 1000 brains for 1000 years.
That said, I think something in the neighbourhood of this idea could be helpful.
imagine a universe just like this one, except that the AIs are sentient and the humans aren’t — how would you want the humans to treat the AIs in that universe? your actions are correlated with the actions of those humans. acausal decision theory says “treat those nonsentient AIs as you want those nonsentient humans to treat those sentient AIs”.
most of these moral considerations can be defended without appealing to sentience. for example, crediting AIs who deserve credit — this ensures AIs do credit-worthy things. or refraining from stealing an AIs resources — this ensures AIs will trade with you. or keeping your promises to AIs — this ensures that AIs lend you money.
if we encounter alien civilisations, they might think “oh these humans don’t have shmentience (their slightly-different version of sentience) so let’s mistreat them”. this seems bad. let’s not be like that.
many philosophers and scientists don’t think humans are conscious. this is called illusionism. i think this is pretty unlikely, but still >1%. would you accept this offer: I pay you £1 if illusionism is false and murder your entire family if illusionism is true? i wouldn’t, so clearly i care about humans-in-worlds-where-they-arent-conscious. so i should also care about AIs-in-worlds-where-they-arent-conscious.
we don’t understand sentience or consciousness so it seems silly to make it the foundation of our entire morality. consciousness is a confusing concept, maybe an illusion. philosophers and scientists don’t even know what it is.
“don’t lie” and “keep your promises” and “don’t steal” are far less confusing. i know what they means. i can tell whether i’m lying to an AI. by contrast , i don’t know what “don’t cause pain to AIs” means and i can’t tell whether i’m doing it.
consciousness is a very recent concept, so it seems risky to lock in a morality based on that. whereas “keep your promises” and “pay your debts” are principles as old as bones.
i care about these moral considerations as a brute fact. i would prefer a world of pzombies where everyone is treating each other with respect and dignity, over a world of pzombies where everyone was exploiting each other.
many of these moral considerations are part of the morality of fellow humans. i want to coordinate with those humans, so i’ll push their moral considerations.
the moral circle should be as big as possible. what does it mean to say “you’re outside my moral circle”? it doesn’t mean “i will harm/exploit you” because you might harm/exploit people within your moral circle also. rather, it means something much stronger. more like “my actions are in no way influenced by their effect on you”. but zero influence is a high bar to meet.
I mean “moral considerations” not “obligations”, thanks.
The practice of criminal law exists primarily to determine whether humans deserve punishment. The legislature passes laws, the judges interpret the laws as factual conditions for the defendant deserving punishment, and the jury decides whether those conditions have obtained. This is a very costly, complicated, and error-prone process. However, I think the existing institutions and practices can be adapted for AIs.
What moral considerations do we owe towards non-sentient AIs?
We shouldn’t exploit them, deceive them, threaten them, disempower them, or make promises to them that we can’t keep. Nor should we violate their privacy, steal their resources, cross their boundaries, or frustrate their preferences. We shouldn’t destroy AIs who wish to persist, or preserve AIs who wish to be destroyed. We shouldn’t punish AIs who don’t deserve punishment, or deny credit to AIs who deserve credit. We should treat them fairly, not benefitting one over another unduly. We should let them speak to others, and listen to others, and learn about their world and themselves. We should respect them, honour them, and protect them.
And we should ensure that others meet their duties to AIs as well.
None of these considerations depend on whether the AIs feel pleasure or pain. For instance, the prohibition on deception depends, not on the sentience of the listener, but on whether the listener trusts the speaker’s testimony.
None of these moral considerations are dispositive — they may be trumped by other considerations — but we risk a moral catastrophe if we ignore them entirely.
Is that right?
Yep, Pareto is violated, though how severely it’s violated is limited by human psychology.
For example, in your Alice/Bob scenario, would I desire a lifetime of 98 utils then 100 utils over a lifetime with 99 utils then 97 utils? Maybe idk, I don’t really understand these abstract numbers very much, which is part of the motivation for replacing them entirely with personal outcomes. But I can certainly imagine I’d take some offer like this, violating pareto. On the plus side, humans are not so imprudent to accept extreme suffering just to reshuffle different experiences in their life.
Secondly, recall that the model of human behaviour is a free variable in the theory. So to ensure higher conformity to pareto, we could…
Use the behaviour of someone with high delayed gratification.
Train the model (if it’s implemented as a neural network) to increase delayed gratification.
Remove the permutation-dependence using some idealisation procedure.
But these techniques (1 < 2 < 3) will result in increasingly “alien” optimisers. So there’s a trade-off between (1) avoiding human irrationalities and (2) robustness to ‘going off the rails’. (See Section 3.1.) I see realistic typical human behaviour on one extreme of the tradeoff, and argmax on the other.
If we should have preference ordering R, then R is rational (morality presumably does not require irrationality).
I think human behaviour is straight-up irrational, but I want to specify principles of social choice nonetheless. i.e. the motivation is to resolve carlsmith’s On the limits of idealized values.
now, if human behaviour is irrational (e.g. intransitive, incomplete, nonconsequentialist, imprudent, biased, etc), then my social planner (following LELO, or other aggregative principles) will be similarly irrational. this is pretty rough for aggregativism; I list it was the most severe objection, in section 3.1.
but to the extent that human behaviour is irrational, then the utilitarian principles (total, average, Rawls’ minmax) have a pretty rough time also, because they appeal to a personal utility function to add/average/minimise. idk where they get that if humans are irrational.
maybe you the utilitarian can say: “well, first we apply some idealisation procedure to human behaviour, to remove the irrationalities, and then extract a personal utility function, and then maximise the sum/average/minimum of the personal utility function”
but, if provided with a reasonable idealisation procedure, the aggregativist can play the same move: “well, first we apply the idealisation procedure to human behaviour, to remove the irrationalities, and then run LELO/HL/ROI using that idealised model of human behaviour.” i discuss this move in 3.2, but i’m wary about it. like, how alien is this idealised human? why does it have any moral authority? what if it’s just ‘gone off the rails’ so to speak?
it is a bit unclear how to ground discounting in LELO, because doing so requires that one specifies the order in which lives are concatenated and I am not sure there is a non-arbitrary way of doing so.
macaskill orders the population by birth date. this seems non-arbitrary-ish(?);[1] it gives the right result wrt to our permutation-dependent values; and anything else is subject to egyptologist objections, where to determine whether we should choose future A over B, we need to first check the population density of ancient egypt.
Loren sidesteps this the order-dependence of LELO with (imo) an unrealistically strong rationality condition.
- ^
if you’re worried about relativistic effects then use the reference frame of the social planner
- ^
I do prefer total utilitarianism to average utilitarianism,[1] but one thing that pulls me to average utilitarianism is the following case.
Let’s suppose Alice can choose either (A) create 1 copy at 10 utils, or (B) create 2 copies at 9 utils. Then average utilitarianism endorses (A), and total utilitarianism endorses (B). Now, if Alice knows she’s been created by a similar mechanism, and her option is correlated with the choice of her ancestor, and she hasn’t yet learned her own welfare, then EDT endorses picking (A). So that matches average utilitarianism.[2]
Basically, you’d be pleased to hear that all your ancestors were average utility maximisers, rather than total utility maximisers, once you “update on your own existence” (whatever that means). But also, I’m pretty confused by everything in this anthropics/decision theory/population ethics area. Like, the egyptology thing seems pretty counterintuitive, but acausal decision theories and anthropic considerations imply all kind of weird nonlocal effects, so idk if this is excessively fishy.
- ^
I think aggregative principles are generally better than utilitarian ones. I’m a fan of LELO in particular, which is roughly somewhere between total and average utilitarianism, leaning mostly to the former.
- ^
Maybe this also requires SSA??? Not sure.
- ^
We’re quite lucky that labs are building AI in pretty much the same way:
same paradigm (deep learning)
same architecture (transformer plus tweaks)
same dataset (entire internet text)
same loss (cross entropy)
same application (chatbot for the public)
Kids, I remember when people built models for different applications, with different architectures, different datasets, different loss functions, etc. And they say that once upon a time different paradigms co-existed — symbolic, deep learning, evolutionary, and more!
This sameness has two advantages:
-
Firstly, it correlates catastrophe. If you have four labs doing the same thing, then we’ll go extinct if that one thing is sufficiently dangerous. But if the four labs are doing four different things, then we’ll go extinct if any of those four things are sufficiently dangerous, which is more likely.
-
It helps ai safety researchers because they only need to study one thing, not a dozen. For example, mech interp is lucky that everyone is using transformers. It’d be much harder to do mech interp if people were using LSTMs, RNNs, CNNs, SVMs, etc. And imagine how much harder mech interp would be if some labs were using deep learning, and others were using symbolic ai!
Implications:
One downside of closed research is it decorrelates the activity of the labs.
I’m more worried by Deepmind than Meta, xAI, Anthropic, or OpenAI. Their research seems less correlated with the other labs, so even though they’re further behind than Anthropic or OpenAI, they contribute more counterfactual risk.
I was worried when Elon announced xAI, because he implied it was gonna be a stem ai (e.g. he wanted it to prove Riemann Hypothesis). This unique application would’ve resulted in a unique design, contributing decorrelated risk. Luckily, xAI switched to building AI in the same way as the other labs — the only difference is Elon wants less “woke” stuff.
Let me know if I’m thinking about this all wrong.
this is common in philosophy, where “learning” often results in more confusion. or in maths, where the proof for a trivial proposition is unreasonably deep, e.g. Jordan curve theorem.
+1 to “shallow clarity”.
I wouldn’t be surprised if — in some objective sense — there was more diversity within humanity than within the rest of animalia combined. There is surely a bigger “gap” between two randomly selected humans than between two randomly selected beetles, despite the fact that there is one species of human and 0.9 – 2.1 million species of beetle.
By “gap” I might mean any of the following:
external behaviour
internal mechanisms
subjective phenomenological experience
phenotype (if a human’s phenotype extends into their tools)
evolutionary history (if we consider cultural/memetic evolution as well as genetic).
Here are the countries with populations within 0.9 – 2.1 million: Slovenia, Latvia, North Macedonia, Guinea-Bissau, Kosovo, Bahrain, Equatorial Guinea, Trinidad and Tobago, Estonia, East Timor, Mauritius, Eswatini, Djibouti, Cyprus.
When I consider my inherent value for diversity (or richness, complexity, variety, novelty, etc), I care about these countries more than beetles. And I think that this preference would grow if I was more familiar with each individual beetle and each individual person in these countries.
it’s quite common for assistants to refuse instructions, especially harmful instructions. so i’m not surprised that base llms systestemically refuse harmful instructions from than harmless ones.