Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
I thought about it some more and realized that this is equivalent to finding a Nash equilibrium. What we do is, we create a player for every call . This player has 2 actions (call them action 0 and action 1). The player’s utilities are set up so that they always get utility for choosing action 0. Also, given fixed actions for the other players, their utility for action 1 is set to be the probability that returns 1 given that is an oracle that returns the action of a different player corresponding to the call it receives. Now, the player’s expected utility is for action 0, and for action 1 it will be ‘s probability of returning 1 where is an oracle that returns answers to calls under the same distribution as the other players’ mixed strategies. So, it will be allowed to choose action 0 iff , action 1 iff , and will be allowed to mix iff , as desired. It should be noted that within a program, repeated calls to the oracle must return the same value, so they should be cached.
Going the other direction is easy: to find the Nash equilibrium for a game where each player has 2 actions, set up a program for each player that queries the oracle for the actions of each other player and then returns the action that maximizes expected utility. For games with more than 2 actions (say, each player has actions ), we can create a different set of programs for each player: program where returns 1 iff some maximum expected utility action’s index is less that or equal to .
It seems to follow that the full problem is equivalent to a game with countably infinite players. I was thinking of how you would solve the full problem using some kind of oracle machine. It would at least be necessary to determine whether a program halts for every infinite input you give it, because this is necessary to detect queries. This interacts weirdly with the requirement that queries almost surely halt; it may require something analogous to for counting the number of inputs a program halts on? The problem of determining if a machine halts for every finite input is computable by a double oracle machine but not a regular oracle machine, but I don’t know what that says about infinite inputs.
Hmm...this seems like less of a problem to me. The thing we need for the Nash equilibrium equivalence is that, within an execution of a single program, equal calls to the oracle return equal results (or equivalently, you can’t give the same call to the oracle twice). But we don’t need to give the same call to the oracle twice in order to model the matching pennies game, because you just need one call (to determine whether your opponent plays heads with greater than 50% probability), and the two players are represented as different programs.
The thing we’re basing the Nash equilibrium equivalence on is the question “given fixed return values of calls, what is the probability that this program returns 1?” in order to write in utility values for our normal form game. Since we’re basing everything on this process, it follows that within an execution of a program, it only make sense to make a given call once. But nothing about this rules out having different programs reason about the same call, and having the oracle give them different answers.
It seems like what is needed is a logical uncertainty model that allows things that PA would consider contradictions (so we can reason about finite agents outputting something they don’t actually output), and that is causal. Some of the models in in Paul’s Non Omniscience, Probabilistic Inference, and Metamathematics paper allow contradictions in a satisfying way, but don’t contain any causation.
It seems that we want “this agent outputs this” to cause “consequences of this decision happen”. Suppose we create a Boolean variable for every logical proposition , and we want to arrange them in something like a causal Bayesian network. Since the consequences of an action are a logical consequence of the agent’s output that can be seen within a small number of steps, perhaps we want to arrange the network so that the premises of an inference rule being true will cause their conclusion to be true (with high probability; we still want some probability of contradiction). But if we naively create causal arrows from the premises of an inference rule (such as modus ponens) to its conclusion (for example, allow and to jointly cause ) then we get cycles. I’m not sure if causation is well-defined in cyclic graphs, but if it’s not then maybe there is a way to fix this by deleting some of the causal arrows?
I discussed this with Benja at a previous MIRIx workshop and I don’t remember exactly what we concluded, but I think it mostly works, it just requires that people behave sensibly when they get scrubbed predictions.
Now that I think about it: to handle cases when people don’t behave that sensibly with scrubbed predictions, maybe we want some kind of sequence of oracles, where oracle 0 outputs nothing, and oracle n+1 outputs what would happen if it were replaced with oracle n. We could take the limit as n approaches infinity, but then we don’t know that much about which fixed point we will get (it will be controlled by subtle feedback loops), so maybe we want something like n=3 being most probable (although we will want to make n random between 0 and 3 so it’s meaningful to condition on n=0, n=1, n=2).
I was confused about this, so I discussed this with Patrick. My current understanding of this framework is something like this:
models are of type X → Y
criteria are of type (X, Y) → Real
F is something like this
def F(model, criteria, data):
return (argmax(m in model_hypothesis_set) score(m, criteria, data),
argmax(c in criteria_hypothesis_set) score(model, c, data))
So we do iterated procedures of optimizing the model to fit the criteria, and optimizing the criteria to make the current model look good. For example, models could be linear regression models (predicting multiple outputs from multiple inputs), and criteria could be weights on outputs that make some errors more costly than other errors. The question is, are there interesting model/criteria sets that don’t “wirehead” by just making the criteria trivial to satisfy? Patrick says this is related to corrigibility, but I don’t really see the correspondence yet.
I don’t have that much to say about the original problem, but would like to note that it is pretty common in machine learning to do the opposite of this: optimize the model to the criteria, and then find new criteria that make the model look as bad as possible:
In boosting: “After a weak learner is added, the data is reweighted: examples that are misclassified gain weight and examples that are classified correctly lose weight”
In optimization (e.g. convex optimization), you can use duality to reframe the original “primal” problem (find a good solution satisfying the constraints) as a “dual” problem (find “weights” on constraints to make the optimal solution as bad as possible). See the part of the Wikipedia article about Lagrange duality to see what I mean by “weights on constraints”. While you can use the primal problem to show lower bounds on the highest possible score (by finding feasible points with that score), you can use the dual problem to show upper bounds on the highest possible score (by finding constraint weights that upper bound the score of the problem with these weights).
I’m not sure why we need to say “from E, F, B(), and v, …” instead of just “from E and v”. It seems like B() is just a generic agent design, and the distribution over F can be determined from E, B(), v.
So my restatement of this is something like: “for each select a distribution , such that if comes from some prior and , then the mutual information ”. has to not change much dependent on , so it has to satisfy many different restrictions (about a portion?). It seems like this will lead to satisfying restriction and also something like a portion of the other restrictions in our prior (specifically, the most convenient ones to jointly satisfy).
Yeah, I think the main problem with active learning is that is either hard to split evenly (with individual points) or is not very general. If we somehow created a nice that is both sufficiently general and easy to split, then we might get exchanges like this (but starting with more than 4 hypotheses, of course):
(AI's current hypotheses: cancer is good, cancer is bad, QALYs are good, QALYs are bad)
AI: Is removing a tumor good?
Programmer: Yes.
(AI's current hypotheses: cancer is bad, QALYs are good)
AI: Is killing everyone good?
Programmer: No.
(AI's only hypothesis: QALYs are good)
Obviously this works very badly if does not contain the hypothesis we really want! Luckily, as long as does contain the correct hypothesis, and we don’t accidentally falsify it, then the system will either determine the correct hypothesis or fail gracefully by reporting that it is uncertain.
Interesting idea! To clarify: does not know or , and further does not know , but will know and .
It seems simplest to assume that each uses CDT, so it accepts iff the existence of is good according to $\epsilon u + v $. So will maximize the probability that will barely change , while approaching the maximum .
The question is whether it is safe to maximize subject to the constraints that does not change much for almost all in our prior. My intuition is that if no in our prior encodes something very close to human values, then it’s possible to destroy human values without changing any . But perhaps this could be used if we already have a prior that assigns some non-negligible probability to human values. This could plausibly be the case if human values have low relative Kolmogorov complexity (relative to the AI’s observations).
A similar thing that might be simpler is: have know and maximize by designing . Then select randomly and have either or get to veto . Alternatively, both could be given veto power.
Now that I think about it more, we might be able to drop the assumption that the maximum is easy to approach by adding a large penalty for getting vetoed.
Certainly. This is why any use of concept learning gets into ontology identification issues.
This seems similar to giving a resource gathering agent veto power.
Suppose a resource gathering agent has a 50% chance of having a goal and 50% chance of . The resource gathering agent is completely indifferent to anything that is not causally connected to the source of information about or . So unless it expects to exist in the future, it will be completely indifferent to the existence of .
if you want to destroy human values on an appreciable scale, you have to destroy (or neutralise) humans
I think this gets into ontology identification issues. If your ontology doesn’t contain high-level mental concepts, then it’s possible to e.g. lobotomize everyone (or do something similar without visibly changing the brain) without any goal expressed in this ontology caring. I think something like this satisficer design could work to conservatively do useful things when we already have the right ontology (or assign significant probability to it).
And the agent does expect (counterfactually or hypothetically or whatever you want to call the false miracle approach) to exist in the future.
I see. I still think this is different from getting vetoed by both and . Suppose will take some action that will increase by one, and will otherwise not affect at all. This will be vetoed by . However, a resource gathering agent will not care. This makes sense given that getting vetoed by both and imposes two constraints, while getting vetoed by only a resource gathering agent imposes one.
If for example you kill everyone (or lobotomize them or whatever) then a putative M(v) (or M(ϵu+v)) agent will find it a lot easier to maximise v, as there is less human opposition to overcome.
That makes sense. I actually didn’t understand the part in the original post where expects to exist in the future, and assumed that it had no abilities other than veto power.
Still, I think it might be possible to fix for almost all while destroying human value. Let be the distribution over that uses. Suppose can compute the default values for each (e.g. because it has a distribution over which value function a future singleton will have). Then it will construct a distribution consisting of a mixture of and for almost all in the support of , sample from this distribution, and allow the resulting utility function to take over the world. To construct the distribution such that each stays almost the same for almost all , it can adjust the probabilities assigned to and appropriately for almost all in the support of . It might be necessary to include convex combinations as well sometimes, but this doesn’t significantly change things. Now with high probability, will take over the world for some that is not close to human values.
Perhaps this leads us to the conclusion that we should set to the distribution over the future singleton’s values, and trust that human values are probable enough that these will be taken into account. But it seems like you are saying we can apply this even when we haven’t constructed a distribution that assigns non-negligible probability to human values.
I think you might be able to use concept learning to extract humans’ native ontology (of the type studied in the ontological crisis paper) and values expressed in this ontology. The next step is to make a more rational version of this ontology (e.g. by mapping it to the AI’s ontology), which does not look like a concept learning problem.
I was actually going to write something like this up, but you beat me to it! My idea was pretty similar. The main difference is that in my setting, a utility function only cares about outcomes (rather than intermediate events such as ). Here’s most of what I had written so far (feel free to just skim it, given that it’s mostly the same thing):
I am interested in the question of when we can substitute changes in beliefs for changes in utility function and vice versa.
Here is a simple setting of utility maximization. There are possible worlds, 2 actions, and outcomes. An agent has some prior over possible worlds, and some utility function over outcomes. The agent can be exposed to different independent experiments. In an experiment, the agent does not know the possible world , but indirectly observes and thereby sees an observation with probability . Then the agent chooses action . Subsequently, some outcome will occur (stochastically) based on the possible world and action, and the agent values this outcome according to .
Let be the “transition difference matrix” defined as It is not unreasonable to believe that can be determined a priori: if possible worlds are (say) probabilistic Turing machines accepting an action as input and returning an outcome as output, then indeed is known a priori.
Let stand for the elementwise product between vectors and , and let stand for the elementwise quotient. Also, define the likelihood to be a vector with . The agent’s posterior distribution over the possible world after seeing evidence will then be proportional to , using Bayes’ rule. We can compute the expected utility difference between actions in an experiment as
The agent’s behavior across experiments is entirely determined by the function . Let us rewrite this as: So in fact, the behavior is determined by the vector . This vector has entries, and entry is equal to the prior probability of possible world times the expected utility difference (between actions 1 and 0) if we are in possible world . We could call this vector the policy of .
Now we can ask two interesting questions:
For what does there exist such that the policy of is the policy of ? That is, can we replace changes in beliefs with changes in utility function?
For what does there exist such that the policy of is the policy of ? That is, can we replace changes in utility function with changes in beliefs?
For question 1:
We can find satisfying this if and only if assigns probability 0 to any outcome that does and is right-invertible. That is, must not have a smaller support than , and we must be able to span with the by-action differences in outcome distributions for each possible world. I expect that will be right-invertible in most practical cases (there will be more possible worlds than outcomes).
For question 2: this will not be true in general. If for some possible world , has a different sign from (that is, and recommend different actions in possible world ), then the policies must be different, unless . So we are not able to replace changes in utility function with changes in beliefs, in general. There is a way to do this by making some possible worlds observationally equivalent, but I haven’t worked through all the details.
It is definitely a problem with infinite buck-passing. It is probably possible to prove optimality if we have a continuous utility function (e.g. we’re using discounting). I think we might actually want a continuous utility function, but maybe not. Is there any time t such that you would consider it almost as good for a wonderful human civilization to exist for t steps and then die, compared to existing indefinitely?
The way I would express the procrastination paradox is something like:
There’s the tiling agents problem: we want AIs to construct successors that they trust to make correct decisions.
It would be desirable to have a system where an infinite sequence of AIs each trust the next one. If it worked, this would solve the tiling agents problem.
But, if we have something like this, then it will be unsound: it will prove that the button will eventually get pressed, even though it will never actually get pressed.
We can construct things that do press the button, but they don’t have the property of trusting successors that is desirable in some ways. Due to their handling of recursion, Paul’s logic and reflective oracles are both candidates for solving the tiling agents problem, however they both fail the procrastination paradox (when it’s set up this way).
This is nice! I actually don’t remember exactly what formulation of this I discussed with Paul at the workshop, but I really like this version. I do think that it may be useful (at least from a pedagogical perspective) to look at what happens in the finite case, where we have a finite set of halting stochastic programs that may query the oracle to ask about the output probabilities of other programs in this set. This restricted version is powerful enough to express the liar’s paradox and some multi-agent problems.
In this case, as long as we are assured that each program always halts, I believe the problem is computable. We can enumerate all possible calls that any program might give to the oracle. Then, we can write the desired 1-returning probabilities for each call as a set-valued function of the oracle’s 1-returning probabilities for each call. This function will be in closed form using only addition, subtraction, multiplication, and a set-valued version of the unit step function. So, it will be possible to find the fixed point of this function using a combination of brute-force search and polynomial solving, similar to how it’s possible to find Nash equilibria.