in this post, i put forth some of my current thoughts about the shape of a formal aligned AI using QACI for its decision — “decision” in the singular here, as this is sufficient when the AI’s decision can be “run me again but with these diffirent inputs”. as it turns out, this doesn’t require solving as many things as i’d thought — it seems like QACI might be general enough to delegate picking a decision theory and solving embedded agency to the counterfactual consideration of the past user.
we’ll posit:
as a convention, we’ll use a prime
to denote counterfactual values, and we’ll be denoting questions
, and answers
(for “response”) to avoid confusion with
for actions.
the AI is denoted
, taking as input an observation as well as the user’s original question and answer, denoted
and
. it returns an action from the set of all possible actions,
finally, we’ll implicitely “cast” mathematical objects as natural numbers wherever appropriate, given that they’re all sampled from countable sets anyways. when they’re cast to or from natural numbers, assume a reasonable bijection between their type and
.
we’ll define the following:
a “carver” function
which returns a set of tuples of:
a function
for extracting a piece of data “in the same way” as
is in
but from any other world
a piece of data
that represents “everything else” than
in the world
a function
for counterfactually putting another piece of data
back in
, alongside
this is done by splitting a world
into the piece of data
, and “everything else”, denoted
. in practice with arbitrary other worlds,
would return “garbage” the immense majority of the time, but the hope is that given a same carving
for the question
, a same carving
for the answer would work enough time to give a signal that would tend to beat the overall noise of the failing cases.
functions together so as to favor them being simple but also similar, but ignores the simplicity of the “rest of the world”
values..
finally, we can define our AI
as a function of
but also an observation
which could, in practice, be anything that lets the AI as well as the user better locate themselves in the set of possible worlds.
what’s going on here, is for each world hypothesis
weighed by its simplicity
we’re finding carvings for
and
that let us inject a counterfactual question and get a counterfactual answer, favoring simpler carvings using
. the counterfactual rerun of the past user is given the world hypothesis
being considered, as well as the action
— we’re not giving the user
because that would make the counterfactual question guaranteed to be strictly larger than the original question, which seems like it might break things, and is not absolutely necessary. the user gets to return an arbitrary function to be evaluated, to which
are actually passed. we are, however, clamping the result of
inside
and then multiplying it by
ourselves, so as to avoid erroneous carvings getting to influence the decision process too much by returning arbitrary large values. the hope here is merely that desired carvings generally outweigh undesired carvings, and especially adverserial carvings.
ways this could go wrong include:
the carvings could fail to sufficiently select the actual instances of the past-user in the world; in particular,
could be returning so much “garbage” when given counterfactual worlds
that are different from
that the signal from the carvings that do work end ups completely drowned out.
the carvings locate question-answering users anywhere in the world, including in the future. this allows for adverserial intelligences to occupy most of the set of simple and coherently-answering carvings, thus steering our AI away from following the actual user’s decisions. a solution to this would be to sufficiently strongly favor carvings that select question-answering processes that are causally upstream of the AI itself; that is to say, causally upstream of
.
weird implementation details as for how turing machines are encoded could dominate most of the signal of values returned by
. in addition, the user could be unsure as to how to return a meaningful scalar given a particular potential action. these points could both be partially addressed by passing to the carvings pairs of actions
for all possible pairs of different actions from
, and have the counterfactual user select a preferred action, rather than relying on a scalar returned about a single action. if
and
agree as to which of
and
is the preferable action, then that’s an actual signal that it is preferable. adding more “symmetries” might make noise cancel itself out.
i don’t think we get to require
to return carvings that work for arbitrarily large counterfactual payloads, nor do we need to. in practice, i expect a constant finite (though large) bit length is to be used for
.
there’s a lot of sets-of-all-programs being sampled from here, leaving a lot of room for demons in the solomonoff prior if those are a concern. typically i tend to favor decision theoritic solutions to those, and maybe a correct
implementation would return action-functions
which would depend on whatever decision theory is correct, such that this can be delegated? but it feels like this system has ways to go wrong before that, in what programs get to control most of the “mass” returned by
to begin with.
this is of course highly uncomputable. the intent here, is to use something like logical induction to approximate good results to this function. what makes me hopeful that a powerful AI can make helpful guesses as to what actions this process would find, if it is indeed aligned, is that even i, a mere human mind, feel like i can make some helpful guesses as to what actions this process would find.
a rough sketch of formal aligned AI using QACI
Link post
in this post, i put forth some of my current thoughts about the shape of a formal aligned AI using QACI for its decision — “decision” in the singular here, as this is sufficient when the AI’s decision can be “run me again but with these diffirent inputs”. as it turns out, this doesn’t require solving as many things as i’d thought — it seems like QACI might be general enough to delegate picking a decision theory and solving embedded agency to the counterfactual consideration of the past user.
we’ll posit:
as a convention, we’ll use a prime
the AI is denoted
all sets of functions
finally, we’ll implicitely “cast” mathematical objects as natural numbers wherever appropriate, given that they’re all sampled from countable sets anyways. when they’re cast to or from natural numbers, assume a reasonable bijection between their type and
we’ll define the following:
a “carver” function
a function
a piece of data
a function
this is done by splitting a world
we’ll define
note how
finally, we can define our AI
what’s going on here, is for each world hypothesis
ways this could go wrong include:
the carvings could fail to sufficiently select the actual instances of the past-user in the world; in particular,
the carvings locate question-answering users anywhere in the world, including in the future. this allows for adverserial intelligences to occupy most of the set of simple and coherently-answering carvings, thus steering our AI away from following the actual user’s decisions. a solution to this would be to sufficiently strongly favor carvings that select question-answering processes that are causally upstream of the AI itself; that is to say, causally upstream of
weird implementation details as for how turing machines are encoded could dominate most of the signal of values returned by
i don’t think we get to require
there’s a lot of sets-of-all-programs being sampled from here, leaving a lot of room for demons in the solomonoff prior if those are a concern. typically i tend to favor decision theoritic solutions to those, and maybe a correct
this is of course highly uncomputable. the intent here, is to use something like logical induction to approximate good results to this function. what makes me hopeful that a powerful AI can make helpful guesses as to what actions this process would find, if it is indeed aligned, is that even i, a mere human mind, feel like i can make some helpful guesses as to what actions this process would find.