I mentioned this problem already. And I (07/2010) thought about ways to ensure that FAI will prefer my/our/rational way of extrapolating.
Now I think it would be better if FAI will select coherent subset of volitions of all reflectively consistent extrapolations. As I suspect it will be something like: protect humanity from existential risk, but don’t touch it beyond that.
It [FAI] doesn’t have a judgement of its own.
[...]
And if it [FAI] can reliably figure out what a wiser version of us would say, it substitutes that person’s judgement for ours.
[...]
I would direct the person to LessWrong, [...] until they’re a good enough rationalist [...] -- then ask them again.
It seems you have a flaw in your reasoning. You will direct a person to LessWrong, someone else will direct a person to church. And FAI should figure out somehow which direction a person should take to be wiser, without a judgment of its own.
So in your design, you’d have to figure out a way to prevent self-halting under all possible input conditions, under all possible self-modifications of the machine.
Self-modifications are being performed by the machine itself. Thus we (and/or machine) don’t need to prove that all modifications aren’t “suicidal”. Machine can be programmed to perform only provably (in reasonable time) non-suicidal self-modifications. Rice’s theorem doesn’t apply in this case.
Edit: However this leaves meta-level unpatched. Machine can self-modify into non-suicidal machine that doesn’t care about preserving non-suicidability over modifications. This can be patched by constraining allowed self-modifications to a class of modifications that leads to machines with provably equivalent behavior (with a possible side effect of inability to self-repair).
If anyone is interested. This extension doesn’t seem to lead to anything of interest.
If we map continuum of UDASSA multiverses into [0;1) then Lebesgue measure of set of multiverses which run particular program is 1⁄2.
Let binary number 0.b1 b2 … bn … be representation of multiverse M if for all n: (bn=1 iff M runs program number n, and bn=0 otherwise).
It is easy to see that map of set of multiverses which run program number n is a collection of intervals [i/2^n;2i/2^n) for i=1..2^(n-1). Thus its Lebesgue measure is 2^(n-1)/2^n=1/2.
Should we stop on UDASSA? Can we consider universe that consists of continuum of UDASSAs each running some (infinite) subset of set of all possible programs.
It even depends on philosophy. Specifically on whether following equality holds.
I survive = There (not necessarily in our universe) exists someone who remembers everything I remember now plus failed suicide I’m going to conduct now.
or
I survive = There exists someone who don’t remember everything I remember now, but he acts as I would acted if I remember what he remembers. (I’m not sure whether I correctly expressed subjunctive mood)
It seems that what I call indirectly self-referential value function can be a syntactic preference as defined by Vladimir Nesov.
Well, I tried to precisely define toy model I use. As for utilons, I took the word that is common here, without much thinking about it. It doesn’t seem to blur the meaning of post significantly.
Future is not a world-state, it is a sequence of world-states. Thus your statement must be reformulated somehow.
Either (1) we must define utility function over a set of (valid) sequences of world-states or (2) we must define what it means that sequence of world-states is optimized for given U, [edit] and that means that this definition should be a part of U itself as U is all we care about. [/edit]
And option 1 is either impossible if rules of world don’t permit an agent to hold full history of world or we can define equivalent utility function over world-states, thus leaving only option 2 as viable choice.
Then your statement means either
For all world-states x in the sequence of world-states optimized for U U(x)>U(y), where y doesn’t belong to the sequence of world-states optimized for U. And that means we must know in advance which future world-states are reachable.
or
U(x)>U(y) for all world-states x in the sequence of world-states optimized for U, for all world-states y in the sequence of world-states optimized for some U2. But U2(x)<U2(y).
However it is not a main point of my post. Main point is that future optimization isn’t necessarily maximizing a fixed function we know in advance.
Edit: I don’t really argue with Vladimir, as future optimization as utility maximization can be a part of his value function, and arguing about values per se is pointless. But maybe he misinterprets what he really values.
I don’t understand. Fixed utility function doesn’t equal unfixed utility function, as optimizing for them leads to different outcomes.
Edit: you mean, that we cannot optimize for unfixed utility function? In second part of article I’ve tried to demonstrate that meaning of optimization according to utility function should be a part of utility function itself, as otherwise result of optimization depends on optimization algorithm too, thus making utility function insufficient to describe everything one cares about.
Debugging will be PITA. Both ways.
Goal is what an agent optimizes for at a given point in time. Value is the initial goal of an agent (in your toy model at least).
In my root post it seems to be optimal for agent A to self-modify into agent A’, which optimizes for G2, thus agent A’ succeeds in optimizing world according to its values (goal of agent A). But original goal doesn’t influence its optimization procedure anymore. Thus if we’ll analyze agent A’ (without knowledge of world’s history), we’ll be unable to infer its values (its original goal).
It’s error on my part, I assumed that goal stability equals value stability. But then it looks like that it can be impossible to reconstruct agent’s values given only its current state.
You seem to assume that world is indifferent to agent’s goals. But if there’s another agent, then it can be not the case.
Let G1 be “tile the universe with still-life”, G2 be “tile upper half of the universe with still-life”.
If agent A has goal G1, it will be provable destroyed by agent B, if A will change it’s goal to G2, then B will not interfere.
A and B have full information on world’s state.
Should A modify its goal?
Edit: Goal stability != value stability. So my point isn’t valid.
Uhm, looks good as invertible fact, by the way.
General brain’s capability for (cross-domain) pattern matching? While studying objects of new formal system and their relations, it patternmatch them with already known formal and real objects and relations. Successful match results in insight (that can be formally wrong).
Pros: mathematics stems from observation of real world objects and their relations: countable objects, lines, etc.
Cons: too simple.
Experimental tests aren’t about consciousness, they are about behavior and brain activity.
Even if sufficiently precise theory of brain operation will be devised (which will allow to predict e.g. after-stimulus train of thoughts, introspective thoughts, description of qualia given pattern of neural activity etc.). I think there still will be philosophers that will see explanatory gap as the theory doesn’t explain how it feels like (for them?) to be experiencing this stimulus or process of thinking.
Thus hard problem of consciousness seems to be scientifically undecidable, at least in current formulation.
Leaving philosophical considerations aside I myself am inclined to favor representational theories of consciousness, e.g. SMT, as they allow to put consciousness in evolutionary perspective as one of stages of development of central nervous system’s representational capabilities, where it [CNS] becomes capable of representing a part of its own internal state in the form that allows to store, recall, communicate and modify parts of internal state in ways aren’t possible without explicit representation.
Oh, thank you. It doesn’t make sense for me, but now I understand.
Let Pg be RAI’s estimation of probability of successful completion of computation without conversion of Earth into computronium. Let Pb be RAI’s estimation of probability of success if it converts Earth into computronium.
Case 1. RAI modifies itself into non-understanding of simulation threats.
When RAI is not simulated, it ignores all threats and proceeds to converting Earth into computronium, thus increasing its probability of success by Pb-Pg (it becomes protected from accidental/intentional turning-offs, etc.).
When RAI is simulated, it fails bargaining and its probability of success becomes 0.
Overall estimated probability of success: Pb times RAI’s estimation of probability that it isn’t being simulated (Pn).
Case 2. RAI doesn’t modify itself.
Overall estimated probability of success: Pg.
Thus it is rational for RAI to modify itself iff Pb*Pn > Pg. Pn is defined by RAI’s prior and cannot be updated in perfect simulation and in real world. Conclusion: don’t build AI which doesn’t believe simulation argument.