jessicat

Karma: 306

jessicat Dec 2, 2014, 3:11 AM
0 points
in reply to: Unknowns’s comment on: Why I will Win my Bet with Eliezer Yudkowsky
I’m talking about the fact that humans can (and sometimes do) sort of optimize the universe. Like, you can reason about the way the universe is and decide to work on causing it to be in a certain state.

So people say they have general goals, but in reality they remain human beings with various tendencies, and continue to act according to those tendencies, and only support that general goal to the extent that it’s consistent with those other behaviors.

This could very well be the case, but humans still sometimes sort of optimize the universe. Like, I’m saying it’s at least possible to sort of optimize the universe in theory, and humans do this somewhat, not that humans directly use universe-optimizing to select their actions. If a way to write universe-optimizing AGIs exists, someone is likely to find it eventually.

I think it is perfectly possible to develop an AI intelligent enough to pass the Turing Test, but which still would not have anything (not even “passing the Turing Test”) as a general goal that would take over its behavior and make it conquer the world.

I agree with this. There are some difficulties with self-modification (as elaborated in my other comment), but it seems probable that this can be done.

And I would expect the first AIs to be of this kind by default, because of the difficulty of ensuring that the whole of the AI’s activity is ordered to one particular goal.

Seems pretty plausible. Obviously it depends on what you mean by “AI”; certainly, most modern-day AIs are this way. At the same time, this is definitely not a reason to not worry about AI risk, because (a) tool AIs could still “accidentally” optimize the universe depending on how search for self-modifications and other actions happens, and (b) we can’t bet on no one figuring out how to turn a superintelligent tool AI into a universe optimizer.

I do agree with a lot of what you say: it seems like a lot of people talk about AI risk in terms of universe-optimization, when we don’t even understand how to optimize functions over the universe given infinite computational power. I do think that non-universe-optimizing AIs are under-studied, that they are somewhat likely to be the first human-level AGIs, and that they will be extraordinary useful for solving some FAI-related problems. But none of this makes the problems of AI risk go away.

jessicat Dec 1, 2014, 9:47 PM
0 points
in reply to: Unknowns’s comment on: Why I will Win my Bet with Eliezer Yudkowsky
In that sense I think the orthogonality thesis will turn out to be false in practice, even if it is true in theory. It is simply too difficult to program a precise goal into an AI, because in order for that to work the goal has to be worked into every physical detail of the thing. It cannot just be a modular add-on.

I find this plausible but not too likely. There are a few things needed for a universe-optimizing AGI:
1. really good mathematical function optimization (which you might be able to use to get approximate Solomonoff induction)
2. a way to specify goals that are still well-defined after an ontological crisis
3. a solution to the Cartesian boundary problem
I think it is likely that (2) and (3) will eventually be solved (or at least worked around) well enough that you can build universe-optimizing AGIs, partially on the basis that humans approximately solve these somehow and we already have tentative hypotheses about what solutions to these problems might look like. It might be the case that we can’t really get (1), we can only get optimizers that work in some domains but not others. Perhaps universe-optimization (when reduced to a mathematical problem using (2) and (3)) is too difficult of a domain: we need to break the problem down into sub-problems in order to feed it to the optimizer, resulting in a tool-AI like design. But I don’t think this is likely.

If we have powerful tool AIs before we get universe optimizers, this will probably be a temporary stage, because someone will figure out how to use a tool AI to design universe-optimizers someday. But your bet was about the first AGI, so this would still be consistent with you winning your bet.

jessicat Nov 27, 2014, 6:58 AM
5 points
on: Why I will Win my Bet with Eliezer Yudkowsky
I, too, think that AIs that don’t optimize a function over the universe (but might optimize one over a domain) are more likely to be safe. This is quite related to the idea of tool AI, proposed by Holden and criticized by Eliezer.

The key here seems to be creating a way to evaluate and search for self-improvements in a way that won’t cause optimization over universe states. In theory, evaluation of a self-improvement might be able to be restricted to a domain: does this modification help me play chess better according to a model of the situation in which a Cartesian boundary exists and I am running on an abstract virtual machine rather than physical computers embedded in the universe?

We could separate this task into generation of possible self-improvements and tests of self-improvements (a suggester/verifier architecture). If the suggester is superintelligent and dangerous (e.g. as a consequence of being universe-optimizing), then we cannot safely test these self-improvements. This is for a few reasons:
1. testing a self-improvement involves executing code. If the virtual machine has security flaws, then this could enable the suggester to cause the AI to rewrite into a universe-optimizer or otherwise do something dangerous.
2. it is possible to create self-improvements that look good according to the verifier but contain information that psychologically manipulates humans (see the AI-box experiment)
3. a self-modification could score fairly well according to the abstract evaluation function, but at the same time also behave in a universe-optimizing manner some of the time. This self-modification would be considered suboptimal by the verifier, but might still be considered relatively good.
So we can’t just use any old suggester and rely on the verifier to do the heavy lifting. Luckily, it may be possible to construct a suggester that acts as if it is running on an abstract virtual machine. If the suggester does not (directly or indirectly) use universe-optimizing as a way to generate self-improvements, then it shouldn’t generate universe-optimizing (therefore dangerous) self-improvements. The key gotcha here is the “indirectly” part: how do we know that the suggester isn’t (e.g.) using many different heuristics to come up with improvements, where some combination of the heuristics ends up expressing something like “try creating improvements by optimizing the universe”. In other words, is universe-optimizing a somewhat useful strategy for finding improvements that good general abstract-mathematical-function-optimizers will pick up on? I don’t know the answer to this question. But if we could design suggesters that don’t directly or indirectly optimize a function over the universe, then maybe this will work.

jessicat Nov 15, 2014, 12:03 AM
3 points
in reply to: Manfred’s comment on: [Link] Physics-based anthropics?

Am working on it—as a placeholder, for many problems, one can use Stuart Armstrong’s proposed algorithm of finding the best strategy according to a non-anthropic viewpoint that adds the utilities of different copies of you, and then doing what that strategy says.

I think this essentially leads to SIA. Since you’re adding utilities over different copies of you, it follows that you care more about universes in which there are more copies of you. So your copies should behave as if they anticipate the probability of being in a universe containing lots of copies to be higher.

However, before assuming [stuff about the universe], you should have [observational data supporting that stuff].

It’s definitely not a completely justified assumption. But we do have evidence that the universe supports arbitrary computations, that it’s extremely large, and that some things are determined randomly, so as a result it will be running many different computations in parallel. This provides some evidence that, if there is a multiverse, it will have similar properties.

jessicat Nov 14, 2014, 7:41 PM
1 point
in reply to: Manfred’s comment on: [Link] Physics-based anthropics?
I’m not sure what you mean by “vanilla anthropics”. Both SSA and SIA are “simple object-level rules for assigning anthropic probabilities”. Vanilla anthropics seems to be vague enough that doesn’t give an answer to the doomsday argument or the presumptuous philosopher problem.

On another note, if you assume that a nonzero percentage of the multiverse’s computation power is spent simulating arbitrary universes with computation power in proportion to the probabilities of their laws of physics, then both SSA and SIA will end up giving you very similar predictions to Brian_Tomasik’s proposal, although I think they might be slightly different.

jessicat Nov 14, 2014, 7:31 PM
2 points
in reply to: Luke_A_Somers’s comment on: Superintelligence 9: The orthogonality of intelligence and goals
Okay. We seem to be disputing definitions here. By your definition, it is totally possible to build a very good cross-domain optimizer without it being an agent (so it doesn’t optimize a utility function over the universe). It seems like we mostly agree on matters of fact.

jessicat Nov 14, 2014, 7:52 AM
1 point
on: [Link] Physics-based anthropics?
I agree with this but I prefer weighting things by computation power instead of physics cells (which may turn out to be somewhat equivalent). It’s easy to justify this model by assuming that some percentage of the multiverse’s computation power is spent simulating all universes in parallel. See Schmidhuber’s paper on this.

jessicat Nov 13, 2014, 5:06 AM
0 points
on: Rodney Brooks talks about Evil AI and mentions MIRI [LINK]
Well, he’s right that intentionally evil AI is highly unlikely to be created:

Malevolent AI would need all these capabilities, and then some. Both an intent to do something and an understanding of human goals, motivations, and behaviors would be keys to being evil towards humans.

which happens to be the exact reason why Friendly AI is difficult. He doesn’t directly address things that don’t care about humans, like paperclip maximizers, but some of his arguments can be applied to them.

Expecting more computation to just magically get to intentional intelligences, who understand the world is similarly unlikely.

He’s totally right that AGI with intentionality is an extremely difficult problem. We haven’t created anything that is even close to practically approximating Solomonoff induction across a variety of situations, and Solomonoff induction is insufficient for the kind of intentionality you would need to build something that cares about universe states while being able to model the universe in a flexible manner. But, you can throw more computation power at a lot of problems to get better solutions, and I expect approximate Solomonoff induction to become practical in limited ways as computation power increases and moderate algorithmic improvements are made. This is true partially because greater computation power allows one to search for better algorithms.

I do agree with him that human-level AGI within the next few decades is unlikely and that significantly slowing down AI research is probably not a good idea right now.

jessicat Nov 12, 2014, 7:45 PM
1 point
in reply to: Luke_A_Somers’s comment on: Superintelligence 9: The orthogonality of intelligence and goals
I don’t think the connotations of “silly” are quite right here. You could still use this program to do quite a lot of useful inference and optimization across a variety of domains, without killing everyone. Sort of like how frequentist statistics can be very accurate in some cases despite being suboptimal by Bayesian standards. Bostrom mostly only talks about agent-like AIs, and while I think that this is mostly the right approach, he should have been more explicit about that. As I said before, we don’t currently know how to build agent-like AGIs at the moment because we haven’t solved the ontology mapping problem, but we do know how to build non-agentlike cross-domain optimizers given enough computation power.

jessicat Nov 11, 2014, 11:13 PM
1 point
in reply to: Luke_A_Somers’s comment on: Superintelligence 9: The orthogonality of intelligence and goals
1) Perhaps you give it one domain and a utility function within that domain, and it returns a good action in this domain. Then you give it another domain and a different utility function, and it returns a good action in this domain. Basically I’m saying that it doesn’t maximize a single unified utility function.

2) You prove too much. This implies that the Unix cat program has a utility function (or else it is wasting effort). Technically you could view it as having a utility function of “1 if I output what the source code of cat outputs, 0 otherwise”, but this really isn’t a useful level of analysis. Also, if you’re going to go the route of assigning a silly utility function to this program, then this is a utility function over something like “memory states in an abstract virtual machine”, not “states of the universe”, so it will not necessarily (say) try to break out of its box to get more computation power.

jessicat Nov 11, 2014, 7:19 AM
4 points
in reply to: KatjaGrace’s comment on: Superintelligence 9: The orthogonality of intelligence and goals
My main problem with this chapter is that Bostrom assumes that an AI has a single utility function, and that it is expressed as a function on possible states of the universe. Theoretically, you could design a program that is given a domain (in the form of, say, a probabilistic program) and outputs a good action within this domain. In principle, you could have asked it to optimize a function over the universe, but if you don’t, then it won’t. So I think that this program can determine near-optimal behavior across a variety of domains without actually having a utility function (especially not one that is over states of the universe). Of course, this would still be dangerous because anyone could ask it to optimize a function over the universe, but I do think that the assumption that AIs have utility functions over universe states should be more clearly stated and discussed.

It should also be noted that we currently do know how to solve domain-specific optimization problems (such as partially observable Markov decision processes) given enough computation power, but we do not know how to optimize a function over universe states in a way that is agnostic about how to model the universe; this is related to the ontological crisis problem.

jessicat Nov 8, 2014, 8:04 AM
4 points
on: more on predicting agents

We would expect that an agent that grows in its understanding would change its utility function—if only because to do so would make it less predictable to adversarial agents that would exploit its simplicity.

Game theory predicts that in some cases, an agent with a fixed utility function will randomize its actions (for example, the Nash equilibrium strategy for rock paper scissors is to randomize equally between all 3). If true randomness is unavailable, an agent may use its computational power to compute expensive pseudorandom numbers that other agents will have difficulty also computing. There is no need for the agent to change its utility function. Changing its utility function would be likely to cause the agent to optimize for different things than the previous utility function would optimize for; therefore, if the agent is acting according to the original utility function, changing the utility function is unlikely to be considered a good action.

In fact, it would be strange for an agent that became more intelligent to have a stable utility function, because the range of possible utility functions available to a more intelligent agent are greater.

Given that changing your utility function is generally a bad thing for a utility maximizer, it does not seem like this will happen. Instead, it seems more likely that the agent’s modeling ability will improve and this will change its observed behavior, possibly making it less predictable. You can often change an agent’s behavior quite a lot by changing its beliefs.

There is certainly the important issue of deciding what the old utility function, defined relative to the agent’s old model of the world, means if the agent’s model of the world changes, as explored in this paper, but this does not lead to the agent taking on a fundamentally different utility function, only a faithful representation of the original one.

jessicat Oct 13, 2014, 7:52 AM
1 point
in reply to: Manfred’s comment on: Baysian conundrum
I think the conflict dissolves if you actually try to use your anticipation to do something useful.

Example. Suppose you can either push button A (before 11AM) so that if you’re still in the room you get a small happiness reward, or you can push button B so if you’re transported to paradise you get a happiness reward. If you value happiness in all your copies equally, you should push button B, which means that you “anticipate” being transported to paradise.

This gets a little weird with the clones and to what extent you should care about them, but there’s an analogous situation where I think the anthropic measure solution is clearly more intuitive: death. Suppose the many worlds interpretation is true and you set up a situation so that you die in 99% of worlds. Then should you “anticipate” death, or anticipate surviving? Anticipating death seems like the right thing. A hedonist should not be willing to sacrifice 1 unit of pleasure before quantum suicide in order to gain 10 units on the off chance that they survive.

So I think that one’s anticipation of the future should not be a probability distribution over sensory input sequences (which sums to 1), but rather a finite non-negative distribution) (which sums to a non-negative real number).