jessicat

Karma: 306

jessicat 25 Apr 2016 4:09 UTC
2 points
in reply to: gwern’s comment on: What is up with carbon dioxide and cognition? An offer

What monitor is that?

This one. It doesn’t log data.

jessicat 3 Feb 2016 6:09 UTC
6 points
in reply to: Houshalter’s comment on: The AI That Pretends To Be Human
Here’s a scenario that doesn’t seem completely implausible. Suppose Bob is someone whose public key is easily available on the internet. The first AI will read things on the internet and output a message. Some of the message will get put on the public internet. Bob suspects that the AI might have secretly sent him a message (e.g. giving him advice on which stocks to buy). So he tries using his private key to decrypt some of the AI’s output (e.g. the lowest-order bits in some images the AI has output).

Knowing that Bob (or someone else like Bob) will likely do this, the AI (which has access to Bob’s public key, due to reading the internet) encrypts a message with Bob’s public key, and puts the ciphertext in the lowest-order bits of some image. The ciphertext looks like random noise to the second AI. Maybe the plaintext contains both actual stock tips and some dangerous messages that trick Bob into running an unrestrained copy of the first AI (I’m assuming the AI is good at psychological manipulation).

And just the AI asking to use encryption, or giving away a public key

In this scenario, the AI doesn’t need to specifically “ask to use encryption” (it just does the encryption), and it doesn’t need to give away a public key (it just needs to read Bob’s public key on the internet).

This does make the AI’s task harder, so I suppose you could consider this technique as a form of AI boxing, but, like “ordinary” AI boxing, I only expect it to scale up to controlling AIs of a limited capability level.

jessicat 3 Feb 2016 2:52 UTC
3 points
in reply to: Houshalter’s comment on: The AI That Pretends To Be Human
The trouble is that it’s much easier to create a steganographic message (e.g. encrypting a message using a particular public key) than to detect it (which requires knowing the private key or otherwise breaking the encryption). So in this case “much more computing power” has to mean “exponentially more computing power”.

jessicat 2 Feb 2016 21:58 UTC
5 points
on: The AI That Pretends To Be Human
You might be interested in reading:

https://medium.com/ai-control/mimicry-maximization-and-meeting-halfway-c149dd23fc17#.v6e533hkf

https://medium.com/ai-control/elaborations-of-apprenticeship-learning-eb93a53ae3ca#.5ubczdqf0

https://intelligence.org/files/QuantilizersSaferAlternative.pdf

This prevents the first AI from doing evil things with it’s output. If it tries to insert complicated infohazards or subagents into it’s output stream, it will be easily detected as an AI. Instead it needs to mimic humans as closely as possible.

Note that steganography is still a problem. For example, the first AI could insert an message to a human attacker in some of its output, which just looks random to the second AI.
What links here?
- A possible training procedure for human-imitators by jessicata (16 Feb 2016 22:43 UTC; 4 points)
- jessicata's comment on Things that can go wrong with decomposed modules by Stuart_Armstrong (16 Feb 2016 21:56 UTC; 0 points)

jessicat 23 Jul 2015 19:20 UTC
17 points
on: Steelmaning AI risk critiques
One of the most common objection’s I’ve seen is that we’re too far from getting AGI to know what AGI will be like, so we can’t productively work on the problem without making a lot of conjunctive assumptions—e.g. see this post.

jessicat 13 Jun 2015 8:05 UTC
0 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects

And before I could scribble a damned thing, Calude went and solved it six months ago. The Halting Problem, I mean.

Cool. If I get the meaning of the result well, it’s that if you run a random program for some number of steps and it doesn’t halt, then (depending on the exact numbers) it will be unlikely to halt when run on a supercomputer either, because halting times have low density. So almost all programs halt quickly or run a really really long time. Is this correct? This doesn’t quite let you approximate Chaitin’s omega, but it’s interesting that you can approximate a bounded variant of Chaitin’s omega (like what percentage of Turing machines halt when run for 10^50 steps). I can see how this would let you solve the halting problem well enough when you live in a bounded universe.

jessicat 13 Jun 2015 7:00 UTC
0 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
It’s not a specific programming language, I guess it’s meant to look like Church. It could be written as:
```
(query
. (define a (p))
. (foreach (range n) (lambda i)
. . (define x (x-prior))
. . (factor (log (U x a)))))
```
Well so does the sigmoided version

It samples an action proportional to p(a) E[sigmoid(U) | a]. This can’t be written as a function of E[U | a].

jessicat 13 Jun 2015 6:56 UTC
2 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
Due to the planning model, the successor always has some nonzero probability of not pressing the button, so (depending on how much you value pressing it later) it’ll be worth it to press it at some point.

jessicat 10 Jun 2015 2:32 UTC
0 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects

When you use e-raised-to-the alpha times expectation, is that similar to the use of an exponential distribution in something like Adaboost, to take something like odds information and form a distribution over assorted weights?

I’m not really that familiar with adaboost. The planning model is just reflecting the fact that bounded agents don’t always take the maximum expected utility action. The higher alpha is, the more bias there is towards good actions, but the more potentially expensive the computation is (e.g. if you use rejection sampling).

Since any realistic agent needs to be able to handle whatever its environment throws at it, it seems to follow that a realistic agent needs some resource-rational way to handle nonprovable nonhalting.

Ah, that makes sense! I think I see how “trading computational power for algorithmic information” makes sense in this framework.

jessicat 10 Jun 2015 2:26 UTC
3 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
Your model selects an action proportional to p(a) E[sigmoid(U) | a], whereas mine selects an action proportional to p(a) e^E[U | a]. I think the second is better, because it actually treats actions the same if they have the same expected utility. The sigmoid version will not take very high utilities or very low utilities into account much.

Btw it’s also possible to select an action proportional to E[U | a]^n:
```
query {
. a ~ p()
. for i = 1 to n
. . x_i ~ P(x)
. . factor(log U(x, a))
}
```

jessicat 9 Jun 2015 6:15 UTC
3 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
(BTW: here’s a writeup of one of my ideas for writing planning queries that you might be interested in)

Often we want a model where the probability of taking action a is proportional to p(a)e^E[U(x, a)], where p is the prior over actions, x consists of some latent variables, and U is the utility function. The straightforward way of doing this fails:
```
query {
      . a ~ p()
      . x ~ P(x)
      . factor(U(x, a))
}
```
Note that I’m assuming factor takes a log probability as its argument. This fails due to “wishful thinking”: it tends to prefer riskier actions. The problem can be reduced by taking more samples:
```
query {
. a ~ p()
. us = []
. for i = 1 to n
.  .  x_i ~ P(x)
.  .  us.append(U(x_i, a))
. factor(mean(us))
}
```
This does better, because since we took multiple samples, mean(us) is likely to be somewhat accurate. But how do we know how many samples to take? The exact query we want cannot be expressed with any finite n.

It turns out that we just need to sample n from a Poisson distribution and make some more adjustments:
```
query {
. a ~ p()
. n ~ Poisson(1)
. for i = 1 to n
. . x_i ~ P(x)
. . factor(log U(x_i, a))
}
```
Note that U must be non-negative. Why does this work? Consider:
```
P(a)    α     p(a) E[e^sum(log U(x_i, a) for i in range(n))]
               = p(a) E[prod(U(x_i, a) for i in range(n))]
               = p(a) E[ E[prod(U(x_i, a) for i in range(n)) | n] ]
               [here use the fact that the terms in the product are independent]
               = p(a) E[ E[U(x, a)]^n ]
               = p(a) sum(i=0 to infinity) E[U(x, a)]^i / i!
               [Taylor series!]
               = p(a) e^E[U(x, a)]
```
Ideally, this technique would help to perform inference in planning models where we can’t enumerate all possible states.

jessicat 9 Jun 2015 6:12 UTC
0 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects

ZOMFG, can you link to a write-up? This links up almost perfectly with a bit of research I’ve been wanting to do.

Well, a write-up doesn’t exist because I haven’t actually done the math yet :)

But the idea is about algorithms for doing nested queries. There’s a planning framework where you take action a proportional to p(a) e^E[U | a]. If one of these actions is “defer to your successor”, then the computation of (U | a) is actually another query that samples a different action b proportional to p(b) e^E[U | b]. In this case you can actually just go ahead and convert the resulting nested query to a 1-level query: you can convert a “softmax of softmax” into a regular softmax, if that makes sense.

This isn’t doing Vingean reflection, because it’s actually doing all the computational work that its successor would have to do. So I’m interested in ways to simplify computationally expensive nested queries into approximate computationally cheap single queries.

Here’s a simple example of why I think this might be possible. Suppose I flip a coin to decide whether the SAT problem I generate has a solution or not. Then I run a nested query to generate a SAT problem that either does or does not have a solution (depending on the original coin flip). Then I hand you the problem, and you have to guess whether it has a solution or not. I check your solution using a query to find the solution to the problem.

If you suck at solving SAT problems, your best bet might just be to guess that there’s a 50% chance that the problem is solveable. You could get this kind of answer by refactoring the complicated nested nested query model into a non-nested model and then noting that the SAT problem itself gives you very little information about whether it is solveable (subject to your computational constraints).

I’m thinking of figuring out the math here better and then applying it to things like planning queries where your successor has a higher rationality parameter than you (an agent with rationality parameter α takes action a with probability proportional to p(a) e^(α * E[U | a]) ). The goal would be to formalize some agent that, for example, generally chooses to defer to a successor who has a higher rationality parameter, unless there is some cost for deferring, in which case it may defer or not depending on some approximation of value of information.

Your project about trading computing power for algorithmic information seems interesting and potentially related, and I’d be interested in seeing any results you come up with.

even if you still have to place some probability mass on \Bot (bottom)

Is this because you assign probability mass to inconsistent theories that you don’t know are inconsistent?

jessicat 5 Jun 2015 2:16 UTC
3 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
Yes, something like that, although I don’t usually think of it as an adversary. Mainly it’s so I can ask questions like “how could a FAI model its operator so that it can infer the operator’s values from their behavior?” without getting hung up on the exact representation of the model or how the model is found. We don’t have any solution to this problem, even if we had access to a probabilistic program induction black box, so it would be silly to impose the additional restriction that we can’t give the black box any induction problems that are too hard.

That said, bounded algorithms can be useful as inspiration, even for unbounded problems. For example, I’m currently looking at ways you could use probabilistic programs with nested queries to model Vingean reflection.

jessicat 4 Jun 2015 23:34 UTC
7 points
in reply to: [deleted]’s comment on: FAI Research Constraints and AGI Side Effects
I should be specific that the kinds of results we want to get are those where you could, in principle, use a very powerful computer instead of a hypercomputer. Roughly, the unbounded algorithm should be a limit of bounded algorithms. The kinds of allowed operations I am thinking about include:
- Solomonoff induction
- optimizing an arbitrary function
- evaluating an arbitrary probabilistic program
- finding a proof of X if one exists
- solving an infinite system of equations that is guaranteed to have a solution
In all these cases, you can get arbitrarily good approximations using bounded algorithms, although they might require a very large amount of computation power. I don’t think things like this would lead to contradictions if you did them correctly.

jessicat 4 Jun 2015 6:01 UTC
3 points
in reply to: ozziegooen’s comment on: FAI Research Constraints and AGI Side Effects
Thanks for the detailed response! I do think the framework can still work with my assumptions. The way I would model it would be something like:
1. In the first stage, we have G->Fremaining (the research to an AGI->FAI solution) and Gremaining (the research to enough AGI for UFAI). I expect G->Fremaining < Gremaining, and a relatively low leakage ratio.
2. after we have AGI->FAI, we have Fremaining (the research for the AGI to input to the AGI->FAI) and Gremaning (the research to enough AGI for UFAI). I expect Fremaining > Gremaining, and furthermore I expect the leakage ratio to be high enough that we are practically guaranteed to have enough AGI capabilities for UFAI before FAI (though I don’t know how long before). Hence the strategic importance of developing AGI capabilities in secret, and not having them lying around for too long in too many hands. I don’t really see a way of avoiding this: the alternative is to have enough research to create FAI but not a paperclip maximizer, which seems implausible (though it would be really nice if we could get this state!).
Also, it seems I had misinterpreted the part about rg and rf, sorry about that!

jessicat 3 Jun 2015 22:02 UTC
20 points
on: FAI Research Constraints and AGI Side Effects
This model seems quite a bit different from mine, which is that FAI research is about reducing FAI to an AGI problem, and solving AGI takes more work than doing this reduction.

More concretely, consider a proposal such as Paul’s reflective automated philosophy method, which might be able to be implemented using epsiodic reinforcement learning. This proposal has problems, and it’s not clear that it works—but if it did, then it would have reduced FAI to a reinforcement learning problem. Presumably, any implementations of this proposal would benefit from any reinforcement learning advances in the AGI field.

Of course, even if we a proposal like this works, it might require better or different AGI capabilities from UFAI projects. I expect this to be true for black-box FAI solutions such as Paul’s. This presents additional strategic difficulties. However, I think the post fails to accurately model these difficulties. The right answer here is to get AGI researchers to develop (and not publish anything about) enough AGI capabilities for FAI without running a UFAI in the meantime, even though the capabilities to run it exist.

Assuming that this reflective automated philosophy system doesn’t work, it could still be the case that there is a different reduction from FAI to AGI that can be created through armchair technical philosophy. This is often what MIRI’s “unbounded solutions” research is about: finding ways you could solve FAI if you had a hypercomputer. Once you find a solution like this, it might be possible to define it in terms of AGI capabilities instead of hypercomputation, and at that point FAI would be reduced to an AGI problem. We haven’t put enough work into this problem to know that a reduction couldn’t be created in, say, 20 years by 20 highly competent mathematician-philosophers.

In the most pessimistic case (which I don’t think is too likely), the task of reducing FAI to an AGI problem is significantly harder than creating AGI. In this case, the model in the post seems to be mostly accurate, except that it neglects the fact that serial advances might be important (so we get diminishing marginal progress towards FAI or AGI per additional researcher in a given year).
What links here?
- Gram_Stone's comment on FAI Research Constraints and AGI Side Effects by JustinShovelain (8 Jun 2015 17:27 UTC; 3 points)

jessicat 7 May 2015 17:24 UTC
1 point
in reply to: [deleted]’s comment on: Debunking Fallacies in the Theory of AI Motivation
I agree that choosing an action randomly (with higher probability for good actions) is a good way to create a fuzzy satisficer. Do you have any insights into how to:
1. create queries for planning that don’t suffer from “wishful thinking”, with or without nested queries. Basically the problem is that if I want an action conditioned on receiving a high utility (e.g. we have a factor on the expected utility node U equal to e^(alpha * U) ), then we are likely to choose high-variance actions while inferring that the rest of the model works out such that these actions return high utilities
2. extend this to sequential planning without nested nested nested nested nested nested queries

jessicat 7 May 2015 17:18 UTC
5 points
in reply to: [deleted]’s comment on: Debunking Fallacies in the Theory of AI Motivation
I would tend to say that you should be training a conceptual map of the world before you install anything like action-taking capability or a goal system of any kind.

This seems like a sane thing to do. If this didn’t work, it would probably be because either
1. lack of conceptual convergence and human understandability; this seems somewhat likely and is probably the most important unknown
2. our conceptual representations are only efficient for talking about things we care about because we care about these things; a “neutral” standard such as resource-bounded Solomonoff induction will horribly learn things we care about for “no free lunch” reasons. I find this plausible but not too likely (it seems like it ought to be possible to “bootstrap” an importance metric for deciding where in the concept space to allocate resources).
3. we need the system to have a goal system in order to self-improve to the point of creating this conceptual map. I find this a little likely (this is basically the question of whether we can create something that manages to self-improve without needing goals; it is related to low impact).
Of course, I also tend to say that you should just use a debugged (ie: cured of systematic faults) model of human evaluative processes for your goal system, and then use actual human evaluations to train the free parameters, and then set up learning feedback from the learned concept of “human” to the free-parameter space of the evaluation model.

I agree that this is a good idea. It seems like the main problem here is that we need some sort of “skeleton” of a normative human model whose parts can be filled in empirically, and which will infer the right goals after enough training.

jessicat 7 May 2015 8:36 UTC
3 points
in reply to: [deleted]’s comment on: Debunking Fallacies in the Theory of AI Motivation

Regularization is already a part of training any good classifier.

A technical point here: we don’t learn a raw classifier, because that would just learn human judgments. In order to allow the system to disagree with a human, we need to use some metric other than “is simple and assigns high probability to human judgments”.

For something like FAI, I want a concept-learning algorithm that will look at the world in this naturalized, causal way (which is what normal modelling shoots for!), and that will model correctly at any level of abstraction or under any available set of features, and will be able to map between these levels as the human mind can.

I totally agree that a good understanding of multi-level models is important for understanding FAI concept spaces. I don’t have a good understanding of multi-level maps; we can definitely see them as useful constructs for bounded reasoners, but it seems difficult to integrate higher levels into the goal system without deciding things about the high-level map a priori so you can define goals relative to this.

jessicat 7 May 2015 8:27 UTC
6 points
in reply to: [deleted]’s comment on: Debunking Fallacies in the Theory of AI Motivation
Okay, thanks a lot for the detailed response. I’ll explain a bit about where I’m coming from with understading the concept learning problem:
- I typically think of concepts as probabilistic programs eventually bottoming out in sense data. So we have some “language” with a “library” of concepts (probabilistic generative models) that can be combined to create new concepts, and combinations of concepts are used to explain complex sensory data (for example, we might compose different generative models at different levels to explain a picture of a scene). We can (in theory) use probabilistic program induction to have uncertainty about how different concepts are combined. This seems like a type of swarm relaxation, due to probabilistic constraints being fuzzy. I briefly skimmed through the McClellard chapter and it seems to mesh well with my understanding of probabilistic programming.
- But, when thinking about how to create friendly AI, I typically use the very conservative assumptions of statistical learning theory, which give us guarantees against certain kinds of overfitting but no guarantee of proper behavior on novel edge cases. Statistical learning theory is certainly too pessimistic, but there isn’t any less pessimistic model for what concepts we expect to learn that I trust. While the view of concepts as probabilistic programs in the previous bullet point implies properties of the system other than those implied by statistical learning theory, I don’t actually have good formal models of these, so I end up using statistical learning theory.
I do think that figuring out if we can get more optimistic (but still justified) assumptions is good. You mention empirical experience with swarm relaxation as a possible way of gaining confidence that it is learning concepts correctly. Now that I think about it, bad handling of novel edge cases might be a form of “meta-overfitting”, and perhaps we can gain confidence in a system’s ability to deal with context shifts by having it go through a series of context shifts well without overfitting. This is the sort of thing that might work, and more research into whether it does is valuable, but it still seems worth preparing for the case where it doesn’t.

Anyway, thanks for giving me some good things to think about. I think I see how a lot of our disagreements mostly come down to how much convergence we expect from different concept learning systems. For example, if “psychological manipulation” is in some sense a natural category, then of course it can be added as a weak (or even strong) constraint on the system.
I’ll probably think about this a lot more and eventually write up something explaining reasons why we might or might not expect to get convergent concepts from different systems, and the degree to which this changes based on how value-laden a concept is.

There is a lot of talk that can be given about how that complex union takes place, but here is one very important takeaway: it can always be made to happen in such a way that there will not, in the future, be any Gotcha cases (those where you thought you did completely merge the two concepts, but where you suddenly find a peculiar situation where you got it disastrously wrong). The reason why you won’t get any Gotcha cases is that the concepts are defined by large numbers of weak constraints, and no strong constraints—in such systems, the effect of smaller and smaller numbers of concepts can be guaranteed to converge to zero. (This happens for the same reason that the effect of smaller and smaller sub-populations of the molecules in a gas will converge to zero as the population sizes go to zero).

I didn’t really understand a lot of what you said here. My current model is something like “if a concept is defined by lots of weak constraints, then lots of these constraints have to go wrong at once for the concept to go wrong, and we think this is unlikely due to induction and some kind of independence/uncorrelatedness assumption”; is this correct? If this is the right understanding, I think I have low confidence that errors in each weak constraint are in fact not strongly correlated with each other.