This seems like a reasonable approach. My main worry is that, if the probability of sampling good advice is very low (less than 1 in 1 billion), then good advice won’t appear at all in the training process, so the training process would result in exactly the same resulting agent as if the advice were never reported. This is essentially the “high variance” problem that you point out. In the case of plagiarism, it’s not clear how to come up with a proposal distribution assigning non-negligible probability to good advice (i.e. a list of references to plagiarized works).
If B is “sufficiently more intelligent” than A, then B can do things like monitor A’s access to the library, read parts of A’s memory state, etc. and this information can be used to detect plagarism directly or yield a better advice distribution. An adversarial model could also be used in this capacity, and increased in intelligence along with A.
If A is “sufficiently intelligent” to “correctly model how B′(a) is generated from B(a)”, I claim that A never needs to see an example of punishment to want to adopt correct behaviour, because it will be able to perform the expected value calculations and arrive at the correct answer. I’m currently convinced of this claim, but I don’t know how to state it more rigorously.
You might be able to use an off-policy learner, and show it a number of generated examples of plagarism and the appropriate (negative) rewards, with Improbable Oversight ensuring that this information matches the true expected values in the setup (though this may not lead to the agent behaving safely). Or maybe you could train the agent initially in more limited environments where Improbable Oversight can be turned on and off, with the agent clearly informed about whether it is turned on or off, and have IO turned on when it moves to more complex environments.
If B is “sufficiently more intelligent” than A, then B can do things like monitor A’s access to the library, read parts of A’s memory state, etc. and this information can be used to detect plagarism directly or yield a better advice distribution.
Yes, but at this point it seems like this proposal is mostly the same as Paul’s original proposal for informed oversight. It has the same uncertainties about it (e.g. is B able to understand A’s architecture, when A’s architecture is not optimized for understandability). Though, I suppose it’s slightly easier to have a 1/1000 chance of detecting plagiarism than a 90% chance, so this could still be useful.
I claim that A never needs to see an example of punishment to want to adopt correct behaviour, because it will be able to perform the expected value calculations and arrive at the correct answer.
It seems like this is based on the assumption that you can specify the agent’s utility function to care about such things. Reinforcement learning agents only acquire their “utility function” through training, so extremely large punishments have to show up in the training process somewhere. In general, the agent will need Var(R|a)/ϵ2 training examples to converge to good beliefs about E[R|a] within ϵ, where R is a random variable indicating how much reward the agent receives and a is the action it takes. It seems like this might work if the agent is able to access low-variance estimates of E[R|a], though.
You might be able to use an off-policy learner, and show it a number of generated examples of plagarism and the appropriate (negative) rewards
Seems like this training data won’t distinguish “you get punished for plagiarizing” from “you get punished for plagiarizing in the exact method used in the off-policy examples”. That is, the agent will be uncertain about whether certain forms of plagiarism will receive punishments, since those types don’t appear in the off-policy training set. This seems insufficient when we want to prevent bad actions that we’ll have trouble thinking of ahead of time (e.g. novel ways of doing steganography).
I don’t quite have the exact specification of what I have in mind yet. Fortunately, this seems like a problem that I could try to address in a toy model with current techniques, so I can think about this a bit more and try to come up with a concrete system which would work.
I think that it should be possible to construct a reinforcement learning which can make use of side information. One proposal would be something like: construct A so that it internally estimates (with good uncertainty models) F(a,r) and B(a), and uses those estimates to predict and maximize E[B′(a)] (perhaps using something as simple as a monte-carlo simulation that can run for a large number of draws). Then allow A to, as an action or a prelude to acting, ask questions about the value of F(a,r) for specific (a,r) pairs, or for estimates of F(a) (with some model that estimates of F(a) might be incorrect). If A performs correct value of information calculations, it should value asking these questions in the training process and correctly learn values, even if it never experiences a situation where it is caught and punished.
This seems like a reasonable approach. My main worry is that, if the probability of sampling good advice is very low (less than 1 in 1 billion), then good advice won’t appear at all in the training process, so the training process would result in exactly the same resulting agent as if the advice were never reported. This is essentially the “high variance” problem that you point out. In the case of plagiarism, it’s not clear how to come up with a proposal distribution assigning non-negligible probability to good advice (i.e. a list of references to plagiarized works).
If B is “sufficiently more intelligent” than A, then B can do things like monitor A’s access to the library, read parts of A’s memory state, etc. and this information can be used to detect plagarism directly or yield a better advice distribution. An adversarial model could also be used in this capacity, and increased in intelligence along with A.
If A is “sufficiently intelligent” to “correctly model how B′(a) is generated from B(a)”, I claim that A never needs to see an example of punishment to want to adopt correct behaviour, because it will be able to perform the expected value calculations and arrive at the correct answer. I’m currently convinced of this claim, but I don’t know how to state it more rigorously.
You might be able to use an off-policy learner, and show it a number of generated examples of plagarism and the appropriate (negative) rewards, with Improbable Oversight ensuring that this information matches the true expected values in the setup (though this may not lead to the agent behaving safely). Or maybe you could train the agent initially in more limited environments where Improbable Oversight can be turned on and off, with the agent clearly informed about whether it is turned on or off, and have IO turned on when it moves to more complex environments.
Yes, but at this point it seems like this proposal is mostly the same as Paul’s original proposal for informed oversight. It has the same uncertainties about it (e.g. is B able to understand A’s architecture, when A’s architecture is not optimized for understandability). Though, I suppose it’s slightly easier to have a 1/1000 chance of detecting plagiarism than a 90% chance, so this could still be useful.
It seems like this is based on the assumption that you can specify the agent’s utility function to care about such things. Reinforcement learning agents only acquire their “utility function” through training, so extremely large punishments have to show up in the training process somewhere. In general, the agent will need Var(R|a)/ϵ2 training examples to converge to good beliefs about E[R|a] within ϵ, where R is a random variable indicating how much reward the agent receives and a is the action it takes. It seems like this might work if the agent is able to access low-variance estimates of E[R|a], though.
Seems like this training data won’t distinguish “you get punished for plagiarizing” from “you get punished for plagiarizing in the exact method used in the off-policy examples”. That is, the agent will be uncertain about whether certain forms of plagiarism will receive punishments, since those types don’t appear in the off-policy training set. This seems insufficient when we want to prevent bad actions that we’ll have trouble thinking of ahead of time (e.g. novel ways of doing steganography).
I don’t quite have the exact specification of what I have in mind yet. Fortunately, this seems like a problem that I could try to address in a toy model with current techniques, so I can think about this a bit more and try to come up with a concrete system which would work.
I think that it should be possible to construct a reinforcement learning which can make use of side information. One proposal would be something like: construct A so that it internally estimates (with good uncertainty models) F(a,r) and B(a), and uses those estimates to predict and maximize E[B′(a)] (perhaps using something as simple as a monte-carlo simulation that can run for a large number of draws). Then allow A to, as an action or a prelude to acting, ask questions about the value of F(a,r) for specific (a,r) pairs, or for estimates of F(a) (with some model that estimates of F(a) might be incorrect). If A performs correct value of information calculations, it should value asking these questions in the training process and correctly learn values, even if it never experiences a situation where it is caught and punished.