While the model is interesting, it is almost irremediably ruined by this line: “since by definition P(A) = Ta/n”, which substantially conflates probability with frequency. From this point of view, the conclusion:
It is clear from Pearl’s work that probability theory provides normatively correct
rules which an AI system must use to reason optimally about uncertain events.
It is equally clear that AI systems (like all other physical systems) are unavoid-
ably subject to a certain degree of random variation and noise in their internal
workings. As we have seen, this random variation does not produce a pattern of
reasoning in which probability estimates vary randomly around the correct value;
instead,it produces systematic biases that push probability estimates in certain
directions and so will produce conservatism, subadditivity, and the conjunction
and disjunction errors in AI reasoning.
does not follow (because the estimation is made by the prior and not by counting). BUT the issue of noise in AI is interesting per se: if we a have stable self-improving friendly AI, could it faultily copies/update itself into an unfriendly version?
Repeated self modification is problematic, because it represents a product of series (though possibly a convergent one, if the AI gets better at maintaining its utility function / rationality with each modification) -- naively, because no projection about the future can have a confidence of 1, there is some chance that each change to the AI will be negative-value.
Right, it’s not only noise that can alter the value of copying/propagating source code, since we can at least imagine that future improvements will be also more stable in this regard: there’s also the possibility that the moral landscape of an AI could be fractal, so that even a small modification might turn friendliness into unfriendliness/hostility.
While the model is interesting, it is almost irremediably ruined by this line: “since by definition P(A) = Ta/n”, which substantially conflates probability with frequency.
Think of P(A) merely as the output of a noiseless version of the same algorithm. Obviously this depends on the prior, but I think this one is not unreasonable in most cases.
Think of P(A) merely as the output of a noiseless version of the same algorithm.
because P(A) is the noiseless parameter. Anyway, the entire paper is based on the counting algorithm to establish that random noise can give rise to structured bias, and that this is a problem for a bayesian AI. But while the mechanism can be an interesting and maybe even correct way to unify the mentioned bias in human mind, it can hardly be posed as a problem for such an artificial intelligence.
A counting algorithm for establishing probabilities basically denies everything bayesian update is designed for (the most trivial example: extraction from a finite urn).
Well, yes, the prior that yields counting algorithms is not universal. But in many cases it’s good idea! And if you decide to use, for example, some rule-of-succession style modifications, the same situation appears.
In the case of a finite urn, you might see different biases (or none at all if your algorithm stubbornly refuses to update because you chose a silly prior).
While the model is interesting, it is almost irremediably ruined by this line: “since by definition P(A) = Ta/n”, which substantially conflates probability with frequency. From this point of view, the conclusion:
does not follow (because the estimation is made by the prior and not by counting).
BUT the issue of noise in AI is interesting per se: if we a have stable self-improving friendly AI, could it faultily copies/update itself into an unfriendly version?
Repeated self modification is problematic, because it represents a product of series (though possibly a convergent one, if the AI gets better at maintaining its utility function / rationality with each modification) -- naively, because no projection about the future can have a confidence of 1, there is some chance that each change to the AI will be negative-value.
Right, it’s not only noise that can alter the value of copying/propagating source code, since we can at least imagine that future improvements will be also more stable in this regard: there’s also the possibility that the moral landscape of an AI could be fractal, so that even a small modification might turn friendliness into unfriendliness/hostility.
Think of P(A) merely as the output of a noiseless version of the same algorithm. Obviously this depends on the prior, but I think this one is not unreasonable in most cases.
I’m not sure I’ve understood the sentence
because P(A) is the noiseless parameter.
Anyway, the entire paper is based on the counting algorithm to establish that random noise can give rise to structured bias, and that this is a problem for a bayesian AI.
But while the mechanism can be an interesting and maybe even correct way to unify the mentioned bias in human mind, it can hardly be posed as a problem for such an artificial intelligence. A counting algorithm for establishing probabilities basically denies everything bayesian update is designed for (the most trivial example: extraction from a finite urn).
Well, yes, the prior that yields counting algorithms is not universal. But in many cases it’s good idea! And if you decide to use, for example, some rule-of-succession style modifications, the same situation appears.
In the case of a finite urn, you might see different biases (or none at all if your algorithm stubbornly refuses to update because you chose a silly prior).