When I try to imagine a safe oracle, what I have in mind is something much more passive and limited than what you describe.
Consider a system that simply accepts input information and integrates it into a huge probability distribution that it maintains. We can then query the oracle by simply examining this distribution. For example, we could use this distribution to estimate the probability of some event in the future conditional on some other event etc. There is nothing in the system that would cause it to “try” to get information, or develop sub-goals, or what ever. It’s very basic in terms of its operation. Nevertheless, if the computer was crazy big enough and feed enough data about the world, it could be quite a powerful device for people wanting to make decisions.
It seems to be that the dangerous part here is what the people then do with it, rather than the machine itself. For example, people looking at the outputs might realise that if they just modified the machine in some small way to collect its own data then its predictions should be much better… and before you know it the machine is no longer such a passive machine.
Perhaps when Bostrom thinks about potentially “safe” oracles, he’s also thinking of something much more limited than what you’re attacking in this post.
Eli:
When I try to imagine a safe oracle, what I have in mind is something much more passive and limited than what you describe.
Consider a system that simply accepts input information and integrates it into a huge probability distribution that it maintains. We can then query the oracle by simply examining this distribution. For example, we could use this distribution to estimate the probability of some event in the future conditional on some other event etc. There is nothing in the system that would cause it to “try” to get information, or develop sub-goals, or what ever. It’s very basic in terms of its operation. Nevertheless, if the computer was crazy big enough and feed enough data about the world, it could be quite a powerful device for people wanting to make decisions.
It seems to be that the dangerous part here is what the people then do with it, rather than the machine itself. For example, people looking at the outputs might realise that if they just modified the machine in some small way to collect its own data then its predictions should be much better… and before you know it the machine is no longer such a passive machine.
Perhaps when Bostrom thinks about potentially “safe” oracles, he’s also thinking of something much more limited than what you’re attacking in this post.