Why would such a system have a goal to acquire more resources? You put some data in, run the algorithm that updates the probability distribution, and it then halts. I would not say that it has “goals”, or a “mind”. It doesn’t “want” to compute more accurately, or want anything else, for that matter. It’s just a really fancy version of GZIP (recall that compression = prediction) running on a thought-experiment-crazy-sized computer and quantities of data.
I accept that such a machine would be dangerous once you put people into the equation, but the machine in itself doesn’t seem dangerous to me. (If you can convince me otherwise… that would be interesting)
Vladimir:
Why would such a system have a goal to acquire more resources? You put some data in, run the algorithm that updates the probability distribution, and it then halts. I would not say that it has “goals”, or a “mind”. It doesn’t “want” to compute more accurately, or want anything else, for that matter. It’s just a really fancy version of GZIP (recall that compression = prediction) running on a thought-experiment-crazy-sized computer and quantities of data.
I accept that such a machine would be dangerous once you put people into the equation, but the machine in itself doesn’t seem dangerous to me. (If you can convince me otherwise… that would be interesting)