A computation hazard is a large negative consequence that may arise merely from vast amounts of computation, such as in a future supercomputer.
Are you including anything in this beyond the hazards of accidental simulation? It sounds to me like you aren’t.
computations that run most algorithms, and computations that are particularly likely to run algorithms that are computation hazards.
I can’t imagine any computation hazard arising from a computer that runs most algorithms, ie Solomonoff induction, actually being a hazard on any size computer and timescale that is commensurate with, say, turning the solar system into computronium and burning all the energy of the Sun. I don’t think the selection power present in “run most algorithms” actually simulates anything sentient for a sufficient length of time for me to care. Now, a computer program that selectively simulates algorithms likely to be sentient might be a different matter, but then you’re having a discussion about the ethics of simulation, not accidental computation hazards or “most algorithms”. In other words, I think the claim expressed above stems from a fundamental misunderstanding of exactly how strong the term “uncomputably complex” is. I suspect you have not yet truly understood the growth curve of the busy beaver function.
Imagine a supercomputer’s power is being tested on a simple game, like chess or Go
I’ve written programs to play games. I cannot possibly see where such a hazard comes from. Have you looked at the proofs that such games can be Turing complete? That is, not just noted their existence, but actually examined the proofs. The level of machine that can be built on a standard size Go or Chess board is trivial. I think the accidental simulation hazard of such a computer is far less than that from noise errors on 8-bit microcontrollers running industrial control programs. Turing completeness relies on assumptions of infinte or very large (for merely approximate completeness) board sizes. And once you let the board size scale like that, you either have something that is exceedingly selective about which paths it explores (and therefore isn’t “accidental” in the sense of running most algorithms), or doesn’t simulate anything for long enough to exhibit sentient behavior, or even to have a very good chance of producing something with anything close to the potential of an unhatched ant.
In other words, I fail to see why this line of inquiry is deserving of anything more than an off-hand assessment of “not a hazard”, let alone the first quarter of the post.
The remainder of the post seems to me to do a poor job of separating out the hazards of simulating people inside an AGI, vs the hazards presented by simply having the AGI. The latter hazards are thoroughly discussed elsewhere, and I don’t find that this discussion adds anything. By mixing those hazards in with computation hazards, you make it very unclear whether those are intended to be included in your definition (which I originally thought excluded the results of the computation), and you make the rest of the post much less clear and much less useful.
I can’t imagine any computation hazard arising from a computer that runs most algorithms actually being a hazard on any size computer and timescale that is commensurate with, say, turning the solar system into computronium and burning all the energy of the Sun.
I’m not sure about this. It’s difficult for me to do the order-of-magnitude calculation, because I don’t know how many flops we can gets from using the solar system as substrate, and because I have no idea how small a program can be before it has moral status.
I suspect you have not yet truly understood the growth curve of the busy beaver function.
From my understanding, the busy beaver function is about the maximum number of steps an n-state Turing machine can take before you know it won’t halt. This doesn’t seem to have anything to do with the probability of simulating people; both halting and non-halting programs could have moral status.
Overall, I agree that the “runs most algorithms” category is not realistic. It’s mostly just for philosophical interest, and completeness.
The remainder of the post seems to me to do a poor job of separating out the hazards of simulating people inside an AGI, vs the hazards presented by simply having the AGI.
Yeah, I’m worried that the concept of “computational hazard” as I’ve used it here isn’t very useful, or rather, doesn’t carve reality at its joints.
The latter hazards are thoroughly discussed elsewhere, and I don’t find that this discussion adds anything.
I wasn’t really trying to add original material to the discussion. This post was supposed to be a summary of ideas already discovered on LW, and I wanted to know if I was missing any, or if this was a good summary of the ideas.
The busy beaver function is the lower bound on how fast a function has to grow before it counts as “uncomputable”. When people say that the Solomonoff induction or Kolmogorov complexity is uncomputable, that’s what they mean. When you say you’re having trouble with an order of magnitude estimate, I have to wonder whether you attempted to come up with an order of magnitude estimate for the exponent. For example, I’m pretty sure that no conceivable amount of computronium will get to 10^10000 territory, and that I can safely ignore ethical problems that only arise once we’re in that territory. I’m rather skeptical of the idea that we could even get to 10^1000 territory by brute force, even if we turned the galaxy into computronium.
Therefore, when you say you’re concerned about computing Solmonoff induction or Kolmogorov priors, I’m left wondering whether you’re worried about 5- or 6-state machines, because there is no conceivable way the program would ever get to the 7-state machines.
And you also can’t simply dismiss the longest-running small Turing machines, in my opinion. If there exists an accidental computational hazard worth worrying about, I would assume it’s in the form of something analogous to “run Conway’s life until the board is empty”—in other words, precisely a small Turing machine that is likely to run a very long time before halting, and whose halting status is difficult to determine.
I wasn’t really trying to add original material to the discussion. This post was supposed to be a summary of ideas already discovered on LW, and I wanted to know if I was missing any, or if this was a good summary of the ideas.
I think the summary is overly confused. If the accidental hazards are of purely philosophical interest, they shouldn’t occupy such a large fraction of the post, and shouldn’t be the first thing discussed. I almost didn’t bother reading the rest.
I think the non-simulation hazards have little in common with the simulation hazards from an ethics standpoint, and less from a practical FAI programming view. If you’re having trouble separating them, then I would take that as a strong clue that you’ve chosen the wrong points to carve at.
Thanks for all your feedback!
You’re welcome! There’s some interesting stuff here, though I’m skeptical that there’s much of interest in anything except the intentional, directed simulations questions.
The busy beaver function is the lower bound on how fast a function has to grow before it counts as “uncomputable”.
Ah, I see. That’s not the definition, but it is a fact about it. Although I think you might be confused about the definition of “uncomputable”. It doesn’t have to do with functions growing. It’s just a separate, awesome fact that all computable functions grow slower than the (uncomputable) busy beaver function. There are many uncomputable functions that grow slower than computable functions.
Hmm. Seems I’ve confused some stuff. What you’ve said is correct, but I think my point is still valid.
The Kolmogorov prior is uncomputable. The time required to approximate it by simulating Turing machines to size n (in other words, the naive brute force approach in question) grows at busy beaver speeds, because it requires simulating all non-halting machines within the relevant size, and there is no generalizable way to shortcut those simulations.
Now, there are ways to approximate Solomonoff / Kolmogorov / AIXI with sane computational limits. However, once you start doing that, you can no longer claim that you run the risk of “running most algorithms”, at least by the mathematical definition of “most” that I assumed you were using. Or rather, you will eventually, as you wait for infinite computation to be expended on the problem. But I’d say that either you’re being selective to a degree that the hazard lies in selecting for simulation, rather than in running “most” algorithms, or you’re in no more danger than in the brute force case. This is related to the point I made above: I think the accidentally dangerous portion of the search space is likely to lie in the algorithms whose runtime is long relative to their complexity, which are precisely the ones that will be avoided by approximations intended to be tractable.
Are you including anything in this beyond the hazards of accidental simulation? It sounds to me like you aren’t.
I can’t imagine any computation hazard arising from a computer that runs most algorithms, ie Solomonoff induction, actually being a hazard on any size computer and timescale that is commensurate with, say, turning the solar system into computronium and burning all the energy of the Sun. I don’t think the selection power present in “run most algorithms” actually simulates anything sentient for a sufficient length of time for me to care. Now, a computer program that selectively simulates algorithms likely to be sentient might be a different matter, but then you’re having a discussion about the ethics of simulation, not accidental computation hazards or “most algorithms”. In other words, I think the claim expressed above stems from a fundamental misunderstanding of exactly how strong the term “uncomputably complex” is. I suspect you have not yet truly understood the growth curve of the busy beaver function.
I’ve written programs to play games. I cannot possibly see where such a hazard comes from. Have you looked at the proofs that such games can be Turing complete? That is, not just noted their existence, but actually examined the proofs. The level of machine that can be built on a standard size Go or Chess board is trivial. I think the accidental simulation hazard of such a computer is far less than that from noise errors on 8-bit microcontrollers running industrial control programs. Turing completeness relies on assumptions of infinte or very large (for merely approximate completeness) board sizes. And once you let the board size scale like that, you either have something that is exceedingly selective about which paths it explores (and therefore isn’t “accidental” in the sense of running most algorithms), or doesn’t simulate anything for long enough to exhibit sentient behavior, or even to have a very good chance of producing something with anything close to the potential of an unhatched ant.
In other words, I fail to see why this line of inquiry is deserving of anything more than an off-hand assessment of “not a hazard”, let alone the first quarter of the post.
The remainder of the post seems to me to do a poor job of separating out the hazards of simulating people inside an AGI, vs the hazards presented by simply having the AGI. The latter hazards are thoroughly discussed elsewhere, and I don’t find that this discussion adds anything. By mixing those hazards in with computation hazards, you make it very unclear whether those are intended to be included in your definition (which I originally thought excluded the results of the computation), and you make the rest of the post much less clear and much less useful.
I’m not sure about this. It’s difficult for me to do the order-of-magnitude calculation, because I don’t know how many flops we can gets from using the solar system as substrate, and because I have no idea how small a program can be before it has moral status.
From my understanding, the busy beaver function is about the maximum number of steps an n-state Turing machine can take before you know it won’t halt. This doesn’t seem to have anything to do with the probability of simulating people; both halting and non-halting programs could have moral status.
Overall, I agree that the “runs most algorithms” category is not realistic. It’s mostly just for philosophical interest, and completeness.
Yeah, I’m worried that the concept of “computational hazard” as I’ve used it here isn’t very useful, or rather, doesn’t carve reality at its joints.
I wasn’t really trying to add original material to the discussion. This post was supposed to be a summary of ideas already discovered on LW, and I wanted to know if I was missing any, or if this was a good summary of the ideas.
Thanks for all your feedback!
The busy beaver function is the lower bound on how fast a function has to grow before it counts as “uncomputable”. When people say that the Solomonoff induction or Kolmogorov complexity is uncomputable, that’s what they mean. When you say you’re having trouble with an order of magnitude estimate, I have to wonder whether you attempted to come up with an order of magnitude estimate for the exponent. For example, I’m pretty sure that no conceivable amount of computronium will get to 10^10000 territory, and that I can safely ignore ethical problems that only arise once we’re in that territory. I’m rather skeptical of the idea that we could even get to 10^1000 territory by brute force, even if we turned the galaxy into computronium.
Therefore, when you say you’re concerned about computing Solmonoff induction or Kolmogorov priors, I’m left wondering whether you’re worried about 5- or 6-state machines, because there is no conceivable way the program would ever get to the 7-state machines.
And you also can’t simply dismiss the longest-running small Turing machines, in my opinion. If there exists an accidental computational hazard worth worrying about, I would assume it’s in the form of something analogous to “run Conway’s life until the board is empty”—in other words, precisely a small Turing machine that is likely to run a very long time before halting, and whose halting status is difficult to determine.
I think the summary is overly confused. If the accidental hazards are of purely philosophical interest, they shouldn’t occupy such a large fraction of the post, and shouldn’t be the first thing discussed. I almost didn’t bother reading the rest.
I think the non-simulation hazards have little in common with the simulation hazards from an ethics standpoint, and less from a practical FAI programming view. If you’re having trouble separating them, then I would take that as a strong clue that you’ve chosen the wrong points to carve at.
You’re welcome! There’s some interesting stuff here, though I’m skeptical that there’s much of interest in anything except the intentional, directed simulations questions.
Ah, I see. That’s not the definition, but it is a fact about it. Although I think you might be confused about the definition of “uncomputable”. It doesn’t have to do with functions growing. It’s just a separate, awesome fact that all computable functions grow slower than the (uncomputable) busy beaver function. There are many uncomputable functions that grow slower than computable functions.
Hmm. Seems I’ve confused some stuff. What you’ve said is correct, but I think my point is still valid.
The Kolmogorov prior is uncomputable. The time required to approximate it by simulating Turing machines to size n (in other words, the naive brute force approach in question) grows at busy beaver speeds, because it requires simulating all non-halting machines within the relevant size, and there is no generalizable way to shortcut those simulations.
Now, there are ways to approximate Solomonoff / Kolmogorov / AIXI with sane computational limits. However, once you start doing that, you can no longer claim that you run the risk of “running most algorithms”, at least by the mathematical definition of “most” that I assumed you were using. Or rather, you will eventually, as you wait for infinite computation to be expended on the problem. But I’d say that either you’re being selective to a degree that the hazard lies in selecting for simulation, rather than in running “most” algorithms, or you’re in no more danger than in the brute force case. This is related to the point I made above: I think the accidentally dangerous portion of the search space is likely to lie in the algorithms whose runtime is long relative to their complexity, which are precisely the ones that will be avoided by approximations intended to be tractable.