It seems like the important thing is how bounded the task is.
For example, in the case of Go, if you just kept training AlphaZero, would you expect it to eventually decide that it needs to break out into the physical world to get more computing power?
It seems to me that it could get to be ultra-super-human at Go without that happening. (Even if there is some theoretical threshold where, with enough computation, it couldn’t help but stumble upon a sequence of moves that causes the program to crash. It seems to me that you’re likely to get crashing behavior long before you get hack-out-of-the-vm behavior, and the threshold for either may be too high to matter.)
If that’s true for Go, then the questions are:
1. How much less bounded of a task can you train a system to do while maintaining the focused-on-the-task property?
and
2. How general of a system can you make by composing such focused systems together?
It seems like the important thing is how bounded the task is.
For example, in the case of Go, if you just kept training AlphaZero, would you expect it to eventually decide that it needs to break out into the physical world to get more computing power?
It seems to me that it could get to be ultra-super-human at Go without that happening. (Even if there is some theoretical threshold where, with enough computation, it couldn’t help but stumble upon a sequence of moves that causes the program to crash. It seems to me that you’re likely to get crashing behavior long before you get hack-out-of-the-vm behavior, and the threshold for either may be too high to matter.)
If that’s true for Go, then the questions are:
1. How much less bounded of a task can you train a system to do while maintaining the focused-on-the-task property?
and
2. How general of a system can you make by composing such focused systems together?