I see, that’s interesting. So you are saying that while the problem as scoped in §2 may take a function of arbitrary complexity, there is a constraint in the superintelligence problem I have missed, which is that the complexity of the objective function has certain computational limits.
I think this is only as extreme a problem as you say in a hard takeoff situation. In a slower takeoff situation, inaccuracies due to missing information could be corrected on-line as computational capacity grows. This is roughly business-as-usual for humanity—powerful entities direct the world according to their current best theories; these are sometimes corrected.
It’s interesting that you are arguing that if we knew what information to include in a full specification of humanity, we’d be making substantial progress towards the value problem. In §3.2 I argued that the value problem need only be solved with a subset of the full specification of humanity. The fullness of that specification was desirable just because it makes it less likely that we’ll be missing the parts that are important to value.
If, on the other hand, that you are right and the full specification of humanity is important to solving the value problem—something I’m secretly very sympathetic to—then
(a) we need a supercomputer capable of processing the full specification in order to solve the value problem, so unless there is an iterative solution here the problem is futile and we should just accept that The End Is Nigh, or else try, as I’ve done, to get something Close Enough and hope for slow takeoff, and
(b) the solution to the value problem is going to be somewhere done the computational path from h and is exactly the sort of thing that would be covered in the scope of g*.
It would be a very nice result, I think, if the indirect normativity problem or CEV or whatever could be expressed in terms of the the depth of computational paths from the present state of humanity for precisely this reason. I don’t think I’ve hit that yet exactly but it’s roughly what I’m going for. I think it may hinge on whether the solution to the value problem is something that involves a halting process, or whether really it’s just to ensure the continuation of human life (i.e. as a computational process). In the latter case, I think the solution is very close to what I’ve been proposing.
While I would agree that not all portions of h are needed to solve the value problem, I think it’s very plausible that it would take all of h to be certain that you’d solved the value problem. As in, you couldn’t know that you had included everything important unless you knew that you had everything unimportant as well.
Also, I don’t think I’m sympathetic to the idea that a slow takeoff buys you time to correct things. How would you check for inaccuracies? You don’t have a less-flawed version to compare things to; if you did, you’d be using that version. Some inaccuracies will be large and obvious, but that’s rarely, if ever, going to catch the kinds of errors that lead to hyperexistential catastrophe, and will miss many existential catastrophes.
On the one hand “close enough” is adequate for horseshoes, but probably not good enough for THE FATE OF THE UNIVERSE (grabs algebra nerd by lapels and shakes vigorously)
On the other hand, supergeniuses like Ben Goertzel have suggested that a takeoff might follow a “semi-hard” trajectory. While others have suggested “firm takeoff” (Voss), and even “tumescent takeoff”
Like most of humanity, I’ll start getting concerned when the computers finally beat us in chess. (...off camera whispering)
On a more serious note, the superhuman AI that polices this site just had a most unwelcome message for me: You are trying to eject messages too fast. Rehydrate and try again in 3 minutes. The machines! …they’re takin’ over! They’re already layin’ down the law!
I see, that’s interesting. So you are saying that while the problem as scoped in §2 may take a function of arbitrary complexity, there is a constraint in the superintelligence problem I have missed, which is that the complexity of the objective function has certain computational limits.
I think this is only as extreme a problem as you say in a hard takeoff situation. In a slower takeoff situation, inaccuracies due to missing information could be corrected on-line as computational capacity grows. This is roughly business-as-usual for humanity—powerful entities direct the world according to their current best theories; these are sometimes corrected.
It’s interesting that you are arguing that if we knew what information to include in a full specification of humanity, we’d be making substantial progress towards the value problem. In §3.2 I argued that the value problem need only be solved with a subset of the full specification of humanity. The fullness of that specification was desirable just because it makes it less likely that we’ll be missing the parts that are important to value.
If, on the other hand, that you are right and the full specification of humanity is important to solving the value problem—something I’m secretly very sympathetic to—then
(a) we need a supercomputer capable of processing the full specification in order to solve the value problem, so unless there is an iterative solution here the problem is futile and we should just accept that The End Is Nigh, or else try, as I’ve done, to get something Close Enough and hope for slow takeoff, and
(b) the solution to the value problem is going to be somewhere done the computational path from h and is exactly the sort of thing that would be covered in the scope of g*.
It would be a very nice result, I think, if the indirect normativity problem or CEV or whatever could be expressed in terms of the the depth of computational paths from the present state of humanity for precisely this reason. I don’t think I’ve hit that yet exactly but it’s roughly what I’m going for. I think it may hinge on whether the solution to the value problem is something that involves a halting process, or whether really it’s just to ensure the continuation of human life (i.e. as a computational process). In the latter case, I think the solution is very close to what I’ve been proposing.
While I would agree that not all portions of h are needed to solve the value problem, I think it’s very plausible that it would take all of h to be certain that you’d solved the value problem. As in, you couldn’t know that you had included everything important unless you knew that you had everything unimportant as well.
Also, I don’t think I’m sympathetic to the idea that a slow takeoff buys you time to correct things. How would you check for inaccuracies? You don’t have a less-flawed version to compare things to; if you did, you’d be using that version. Some inaccuracies will be large and obvious, but that’s rarely, if ever, going to catch the kinds of errors that lead to hyperexistential catastrophe, and will miss many existential catastrophes.
On the one hand “close enough” is adequate for horseshoes, but probably not good enough for THE FATE OF THE UNIVERSE (grabs algebra nerd by lapels and shakes vigorously)
On the other hand, supergeniuses like Ben Goertzel have suggested that a takeoff might follow a “semi-hard” trajectory. While others have suggested “firm takeoff” (Voss), and even “tumescent takeoff”
Like most of humanity, I’ll start getting concerned when the computers finally beat us in chess. (...off camera whispering)
On a more serious note, the superhuman AI that polices this site just had a most unwelcome message for me: You are trying to eject messages too fast. Rehydrate and try again in 3 minutes. The machines! …they’re takin’ over! They’re already layin’ down the law!