A long time ago you described what you perceived as the difficulties for FAI:
Solving the technical problems required to maintain a well-specified abstract invariant in a self-modifying goal system. (Interestingly, this problem is relatively straightforward from a theoretical standpoint.)
Choosing something nice to do with the AI. This is about midway in theoretical hairiness between problems 1 and 3.
Designing a framework for an abstract invariant that doesn’t automatically wipe out the human species. This is the hard part.
I know that was a long time ago, but people here still link to it, presumably because they don’t know of any more up to date statement with similar content. Hopefully you can see why I was confused about which part of this problem was supposed to be hard. I now see that I probably misinterpreted it, but the examples which come directly afterwards reaffirm my incorrect interpretation.
So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don’t understand, already requires solving the hard part?
Either my views changed since that time, or what I was trying to communicate by it was that 3 was the most inscrutable part of the problem to people who try to tackle it, rather than that it was the blocker problem. 1 is the blocker problem, I think. I probably realize that now to a greater degree than I did at that time, and probably also made more progress on 3 relative to 1, but I don’t know how much my opinions actually changed (there’s some well-known biases about that).
So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don’t understand, already requires solving the hard part?
Current estimate says yes. There would still be an inscrutable problem to solve too, but I don’t think it would have quite the same impenetrability about it.
Would it be fair to say that even developing a formalism which is capable of precisely expressing the idea that something is a good paperclipper is significantly beyond current techniques, and that substantial progress on this problem probably represents substantial progress towards FAI?
A long time ago you described what you perceived as the difficulties for FAI:
I know that was a long time ago, but people here still link to it, presumably because they don’t know of any more up to date statement with similar content. Hopefully you can see why I was confused about which part of this problem was supposed to be hard. I now see that I probably misinterpreted it, but the examples which come directly afterwards reaffirm my incorrect interpretation.
So would it be fair to say that figuring out how to build a good paperclipper, as opposed to a process that does something we don’t understand, already requires solving the hard part?
Either my views changed since that time, or what I was trying to communicate by it was that 3 was the most inscrutable part of the problem to people who try to tackle it, rather than that it was the blocker problem. 1 is the blocker problem, I think. I probably realize that now to a greater degree than I did at that time, and probably also made more progress on 3 relative to 1, but I don’t know how much my opinions actually changed (there’s some well-known biases about that).
Current estimate says yes. There would still be an inscrutable problem to solve too, but I don’t think it would have quite the same impenetrability about it.
Would it be fair to say that even developing a formalism which is capable of precisely expressing the idea that something is a good paperclipper is significantly beyond current techniques, and that substantial progress on this problem probably represents substantial progress towards FAI?
Yes.