What we actually said about Nanny AI is that it may be FAI-complete, and that it is thus really full-blown Friendly AI even though when Ben Goertzel talks about it in English it might sound like not-FAI.
It’s worth distinguishing between two claims: (1) If you can build Nanny AI, you can build FAI and (2) If you’ve built Nanny AI, you’ve built FAI.
(2) is compatible with and in fact entails (1). (1) does not, however, entail (2). In fact, (1) seems pointless to say if you also believe (2) because the entailment is so obvious. Because your paper explicitly asserts (1), I inferred you did not believe (2). Your comment seems to explicitly assert both (1) and (2), making me somewhat confused about what your view is.
EDIT: Part of what is confusing about your comment is that it seems to say “(1), thus (2)” which does not follow. Also, to save people the trouble of looking up the relevant section of the paper, the term “FAI complete” is explained in this way: “That is, in order to build Nanny AI, you may need to solve all the problems required to build full-blown Friendly AI.”
Here’s an example of why “Friendly AI may be incoherent and impossible.” Suppose that the only way to have a superintelligent AI beneficial to humanity is something like CEV, but nobody is ever able to make sense of the idea of combining and extrapolating human values. “Can we extrapolate the coherent convergence of human values?” sounds suspiciously like a Wrong Question. Maybe there’s a Right Question somewhere near that space, and we’ll be able to find the answer, but right now we are fundamentally philosophically confused about what these English words could usefully mean.
I’m not sure I understand what you mean by this either. Maybe, going off the “beneficial to humanity” definition of FAI, you mean to say that it’s possible that right now, we are fundamentally philosophically confused about what “beneficial to humanity” might mean?
It’s worth distinguishing between two claims: (1) If you can build Nanny AI, you can build FAI and (2) If you’ve built Nanny AI, you’ve built FAI.
(2) is compatible with and in fact entails (1). (1) does not, however, entail (2). In fact, (1) seems pointless to say if you also believe (2) because the entailment is so obvious. Because your paper explicitly asserts (1), I inferred you did not believe (2). Your comment seems to explicitly assert both (1) and (2), making me somewhat confused about what your view is.
EDIT: Part of what is confusing about your comment is that it seems to say “(1), thus (2)” which does not follow. Also, to save people the trouble of looking up the relevant section of the paper, the term “FAI complete” is explained in this way: “That is, in order to build Nanny AI, you may need to solve all the problems required to build full-blown Friendly AI.”
I’m not sure I understand what you mean by this either. Maybe, going off the “beneficial to humanity” definition of FAI, you mean to say that it’s possible that right now, we are fundamentally philosophically confused about what “beneficial to humanity” might mean?