Comment by Eliezer Yudkowsky Jun 18, 2007 12:52 pm: I furthermore agree that it is not the most elegant idea I have ever had, but then it is trying to solve what appears to be an inherently inelegant problem.
I strongly agree with this: the problem that CEV is the solution to is urgent but it isn’t elegant. Absolutes like “There isn’t a beliefs/desires separation” are unhelpful when solving such inelegant but important problems. There is, in any given person, some kind of separation, and in some people that separation is sufficiently strong that there is a fairly clear and unique way to help them.
I strongly agree with this: the problem that CEV is the solution to is urgent but it isn’t elegant. Absolutes like “There isn’t a beliefs/desires separation” are unhelpful when solving such inelegant but important problems.
One lesson of reductionism and success of simple-laws-based science and technology is that for the real-world systems, there might be no simple way of describing them, but there could be a simple way of manipulating their data-rich descriptions. (What’s the yield strength of a car? -- Wrong question!) Given a gigabyte’s worth of problem statement and the right simple formula, you could get an answer to your query. There is a weak analogy with misapplication of Occam’s razor where one tries to reduce the amount of stuff rather than the amount of detail in the ways of thinking about this stuff.
In the case of beliefs/desires separation, you are looking for a simple problem statement, for a separation in the data describing the person itself. But what you should be looking for is a simple way of implementing the make-smarter-and-better extrapolation on a given pile of data. The beliefs/desires separation, if it’s ever going to be made precise, is going to reside in the structure of this simple transformation, not in the people themselves.
Of course, it would be nice if we could find a general “make-smarter-and-better extrapolation on a given pile of data” algorithm.
But on the other hand, a set of special cases to deal with merely human minds might be the way forward. Even medieval monks had a collection of empirically validated medical practices that worked to an extent, e.g. herbal medicine, but they had no unified theory. Really there is no “unified theory” for healing someone’s body: there are lots of ideas and techniques, from surgery to biochemistry to germ theory. I think that this CEV problem may well turn out to be rather like medicine. Of course, it could look more like wing design, where there is really just one fundamental set of laws, and all else is approximation.
Furthermore, from the CEV thread on SIAI blog:
I strongly agree with this: the problem that CEV is the solution to is urgent but it isn’t elegant. Absolutes like “There isn’t a beliefs/desires separation” are unhelpful when solving such inelegant but important problems. There is, in any given person, some kind of separation, and in some people that separation is sufficiently strong that there is a fairly clear and unique way to help them.
One lesson of reductionism and success of simple-laws-based science and technology is that for the real-world systems, there might be no simple way of describing them, but there could be a simple way of manipulating their data-rich descriptions. (What’s the yield strength of a car? -- Wrong question!) Given a gigabyte’s worth of problem statement and the right simple formula, you could get an answer to your query. There is a weak analogy with misapplication of Occam’s razor where one tries to reduce the amount of stuff rather than the amount of detail in the ways of thinking about this stuff.
In the case of beliefs/desires separation, you are looking for a simple problem statement, for a separation in the data describing the person itself. But what you should be looking for is a simple way of implementing the make-smarter-and-better extrapolation on a given pile of data. The beliefs/desires separation, if it’s ever going to be made precise, is going to reside in the structure of this simple transformation, not in the people themselves.
This is a good point.
Of course, it would be nice if we could find a general “make-smarter-and-better extrapolation on a given pile of data” algorithm.
But on the other hand, a set of special cases to deal with merely human minds might be the way forward. Even medieval monks had a collection of empirically validated medical practices that worked to an extent, e.g. herbal medicine, but they had no unified theory. Really there is no “unified theory” for healing someone’s body: there are lots of ideas and techniques, from surgery to biochemistry to germ theory. I think that this CEV problem may well turn out to be rather like medicine. Of course, it could look more like wing design, where there is really just one fundamental set of laws, and all else is approximation.