A solution to the problem of preference aggregation
These need seed content, but seem like they can be renormalized.
A way to choose what subset of humanity gets included in CEV that doesn’t include too many superstitious/demented/vengeful/religious nutjobs and land those who implement it in infinite perfect hell.
This may be a problem, but it seems to me that choosing this particular example, and being as confident of it as you appear to be, are symptomatic of an affective death spiral.
All of the above working first time, without testing the entire superintelligence.
The original CEV proposal appears to me to endorse using something like a CFAI-style controlled ascent rather than blind FOOM: “A key point in building a young Friendly AI is that when the chaos in the system grows too high (spread and muddle both add to chaos), the Friendly AI does not guess. The young FAI leaves the problem pending and calls a programmer, or suspends, or undergoes a deterministic controlled shutdown.”
Some quibbles:
These need seed content, but seem like they can be renormalized.
This may be a problem, but it seems to me that choosing this particular example, and being as confident of it as you appear to be, are symptomatic of an affective death spiral.
The original CEV proposal appears to me to endorse using something like a CFAI-style controlled ascent rather than blind FOOM: “A key point in building a young Friendly AI is that when the chaos in the system grows too high (spread and muddle both add to chaos), the Friendly AI does not guess. The young FAI leaves the problem pending and calls a programmer, or suspends, or undergoes a deterministic controlled shutdown.”