Some language to simplify some of the places where the debate got stuck.
Is-Ought
Analyzing how to preserve or act on preferences is a coherent thing to do, and it’s possible to do so without assuming a one true universal morality. Assume a preference ordering, and now you’re in the land of is, not ought, where there can be a correct answer (highest expected value).
Is There One Reality?
Let existence be defined to mean everything, all the math, all the indexical facts. “Ah, but you left out-” Nope, throw that in too. Everything. Existence is a pretty handy word for that; let’s reserve it for that purpose.As for any points about how our observations are compatible with multiple implementations: we’ve already lumped those into our description of a “unique reality”.
Noise, In MY Conformal Geometry?!
Noise is noise with respect to a prediction, and so is coherent to discuss. One can abstract away from certain details for the purpose of making a specific prediction; call the stuff that can be abstracted away from noise relative to that prediction.
Decoupled Outer And Inner Optimization Targets
Inclusive genetic fitness led to weirdos that like ice cream, but predictive loss may be a purer target than IGF. If we don’t press down on that insanely hard, it’s quite plausible that we get all the way to significantly superhuman generality without any unfortunate parallels to that issue. If you work at a frontier AI lab, probably don’t build agents in stupid ways or enable their being built too quickly; that seems like the greatest liability at present.
Yudkowsky + Wolfram Debate
Some language to simplify some of the places where the debate got stuck.
Is-Ought
Analyzing how to preserve or act on preferences is a coherent thing to do, and it’s possible to do so without assuming a one true universal morality. Assume a preference ordering, and now you’re in the land of is, not ought, where there can be a correct answer (highest expected value).
Is There One Reality?
Let existence be defined to mean everything, all the math, all the indexical facts. “Ah, but you left out-” Nope, throw that in too. Everything. Existence is a pretty handy word for that; let’s reserve it for that purpose. As for any points about how our observations are compatible with multiple implementations: we’ve already lumped those into our description of a “unique reality”.
Noise, In MY Conformal Geometry?!
Noise is noise with respect to a prediction, and so is coherent to discuss. One can abstract away from certain details for the purpose of making a specific prediction; call the stuff that can be abstracted away from noise relative to that prediction.
Decoupled Outer And Inner Optimization Targets
Inclusive genetic fitness led to weirdos that like ice cream, but predictive loss may be a purer target than IGF. If we don’t press down on that insanely hard, it’s quite plausible that we get all the way to significantly superhuman generality without any unfortunate parallels to that issue. If you work at a frontier AI lab, probably don’t build agents in stupid ways or enable their being built too quickly; that seems like the greatest liability at present.