I was led to this comment by your request for assistance here. You seem to be asking about the relationship between our intuitive values and our attempts to systematize those values rationally. To what extent should we let our intuitions guide the construction of our theories? To what extent should we allow our theories to reform and override our intuitions?
My own take on this is that there can be levels (degrees of stability) of equilibria. For example, the foundational idea of utility and expected utility maximization (as axiomatized by Savage or Aumann) strikes me as pretty solid. But when you add on additional superstructure (such as interpersonal comparison of utilities, or universalist, consequentialist ethics) it becomes more and more difficult to bring the axiomatic structure into equilibrium with the raw intuitions of everyone.
I was led to this comment by your request for assistance here. You seem to be asking about the relationship between our intuitive values and our attempts to systematize those values rationally. To what extent should we let our intuitions guide the construction of our theories? To what extent should we allow our theories to reform and override our intuitions?
This is the very important and difficult issue of reflective equilibrium as expounded upon by Goodman and Rawls, to say nothing of Yudkowsky. I hope the links are helpful.
My own take on this is that there can be levels (degrees of stability) of equilibria. For example, the foundational idea of utility and expected utility maximization (as axiomatized by Savage or Aumann) strikes me as pretty solid. But when you add on additional superstructure (such as interpersonal comparison of utilities, or universalist, consequentialist ethics) it becomes more and more difficult to bring the axiomatic structure into equilibrium with the raw intuitions of everyone.