why do we even believe that human values are good?
Because they constitute, by definition, our goodness criterion? It’s not like we have two separate modules—one for “human values”, and one for “is this good?”. (ETA or are you pointing out how our values might shift over time as we reflect on our meta-ethics?)
Perhaps the typical human behaviour amplified by possibilities of a super-intelligence would actually destroy the universe.
If I understand correctly, this is “are human behaviors catastrophic?”—not “are human values catastrophic?”.
On the latter: yes, this is part of the question but not the whole question. See addendum.
On the former: technically not true. If we take “human values” as “values averaged between different humans” (not necessarily by arithmetical mean, of course) they may be vastly different from “is this good from my viewpoint?”.
On the bracketed part: yeah, that too. And our current morals may not be that good judging by our metamorals.
Again, I want to underscore that I mention this as a theoretical possibility not so improbable as to make it not worth considering—not as an unavoidable fact.
Because they constitute, by definition, our goodness criterion? It’s not like we have two separate modules—one for “human values”, and one for “is this good?”. (ETA or are you pointing out how our values might shift over time as we reflect on our meta-ethics?)
If I understand correctly, this is “are human behaviors catastrophic?”—not “are human values catastrophic?”.
On the latter: yes, this is part of the question but not the whole question. See addendum.
On the former: technically not true. If we take “human values” as “values averaged between different humans” (not necessarily by arithmetical mean, of course) they may be vastly different from “is this good from my viewpoint?”.
On the bracketed part: yeah, that too. And our current morals may not be that good judging by our metamorals.
Again, I want to underscore that I mention this as a theoretical possibility not so improbable as to make it not worth considering—not as an unavoidable fact.