I strongly agree. I think there are vastly better sources of evidence on how inner goals relate to outer selection criteria than “evolution → human values”, and that most of those better sources of evidence paint a much more optimistic picture of how things are likely to go with AI. I think there are lots of reasons to be skeptical of using “evolution → human values” as an informative reference class for AIs, some of which I’ve described in my reply to Ben’s comment.
I strongly agree. I think there are vastly better sources of evidence on how inner goals relate to outer selection criteria than “evolution → human values”, and that most of those better sources of evidence paint a much more optimistic picture of how things are likely to go with AI. I think there are lots of reasons to be skeptical of using “evolution → human values” as an informative reference class for AIs, some of which I’ve described in my reply to Ben’s comment.