This is a very good point. I’d sorta defend myself by claiming that “what would you do with the galaxy” (and how you rate that) is unusually determined by memetics compared to what you eat for breakfast (and how you rate that). What you eat for breakfast currently has a way bigger impact on your QOL, but it’s more closely tied to supervisory signals shared across humans.
On the one hand, this means I’m picking on a special case, on the other hand, I think that special case is a pretty good analogy for building AI that becomes way more powerful after training.
This is a very good point. I’d sorta defend myself by claiming that “what would you do with the galaxy” (and how you rate that) is unusually determined by memetics compared to what you eat for breakfast (and how you rate that). What you eat for breakfast currently has a way bigger impact on your QOL, but it’s more closely tied to supervisory signals shared across humans.
On the one hand, this means I’m picking on a special case, on the other hand, I think that special case is a pretty good analogy for building AI that becomes way more powerful after training.