I don’t disagree with what you write here, but I think if you say something like “You use the art of rationality to determine what you value” you’ll raise the expectation that there is already an art of rationality that can be used to determine what someone values, and then people will be disappointed when they look closer and find out that’s not the case.
Ah, I see your point. So the less misleading thing to say might be something roughly like: “We don’t yet know how to find or reason about our values, but we have notions of where we might start, and we can expect that whatever methods do end up making headway are going to have to be non-stupid in at least as many ways as our existing methods of solving hard problems are non-stupid.”
I don’t disagree with what you write here, but I think if you say something like “You use the art of rationality to determine what you value” you’ll raise the expectation that there is already an art of rationality that can be used to determine what someone values, and then people will be disappointed when they look closer and find out that’s not the case.
Ah, I see your point. So the less misleading thing to say might be something roughly like: “We don’t yet know how to find or reason about our values, but we have notions of where we might start, and we can expect that whatever methods do end up making headway are going to have to be non-stupid in at least as many ways as our existing methods of solving hard problems are non-stupid.”