Well ok, agreed, but even if we were Cartesian, we would still have questions about what is the right way to link up our machines with this place where agentiness is coming from, how we discern whether we are in fact Cartesian or embedded, and so on down to the problem of the criterion as you described it.
One common response to any such difficult philosophical problems seems to be to just build AI that uses some form of indirect normativity such as CEV or HCH or AI debate to work out what wise humans would do about those philosophical problems. But I don’t think it’s so easy to sidestep the problem of the criterion.
Oh, I don’t think those things exactly sidestep the problem of the criterion so much as commit to a response to it without necessarily realizing that’s what they’re doing. All of them sort of punt on it by saying “let humans figure out that part”, which at the end of the day is what any solution is going to do because we’re the ones trying to build the AI and making the decisions, but we can be more or less deliberate about how we do this part.
Well ok, agreed, but even if we were Cartesian, we would still have questions about what is the right way to link up our machines with this place where agentiness is coming from, how we discern whether we are in fact Cartesian or embedded, and so on down to the problem of the criterion as you described it.
One common response to any such difficult philosophical problems seems to be to just build AI that uses some form of indirect normativity such as CEV or HCH or AI debate to work out what wise humans would do about those philosophical problems. But I don’t think it’s so easy to sidestep the problem of the criterion.
Oh, I don’t think those things exactly sidestep the problem of the criterion so much as commit to a response to it without necessarily realizing that’s what they’re doing. All of them sort of punt on it by saying “let humans figure out that part”, which at the end of the day is what any solution is going to do because we’re the ones trying to build the AI and making the decisions, but we can be more or less deliberate about how we do this part.