To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn’t it be equivalent to Alice-who-values-copies-of-herself?
my priors regarding the prevalence of godlike agents is currently extremely low,
And what are those priors based on? Evidence! Evidence that a godlike being would be motivated to falsify!
To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn’t it be equivalent to Alice-who-values-copies-of-herself?
Sure, but the result you describe is equivalent to Alice being an average utilitarian with respect to copies of herself. What if Alice is a total utilitarian with respect to copies of herself?
To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn’t it be equivalent to Alice-who-values-copies-of-herself?
And what are those priors based on? Evidence! Evidence that a godlike being would be motivated to falsify!
Sure, but the result you describe is equivalent to Alice being an average utilitarian with respect to copies of herself. What if Alice is a total utilitarian with respect to copies of herself?
Actually, she should still make the same choice, although she would choose differently in other scenarios.