I’d only add this: if a genie is to be capable of granting this wish, it would need to know what your judgements were. It would need to understand them, at least as well as you do. This pretty much resolves to the same problem that Eliezer already discussed.
To create such a genie, you would either need to explain to the genie how you would feel about every possible circumstance, or you would need to program the genie so as to be able to correctly figure it out. Both of these tasks are probably a lot harder than they sound.
What if your judgement criteria are fluid—depending, perhaps, on your current hormonal state, your available knowledge, and your particular position in society?
“I wish for a genie that shares all my judgment criteria” is probably the only safe way.
This might be done by picking an arbitrary genie, and then modifying your judgement criteria to match that genie’s.
Which is perhaps most efficiently achieved by killing the wisher and returning an arbitrary inanimate object.
I see where you’re coming from on this one.
I’d only add this: if a genie is to be capable of granting this wish, it would need to know what your judgements were. It would need to understand them, at least as well as you do. This pretty much resolves to the same problem that Eliezer already discussed.
To create such a genie, you would either need to explain to the genie how you would feel about every possible circumstance, or you would need to program the genie so as to be able to correctly figure it out. Both of these tasks are probably a lot harder than they sound.
What if your judgement criteria are fluid—depending, perhaps, on your current hormonal state, your available knowledge, and your particular position in society?