I think this is the right answer. Just to expand on this a bit: The problem isn’t necessarily that 222+222=555 leads to a contradiction with the rest of arithmetic. One can imagine that instead of defining “+” using “x+Sy=y+Sx”, we could give it a much more complex definition where there is a special case carved out for certain values like 222. The issue is that the AI has no reason to use this version of “+” and will define some other operation that works just like actual addition. Even if we ban the AI from using “x+Sy=y+Sx” to define any operations, it will choose the nearest thing isomorphic to addition that we haven’t blocked, because addition is so common and useful. Or maybe it will use the built-in addition, but whenever it wants to add n+m, it instead adds 4n+4m, since our weird hack doesn’t affect the subgroup consisting of integers divisible by 4.
I think this is the right answer. Just to expand on this a bit: The problem isn’t necessarily that 222+222=555 leads to a contradiction with the rest of arithmetic. One can imagine that instead of defining “+” using “x+Sy=y+Sx”, we could give it a much more complex definition where there is a special case carved out for certain values like 222. The issue is that the AI has no reason to use this version of “+” and will define some other operation that works just like actual addition. Even if we ban the AI from using “x+Sy=y+Sx” to define any operations, it will choose the nearest thing isomorphic to addition that we haven’t blocked, because addition is so common and useful. Or maybe it will use the built-in addition, but whenever it wants to add n+m, it instead adds 4n+4m, since our weird hack doesn’t affect the subgroup consisting of integers divisible by 4.