Thanks for the answer! As you suspected, I don’t think wireheading is a good thing, but after reading about infinite ethics and the repugnant conclusion I’m not entirely sure that there exists a stable mathematically expressible form of ethics we could give to an AGI. Obviously I think it’s possible if you specify exactly what you want and tell the AGI not to extrapolate. However I feel that realistically, it’s going to take our ethics and take it to its logical end, and there exists no ethical theory that really expresses how utility should be valued without causing paradoxes or problems we can’t solve. Unless we manage to build AGI using an evolutionary method to mimick human evolution, I believe that any training or theory given to it would subtly fail.
Thanks for the answer! As you suspected, I don’t think wireheading is a good thing, but after reading about infinite ethics and the repugnant conclusion I’m not entirely sure that there exists a stable mathematically expressible form of ethics we could give to an AGI. Obviously I think it’s possible if you specify exactly what you want and tell the AGI not to extrapolate. However I feel that realistically, it’s going to take our ethics and take it to its logical end, and there exists no ethical theory that really expresses how utility should be valued without causing paradoxes or problems we can’t solve. Unless we manage to build AGI using an evolutionary method to mimick human evolution, I believe that any training or theory given to it would subtly fail.