The ought is, You ought to do whatever the very credible Mugger tells you to do if you find yourself in a situation with all the properties I list above. Blind obedience does not have a very good reputation; please remember, reader, that the fact that the Nazis enthusiastically advocated and built an interstate highway system does not mean that an interstate highway system is always a bad idea. Every ethical intelligent agent should do his best to increase his intelligence, his knowledge of reality and to help other ethical intelligent agents do the same. That entails consistently resisting tyranny and exploitation. But intelligence can be defined as the ability to predict and control reality or to put it another way to achieve goals. So, if your only goal is to increase intelligence, you set up a recursion that has to bottom out somehow. You cannot increase intelligence indefinitely without eventually confronting the question of what other goals the intelligence you have helped to create will be applied to. That is a tricky question that our civilization does not have much success answering, and I am trying to do better.
The ought is, You ought to do whatever the very credible Mugger tells you to do if you find yourself in a situation with all the properties I list above. Blind obedience does not have a very good reputation; please remember, reader, that the fact that the Nazis enthusiastically advocated and built an interstate highway system does not mean that an interstate highway system is always a bad idea. Every ethical intelligent agent should do his best to increase his intelligence, his knowledge of reality and to help other ethical intelligent agents do the same. That entails consistently resisting tyranny and exploitation. But intelligence can be defined as the ability to predict and control reality or to put it another way to achieve goals. So, if your only goal is to increase intelligence, you set up a recursion that has to bottom out somehow. You cannot increase intelligence indefinitely without eventually confronting the question of what other goals the intelligence you have helped to create will be applied to. That is a tricky question that our civilization does not have much success answering, and I am trying to do better.