I think it’s cleaner to build the foundation around the basic laws of physics. You don’t need the laws of reasoning in the description of physics, they define the way reality flows. Leaving the margin for doubt on the laws themselves, all the higher-level things like minds and principles of reasoning within the current framework can be built on top of that, like special kind of physical structures with a certain kind of dynamics. I think about the goal-driven intelligence (optimization process) as an attractor in the dynamics of reality, which drives it towards a certain kind of dynamics. When morality of the intelligent agent is changed, either by being broken significantly or by being influenced, it means that the high-level description that fits the process that is reality is changed. Where before the dynamics looked like it was heading in this direction, now it jumped to a form where it’s heading in another direction, or is changed the course, so that it’s heading in a slightly different way. The laws of physics are the substrate, on which one has to figure a way of building the attractor.
Another question is that if the mind is running inside a simulation, it is not semantically bound to the structure of real world, its goal gets hijacked by substituting the reality by the simulated happy button, the limited simulated reality that is accessible to inspection. Such mind can’t have a global attractor effect on the flow of reality, but it can bind it locally, just as an ordinary human optimizer, so I guess it’s not fundamentally different.
I think it’s cleaner to build the foundation around the basic laws of physics. You don’t need the laws of reasoning in the description of physics, they define the way reality flows. Leaving the margin for doubt on the laws themselves, all the higher-level things like minds and principles of reasoning within the current framework can be built on top of that, like special kind of physical structures with a certain kind of dynamics. I think about the goal-driven intelligence (optimization process) as an attractor in the dynamics of reality, which drives it towards a certain kind of dynamics. When morality of the intelligent agent is changed, either by being broken significantly or by being influenced, it means that the high-level description that fits the process that is reality is changed. Where before the dynamics looked like it was heading in this direction, now it jumped to a form where it’s heading in another direction, or is changed the course, so that it’s heading in a slightly different way. The laws of physics are the substrate, on which one has to figure a way of building the attractor.
Another question is that if the mind is running inside a simulation, it is not semantically bound to the structure of real world, its goal gets hijacked by substituting the reality by the simulated happy button, the limited simulated reality that is accessible to inspection. Such mind can’t have a global attractor effect on the flow of reality, but it can bind it locally, just as an ordinary human optimizer, so I guess it’s not fundamentally different.