the happiness maximizer is going to need to be able to find happiness inside an unfamiliar ontology.
But the module for predicting human behaviour/preferences should surely be the same in a different ontology? The module is a model, and the model is likely not grounded in the fine detail of the ontology.
Example: the law of comparative advantage in economics is a high level model, which won’t collapse because the fundamental ontology is relativity rather than newtonian mechanics. Even in a different ontology, humans should remain (by far) the best things in the world that approximate the “human model”.
If there is a module that specifically requires prediction of human behavior, sure. My claim in the second part of my comment is that if the model predicts the number of paperclips, it’s not necessary that the closest match to things that function like human decisions will actually be a useful predictive model of human decisions.
But the module for predicting human behaviour/preferences should surely be the same in a different ontology? The module is a model, and the model is likely not grounded in the fine detail of the ontology.
Example: the law of comparative advantage in economics is a high level model, which won’t collapse because the fundamental ontology is relativity rather than newtonian mechanics. Even in a different ontology, humans should remain (by far) the best things in the world that approximate the “human model”.
If there is a module that specifically requires prediction of human behavior, sure. My claim in the second part of my comment is that if the model predicts the number of paperclips, it’s not necessary that the closest match to things that function like human decisions will actually be a useful predictive model of human decisions.