So you’re searching for “the most important thing”, and reason that this is the same as searching for some utility function, and then you note that one reason this question seems worth thinking about is because it’s interesting, and then you refer to Schmidhuber’s definition of interestingness (which would yield a utility function), and note that it is itself interesting, so maybe importance is the same as interestingness, because importance has to be itself important and (Schmidhuberian) interestingness satisfies this requirement by being itself interesting
At this point I’m not very impressed. This seems to be the same style of reasoning that gets people obsessed with “complexity” or “universal instrumental values” as the ultimate utility functions.
At the end you say you doubt that interestingness is the ultimate utility function, too, but apparently you still think engaging in this style of reasoning is a good idea, we just have to take it even further.
At this point I’m thinking that it could go either way: you could come up with an interesting proposal in the class of CEV or “Indirect Normativity”, which definitely are in some sense the result of going meta about values, or you could come up with something that turns out to be just another fake utility function in the class of “complexity” and “universal instrumental values”.
So you’re searching for “the most important thing”, and reason that this is the same as searching for some utility function, and then you note that one reason this question seems worth thinking about is because it’s interesting, and then you refer to Schmidhuber’s definition of interestingness (which would yield a utility function), and note that it is itself interesting, so maybe importance is the same as interestingness, because importance has to be itself important and (Schmidhuberian) interestingness satisfies this requirement by being itself interesting
At this point I’m not very impressed. This seems to be the same style of reasoning that gets people obsessed with “complexity” or “universal instrumental values” as the ultimate utility functions.
At the end you say you doubt that interestingness is the ultimate utility function, too, but apparently you still think engaging in this style of reasoning is a good idea, we just have to take it even further.
At this point I’m thinking that it could go either way: you could come up with an interesting proposal in the class of CEV or “Indirect Normativity”, which definitely are in some sense the result of going meta about values, or you could come up with something that turns out to be just another fake utility function in the class of “complexity” and “universal instrumental values”.