There are no free lunch theorems “proving” that intelligence is impossible. There is no algorithm that can optimize an arbitrary environment. We display intelligence. The problem with the theorem comes from the part where you assume an arbitrary max-entropy environment, rather than inductive priors. If you assume that human values are simple (low komelgorov complexity) and that human behavior is quite good at fulfilling those values, then you can deduce non trivial values for humans.
If you assume that human values are simple (low komelgorov complexity) and that human behavior is quite good at fulfilling those values, then you can deduce non trivial values for humans.
And if instead you mean a proper accounting of bounded rationality, of the difference between anchoring bias and taste, of the difference between system 1 and system 2, of the whole collection of human biases… well, then, yes, I might agree with you. But that’s because you’ve already put all the hard work in.
I should have been clearer, the point isn’t that you get correct values, the point is that you get out of the swath of null or meaningless values and into the just wrong. While the values gained will be wrong, they would be significantly correlated, its the sort of AI to produce drugged out brains in vats, or something else that’s not what we want, but closer than paperclips. One measure you could use of human effectiveness is given all possible actions ordered by util, what percentile are the actions we took in.
Once we get into this region, it becomes clear that the next task is to fine tune our model of the bounds on human rationality, or figure out how to get an AI to do it for us.
I disagree. I think that if we put a complexity upper bound on human rationality, and assume noisy rationality, then we will get values that are “meaningless” from your perspective.
I’m trying to think of ways how of we could test this....
There are no free lunch theorems “proving” that intelligence is impossible. There is no algorithm that can optimize an arbitrary environment. We display intelligence. The problem with the theorem comes from the part where you assume an arbitrary max-entropy environment, rather than inductive priors. If you assume that human values are simple (low komelgorov complexity) and that human behavior is quite good at fulfilling those values, then you can deduce non trivial values for humans.
And you will deduce them wrong. “Human values are simple” pushes you towards “humans have no preferences”, and if by “human behavior is quite good at fulfilling those values” you mean something like noisy rationality, then it will go very wrong, see for example https://www.lesswrong.com/posts/DuPjCTeW9oRZzi27M/bounded-rationality-abounds-in-models-not-explicitly-defined
And if instead you mean a proper accounting of bounded rationality, of the difference between anchoring bias and taste, of the difference between system 1 and system 2, of the whole collection of human biases… well, then, yes, I might agree with you. But that’s because you’ve already put all the hard work in.
I should have been clearer, the point isn’t that you get correct values, the point is that you get out of the swath of null or meaningless values and into the just wrong. While the values gained will be wrong, they would be significantly correlated, its the sort of AI to produce drugged out brains in vats, or something else that’s not what we want, but closer than paperclips. One measure you could use of human effectiveness is given all possible actions ordered by util, what percentile are the actions we took in.
Once we get into this region, it becomes clear that the next task is to fine tune our model of the bounds on human rationality, or figure out how to get an AI to do it for us.
I disagree. I think that if we put a complexity upper bound on human rationality, and assume noisy rationality, then we will get values that are “meaningless” from your perspective.
I’m trying to think of ways how of we could test this....