What, no-one taken the essay apart yet? Please go ahead!
Ok, I’ll take a shot at it.
As far as I can tell, this article should be titled “why overfitting is a bad idea”. You are absolutely right in saying that, lacking perfect knowledge, we are unable to produce a perfectly optimal solution to any given problem, and that any attempts to do so are bound to end in failure due to unintended consequences. But that is obsession, not efficiency.
Efficiency is a ratio of your expenditure of resources (which include money, time, oxygen, etc.) to the number of your true goals which you are able to achieve. Since none of us have perfect knowledge of our true goals, any strategy we implement must, by necessity, take uncertainty into account. This means that we must consider all the virtues you mentioned—stability, friction, conservation, etc. -- as instrumental goals, and include them in our calculations. If we fail to do so, we risk producing a solution that is vastly inefficient: instead of helping us reach our true goals, the solution will reach some other goals which we mistakenly believe to be valuable.
The problem with standardized tests that you mentioned is a classic example of such overfitting. Our goal is to produce smarter students. We measure progress toward this goal by using a questionnaire, and we build our educational system to maximize scores on the questionnaire. After a few iterations, we find that our students get perfect scores, yet they are actually dumber than they were before we embarked on the project.
You claim that efficiency is the culprit here, but in fact, our solution turned out to be grossly inefficient. We wanted to make our students as smart as possible as quickly (and cheaply) as possible, but we have failed. The reason we failed is because we placed total trust in one questionnaire: we believed that this one test is 100% accurate at predicting future student performance, whereas in reality it is (for example) only 60% accurate. Thus, a more efficient solution should’ve included multiple educational strategies, evaluated over several generations, with the assistance of multiple metrics. Such a solution would’ve been slower, but efficiency is not synonymous with speed.
The fully general efficiency counterargument does not exactly apply here. It says:
If efficiency falls short, then we must estimate the amount to which it falls short, analyse the implementation, improve incentives, etc… Do you see what’s going on there? The solution to a badly implemented system of measurement, is to add extra complications to the system, to measure even more things, the add more constraints, more boxes to tick.
While obtaining more accurate metrics can sometimes be good, the quest for too much accuracy can lead to the same kind of overfitting as the one I described previously. The solution, once again, is not to keep striving for more accurate metrics, but to start taking uncertainty into account.
To use the educational example again, if you launched into educational reforms knowing absolutely nothing about your students’ performance, then you would most likely waste a bunch of resources, because you’re operating in the dark, and your efficiency is very low. You could build up some metrics, such as standardized tests, that will allow you to estimate the performance of each student; but because these metrics are merely estimates, if you focus solely on optimizing the metrics you will most likely waste a bunch of resources (as per my previous comment).
Yes, you could collect increasingly more detailed metrics. You could begin collecting information about each student’s genetics, diet, real-time GPS coordinates, etc. etc. However, these metrics are not free. The time, money, and CPU cycles you are spending on them could be spent on something else, and that something else would most likely get you a better payoff. In fact, at the extreme end of the spectrum, you could end up in a situation where all of your resources are going toward collecting and analyzing terabytes of data; and none of your resources are actually going toward teaching the students. This is not efficiency, this is waste.
Remember, efficiency is not defined as “measuring everything about your problem domain as accurately as possible”, but rather as something like, “solving as many of your true goals as possible using as few resources as possible, while operating under uncertainty regarding your true goals, your resources, and your performance”.
Hmm, no, I don’t see that. If anything, that section is more of a straw-man. It cautions against excessive obsession with collecting data—which can be a real problem—but it assumes that collecting data is the only thing that efficiency is all about (as opposed to actually achieving your true goals).
Ok, I’ll take a shot at it.
As far as I can tell, this article should be titled “why overfitting is a bad idea”. You are absolutely right in saying that, lacking perfect knowledge, we are unable to produce a perfectly optimal solution to any given problem, and that any attempts to do so are bound to end in failure due to unintended consequences. But that is obsession, not efficiency.
Efficiency is a ratio of your expenditure of resources (which include money, time, oxygen, etc.) to the number of your true goals which you are able to achieve. Since none of us have perfect knowledge of our true goals, any strategy we implement must, by necessity, take uncertainty into account. This means that we must consider all the virtues you mentioned—stability, friction, conservation, etc. -- as instrumental goals, and include them in our calculations. If we fail to do so, we risk producing a solution that is vastly inefficient: instead of helping us reach our true goals, the solution will reach some other goals which we mistakenly believe to be valuable.
The problem with standardized tests that you mentioned is a classic example of such overfitting. Our goal is to produce smarter students. We measure progress toward this goal by using a questionnaire, and we build our educational system to maximize scores on the questionnaire. After a few iterations, we find that our students get perfect scores, yet they are actually dumber than they were before we embarked on the project.
You claim that efficiency is the culprit here, but in fact, our solution turned out to be grossly inefficient. We wanted to make our students as smart as possible as quickly (and cheaply) as possible, but we have failed. The reason we failed is because we placed total trust in one questionnaire: we believed that this one test is 100% accurate at predicting future student performance, whereas in reality it is (for example) only 60% accurate. Thus, a more efficient solution should’ve included multiple educational strategies, evaluated over several generations, with the assistance of multiple metrics. Such a solution would’ve been slower, but efficiency is not synonymous with speed.
Cool, we’re getting somewhere ^_^ and if I accused you of using the fully general efficiency counterargument, what would you say?
The fully general efficiency counterargument does not exactly apply here. It says:
While obtaining more accurate metrics can sometimes be good, the quest for too much accuracy can lead to the same kind of overfitting as the one I described previously. The solution, once again, is not to keep striving for more accurate metrics, but to start taking uncertainty into account.
To use the educational example again, if you launched into educational reforms knowing absolutely nothing about your students’ performance, then you would most likely waste a bunch of resources, because you’re operating in the dark, and your efficiency is very low. You could build up some metrics, such as standardized tests, that will allow you to estimate the performance of each student; but because these metrics are merely estimates, if you focus solely on optimizing the metrics you will most likely waste a bunch of resources (as per my previous comment).
Yes, you could collect increasingly more detailed metrics. You could begin collecting information about each student’s genetics, diet, real-time GPS coordinates, etc. etc. However, these metrics are not free. The time, money, and CPU cycles you are spending on them could be spent on something else, and that something else would most likely get you a better payoff. In fact, at the extreme end of the spectrum, you could end up in a situation where all of your resources are going toward collecting and analyzing terabytes of data; and none of your resources are actually going toward teaching the students. This is not efficiency, this is waste.
Remember, efficiency is not defined as “measuring everything about your problem domain as accurately as possible”, but rather as something like, “solving as many of your true goals as possible using as few resources as possible, while operating under uncertainty regarding your true goals, your resources, and your performance”.
Good ^_^
Did you notice that the section criticising efficiency fully general counterarguments… was itself a fully general counterargument?
Hmm, no, I don’t see that. If anything, that section is more of a straw-man. It cautions against excessive obsession with collecting data—which can be a real problem—but it assumes that collecting data is the only thing that efficiency is all about (as opposed to actually achieving your true goals).