But how do those skew the result? It’s not enough to say that there’s bias; you would have to show that the bias works against the claimed conclusion. So with respect to your points:
1) If you don’t have a reason to believe the bias skews one way or the other, knowledge that it could be higher or lower doesn’t matter.
2) The wealthier you are, the easier you are to track down, so the survey would tend to over-state the average income.
3) True, but again, do you have a reason to believe it goes one way rather than the other? (I admit I will never understand what kind of person actually responds to surveys. For people in my demographic, responding to a pollster is a mark of shame, and my parents, less reliant on cell phones, refuse to answer pollsters as well.)
But how do those skew the result? It’s not enough to say that there’s bias; you would have to show that the bias works against the claimed conclusion. So with respect to your points:
1) If you don’t have a reason to believe the bias skews one way or the other, knowledge that it could be higher or lower doesn’t matter.
2) The wealthier you are, the easier you are to track down, so the survey would tend to over-state the average income.
3) True, but again, do you have a reason to believe it goes one way rather than the other? (I admit I will never understand what kind of person actually responds to surveys. For people in my demographic, responding to a pollster is a mark of shame, and my parents, less reliant on cell phones, refuse to answer pollsters as well.)
If you have reason to believe that there are significant sources of sample bias in a poll, you will have to discount the results severely.