I haven’t read the article, but the more noise there is, the more samples you should get before giving the feedback significance.
A recent Science article showed this graphically. They were using a two-deck game, where you give subjects two decks of cards, each with 2 types of cards, winning and losing. You tell them that one deck has more winning cards than the other. Then they get a certain number of card draws, and can draw each card from either deck.
What’s amazing is how lousy people do on this test. You would win by drawing 10 cards from each deck, choosing the better deck, and sticking with it. People don’t. (Received wisdom is that they want to draw from each deck in proportion to its probability of a winning card, which is not a good strategy.)
In this paper, they posited that people estimate the probability of a deck being the better deck by exponentially discounting older evidence. (The most recently drawn card influences them most.) They plotted a graph of the probability over time that this would give you of one deck being the winning deck. Amazingly, this graph showed the probability hovering around .5 over 90 card draws, when actually the difference between the decks was dramatic (something like .6 win vs. .4 win).
I haven’t read the article, but the more noise there is, the more samples you should get before giving the feedback significance.
A recent Science article showed this graphically. They were using a two-deck game, where you give subjects two decks of cards, each with 2 types of cards, winning and losing. You tell them that one deck has more winning cards than the other. Then they get a certain number of card draws, and can draw each card from either deck.
What’s amazing is how lousy people do on this test. You would win by drawing 10 cards from each deck, choosing the better deck, and sticking with it. People don’t. (Received wisdom is that they want to draw from each deck in proportion to its probability of a winning card, which is not a good strategy.)
In this paper, they posited that people estimate the probability of a deck being the better deck by exponentially discounting older evidence. (The most recently drawn card influences them most.) They plotted a graph of the probability over time that this would give you of one deck being the winning deck. Amazingly, this graph showed the probability hovering around .5 over 90 card draws, when actually the difference between the decks was dramatic (something like .6 win vs. .4 win).