Question 11 of 21: Imagine that a fast food company called BurgerCorp has launched a new sandwich called the BacoNation: a bacon patty slathered in bacon jack cheese, pork drippings, and bacon-fat-fried onions, served on a bacon-bran bun.
The BacoNation was introduced in March of this year. So far, its per-month sales revenues have been inconsistent:
March: $55.0 million
April: $43.8 million
May: $59.4 million
June: $49.6 million
July: $46.1 million
August: $54.9 million
September: $44.5 million
As BurgerCorp heads into the holidays, its executives hope to boost sales of the BacoNation by having their advertising team create an ad campaign featuring a wacky animated mascot called the BacoNutcase—a wild-eyed bacon fanatic who is willing to commit unspeakably immoral and degrading acts in the pursuit of BacoNation sandwiches.
BurgerCorp launches the BacoNutcase ad campaign at the beginning of October. When the sales figures come in at the end of the month, the executives are ecstatic: it turns out that the BacoNation sandwich earned $60.5 million in sales during the month of October. This figure represents an increase of $16 million over the September sales figure, and a new record high for the sandwich.
How likely is it that the introduction of the BacoNutcase largely caused the increase in the BacoNation’s sales?
They claim that the correct answer is “somewhat likely,” arguing that the prize has fluctuated as much in past months as it did from September to October, therefore there is no strong evidence for the ads to have contributed. I think this is plain wrong, because
the a priori chance of ads increasing sales is high
October beat the past 7 months, therefore Bayesian updating increases the odds further
The way I see it, their explanation would possibly be correct if we only knew that there was a 50% probability that the ad campaign had taken place, and were to decide whether or not it did based on the results. But since we already know that it did take place, the correct answer seems to be “very likely.”
Who is correct? The test or me? (I’m asking because if I’m wrong I really want to know why).
ok, there’s not really enough data points to do proper stats but lets give it a go anyway.
Lets consider the possibility that the ad campaign did nothing. Some ad campaigns are actually damaging so lets try to get an idea of how much it varies from month to month.
Mean = 50.5
Standard Deviation = 6.05
So about 1 and 2/3rds SD’s above the mean.
Sure, October is a little higher than normal but not by much.
Or put another way, imagine that the ad campaign had been put into effect in April but actually did absolutely nothing. They would have seen an increase of 15.6 million along with a new record high.
The priori chance of ads increasing sales is high for good ad campaigns but as countless dot com bubble companies learned: it’s entirely possible for advertising to get you absolutely nothing.
Remember that the priori is a fancy way of encoding your expectations into how you do calculations.
If you’re trying to decide whether an ad campaign you’ve paid for actually worked a system of assessment which involves saying “well, I believed it should work in principle so I spent money on doing it in the first place and now I can confirm it worked partly because I believe it should work in principle”
Hm, thanks. It seems like I was misinformed about ads – I had the belief that they increase sales almost all of the time, which, based on what you said and a quick search. appears to have been totally false. With that and the ‘largely’ I missed, I’d now say the test was mostly correct.
It reduces the accuracy of the test if people read the test questions before tacking it. In general ClearThinking is a valuble project, where I don’t think we act in a way to reduce their accuracy.
I would buy that argument if this is a ClearThinking forum. But it is not. rot13 just gets in the way, prevents search from working properly, etc. We shouldn’t have an obligation to complicate our discussions for whatever small potential increase in accuracy.
In the question there is a key word: “largely”. I interpret the question to mean “Was the ad campaign responsible for most of the increase in October?” and specifically NOT to mean “Did the ad campaign have any positive effect on the sales?”.
The answer to the latter question is “very likely”, but to the former is merely “somewhat likely” because you have high variation in monthly revenues and just on the basis of data provided you cannot confidently assert that the difference between September and October was driven by the ad campaign.
It’s a stupid question. It wouldn’t be too hard to give 10 methodologists this question, then tell them the side to support, and watch them all build great cases. Obviously that’s an assertion, I can’t imagine evidence then claim it proves me right :P, but I strongly suspect this would be true.
The question is so dumb. Even if they got rid of the business story-line, and abstracted it to pure statistics, it’s still stupid. What distribution characterizes it? If they got rid of the business, gave the data, AND gave info on the generative distribution, AND made it a numerical answer… Then I guess it’s a fair question, but at that point it’s just a pure stats question.
I think it’s not stupid. Often in real cases and applied rationality you don’t have cleanly cooked up priors and distributions. You only have data like the series above, and it’s up to you to draw conclusions. Success happens when you are modest in your suppositions and able to change idea based on future evidence.
I’ll cross-post this from here because no-one responded and I’m still interested in an answer.
There is a question in this rationality test which goes like this:
They claim that the correct answer is “somewhat likely,” arguing that the prize has fluctuated as much in past months as it did from September to October, therefore there is no strong evidence for the ads to have contributed. I think this is plain wrong, because
the a priori chance of ads increasing sales is high
October beat the past 7 months, therefore Bayesian updating increases the odds further
The way I see it, their explanation would possibly be correct if we only knew that there was a 50% probability that the ad campaign had taken place, and were to decide whether or not it did based on the results. But since we already know that it did take place, the correct answer seems to be “very likely.”
Who is correct? The test or me? (I’m asking because if I’m wrong I really want to know why).
ok, there’s not really enough data points to do proper stats but lets give it a go anyway.
Lets consider the possibility that the ad campaign did nothing. Some ad campaigns are actually damaging so lets try to get an idea of how much it varies from month to month.
Mean = 50.5 Standard Deviation = 6.05
So about 1 and 2/3rds SD’s above the mean.
Sure, October is a little higher than normal but not by much.
Or put another way, imagine that the ad campaign had been put into effect in April but actually did absolutely nothing. They would have seen an increase of 15.6 million along with a new record high.
The priori chance of ads increasing sales is high for good ad campaigns but as countless dot com bubble companies learned: it’s entirely possible for advertising to get you absolutely nothing.
Remember that the priori is a fancy way of encoding your expectations into how you do calculations.
If you’re trying to decide whether an ad campaign you’ve paid for actually worked a system of assessment which involves saying “well, I believed it should work in principle so I spent money on doing it in the first place and now I can confirm it worked partly because I believe it should work in principle”
Hm, thanks. It seems like I was misinformed about ads – I had the belief that they increase sales almost all of the time, which, based on what you said and a quick search. appears to have been totally false. With that and the ‘largely’ I missed, I’d now say the test was mostly correct.
I would advocate for rot13. There’s no reason to feature the test questions in the open.
Why?
It reduces the accuracy of the test if people read the test questions before tacking it. In general ClearThinking is a valuble project, where I don’t think we act in a way to reduce their accuracy.
I would buy that argument if this is a ClearThinking forum. But it is not. rot13 just gets in the way, prevents search from working properly, etc. We shouldn’t have an obligation to complicate our discussions for whatever small potential increase in accuracy.
In the question there is a key word: “largely”. I interpret the question to mean “Was the ad campaign responsible for most of the increase in October?” and specifically NOT to mean “Did the ad campaign have any positive effect on the sales?”.
The answer to the latter question is “very likely”, but to the former is merely “somewhat likely” because you have high variation in monthly revenues and just on the basis of data provided you cannot confidently assert that the difference between September and October was driven by the ad campaign.
It’s a stupid question. It wouldn’t be too hard to give 10 methodologists this question, then tell them the side to support, and watch them all build great cases. Obviously that’s an assertion, I can’t imagine evidence then claim it proves me right :P, but I strongly suspect this would be true.
The question is so dumb. Even if they got rid of the business story-line, and abstracted it to pure statistics, it’s still stupid. What distribution characterizes it? If they got rid of the business, gave the data, AND gave info on the generative distribution, AND made it a numerical answer… Then I guess it’s a fair question, but at that point it’s just a pure stats question.
I think it’s not stupid. Often in real cases and applied rationality you don’t have cleanly cooked up priors and distributions. You only have data like the series above, and it’s up to you to draw conclusions.
Success happens when you are modest in your suppositions and able to change idea based on future evidence.
Gurer ab fgngrzrag gung vzcyvrf gung gur pbzcnal qvqa’g eha nal nqf orsber Bpgbore naq gung gurl eha enqvpnyyl zber nqf.