Well, that’s just the entire point of this LW post—what prior to choose matters a lot. It even matters specifically in the case of BeerAdvocate, who apparently got appreciably different results from changing their value of 10 reviews.
I think I specifically said that variance matters. I’ll also say that your application matters—when choosing beers, I would be OK with p much worse than 0.05 since I can afford to order another beer. When choosing charities, it is a harder question.
Well, that’s just the entire point of this LW post—what prior to choose matters a lot. It even matters specifically in the case of BeerAdvocate, who apparently got appreciably different results from changing their value of 10 reviews.
And the solution you suggest is to just go with p=0.05 and pretend this problem isn’t unavoidable, right?
I think I specifically said that variance matters. I’ll also say that your application matters—when choosing beers, I would be OK with p much worse than 0.05 since I can afford to order another beer. When choosing charities, it is a harder question.