Alice says: the p-value for the drug effectiveness is X. This means that there is X% probability that the results we see arose entirely by chance while the drug has no effect at all.
No. You don’t understand null hypotheis testing. It doesn’t measure whether the results arose entirely by chance. It measures whether a specifc null hypothsis can be rejected.
I hate to disappoint you, but I do understand null hypothesis testing. In this particular example the specific null hypothesis is that the drug has no effect and therefore all observable results arose entirely by chance.
You are really determined to fight they hypothetical, aren’t you? :-) Let me quote myself with the relevant part emphasized: “You want to find out whether the drug has certain (specific, detectable) effects.”
I could simply run n=1 experiments
And how would they help you? There is the little issue of noise. You cannot detect any effects below the noise floor and for n=1 that floor is going to be pretty high.
“You want to find out whether the drug has certain (specific, detectable) effects.”
A p-value isn’t the probability that a drug has certain (specific, detectable) effects. 1-p isn’t either.
You are really determined to fight they hypothetical, aren’t you?
No, I’m accepting it. The probability of a drug having zero effects is 0. If your statistics give you an answer that a drug has a probability other than 0 for a drug having zero effects your statistics are wrong.
I think your answer suggests the idea that an experiment might provide actionable information.
And how would they help you? There is the little issue of noise. You cannot detect any effects below the noise floor and for n=1 that floor is going to be pretty high.
But you still claim that every experiment provides an actionable probability when interpreted by a frequentist.
If you give a bayesian your priors and then get a posterior probability from the bayesian that probability is in every case actionable.
Again: the probability that a drug has no specific, detectable effects is NOT zero.
I don’t care about detectability when I take a drug. I care about whether it helps me.
I want a number that tell me the probability of the drug helping me. I don’t want the statisician to answer a different question.
Detectability depends on the power of a trial.
If a frequentist gives you some number after he analysed an experiment you can’t just fit that number in a decision function.
You have to think about issues such as whether the experiment had enough power to pick up an effect.
If a bayesian gives you a probability you don’t have to think about such issues because the bayesian already integrates your prior knowledge. The probability that the bayesian gives you can be directly used.
Drug trials are neither designed to, nor capable of answering questions like this.
Whether a drug will help you is a different probability that comes out of a complicated evaluation for which the drug trial results serve as just one of the inputs.
If a bayesian gives you a probability you don’t have to think about such issues
Whether a drug will help you is a different probability that comes out of a complicated evaluation for which the drug trial results serve as just one of the inputs.
That evaluation is in it’s nature bayesian. Bayes rule is about adding together different probabilities.
At the moment there no systematic way of going about it. That’s where theory development is needed. I would that someone like the FDA writes down all their priors and then provides some computer analysis tool that actually calculates that probability.
I am sorry, you’re speaking nonsense.
If the priors are correct then a correct bayesian analysis provides me exactly the probability in which I should believe after I read the study.
No. You don’t understand null hypotheis testing. It doesn’t measure whether the results arose entirely by chance. It measures whether a specifc null hypothsis can be rejected.
I hate to disappoint you, but I do understand null hypothesis testing. In this particular example the specific null hypothesis is that the drug has no effect and therefore all observable results arose entirely by chance.
Almost no drug has no effect. Most drug changes the patient and produces either a slight advantage or disadvantage.
If what you saying is correct I could simply run n=1 experiments.
You are really determined to fight they hypothetical, aren’t you? :-) Let me quote myself with the relevant part emphasized: “You want to find out whether the drug has certain (specific, detectable) effects.”
And how would they help you? There is the little issue of noise. You cannot detect any effects below the noise floor and for n=1 that floor is going to be pretty high.
A p-value isn’t the probability that a drug has certain (specific, detectable) effects. 1-p isn’t either.
No, I’m accepting it. The probability of a drug having zero effects is 0. If your statistics give you an answer that a drug has a probability other than 0 for a drug having zero effects your statistics are wrong.
I think your answer suggests the idea that an experiment might provide actionable information.
But you still claim that every experiment provides an actionable probability when interpreted by a frequentist.
If you give a bayesian your priors and then get a posterior probability from the bayesian that probability is in every case actionable.
Again: the probability that a drug has no specific, detectable effects is NOT zero.
Huh? What? I don’t even… Please quote me.
What do you call an “actionable” probability? What would be an example of a “non-actionable” probability?
I don’t care about detectability when I take a drug. I care about whether it helps me. I want a number that tell me the probability of the drug helping me. I don’t want the statisician to answer a different question.
Detectability depends on the power of a trial.
If a frequentist gives you some number after he analysed an experiment you can’t just fit that number in a decision function. You have to think about issues such as whether the experiment had enough power to pick up an effect.
If a bayesian gives you a probability you don’t have to think about such issues because the bayesian already integrates your prior knowledge. The probability that the bayesian gives you can be directly used.
Drug trials are neither designed to, nor capable of answering questions like this.
Whether a drug will help you is a different probability that comes out of a complicated evaluation for which the drug trial results serve as just one of the inputs.
I am sorry, you’re speaking nonsense.
That evaluation is in it’s nature bayesian. Bayes rule is about adding together different probabilities.
At the moment there no systematic way of going about it. That’s where theory development is needed. I would that someone like the FDA writes down all their priors and then provides some computer analysis tool that actually calculates that probability.
If the priors are correct then a correct bayesian analysis provides me exactly the probability in which I should believe after I read the study.