I found this post useful, because it contains practical ideas about how to perform personal experiments in dating.
I think you are confused about how statistics work. Student’s T-test is a frequentist thing, while in some places you say that you have aprior which suggests you want to do bayesian analysis.
The sentence
My prior is 70% that the H1 is true (conditional on null being false).
makes no sense because of the part in parentheses.
So all I need from the experiment is the ratio of how likely I am to observe this result given H1 vs. given H2. At first I thought calculating that with t-distributions would be trivial, but I’m noticing a problem now. If I have 7 successes in treatment and 3 in control, I can use the t-dist to calculate how often I would observe that in the null. But since the null is decision irrelevant, that doesn’t get me the likelihood ratio which I want. I see that now. I’ll circle back to this problem when I have more time later.
What is your null hypothesis? Nowhere does your post says that. I suspect you don’t know what a null hypothesis is.
From your post and your comment, I infer that you want to find the probability of “intentionally reducing reactivity and affection for the first three dates will increase attraction in partners”. That doesn’t work well with bayesian analysis. Instead you should try to get a posterior distribution over the value of how much it increases attraction.
I think if you want to do the bayesian data analysis, then one of the simplest ways you could model your situation is as follows.
If you PHTG, you achieve sex (or whatever it is you’re after, but I’ll just say sex for simplicity) with probability p∈(0, 1). If you don’t PHTG, you achieve sex with probability q∈(0, 1). Currently, you don’t know the values of q and r but you have a prior distribution p(q, r) over them. In this prior p(q, r), q and r are not necessarily independent. On the opposite, I would expect that they correlate (with respect to the prior p(q, r)) very strongly, because if you often achieve sex with one strategy, probably you’ll also be able to do that with the other strategy, and if you can’t achieve sex with one strategy, probably you can’t with the other. Next, you will go and do the experiments (go on dates and randomly choose whether to PHTG). An experiment is like tossing a biased coin. If you are PHTG, you are tossing a coin which lands on heads with probability q. If you are not PHTG, you are tossing a coin which lands on heads with probability r. After n experimental results e1,…,en, you update your distribution over the values q and r: p(q,r∣e1,…,en)=p(e1,…,en∣q,r)p(q,r)p(e1,…,en)=p(e1∣q,r)p(e2∣q,r)…p(en∣q,r)p(q,r)p(e1,…,en) and this is the result you get. I think this models represents your situation fairly well.
I don’t know what prior p(q, r) you should choose in order to have it fairly close to your actualy beliefs while at the same time making the computation tractable. A simplification you can try is imagining that prior to the experimental data, q and r are totally independent from each other. Then your situation is simply two separate situations, in each you are trying to estimate the biasedness of a coin. Then you take the prior of q to be a beta distribution, and the prior of r to be a beta distribution as well. Then you open “Data analysis a bayesian tutorial—Sivia Skilling” (can be found on libgen) page 14 example 1 “is this a fair coin?” and do whatever it says. Another thing you could probably do is come up with some kind piecewise-constant prior p(q, r) manually and perform the bayesian analysis by simulating everything on the computer rather than tinkering with integrals on paper. Formally, this is called Monte Carlo integration.
Also, instead of treating the outcomes as binary (sex or no-sex), you could treat them as real numbers which represent how well it went. I think this way you’ll need less experiments to get a conclusion. For this case, you can read “Bayesian Estimation Supersedes the t Test—Kruschke 2012”. That paper describes how to do bayesian analysis when you have two groups (treatment and control) and you want to measure what the treatment does if it does anything.
Thanks for pointing me in the right direction with these! My degree is really frequentist and slow paced, excited to get to work on this analysis.
Clarification on null hypothesis:
The null hypothesis is that there is no difference in effect on the dependent variable from the treatment and control variables. I am not assessing the truth of the null hypothesis because if it is true, then I can pick whichever one I want. If control is better, then picking treatment is negative utility. If treatment is better then picking control is negative utility. If the null is true, then I am free to do treatment or control without suffering in either case. Therefore I gain no utility from a test to see if the null is true or not.
Consider shifting your tone when people know less stats than you. Saying ” I suspect you don’t know what a null hypothesis is” makes people feel defensive and not willing to take your useful advice. Try saying “can you clarify what you mean by ____” or “here’s a common definition of a null hypothesis”.
Clarification on dependent variables:
I was going to code the outcome as 1 if sex|second date. The thinking is that only women who are attracted to me have sex with me (but may not want to date, for lots of good reasons). Meanwhile many women who are attracted to me do go on a second date. But few women are attracted to me but do neither sex nor a second date. Since attraction is the concept I want to explain, this should have the best specificity and sensitivity of available measures.
I am considering a second DV using eye contact during date (qualitative) as a robustness check. I think some people do lots of eye contact on all dates as a subconscious influence strategy, so it has more false positives than the other two.
I found this post useful, because it contains practical ideas about how to perform personal experiments in dating.
I think you are confused about how statistics work. Student’s T-test is a frequentist thing, while in some places you say that you have aprior which suggests you want to do bayesian analysis.
The sentence
makes no sense because of the part in parentheses.
I’m glad I could help!
Here’s my plan for analyzing the data, let me know what changes your would make.
Bayes rule odds form: P(H1)priorP(H2)PriorxP(E|H1)P(E|H2)=P(H1)PosteriorP(H2)Posterior
So all I need from the experiment is the ratio of how likely I am to observe this result given H1 vs. given H2. At first I thought calculating that with t-distributions would be trivial, but I’m noticing a problem now. If I have 7 successes in treatment and 3 in control, I can use the t-dist to calculate how often I would observe that in the null. But since the null is decision irrelevant, that doesn’t get me the likelihood ratio which I want. I see that now. I’ll circle back to this problem when I have more time later.
Found a guide
I don’t care about the null because if the null is true and I get the wrong answer, it doesn’t matter.
What is your null hypothesis? Nowhere does your post says that. I suspect you don’t know what a null hypothesis is.
From your post and your comment, I infer that you want to find the probability of “intentionally reducing reactivity and affection for the first three dates will increase attraction in partners”. That doesn’t work well with bayesian analysis. Instead you should try to get a posterior distribution over the value of how much it increases attraction.
I think if you want to do the bayesian data analysis, then one of the simplest ways you could model your situation is as follows.
If you PHTG, you achieve sex (or whatever it is you’re after, but I’ll just say sex for simplicity) with probability p∈(0, 1). If you don’t PHTG, you achieve sex with probability q∈(0, 1). Currently, you don’t know the values of q and r but you have a prior distribution p(q, r) over them. In this prior p(q, r), q and r are not necessarily independent. On the opposite, I would expect that they correlate (with respect to the prior p(q, r)) very strongly, because if you often achieve sex with one strategy, probably you’ll also be able to do that with the other strategy, and if you can’t achieve sex with one strategy, probably you can’t with the other. Next, you will go and do the experiments (go on dates and randomly choose whether to PHTG). An experiment is like tossing a biased coin. If you are PHTG, you are tossing a coin which lands on heads with probability q. If you are not PHTG, you are tossing a coin which lands on heads with probability r. After n experimental results e1,…,en, you update your distribution over the values q and r: p(q,r∣e1,…,en)=p(e1,…,en∣q,r)p(q,r)p(e1,…,en)=p(e1∣q,r)p(e2∣q,r)…p(en∣q,r)p(q,r)p(e1,…,en) and this is the result you get. I think this models represents your situation fairly well.
I don’t know what prior p(q, r) you should choose in order to have it fairly close to your actualy beliefs while at the same time making the computation tractable. A simplification you can try is imagining that prior to the experimental data, q and r are totally independent from each other. Then your situation is simply two separate situations, in each you are trying to estimate the biasedness of a coin. Then you take the prior of q to be a beta distribution, and the prior of r to be a beta distribution as well. Then you open “Data analysis a bayesian tutorial—Sivia Skilling” (can be found on libgen) page 14 example 1 “is this a fair coin?” and do whatever it says. Another thing you could probably do is come up with some kind piecewise-constant prior p(q, r) manually and perform the bayesian analysis by simulating everything on the computer rather than tinkering with integrals on paper. Formally, this is called Monte Carlo integration.
Also, instead of treating the outcomes as binary (sex or no-sex), you could treat them as real numbers which represent how well it went. I think this way you’ll need less experiments to get a conclusion. For this case, you can read “Bayesian Estimation Supersedes the t Test—Kruschke 2012”. That paper describes how to do bayesian analysis when you have two groups (treatment and control) and you want to measure what the treatment does if it does anything.
Thanks for pointing me in the right direction with these! My degree is really frequentist and slow paced, excited to get to work on this analysis.
Clarification on null hypothesis:
The null hypothesis is that there is no difference in effect on the dependent variable from the treatment and control variables. I am not assessing the truth of the null hypothesis because if it is true, then I can pick whichever one I want. If control is better, then picking treatment is negative utility. If treatment is better then picking control is negative utility. If the null is true, then I am free to do treatment or control without suffering in either case. Therefore I gain no utility from a test to see if the null is true or not.
Consider shifting your tone when people know less stats than you. Saying ” I suspect you don’t know what a null hypothesis is” makes people feel defensive and not willing to take your useful advice. Try saying “can you clarify what you mean by ____” or “here’s a common definition of a null hypothesis”.
Clarification on dependent variables:
I was going to code the outcome as 1 if sex|second date. The thinking is that only women who are attracted to me have sex with me (but may not want to date, for lots of good reasons). Meanwhile many women who are attracted to me do go on a second date. But few women are attracted to me but do neither sex nor a second date. Since attraction is the concept I want to explain, this should have the best specificity and sensitivity of available measures.
I am considering a second DV using eye contact during date (qualitative) as a robustness check. I think some people do lots of eye contact on all dates as a subconscious influence strategy, so it has more false positives than the other two.
Alright, it seems you do know what a null hypothesis is. Glad I could be of help.