Obviously this correction is relatively most important when the point estimate of the sensitivity/specificity is 100%, making the corresponding Bayes factor meaningless. Do you have a sense of how important the correction is for smaller values / how small the value can be before it’s fine to just ignore the correction? I assume everything is pulled away from extreme values slightly, but in general not by enough to matter.
Simple answer first: If the sensitivity and specificity are estimated with data from studies with large (>1000) sample sizes it mostly won’t matter.
Various details:
Avoiding point estimates altogether will get you broader estimates of the information content of the tests, regardless of whether you arrive at those point estimates from Bayesian or frequentist methods.
Comparing the two methods, the Bayesian one will pull very slightly towards 50% relative to simply taking the sample rate as the true rate. Indeed, it’s equivalent to adding a single success and failure to the sample and just computing the rate of correct identification in the sample.
The parameters of a Beta distribution can be interpreted as the total number of successes and failures, combining the prior and observed data to get you the posterior.
Thanks, I was wondering if the answer would be something like this (basically that I should be using a distribution rather than a point estimate, something that @gwillen also mentioned when he reviewed the draft version of this point).
If the sensitivity and specificity are estimated with data from studies with large (>1000) sample sizes it mostly won’t matter.
That’s the case for the antigen test data; the sample sizes are >1000 for each subgroup analyzed (asymptomatic, symptoms developed <1 week ago, symptoms developed >1 week ago).
The sample size for all NAATs was 4351, but the sample size for the subgroups of Abbot ID Now and Cepheid Xpert Xpress were only 812 and 100 respectively. Maybe that’s a small enough sample size that I should be suspicious of the subgroup analyses? (@JBlack mentioned this concern below and pointed out that for the Cepheid test, there were only 29 positive cases total).
Obviously this correction is relatively most important when the point estimate of the sensitivity/specificity is 100%, making the corresponding Bayes factor meaningless. Do you have a sense of how important the correction is for smaller values / how small the value can be before it’s fine to just ignore the correction? I assume everything is pulled away from extreme values slightly, but in general not by enough to matter.
Simple answer first: If the sensitivity and specificity are estimated with data from studies with large (>1000) sample sizes it mostly won’t matter.
Various details:
Avoiding point estimates altogether will get you broader estimates of the information content of the tests, regardless of whether you arrive at those point estimates from Bayesian or frequentist methods.
Comparing the two methods, the Bayesian one will pull very slightly towards 50% relative to simply taking the sample rate as the true rate. Indeed, it’s equivalent to adding a single success and failure to the sample and just computing the rate of correct identification in the sample.
The parameters of a Beta distribution can be interpreted as the total number of successes and failures, combining the prior and observed data to get you the posterior.
Thanks, I was wondering if the answer would be something like this (basically that I should be using a distribution rather than a point estimate, something that @gwillen also mentioned when he reviewed the draft version of this point).
That’s the case for the antigen test data; the sample sizes are >1000 for each subgroup analyzed (asymptomatic, symptoms developed <1 week ago, symptoms developed >1 week ago).
The sample size for all NAATs was 4351, but the sample size for the subgroups of Abbot ID Now and Cepheid Xpert Xpress were only 812 and 100 respectively. Maybe that’s a small enough sample size that I should be suspicious of the subgroup analyses? (@JBlack mentioned this concern below and pointed out that for the Cepheid test, there were only 29 positive cases total).