Your approach to blinding makes sense, and works. I thought we were trying for a zero third party approach though?
I was giving more thought to a distributed solution during dinner, and I think I see how to solve the physical shipments problem in a scalable way. I’m still not 100% sold on it, but consider these two options:
You ship both a placebo package and a non-placebo package to the participant, and have them flip a coin to decide which one to use. They either throw away or disregard the other package for the duration of the study.
You ship N packages to Total/N participants. The participants which receive N packages then randomly assigns himself a package, and randomly distributes the remaining (N-1) packages to other participants.
They both require trusting the participant with assignment. Which feels wrong to me, but I’m not sure why...
I have thought a bit more about the blinding issue.
One question that comes to mind:
How do we trust that the placebo is a real placebo and substantially different then the drug? The company producing the product wants to show a difference. Therefore they have no incentive to give both parties the same product.
On the other hand the company could mix some slight poison into the placebo. Even an ineffective drug beats a poison.
Therefore the placebo has to be produced or purchased by a trusted organisation and that organisation has to package the placebo in the same box that it’s packaging the drug.
This is awesomely paranoid. Thank you for pointing this out.
I’m a little worried a solution here will call for whoever controls the webapp to also be an expert at creating placebos for every product type. (If we trust contract manufacturers to be honest, then the issue of adding poisons to a placebo can be handled by having them ship directly to the third party for mailing… but I that’s already the default case).
Perhaps poisons can be discovered by looking at other products which performed the same protocol? “This experiment has to be re-done because the control group mysteriously got sick” doesn’t seem like a good solution though...
I’ll wrestle with this. Maybe something with MaxL’s answer to #8 might be possible?
I’m a little worried a solution here will call for whoever controls the webapp to also be an expert at creating placebos for every product type.
How about… company with product type X suggests placebo Y. Webapp/process owner confirms suitability of placebo Y with unaffiliated/blinded subject matter expert in the field of product X. If confirmed as suitable, placebo is produced by unaffiliated external company (who doesn’t know what the placebo is intended for, only the formulation of requested items).
Alternately, the webapp/process owner could produce the confirmed placebo, but I’m not sure if this makes sense cost-wise, and also it may open the company up to accusations of corruption, because the webapp/process owner is not blinded to who the recipient company is, and therefore might collude.
You ship both a placebo package and a non-placebo package to the participant, and have them flip a coin to decide which one to use. They either throw away or disregard the other package for the duration of the study.
That’s possible but it means that you double your product costs. The advantage would be that you can do crossover trials.
In the case of your probiotic a crossover trial might to worthwhile even without this reason.
In this case you could encrypt a list with products ID for placebos and non-placebos before you ship your product. The participant has no opportunity to know which of the two products are placebos.
When he starts the study the participant puts the product ID of the product he decides to use into the webapp. He also puts the ID of the product he doesn’t want to use in the webapp.
The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn’t need to go through the process of flipping a real coin.
Once the study is finished you releases the decryption key that can be used to distinguish placebos from real products. You can give the decryption key to a trusted third-party organisation so that your company can’t prevent the results of the study from being published.
You ship N packages to Total/N participants. The participants which receive N packages then randomly assigns himself a package, and randomly distributes the remaining (N-1) packages to other participants.
That means the participants has to do the work of going to the post office and remailing packages. Some of them will require additional time to remail and it might produce complications.
About the shipping of products and placebos to people, I see a physical way of doing it, but it is definitely not scalable.
Maybe you can win a company such as Amazon as a partner for distribution. Amazon warehouses can store both products and placebos and ship randomly.
For Amazon it should be relatively little work and it could be good PR.
Labels don’t have to be printed on the bottle. Amazon has the capabilities of adding a paper with study instructions to a shipment and that paper can contain a unique ID for the product. Amazon could again publish an encrypted version of the ID at the beginning of the study and release the decryption key at the end of the study.
The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn’t need to go through the process of flipping a real coin.
I am worried that standardized shipping will come with standardized package layout, and I’m guessing “preference of left vs right identical thing” correlates with something the system will eventually test. Having thought about it more, this is the real issue with allowing customers to choose which product they’ll use: that decision has to be purely random if you want the math to be simple / understandable. I agree people are unlikely to actually flip a coin :/
Thankfully the fix is easy: you have the testing webapp decide for the participant. They receive the product, enter the numbers online, and are told which to use.
For non-crossover trials I agree this needlessly increases the cost. It’s almost surely better to use a trusted third party.
That means the participants has to do the work of going to the post office and remailing packages. Some of them will require additional time to remail and it might produce complications.
Agree. I think having a trusted third party handle the shipments is cleaner at the moment. I’m still curious what blogospheroid’s thread comes to. It seems like the paranoia of cryptoland is helping us see some more holes in modern experiment design (ie your thread on poisonous placebos).
Maybe you can win a company such as Amazon as a partner for distribution. Amazon warehouses can store both products and placebos and ship randomly.
Thanks for saying this. Looking closer, I actually think their existing Fulfillment APIs would just work for this (ie the webapp controls an Amazon fulfillment account, the person seeking a test ships two pallets of physical product there, the webapp says where to send them).
Looking closer, I actually think their existing Fulfillment APIs would just work for this (ie the webapp controls an Amazon fulfillment account, the person seeking a test ships two pallets of physical product there, the webapp says where to send them).
You are right, if we already have hosted our webapp at trust-place we should be able to use the existing Amazon API.
If the company whose product is tested simply ships additional copies to the Amazon warehouse, those copies could by achieved by the trusted organisation. If anybody doubts that the products are real the trusted organisation has copies that they can analyse.
If the whole things scales the trusted organisation also can randomly inspect products to see if they contain what they should contain.
Agree. I think having a trusted third party handle the shipments is cleaner at the moment. I’m still curious what blogospheroid’s thread comes to. It seems like the paranoia of cryptoland is helping us see some more holes in modern experiment design
Yes cryptoparanoia is always fun ;) The web app could regularly publish hashes of the data of specific studies to a public block chain. That way any tempering that happens afterwards can be detected and you only need to trust that the web app is temper proof the moment the data gets transmitted.
If anybody doubts that the products are real the trusted organisation has copies that they can analyse.
This is a great point. Maybe community members could bet karma on the outcome of a tox screening? This could create a prioritized list.
One problem with my earlier suggestion is that some companies will want narrowly selected participant pools. These will necessarily differ from the population at large, and might create data that looks like a poison placebo is being used. I see two possible solutions to this problem:
Log baseline data before the treatment is used. If people do worse on the placebo, that would be very suspicious.
Include an additional group of testers that do something different not related to the placebo/product. “Eat an apple every day for the next week”. If the placebo group did worse than the apple group, that would be very suspicious.
I feel like #2 from above is unsatisfying though, if we think it works then why are we using normal placebos?
The web app could regularly publish hashes of the data of specific studies to a public block chain. That way any tempering that happens afterwards can be detected and you only need to trust that the web app is temper proof the moment the data gets transmitted.
This would actually be really easy to implement. (Not the block chain portion, the per-study rolling checksums).
Your approach to blinding makes sense, and works. I thought we were trying for a zero third party approach though?
I was giving more thought to a distributed solution during dinner, and I think I see how to solve the physical shipments problem in a scalable way. I’m still not 100% sold on it, but consider these two options:
You ship both a placebo package and a non-placebo package to the participant, and have them flip a coin to decide which one to use. They either throw away or disregard the other package for the duration of the study.
You ship N packages to Total/N participants. The participants which receive N packages then randomly assigns himself a package, and randomly distributes the remaining (N-1) packages to other participants.
They both require trusting the participant with assignment. Which feels wrong to me, but I’m not sure why...
I have thought a bit more about the blinding issue.
One question that comes to mind: How do we trust that the placebo is a real placebo and substantially different then the drug? The company producing the product wants to show a difference. Therefore they have no incentive to give both parties the same product.
On the other hand the company could mix some slight poison into the placebo. Even an ineffective drug beats a poison.
Therefore the placebo has to be produced or purchased by a trusted organisation and that organisation has to package the placebo in the same box that it’s packaging the drug.
This is awesomely paranoid. Thank you for pointing this out.
I’m a little worried a solution here will call for whoever controls the webapp to also be an expert at creating placebos for every product type. (If we trust contract manufacturers to be honest, then the issue of adding poisons to a placebo can be handled by having them ship directly to the third party for mailing… but I that’s already the default case).
Perhaps poisons can be discovered by looking at other products which performed the same protocol? “This experiment has to be re-done because the control group mysteriously got sick” doesn’t seem like a good solution though...
I’ll wrestle with this. Maybe something with MaxL’s answer to #8 might be possible?
How about… company with product type X suggests placebo Y. Webapp/process owner confirms suitability of placebo Y with unaffiliated/blinded subject matter expert in the field of product X. If confirmed as suitable, placebo is produced by unaffiliated external company (who doesn’t know what the placebo is intended for, only the formulation of requested items).
Alternately, the webapp/process owner could produce the confirmed placebo, but I’m not sure if this makes sense cost-wise, and also it may open the company up to accusations of corruption, because the webapp/process owner is not blinded to who the recipient company is, and therefore might collude.
That’s possible but it means that you double your product costs. The advantage would be that you can do crossover trials.
In the case of your probiotic a crossover trial might to worthwhile even without this reason.
In this case you could encrypt a list with products ID for placebos and non-placebos before you ship your product. The participant has no opportunity to know which of the two products are placebos.
When he starts the study the participant puts the product ID of the product he decides to use into the webapp. He also puts the ID of the product he doesn’t want to use in the webapp.
The participant has no way to decide between the two packages or know which one is the placebo and which one is the real thing so he doesn’t need to go through the process of flipping a real coin.
Once the study is finished you releases the decryption key that can be used to distinguish placebos from real products. You can give the decryption key to a trusted third-party organisation so that your company can’t prevent the results of the study from being published.
That means the participants has to do the work of going to the post office and remailing packages. Some of them will require additional time to remail and it might produce complications.
Maybe you can win a company such as Amazon as a partner for distribution. Amazon warehouses can store both products and placebos and ship randomly.
For Amazon it should be relatively little work and it could be good PR.
Labels don’t have to be printed on the bottle. Amazon has the capabilities of adding a paper with study instructions to a shipment and that paper can contain a unique ID for the product. Amazon could again publish an encrypted version of the ID at the beginning of the study and release the decryption key at the end of the study.
I am worried that standardized shipping will come with standardized package layout, and I’m guessing “preference of left vs right identical thing” correlates with something the system will eventually test. Having thought about it more, this is the real issue with allowing customers to choose which product they’ll use: that decision has to be purely random if you want the math to be simple / understandable. I agree people are unlikely to actually flip a coin :/
Thankfully the fix is easy: you have the testing webapp decide for the participant. They receive the product, enter the numbers online, and are told which to use.
For non-crossover trials I agree this needlessly increases the cost. It’s almost surely better to use a trusted third party.
Agree. I think having a trusted third party handle the shipments is cleaner at the moment. I’m still curious what blogospheroid’s thread comes to. It seems like the paranoia of cryptoland is helping us see some more holes in modern experiment design (ie your thread on poisonous placebos).
Thanks for saying this. Looking closer, I actually think their existing Fulfillment APIs would just work for this (ie the webapp controls an Amazon fulfillment account, the person seeking a test ships two pallets of physical product there, the webapp says where to send them).
You are right, if we already have hosted our webapp at trust-place we should be able to use the existing Amazon API.
If the company whose product is tested simply ships additional copies to the Amazon warehouse, those copies could by achieved by the trusted organisation. If anybody doubts that the products are real the trusted organisation has copies that they can analyse. If the whole things scales the trusted organisation also can randomly inspect products to see if they contain what they should contain.
Yes cryptoparanoia is always fun ;) The web app could regularly publish hashes of the data of specific studies to a public block chain. That way any tempering that happens afterwards can be detected and you only need to trust that the web app is temper proof the moment the data gets transmitted.
This is a great point. Maybe community members could bet karma on the outcome of a tox screening? This could create a prioritized list.
One problem with my earlier suggestion is that some companies will want narrowly selected participant pools. These will necessarily differ from the population at large, and might create data that looks like a poison placebo is being used. I see two possible solutions to this problem:
Log baseline data before the treatment is used. If people do worse on the placebo, that would be very suspicious.
Include an additional group of testers that do something different not related to the placebo/product. “Eat an apple every day for the next week”. If the placebo group did worse than the apple group, that would be very suspicious.
I feel like #2 from above is unsatisfying though, if we think it works then why are we using normal placebos?
This would actually be really easy to implement. (Not the block chain portion, the per-study rolling checksums).