The dedicated employees of Google and dedicated editors of Wikipedia do a lot of work to make sure that it is as little of a problem as it is—and there are still persistent issues. If it takes off, the dedicated employees of PatientsLikeMe will have to expect to do the same, and I will expect problems to fall through the cracks the same way they do on Wikipedia and Google.
Strategies for gaming search engines are relatively easy to automatically detect and counteract, once you know what they are. On Wikipedia, a person who knows the subject in question will easily notice fradulent or incorrect information. But I’m not sure what criteria you could use to clearly detect cheaters on a site like this. You could ignore outliers, but that risks losing information from real people who actually have an unusual reaction to the medication. Even if you accepted that as necessary sacrifice, nothing’s to prevent the cheaters from creating enough accounts to make them into non-outliers.
I intuitively agree with your fears, with a mild caveat or two; how much of my fear comes from not respecting doctors? How much comes from boo lights for social networking? Or did I have applause lights and am overcompensating for them?
Since I have no idea what your sentence means past “I fear...Google’s problem” I think a lot of my agreement is bias. What might help is a better explanation of what the persistent factor is and how it would get corrupted.
Of course generally treating information as information and science as science is good, but making fast medical decisions involving lots of money and major health issues seems like a good place to introduce new information that could help lots of people. Skepticism is good but barriers to use and acceptance could be harmful.
“Google’s problem” = “Search engine optimization” = people with an agenda trying to game the algorithm (so their site gets ranked higher than better ones). For example, linkspam makes “number of inbound links” a less reliable metric of site quality than it would otherwise be.
If something is known to be used as a proxy for quality and people are rewarded accordingly, then you’ll end up with people trying to achieve the proxy for quality at the expense of actual quality.
I’m not sure what that has to do with the original topic, though. Are you anticipating that quacks will go on sites like this and say “You should buy my snake oil—look at all these sockpuppets that it’s helped!”
If something is known to be used as a proxy for quality and people are rewarded accordingly, then you’ll end up with people trying to achieve the proxy for quality at the expense of actual quality.
But isn’t that exactly what happens in science too? Citation stat gaming, ghostwritten papers, senior person’s name first, etc.?
And those kinds of gaming are much harder for outsiders to detect, as well as harder to fix or regulate, especially when they’re the commonly accepted official practices.
So even if the public data aggregation approach is more open to gaming, it doesn’t necessarily follow that it’s more vulnerable to gaming. Gaming might also be much easier to detect and/or eliminate by automated or semi-automated means.
I’m not sure what that has to do with the original topic, though. Are you anticipating that quacks will go on sites like this and say “You should buy my snake oil—look at all these sockpuppets that it’s helped!”
I fear this might suffer from Google’s problem: once it becomes relied on, the persistent factor which was relied upon gets corrupted.
You can treat it as information, but not as science—it’s not protected from malice the same way scientific studies are.
Google is still pretty reliable—as is Wikipedia, despite warnings that “anybody can edit” would lead to noise and inaccuracy.
The dedicated employees of Google and dedicated editors of Wikipedia do a lot of work to make sure that it is as little of a problem as it is—and there are still persistent issues. If it takes off, the dedicated employees of PatientsLikeMe will have to expect to do the same, and I will expect problems to fall through the cracks the same way they do on Wikipedia and Google.
Strategies for gaming search engines are relatively easy to automatically detect and counteract, once you know what they are. On Wikipedia, a person who knows the subject in question will easily notice fradulent or incorrect information. But I’m not sure what criteria you could use to clearly detect cheaters on a site like this. You could ignore outliers, but that risks losing information from real people who actually have an unusual reaction to the medication. Even if you accepted that as necessary sacrifice, nothing’s to prevent the cheaters from creating enough accounts to make them into non-outliers.
Scientific studies aren’t protected from malice the way PatientsLikeMe is either.
I intuitively agree with your fears, with a mild caveat or two; how much of my fear comes from not respecting doctors? How much comes from boo lights for social networking? Or did I have applause lights and am overcompensating for them?
Since I have no idea what your sentence means past “I fear...Google’s problem” I think a lot of my agreement is bias. What might help is a better explanation of what the persistent factor is and how it would get corrupted.
Of course generally treating information as information and science as science is good, but making fast medical decisions involving lots of money and major health issues seems like a good place to introduce new information that could help lots of people. Skepticism is good but barriers to use and acceptance could be harmful.
“Google’s problem” = “Search engine optimization” = people with an agenda trying to game the algorithm (so their site gets ranked higher than better ones). For example, linkspam makes “number of inbound links” a less reliable metric of site quality than it would otherwise be.
If something is known to be used as a proxy for quality and people are rewarded accordingly, then you’ll end up with people trying to achieve the proxy for quality at the expense of actual quality.
I’m not sure what that has to do with the original topic, though. Are you anticipating that quacks will go on sites like this and say “You should buy my snake oil—look at all these sockpuppets that it’s helped!”
But isn’t that exactly what happens in science too? Citation stat gaming, ghostwritten papers, senior person’s name first, etc.?
And those kinds of gaming are much harder for outsiders to detect, as well as harder to fix or regulate, especially when they’re the commonly accepted official practices.
So even if the public data aggregation approach is more open to gaming, it doesn’t necessarily follow that it’s more vulnerable to gaming. Gaming might also be much easier to detect and/or eliminate by automated or semi-automated means.
I’m saying it’s definitely a concern.
Okay, that clarifies things.