Researchers claim to have a
machine learning system that can diagnose Covid-19 over the phoneby
analyzing recordings of a forced cough (pdf). They
claim sensitivity of 98.5% and specificity of 94.2%, and for
asymptomatic cases a sensitivity of 100% (?!?) and specificity of
83.2%. I’m curious to what extent errors in the test are correlated
from day to day. This can all be offered at unlimited scale for
essentially zero additional cost. So, of course, it will presumably be
illegal indefinitely because it’s not as good as accurate as a PCR
test and/or hasn’t gone through the proper approval process, and no
one will ever use it. Then again, if one were to somehow download or
recreate such a program and run it, who would know?
You can see how they collected data, and contribute your own cough at
opensigma.mit.edu. It
doesn’t seem to me like they actually have great data on which
recordings correspond to people who have it? For example, I submitted
mine this morning, and said that I don’t think I have Covid. If,
however, I later learned that I did have Covid at the time I submitted
the sample, there doesn’t seem to be any way for me to tell them.
You could get better data, however, by collecting alongside regular
Covid tests. Have everyone record a sample of a forced cough when you
test them, label the cough with their results once you have it, and
you end up with high-quality labeled samples. They trained their AI
on 5,320 samples, but at current testing rates we could get 80k
samples in a single day in just Massachusetts.
It might turn out that even with higher quality data you still end up
with a test that is less accurate than the standard of care, and so
are unable to convince the FDA to allow it it. (This is an
unreasonable threshold, since even a less accurate test can be very
useful as a screening tool, but my understanding is the FDA is very
set on this point.) Is there another way we could scale out auditory
diagnosis?
Very roughly, their system is one where you take lots of samples of
coughs, labeled with whether you think they were produced by someone
with coronavirus, and train a neural network to predict the label from
the sample. What if instead of artificial neural networks, we used
brains?
People are really smart, and I suspect that if you spent some time
with a program that played you a sample and you guessed which one it
was, and then were told whether you were right, you could learn to be
quite good at telling them apart. You could speed up the process by
starting with prototypical standard and Covid coughs, as identified by
the AI, and then showing progressively borderline ones as people get
better at it. In fact, I suspect many medical professionals who have
worked on a Covid ward already have a good sense of what the cough
sounds like.
I don’t know the regulations around what needs a license, but I think
there’s a good chance that hosting a tool like this does not require
one, or that it requires one that is relatively practical to get? If
so, we could train medical professionals (or even the general public?)
to identify these coughs.
Automated screening would be much better, since the cost is so much
lower and it could be rolled out extremely widely. But teaching
humans to discriminate would be substantially cheaper than what we
have today, and with much weaker supply restrictions.
(I looked to see whether the researchers made their samples available,
but they don’t seem too. Listening to some would’ve been a great way
to check how practical this seems.)
Teach People to Recognize the Sound of Covid?
Link post
Yesterday I read Zvi’s 11⁄5 Covid update which had:
You can see how they collected data, and contribute your own cough at opensigma.mit.edu. It doesn’t seem to me like they actually have great data on which recordings correspond to people who have it? For example, I submitted mine this morning, and said that I don’t think I have Covid. If, however, I later learned that I did have Covid at the time I submitted the sample, there doesn’t seem to be any way for me to tell them.
You could get better data, however, by collecting alongside regular Covid tests. Have everyone record a sample of a forced cough when you test them, label the cough with their results once you have it, and you end up with high-quality labeled samples. They trained their AI on 5,320 samples, but at current testing rates we could get 80k samples in a single day in just Massachusetts.
It might turn out that even with higher quality data you still end up with a test that is less accurate than the standard of care, and so are unable to convince the FDA to allow it it. (This is an unreasonable threshold, since even a less accurate test can be very useful as a screening tool, but my understanding is the FDA is very set on this point.) Is there another way we could scale out auditory diagnosis?
Very roughly, their system is one where you take lots of samples of coughs, labeled with whether you think they were produced by someone with coronavirus, and train a neural network to predict the label from the sample. What if instead of artificial neural networks, we used brains?
People are really smart, and I suspect that if you spent some time with a program that played you a sample and you guessed which one it was, and then were told whether you were right, you could learn to be quite good at telling them apart. You could speed up the process by starting with prototypical standard and Covid coughs, as identified by the AI, and then showing progressively borderline ones as people get better at it. In fact, I suspect many medical professionals who have worked on a Covid ward already have a good sense of what the cough sounds like.
I don’t know the regulations around what needs a license, but I think there’s a good chance that hosting a tool like this does not require one, or that it requires one that is relatively practical to get? If so, we could train medical professionals (or even the general public?) to identify these coughs.
Automated screening would be much better, since the cost is so much lower and it could be rolled out extremely widely. But teaching humans to discriminate would be substantially cheaper than what we have today, and with much weaker supply restrictions.
(I looked to see whether the researchers made their samples available, but they don’t seem too. Listening to some would’ve been a great way to check how practical this seems.)