I’m still fairly skeptical that algorithmically fact-checking anything complex is tractable today. The Google article states that “this is 100 percent theoretical: It’s a research paper, not a product announcement or anything equally exciting.” Also, no real insights into nlp are presented; the article only suggests that an algorithm could fact check relatively simple statements that have clear truth values by checking a large database of information. So if the database has nothing to say about the statement, the algorithm is useless. In particular, such an approach would be unable to fact-check the Fiorina quote you used as an example.
It depends what level of fact checking is needed. Watson is well-suited for answering questions like “What year was Obama born?”, because the answer is unambiguous and also fairly likely to be found in a database. I would be very surprised if Watson could fact check a statement like “Putin has absolutely no respect for President Obama”, because the context needed to evaluate such a statement is not so easy to search for and interpret.
“Putin has absolutely no respect for President Obama”, because the context needed to evaluate such a statement is not so easy to search for and interpret.
I’m not sure that a statement like that has to tagged as a falsehood. I would be fine with a fact checker that focuses on statements that are more clearly false.
I think the standard for accuracy would be very different. If Watson gets something right you think “Wow that was so clever”, if it’s wrong you’re fairly forgiving. On that other hand, I feel like if an automated fact checker got even 1⁄10 things wrong it would be subject to insatiable rage for doing so. I think specifically correcting others is the situation in which people would have the highest standard for accuracy.
And that’s before you get into the levels of subjectivity and technicality in the subject matter which something like Watson would never be subjected to.
I think the standard for accuracy would be very different. If Watson gets something right you think “Wow that was so clever”, if it’s wrong you’re fairly forgiving.
Given that Watson get’s used to make medical decisions about how to cure cancer I don’t think people are strongly forgiving.
Yes, because Watson’s corpus doesn’t contain people lying. On the other hand, for political fact-checking the corpus is going to have tons of lies, half-truth, and BS.
It would still be helpful to have automatic fact-checking of simple statements. Consider this Hacker News thread—two people are arguing about crime rates in the UK and USA. Someone says “The UK is a much more violent society than the US” and they argue about that, neither providing citations. That might be simple enough that natural language processing could parse it and check it against various interpretations of it. For example, one could imagine a bot that notices when people are arguing over something like that (whether on the internet or in a national election. It would provide useful relevant statistics, like the total violent crime rates in each country, or the murder rate, or whatever it thinks is relevant. If it were an ongoing software project, the programmers could notice when it’s upvoted and downvoted, and improve it.
I’m still fairly skeptical that algorithmically fact-checking anything complex is tractable today. The Google article states that “this is 100 percent theoretical: It’s a research paper, not a product announcement or anything equally exciting.” Also, no real insights into nlp are presented; the article only suggests that an algorithm could fact check relatively simple statements that have clear truth values by checking a large database of information. So if the database has nothing to say about the statement, the algorithm is useless. In particular, such an approach would be unable to fact-check the Fiorina quote you used as an example.
Do you think fact checking is an inherently more difficult problem then what Watson can do?
It depends what level of fact checking is needed. Watson is well-suited for answering questions like “What year was Obama born?”, because the answer is unambiguous and also fairly likely to be found in a database. I would be very surprised if Watson could fact check a statement like “Putin has absolutely no respect for President Obama”, because the context needed to evaluate such a statement is not so easy to search for and interpret.
I’m not sure that a statement like that has to tagged as a falsehood. I would be fine with a fact checker that focuses on statements that are more clearly false.
I think the standard for accuracy would be very different. If Watson gets something right you think “Wow that was so clever”, if it’s wrong you’re fairly forgiving. On that other hand, I feel like if an automated fact checker got even 1⁄10 things wrong it would be subject to insatiable rage for doing so. I think specifically correcting others is the situation in which people would have the highest standard for accuracy.
And that’s before you get into the levels of subjectivity and technicality in the subject matter which something like Watson would never be subjected to.
Given that Watson get’s used to make medical decisions about how to cure cancer I don’t think people are strongly forgiving.
Yes, because Watson’s corpus doesn’t contain people lying. On the other hand, for political fact-checking the corpus is going to have tons of lies, half-truth, and BS.
It would still be helpful to have automatic fact-checking of simple statements. Consider this Hacker News thread—two people are arguing about crime rates in the UK and USA. Someone says “The UK is a much more violent society than the US” and they argue about that, neither providing citations. That might be simple enough that natural language processing could parse it and check it against various interpretations of it. For example, one could imagine a bot that notices when people are arguing over something like that (whether on the internet or in a national election. It would provide useful relevant statistics, like the total violent crime rates in each country, or the murder rate, or whatever it thinks is relevant. If it were an ongoing software project, the programmers could notice when it’s upvoted and downvoted, and improve it.
This is harder than it seems. The two countries use different methodologies to collect their crime statistics.
Yes, you’d want to use the International Crime Victims Survey. It’s the standard way to compare crime rates between countries.