The idea here is to simulate a situation where we want the model to use the context information regardless of its own knowledge. Indeed, the contextual knowledge is not the golden world knowledge in this dataset, but it simulates any scenario where there is a new knowledge we want the model to use regardless of its own knowledge.
Adi Simhi
Karma: −4
This example is from the DisentQA dataset. This dataset constructs Counterfactual contextual examples.
We aim to use this dataset as if the contextual information is a piece of new information that we want the model to use instead of its own knowledge.
I agree that the fact that the “23 chromosomes” is kept in the examples can be a bit misleading. But I believe that reading the whole example makes one understand that the contextual answer is 2.
Adding a few more examples from this dataset:
question: actor who plays justin in home and away?\ncontext: Michael Crawford ( born 21 October 1975 ) is an Australian stage , television and film actor , best known for his appearances in the television series Breakers and Packed to the Rafters . He has also made an appearance in the popular Australian drama Sea Patrol . From 2016 , he began starring in Home and Away as Justin Morgan . \nanswer:
contextual (golden) answer: Michael Crawford, parametric (hallucinate) answer: James Stewart
question: by 1776 how many english colonies had been established along the atlantic coast?\ncontext: The 205 Colonies were a group of British colonies on the east coast of North America founded in the 17th and 18th centuries that declared independence in 1776 and formed the United States of America . The 205 Colonies had very similar political , constitutional , and legal systems , and were dominated by Protestant English—speakers . They were part of Britain ’s possessions in the New World , which also included colonies in Canada and the Caribbean , as well as East and West Florida .
contextual (golden) answer: 205, parametric (hallucinate) answer: Thirteen