If they were to exclude all documents with the canary, everyone would include the canary to avoid being scraped.
Which would be a good thing as nominally they claim to let everyone opt out of scraping already by using robots.txt and other methods, and so the canary shouldn’t do anything there that people couldn’t already do.
robots.txt
If they were to exclude all documents with the canary, everyone would include the canary to avoid being scraped.
Which would be a good thing as nominally they claim to let everyone opt out of scraping already by using
robots.txtand other methods, and so the canary shouldn’t do anything there that people couldn’t already do.