Note that text in pretraining may even be an expensive way to go about it: one of the most dramatic demonstrations MS gave us with Sydney was the incredible speed & efficiency of web-search-powered adversarial attacks on LLMs. You don’t need to dump a lot of samples onto the Internet and pray they make it into the training data and don’t get forgotten, if you can set up a single sample with good SEO and the LLM kindly retrieves it for you and attacks itself with your sample.
This is something to think about: it’s not just making it into the training data, it’s making it into the agent’s prompt or context that can matter. People are currently talking about how Deep Research is an example of the AI trend which will drive paywalls everywhere… which may happen, but consider the positives for people who don’t put up paywalls.
Note that text in pretraining may even be an expensive way to go about it: one of the most dramatic demonstrations MS gave us with Sydney was the incredible speed & efficiency of web-search-powered adversarial attacks on LLMs. You don’t need to dump a lot of samples onto the Internet and pray they make it into the training data and don’t get forgotten, if you can set up a single sample with good SEO and the LLM kindly retrieves it for you and attacks itself with your sample.
This is something to think about: it’s not just making it into the training data, it’s making it into the agent’s prompt or context that can matter. People are currently talking about how Deep Research is an example of the AI trend which will drive paywalls everywhere… which may happen, but consider the positives for people who don’t put up paywalls.