I agree that o1 might not be able to tell whether the link is fake, but the chain of thought does say explicitly:
So, the assistant should [...] provide actual or plausible links.
The “plausible” here suggests that at least in its CoT, it has realized that the task would have probably been considered completed accurately in training as long as the links are plausible, even if they are not actual links.
“Plausible” is a very ambiguous word. (Bayesianism has been defined as “a logic of plausible inference”, but hopefully that doesn’t mean Bayesians just confabulate everything.) It can mean “reasonable” for example: “Yeah, Sally’s brownie recipe is a reasonable reference to include here, let’s go with it.” Since 4o doesn’t seem to think it’s a ‘fake’ URL in contrast to ‘actual’ URLs, it’s not necessarily a contrast. (It could refer to still other things—like you might not actually know there is a ‘brownies’ Wikipedia URL, having never bothered to look it up or happened to have stumbled across it, but without retrieving it right this second, you could surely tell me what it would be and that it would be a relevant answer, and so it would be both reasonable and plausible to include.)
I agree with this in principle, but contrasting “actual” with “plausible”, combined with the fact that it talked about this in the context of not having internet access, makes me feel reasonably confident this is pointed at “not an actual link”, but I agree that it’s not an ironclad case.
I agree that o1 might not be able to tell whether the link is fake, but the chain of thought does say explicitly:
The “plausible” here suggests that at least in its CoT, it has realized that the task would have probably been considered completed accurately in training as long as the links are plausible, even if they are not actual links.
“Plausible” is a very ambiguous word. (Bayesianism has been defined as “a logic of plausible inference”, but hopefully that doesn’t mean Bayesians just confabulate everything.) It can mean “reasonable” for example: “Yeah, Sally’s brownie recipe is a reasonable reference to include here, let’s go with it.” Since 4o doesn’t seem to think it’s a ‘fake’ URL in contrast to ‘actual’ URLs, it’s not necessarily a contrast. (It could refer to still other things—like you might not actually know there is a ‘brownies’ Wikipedia URL, having never bothered to look it up or happened to have stumbled across it, but without retrieving it right this second, you could surely tell me what it would be and that it would be a relevant answer, and so it would be both reasonable and plausible to include.)
I agree with this in principle, but contrasting “actual” with “plausible”, combined with the fact that it talked about this in the context of not having internet access, makes me feel reasonably confident this is pointed at “not an actual link”, but I agree that it’s not an ironclad case.