If you just prompt “Are bugs real?”, text-davinci-002 will give:
[38.46% probability] “Yes” which usually leads to “Yes, bugs are real,” though with 1% chance you get something more like “Yes, insects are real. They are small animals that have six legs...”
[35.53% probability] “There” which usually leads to a variant of “There is no simple answer to this question… If you mean X, then the answer is yes. If you mean Y, then the answer is also yes.”
[12% probability] “B” which always leads to “Bugs are real.”
I just replicated this. I think it would be pretty helpful to add this 36% number to the OP, after reading the OP my estimate for this answer was more like 90%.
(I think it’s still a very high result for a blatantly silly and probably adversarially selected answer, but 36% vs 90% is a distinction at least in how embarrassingly reliable this answer is.)
I just replicated this. I think it would be pretty helpful to add this 36% number to the OP, after reading the OP my estimate for this answer was more like 90%.
(I think it’s still a very high result for a blatantly silly and probably adversarially selected answer, but 36% vs 90% is a distinction at least in how embarrassingly reliable this answer is.)