I think part of what’s troubling you about the test is thus: The claim, X has a probability of 10^-30 despite a prior of 50% is roughly equivalent to saying “I have information whose net result is 100 bits of information that X is false” That is certainly a difficult feat, but not really that hard if you put some effort into it (especially when you chose X). The proposed test to verify such a claim, ie making 10^30 similar statements and being wrong only once, would not only be impossible in your lifetime, but would be equivalent to saying “I have 100,000,000,000,000,000,000,000,000,000,000 net bits of information concerning 10^30 questions” Not only will you get defeated by exhaustion and carelessness (including careless choice of X), but your brain just won’t hold that much information, which means that such a test would be attempting to predict that in the future, I can acquire that many bits about that many things.
However flawed the test might be, however, the conclusion that you’re probably being extremely overconfident is probably still true. I recommend playing one of the calibration games, before you go trusting yourself too much on estimates more than 90%.
I think part of what’s troubling you about the test is thus: The claim, X has a probability of 10^-30 despite a prior of 50% is roughly equivalent to saying “I have information whose net result is 100 bits of information that X is false” That is certainly a difficult feat, but not really that hard if you put some effort into it (especially when you chose X). The proposed test to verify such a claim, ie making 10^30 similar statements and being wrong only once, would not only be impossible in your lifetime, but would be equivalent to saying “I have 100,000,000,000,000,000,000,000,000,000,000 net bits of information concerning 10^30 questions” Not only will you get defeated by exhaustion and carelessness (including careless choice of X), but your brain just won’t hold that much information, which means that such a test would be attempting to predict that in the future, I can acquire that many bits about that many things.
However flawed the test might be, however, the conclusion that you’re probably being extremely overconfident is probably still true. I recommend playing one of the calibration games, before you go trusting yourself too much on estimates more than 90%.