I estimate, that a currently working and growing superintelligence has a probability in a range of 1/million to 1/1000. I am at least 50% confident that it is so.
The probability is already just an expression of your own uncertainty. Giving a confidence interval over the probability does not make sense.
Well, the probability is computed by an algorithm that is itself imperfect. “I’m 50% confident that the probability is 1/1000” means something like “My computation gives a probably of 1/1000, but I’m only 50% confident that I did it right”. For example, if given a complex maths problem about probabilities of getting some card patterns from a deck with twisted rules of drawing and shuffling, you can do that maths, ends up with a probability of 1⁄5 that you’ll get the pattern, but not be confident you didn’t make a mistake in applying the laws of probability, so you’ll only give a 50% confidence to that answer.
And there is also the difference between belief and belief in belief. I can something “I believe the probability to be of 1/1000, but I’m only 50% confident that this is my real belief, and not just a belief in belief”.
Maybe when you say it, that’s what you mean (do you say it?), but that’s pretty weak evidence about what Thomas means. Added: oops, I didn’t mean to revive something two years old.
It sounds to me like he is describing his distribution over probabilities, and estimates at least 50% of the mass of his distribution is between 1⁄1,000 and 1⁄1,000,000. Is that a convenient way to store or deal with probabilities? Not really, no, but I can see why someone would pick it.
The problem with this interpretation is that it renders the initial statement pretty meaningless. Assuming he’s decided to give us a centered 50% confidence interval, which is the only one that really makes sense, that means that 25% of his probability distribution over probabilities is more likely than 1/1000, and this part of the probability mass is going to dominate the rest.
For example, if you think there’s a 25% chance that the “actual probability” (whatever that means) is 0.01, then your best estimate of the “actual probability” has to be at least 0.004, which is significantly more than 1/1000, and even a 1% chance of it being 0.1 would already be enough to move your best estimate above 0.001, so it’s not just that I’m not sure the concept makes sense, it’s that the statement gives us basically no information in the only interpretation in which it does make sense.
Suppose you wanted to make a decision that is equally sensible for P values above X, and not sensible for P values below X. Then, knowing that a chunk of the pdf is below or above X is valuable. (If you only care about whether or not the probability is greater than 1e-3; he’s suggested there’s a less than 50% chance that’s the case).
To elaborate a little more: he’s answered one of the first questions you would ask to determine someone’s pdf for a variable. One isn’t enough; we need two (or hopefully more) answers. But it’s still a good place to start.
I basically agree that the part of the original comment that you quote doesn’t make any sense at all, and am not attempting to come to the defence of confidence intervals over probabilities, but it does feel like there should be some way of giving statements of probability and indicating how sure one is about the statement at the same time. I think, in some sense, I want to be able to say how likely I think it is that I will get new information that will cause me to update away from my current estimate, or give a second-derivative of my uncertainty, if you will.
Let’s say we have two bags, one contains 1 million normal coins, one contains 500,000 2-headed coins and 500,000 2-tailed coins. Now, I draw a coin from the first bag and toss it—I have a 50% chance of getting a head. I draw a coin from the second bag and toss it—I also have a 50% chance of getting a head, but it does feel like there’s some meaningful difference between the two situations. I will admit, though, that I have basically no idea how to formalise this—I assume somebody, somewhere, does.
I agree. Perhaps he means to say that his opinion is based on very little evidence and is “just a hunch”.
I do think that in fitting a model to data, you can give meaningful confidence intervals for parameters of those models which correspond to probabilities (e.g. p(heads) for a particular coin flipping device). But that’s not relevant here.
The probability is already just an expression of your own uncertainty. Giving a confidence interval over the probability does not make sense.
Well, the probability is computed by an algorithm that is itself imperfect. “I’m 50% confident that the probability is 1/1000” means something like “My computation gives a probably of 1/1000, but I’m only 50% confident that I did it right”. For example, if given a complex maths problem about probabilities of getting some card patterns from a deck with twisted rules of drawing and shuffling, you can do that maths, ends up with a probability of 1⁄5 that you’ll get the pattern, but not be confident you didn’t make a mistake in applying the laws of probability, so you’ll only give a 50% confidence to that answer.
And there is also the difference between belief and belief in belief. I can something “I believe the probability to be of 1/1000, but I’m only 50% confident that this is my real belief, and not just a belief in belief”.
Maybe when you say it, that’s what you mean (do you say it?), but that’s pretty weak evidence about what Thomas means. Added: oops, I didn’t mean to revive something two years old.
It sounds to me like he is describing his distribution over probabilities, and estimates at least 50% of the mass of his distribution is between 1⁄1,000 and 1⁄1,000,000. Is that a convenient way to store or deal with probabilities? Not really, no, but I can see why someone would pick it.
The problem with this interpretation is that it renders the initial statement pretty meaningless. Assuming he’s decided to give us a centered 50% confidence interval, which is the only one that really makes sense, that means that 25% of his probability distribution over probabilities is more likely than 1/1000, and this part of the probability mass is going to dominate the rest.
For example, if you think there’s a 25% chance that the “actual probability” (whatever that means) is 0.01, then your best estimate of the “actual probability” has to be at least 0.004, which is significantly more than 1/1000, and even a 1% chance of it being 0.1 would already be enough to move your best estimate above 0.001, so it’s not just that I’m not sure the concept makes sense, it’s that the statement gives us basically no information in the only interpretation in which it does make sense.
Suppose you wanted to make a decision that is equally sensible for P values above X, and not sensible for P values below X. Then, knowing that a chunk of the pdf is below or above X is valuable. (If you only care about whether or not the probability is greater than 1e-3; he’s suggested there’s a less than 50% chance that’s the case).
To elaborate a little more: he’s answered one of the first questions you would ask to determine someone’s pdf for a variable. One isn’t enough; we need two (or hopefully more) answers. But it’s still a good place to start.
If you can have a 95% confidence interval, why can’t you have a >50% confidence interval as well?
50% confidence intervals are standard practice. But not the point and not what I questioned.
There is no way in which my comment can be read which would make your reply make sense in the context.
I basically agree that the part of the original comment that you quote doesn’t make any sense at all, and am not attempting to come to the defence of confidence intervals over probabilities, but it does feel like there should be some way of giving statements of probability and indicating how sure one is about the statement at the same time. I think, in some sense, I want to be able to say how likely I think it is that I will get new information that will cause me to update away from my current estimate, or give a second-derivative of my uncertainty, if you will.
Let’s say we have two bags, one contains 1 million normal coins, one contains 500,000 2-headed coins and 500,000 2-tailed coins. Now, I draw a coin from the first bag and toss it—I have a 50% chance of getting a head. I draw a coin from the second bag and toss it—I also have a 50% chance of getting a head, but it does feel like there’s some meaningful difference between the two situations. I will admit, though, that I have basically no idea how to formalise this—I assume somebody, somewhere, does.
I agree. Perhaps he means to say that his opinion is based on very little evidence and is “just a hunch”.
I do think that in fitting a model to data, you can give meaningful confidence intervals for parameters of those models which correspond to probabilities (e.g. p(heads) for a particular coin flipping device). But that’s not relevant here.