This doesn’t require omniscience, or AI: people do this now (based on info they have). If you have more info, we know how to use it (there is theory). Why are we talking about AI, this is a math problem.
Slightly harshly worded suggestion (not to you specifically): maybe more reading, less invocation of robo-Jesus in vain.
Eliezer claims that randomness is always bad; many other people claim that one way randomness is good is that it is unbiased. Partitioning subjects into experimental conditions must be unbiased. Using an algorithm and knowing that its biases are orthogonal to the phenomenon being investigated requires omniscience. Besides, if you knew in advance what was relevant, you wouldn’t need to do the experiment.
That is what the comment means. The use of the term “AI” is just to show that the claim is that no real-world agent can be smart enough to do unbiased partitioning in all cases, not just that we’re not smart enough to do it.
In practice, a possibly biased but intelligent partitioning is better when the sample size is small.
We know what property we want (that randomization will give you), good balance in relevant covariates between two groups. I can use a deterministic algorithm for this, and in fact people do, e.g. matching algorithms. Another thing people do is try all possible assignments (see: permutation tests for the null).
Discussion of AI and omniscience is a complete red herring, you don’t need that to show that you don’t need randomness for this. We aren’t randomizing for the sake of randomizing, we are doing it because we want some property that we can directly target deterministically.
I don’t think EY can possibly know enough math to make his claim go through, I think this is an “intellectual marketing” claim. People do this a lot, if we are talking about your claim, you won the game.
If you sort all the subjects on one criteria, it may be correlated in an unexpected way with another criteria you’re unaware of. Suppose you want to study whether licorice causes left-handedness in a population from Tonawanda, NY. So you get a list of addresses from Tonawanda New York, sort them by address, and go down the list throwing them alternately into control and experimental group. Then you mail the experimental group free licorice for a ten years. Voila, after 10 years there are more left-handers in the experimental group.
But even and odd addresses are on opposite sides of the street. And it so happens that in Tonawanda, NY, the screen doors on the front of every house are hinged on the west side, regardless of which way the house faces, because the west wind is so strong it would rip the door off its hinges otherwise. So people on the north side of the street, who are mostly in your experimental group, open the door with their left hand, getting a lot of exercise from this (the wind is very strong), while people on the south side open the screen door with their right hand.
It seems unlikely to me that many hidden correlations would survive alternating picks from a sorted list like this rigged example, but if the sample size is large enough, you’d still be better off randomizing than following any deterministic algorithm, because “every other item from a list sorted on X” has low Kolmogorov complexity and can be replicated by an unknown correlate of your observable variable by chance.
Eliezer claims that randomness is always bad; many other people claim that one way randomness is good is that it is unbiased. Partitioning subjects into experimental conditions must be unbiased.
This is perhaps a useful place to illustrate the “randomness hath no power” argument: randomness is unbiased in expectation but we actually expect the absolute amount of biasedness for a randomly selected assignment to be nonzero. When biasing factors are known ahead of time, we do better by controlling for it directly (with, say, a paired assignment).
Exactly! This is a math problem! And it becomes a very complicated math problem very quickly as the prior information gets interesting.
There’s nothing magical about an AI; it can’t figure out anything a human couldn’t figure out in principle. The difference is the “superintelligence” bit: a superintelligent AI could efficiently use much more complicated prior information for experiment design.
I don’t understand the improvement you think is possible here. In a lot of cases, the math isn’t the problem, the theory is known. The difficulty is usually finding a large enough sample size,etc.
This doesn’t require omniscience, or AI: people do this now (based on info they have). If you have more info, we know how to use it (there is theory). Why are we talking about AI, this is a math problem.
Slightly harshly worded suggestion (not to you specifically): maybe more reading, less invocation of robo-Jesus in vain.
Eliezer claims that randomness is always bad; many other people claim that one way randomness is good is that it is unbiased. Partitioning subjects into experimental conditions must be unbiased. Using an algorithm and knowing that its biases are orthogonal to the phenomenon being investigated requires omniscience. Besides, if you knew in advance what was relevant, you wouldn’t need to do the experiment.
That is what the comment means. The use of the term “AI” is just to show that the claim is that no real-world agent can be smart enough to do unbiased partitioning in all cases, not just that we’re not smart enough to do it.
In practice, a possibly biased but intelligent partitioning is better when the sample size is small.
We know what property we want (that randomization will give you), good balance in relevant covariates between two groups. I can use a deterministic algorithm for this, and in fact people do, e.g. matching algorithms. Another thing people do is try all possible assignments (see: permutation tests for the null).
Discussion of AI and omniscience is a complete red herring, you don’t need that to show that you don’t need randomness for this. We aren’t randomizing for the sake of randomizing, we are doing it because we want some property that we can directly target deterministically.
I don’t think EY can possibly know enough math to make his claim go through, I think this is an “intellectual marketing” claim. People do this a lot, if we are talking about your claim, you won the game.
If you sort all the subjects on one criteria, it may be correlated in an unexpected way with another criteria you’re unaware of. Suppose you want to study whether licorice causes left-handedness in a population from Tonawanda, NY. So you get a list of addresses from Tonawanda New York, sort them by address, and go down the list throwing them alternately into control and experimental group. Then you mail the experimental group free licorice for a ten years. Voila, after 10 years there are more left-handers in the experimental group.
But even and odd addresses are on opposite sides of the street. And it so happens that in Tonawanda, NY, the screen doors on the front of every house are hinged on the west side, regardless of which way the house faces, because the west wind is so strong it would rip the door off its hinges otherwise. So people on the north side of the street, who are mostly in your experimental group, open the door with their left hand, getting a lot of exercise from this (the wind is very strong), while people on the south side open the screen door with their right hand.
It seems unlikely to me that many hidden correlations would survive alternating picks from a sorted list like this rigged example, but if the sample size is large enough, you’d still be better off randomizing than following any deterministic algorithm, because “every other item from a list sorted on X” has low Kolmogorov complexity and can be replicated by an unknown correlate of your observable variable by chance.
This is perhaps a useful place to illustrate the “randomness hath no power” argument: randomness is unbiased in expectation but we actually expect the absolute amount of biasedness for a randomly selected assignment to be nonzero. When biasing factors are known ahead of time, we do better by controlling for it directly (with, say, a paired assignment).
Exactly! This is a math problem! And it becomes a very complicated math problem very quickly as the prior information gets interesting.
There’s nothing magical about an AI; it can’t figure out anything a human couldn’t figure out in principle. The difference is the “superintelligence” bit: a superintelligent AI could efficiently use much more complicated prior information for experiment design.
I don’t understand the improvement you think is possible here. In a lot of cases, the math isn’t the problem, the theory is known. The difficulty is usually finding a large enough sample size,etc.