The idea behind the scientific method is to design procedures that are robust to the scientist being biased or incompetent or even corrupt. Any approach that starts with “assume a perfect scientist” is not going to work in reality.
Science is a set of hacks to get usable modelling out of humans, accepting that
There are things that humans do which are critical to modelling reality, and which you do not understand to the point of being able to reimplement them, but
you also can’t just leave humans to do free-form theorizing, because that has been conclusively shown to lead to all kinds of problems.
The critical black box in this specific case is about how to judge a theory’s simplicity, and what the best way to build a prior from that is.
As long as either of these things is a black box to you, you won’t be able to do much better than using high-level heuristical hacks of the sort science is made out of. But that’s going to bite you every time you don’t have the luxury of being able to apply these hacks—say because you’re modelling (some aspect of) human history, and can’t rerun the experiment. Also, you won’t be able to build an AGI.
In addition, if you’re really worried about corruption, the holding-back-data-on-purpose thing is setting up great profits to be made this way:
Corrupt scientist takes out a loan for BIGNUM $.
Corrupt scientist pays this money to someone with access to the still-secret data.
Bribed data keeper gives corrupt scientist a copy of the data.
Corrupt scientist fits their hypothesis to the whole data set.
Corrupt scientist publishes hypothesis.
Full data set is released officially.
Hypothesis of corrupt scientist is verified to match whole data set. Corrupt scientist gains great prestige, and uses that to obtain sufficient money to pay off the loan from 1, and then some.
You could try to set up the data keeper organization so that a premature limited data release is unlikely even in the face of potentially large bribes, but that seems like a fairly tough problem (and are they even thinking about it seriously?). Data is very easy to copy, preventing it from being copied is hard. And in this case, more so than in most cases where you’re worried about leaks, figuring out that a leak has in fact happened might be extremely difficult—at least if you really are ignorant about what hypothesis simplicity looks like.
But that’s going to bite you every time you don’t have the luxury of being able to apply these hacks—say because you’re modelling (some aspect of) human history, and can’t rerun the experiment.
? History sounds like exactly the situation where “hold back half the data, hypothsise on the other half, then look at the whole” is the only way of reasonably going about this.
Also, you won’t be able to build an AGI.
Don’t follow that argument at all—in the worst case scenario, you can brute force it by scanning and moddelling a human brain. But even if true, it’s not really an issue for social scientists and their ilk. And there the “look at half the data” would cause definite improvements in their proceedures. It would make science work for the “flawed but honest” crowd.
As for deliberately holding back half the data from other scientists (as opposed to one guy simply choosing to only look at half), that’s a different issue. I’ve got no really strong feelings on that. It could go either way.
It’s an ok hack for someone in the “flawed but honest” crowd, individually. But note that it really doesn’t scale to allowing you to deal with corruption (which was one of the problems I assumed in the post you replied to).
Extended to an entire field, this means that you may end up with N papers, all about the same data set, all proposing a different hypothesis that produces a good match on the set, and all of them claiming that their hypothesis was formulated using this procedure. IOW, you end up with unverifiable “trust us, we didn’t cheat” claims for each of those hypotheses. Which is not a good basis for arriving at a consensus in the field.
Re AI design, assuming you actually understand what you implemented (as opposed to just blindly copying algorithms from the human brain without understanding what they do), the reason this method would work is that you’ve successfully extracted the human built-in simplicity prior (and I don’t know how good that one is exactly, but it has to be a halfway workable approximation; otherwise humans couldn’t model reality at all).
The idea behind the scientific method is to design procedures that are robust to the scientist being biased or incompetent or even corrupt. Any approach that starts with “assume a perfect scientist” is not going to work in reality.
Science is a set of hacks to get usable modelling out of humans, accepting that
There are things that humans do which are critical to modelling reality, and which you do not understand to the point of being able to reimplement them, but
you also can’t just leave humans to do free-form theorizing, because that has been conclusively shown to lead to all kinds of problems.
The critical black box in this specific case is about how to judge a theory’s simplicity, and what the best way to build a prior from that is. As long as either of these things is a black box to you, you won’t be able to do much better than using high-level heuristical hacks of the sort science is made out of. But that’s going to bite you every time you don’t have the luxury of being able to apply these hacks—say because you’re modelling (some aspect of) human history, and can’t rerun the experiment. Also, you won’t be able to build an AGI.
In addition, if you’re really worried about corruption, the holding-back-data-on-purpose thing is setting up great profits to be made this way:
Corrupt scientist takes out a loan for BIGNUM $.
Corrupt scientist pays this money to someone with access to the still-secret data.
Bribed data keeper gives corrupt scientist a copy of the data.
Corrupt scientist fits their hypothesis to the whole data set.
Corrupt scientist publishes hypothesis.
Full data set is released officially.
Hypothesis of corrupt scientist is verified to match whole data set. Corrupt scientist gains great prestige, and uses that to obtain sufficient money to pay off the loan from 1, and then some.
You could try to set up the data keeper organization so that a premature limited data release is unlikely even in the face of potentially large bribes, but that seems like a fairly tough problem (and are they even thinking about it seriously?). Data is very easy to copy, preventing it from being copied is hard. And in this case, more so than in most cases where you’re worried about leaks, figuring out that a leak has in fact happened might be extremely difficult—at least if you really are ignorant about what hypothesis simplicity looks like.
? History sounds like exactly the situation where “hold back half the data, hypothsise on the other half, then look at the whole” is the only way of reasonably going about this.
Don’t follow that argument at all—in the worst case scenario, you can brute force it by scanning and moddelling a human brain. But even if true, it’s not really an issue for social scientists and their ilk. And there the “look at half the data” would cause definite improvements in their proceedures. It would make science work for the “flawed but honest” crowd.
As for deliberately holding back half the data from other scientists (as opposed to one guy simply choosing to only look at half), that’s a different issue. I’ve got no really strong feelings on that. It could go either way.
It’s an ok hack for someone in the “flawed but honest” crowd, individually. But note that it really doesn’t scale to allowing you to deal with corruption (which was one of the problems I assumed in the post you replied to).
Extended to an entire field, this means that you may end up with N papers, all about the same data set, all proposing a different hypothesis that produces a good match on the set, and all of them claiming that their hypothesis was formulated using this procedure. IOW, you end up with unverifiable “trust us, we didn’t cheat” claims for each of those hypotheses. Which is not a good basis for arriving at a consensus in the field.
Re AI design, assuming you actually understand what you implemented (as opposed to just blindly copying algorithms from the human brain without understanding what they do), the reason this method would work is that you’ve successfully extracted the human built-in simplicity prior (and I don’t know how good that one is exactly, but it has to be a halfway workable approximation; otherwise humans couldn’t model reality at all).