This sounds a lot like the concept of cross-validation in machine learning. Suppose you have a bunch of data—emails that are either spam or nonspam, for example—and you want to make a model that will allow you to predict whether an email is spam or not. You split the data into n subsets, and for each subset S, do this test:
Create a model from all the data that are not in this set S.
Test the model on the data from S. How good are the predictions?
This method lets you test how well your machine learning methods are doing, so you can tweak them or go with a different method. Google recently got some publicity for offering a service that will apply a bunch of machine learning methods to your data and pick the best one; it picks the best one by running cross-validation.
Applying this to scientists, we get a variation on what you proposed in your penultimate paragraph: give part of the data to separate individuals or groups, stick them in a Science Temple in the mountains somewhere without Internet access, and then once they’ve come up with theories, test them on the data that they weren’t given. For extra hilarity, split the data 50⁄50 among two large research groups, let them think of theories for a year, then have them exchange theories but not data. Then sit back and watch the argument.
You could also combine this sort of thing with the method suggested in VoF, by giving data out slowly to separate groups, in different orders. I’m not sure which of these ideas is best.
This sounds a lot like the concept of cross-validation in machine learning. Suppose you have a bunch of data—emails that are either spam or nonspam, for example—and you want to make a model that will allow you to predict whether an email is spam or not. You split the data into n subsets, and for each subset S, do this test:
Create a model from all the data that are not in this set S.
Test the model on the data from S. How good are the predictions?
This method lets you test how well your machine learning methods are doing, so you can tweak them or go with a different method. Google recently got some publicity for offering a service that will apply a bunch of machine learning methods to your data and pick the best one; it picks the best one by running cross-validation.
Applying this to scientists, we get a variation on what you proposed in your penultimate paragraph: give part of the data to separate individuals or groups, stick them in a Science Temple in the mountains somewhere without Internet access, and then once they’ve come up with theories, test them on the data that they weren’t given. For extra hilarity, split the data 50⁄50 among two large research groups, let them think of theories for a year, then have them exchange theories but not data. Then sit back and watch the argument.
You could also combine this sort of thing with the method suggested in VoF, by giving data out slowly to separate groups, in different orders. I’m not sure which of these ideas is best.