Thanks, I really like this comment. Here are some points about metascience I agree and disagree with, and how this fits into my framework for thinking about deconfusion vs data collection in AI alignment.
I tentatively think you’re right about relativity, though I also feel out of my depth here. [1]
David Bau must have mentioned the Schrödinger book but I forgot about it, thanks for the correction. The fact that ideas like this told Watson and Crick where to look definitely seems important.
Overall, I agree that a good theoretical understanding guides further experiment and discovery early on in a field.
However, I don’t think curing cancer or defining life are bottlenecked on deconfusion. [2]
For curing cancer, we know the basic mechanisms behind cancer and understand that they’re varied and complex. We have categorized dozens of oncogenes of about 7 different types, and equally many ways that organisms defend against cancer. It seems unlikely that the the cure for cancer will depend on some unified theory of cancer, and much more plausible that it’ll be due to investments in experiment and engineering. It was mostly engineering that gave us mRNA vaccines, and a mix of all three that allowed CRISPR.
For defining life, we already have edge cases like viruses and endosymbiotic organisms, and understand pretty well which things can maintain homeostasis, reproduce, etc. in what circumstances. It also seems unlikely that someone will draw a much sharper boundary around life, especially without lots more useful data.
My model of the basic process of science currently looks like this:
Note that I distinguish deconfusion (what you can invest in) from theory (the output). At any time, there are various returns to investing in data collection tech, experiments, and deconfusion, and returns diminish with the amount invested. I claim that in both physics and biology, we went through an early phase where the bottleneck was theory and returns to deconfusion were high, and currently the fields are relatively mature, such that the bottlenecks have shifted to experiment and engineering, but with theory still playing a role.
In AI, we’re in a weird situation:
We feel pretty confused about basic concepts like agency, suggesting that we’re early and deconfusion is valuable.
Machine learning is a huge field. If there are diminishing returns to deconfusion, this means experiment and data collection are more valuable than deconfusion.
Machine learning is already doing impressive things primarily on the back of engineering, without much reliance on the type of theory that deconfusion generates alone (deep, simple relationships between things).
But even if engineering alone is enough to build AGI, we need theory for alignment.
In biology, we know cancer is complex and unlikely to be understood by a deep simple theory, but in AI, we don’t know whether intelligence is complex.
I’m not sure what to make of all this, and this comment is too long already, but hopefully I’ve laid out a frame that we can roughly agree on.
[1] When writing the dialogue I thought the hard part of special relativity was discovering the Lorentz transformations (which GPS clock drift observations would make obvious), but Lorentz did this between 1892-1904 and it took until 1905 for Einstein to discover the theory of special relativity. I missed the point about theory guiding experiment earlier, and without relativity we would not have built gravitational wave detectors. I’m not sure whether this also applies to gravitational lensing or redshift.
[2] I also disagree with the idea that “practically no one is trying to find theories in biology”. Theoretical biology seems like a decently large field—probably much larger than it was in 1950-- and biologists use mathematical models all the time.
Thanks, I really like this comment. Here are some points about metascience I agree and disagree with, and how this fits into my framework for thinking about deconfusion vs data collection in AI alignment.
I tentatively think you’re right about relativity, though I also feel out of my depth here. [1]
David Bau must have mentioned the Schrödinger book but I forgot about it, thanks for the correction. The fact that ideas like this told Watson and Crick where to look definitely seems important.
Overall, I agree that a good theoretical understanding guides further experiment and discovery early on in a field.
However, I don’t think curing cancer or defining life are bottlenecked on deconfusion. [2]
For curing cancer, we know the basic mechanisms behind cancer and understand that they’re varied and complex. We have categorized dozens of oncogenes of about 7 different types, and equally many ways that organisms defend against cancer. It seems unlikely that the the cure for cancer will depend on some unified theory of cancer, and much more plausible that it’ll be due to investments in experiment and engineering. It was mostly engineering that gave us mRNA vaccines, and a mix of all three that allowed CRISPR.
For defining life, we already have edge cases like viruses and endosymbiotic organisms, and understand pretty well which things can maintain homeostasis, reproduce, etc. in what circumstances. It also seems unlikely that someone will draw a much sharper boundary around life, especially without lots more useful data.
My model of the basic process of science currently looks like this:
Note that I distinguish deconfusion (what you can invest in) from theory (the output). At any time, there are various returns to investing in data collection tech, experiments, and deconfusion, and returns diminish with the amount invested. I claim that in both physics and biology, we went through an early phase where the bottleneck was theory and returns to deconfusion were high, and currently the fields are relatively mature, such that the bottlenecks have shifted to experiment and engineering, but with theory still playing a role.
In AI, we’re in a weird situation:
We feel pretty confused about basic concepts like agency, suggesting that we’re early and deconfusion is valuable.
Machine learning is a huge field. If there are diminishing returns to deconfusion, this means experiment and data collection are more valuable than deconfusion.
Machine learning is already doing impressive things primarily on the back of engineering, without much reliance on the type of theory that deconfusion generates alone (deep, simple relationships between things).
But even if engineering alone is enough to build AGI, we need theory for alignment.
In biology, we know cancer is complex and unlikely to be understood by a deep simple theory, but in AI, we don’t know whether intelligence is complex.
I’m not sure what to make of all this, and this comment is too long already, but hopefully I’ve laid out a frame that we can roughly agree on.
[1] When writing the dialogue I thought the hard part of special relativity was discovering the Lorentz transformations (which GPS clock drift observations would make obvious), but Lorentz did this between 1892-1904 and it took until 1905 for Einstein to discover the theory of special relativity. I missed the point about theory guiding experiment earlier, and without relativity we would not have built gravitational wave detectors. I’m not sure whether this also applies to gravitational lensing or redshift.
[2] I also disagree with the idea that “practically no one is trying to find theories in biology”. Theoretical biology seems like a decently large field—probably much larger than it was in 1950-- and biologists use mathematical models all the time.