Lacking time right now for a long reply: The main thrust of my reaction is that this seems like a style of thought which would have concluded in 2008 that it’s incredibly unlikely for superintelligences to be able to solve the protein folding problem. People did, in fact, claim that to me in 2008. It furthermore seemed to me in 2008 that protein structure prediction by superintelligence was the hardest or least likely step of the pathway by which a superintelligence ends up with nanotech; and in fact I argued only that it’d be solvable for chosen special cases of proteins rather than biological proteins because the special-case proteins could be chosen to have especially predictable pathways. All those wobbles, all those balanced weak forces and local strange gradients along potential energy surfaces! All those nonequilibrium intermediate states, potentially with fragile counterfactual dependencies on each interim stage of the solution! If you were gonna be a superintelligence skeptic, you might have claimed that even chosen special cases of protein folding would be unsolvable. The kind of argument you are making now, if you thought this style of thought was a good idea, would have led you to proclaim that probably a superintelligence could not solve biological protein folding and that AlphaFold 2 was surely an impossibility and sheer wishful thinking.
If you’d been around then, and said, “Pre-AGI ML systems will be able to solve general biological proteins via a kind of brute statistical force on deep patterns in an existing database of biological proteins, but even superintelligences will not be able to choose special cases of such protein folding pathways to design de novo synthesis pathways for nanotechnological machinery”, it would have been a very strange prediction, but you would now have a leg to stand on. But this, I most incredibly doubt you would have said—the style of thinking you’re using would have predicted much more strongly, in 2008 when no such thing had been yet observed, that pre-AGI ML could not solve biological protein folding in general, than that superintelligences could not choose a few special-case solvable de novo folding pathways along sharper potential energy gradients and with intermediate states chosen to be especially convergent and predictable.
When you describe the “emailing protein sequences → nanotech” route, are you imagining an AGI with computers on which it can run code (like simulations)? Or do you claim that the AGI could design the protein sequences without writing simulations, by simply thinking about it “in its head”?
I’m confused as to why there necessarily wouldn’t be a difference between the two.
I can think of a classes problems without better than brute force or simulation solutions, e.g. Bitcoin mining and there are plenty of these solutions that explode enough in complexity that they are infeasible given the resource of the universe for certain input sizes.
Alphafold2 also did not fully solve the protein folding problem. Rather, it still has a significant error rate as reported by google themselves with regions of low confidence. It also seems to go up significantly in certain classes of inputs such as longer loop structures.
Further, it was possibly not solved to the extent of what people in 2008 were arguing against, given we are nowhere near predicting how protein folds in interactions. I’m unsure whether the predicted value of solving protein folding back then is equal to the predicted value of extent to which AlphaFold solves it.
My next conclusion is mostly an inference, but broadly it seems like DL models hit a sigmoid curve when it comes to eeking out the final percentages of accuracy, and this can also be seen in self driving cars and LLMs. This makes sense given the world largely follows predictable laws with a small but frequent number of exceptions. In a one shot experiment, this uncertainty would accumulate exponentially making it seem unlikely that it would succeed.
I think clearly it can reduce the amount of experiments needed, but one shot seems like too high of a bar to hold through only generalizations that take less than 1 OOM of compute than the universe uses to “simulate” results as you experiment due to compounding errors.
Lacking time right now for a long reply: The main thrust of my reaction is that this seems like a style of thought which would have concluded in 2008 that it’s incredibly unlikely for superintelligences to be able to solve the protein folding problem. People did, in fact, claim that to me in 2008. It furthermore seemed to me in 2008 that protein structure prediction by superintelligence was the hardest or least likely step of the pathway by which a superintelligence ends up with nanotech; and in fact I argued only that it’d be solvable for chosen special cases of proteins rather than biological proteins because the special-case proteins could be chosen to have especially predictable pathways. All those wobbles, all those balanced weak forces and local strange gradients along potential energy surfaces! All those nonequilibrium intermediate states, potentially with fragile counterfactual dependencies on each interim stage of the solution! If you were gonna be a superintelligence skeptic, you might have claimed that even chosen special cases of protein folding would be unsolvable. The kind of argument you are making now, if you thought this style of thought was a good idea, would have led you to proclaim that probably a superintelligence could not solve biological protein folding and that AlphaFold 2 was surely an impossibility and sheer wishful thinking.
If you’d been around then, and said, “Pre-AGI ML systems will be able to solve general biological proteins via a kind of brute statistical force on deep patterns in an existing database of biological proteins, but even superintelligences will not be able to choose special cases of such protein folding pathways to design de novo synthesis pathways for nanotechnological machinery”, it would have been a very strange prediction, but you would now have a leg to stand on. But this, I most incredibly doubt you would have said—the style of thinking you’re using would have predicted much more strongly, in 2008 when no such thing had been yet observed, that pre-AGI ML could not solve biological protein folding in general, than that superintelligences could not choose a few special-case solvable de novo folding pathways along sharper potential energy gradients and with intermediate states chosen to be especially convergent and predictable.
When you describe the “emailing protein sequences → nanotech” route, are you imagining an AGI with computers on which it can run code (like simulations)? Or do you claim that the AGI could design the protein sequences without writing simulations, by simply thinking about it “in its head”?
At the superintelligent level there’s not a binary difference between those two clusters. You just compute each thing you need to know efficiently.
I’m confused as to why there necessarily wouldn’t be a difference between the two.
I can think of a classes problems without better than brute force or simulation solutions, e.g. Bitcoin mining and there are plenty of these solutions that explode enough in complexity that they are infeasible given the resource of the universe for certain input sizes.
Alphafold2 also did not fully solve the protein folding problem. Rather, it still has a significant error rate as reported by google themselves with regions of low confidence. It also seems to go up significantly in certain classes of inputs such as longer loop structures.
Further, it was possibly not solved to the extent of what people in 2008 were arguing against, given we are nowhere near predicting how protein folds in interactions. I’m unsure whether the predicted value of solving protein folding back then is equal to the predicted value of extent to which AlphaFold solves it.
My next conclusion is mostly an inference, but broadly it seems like DL models hit a sigmoid curve when it comes to eeking out the final percentages of accuracy, and this can also be seen in self driving cars and LLMs. This makes sense given the world largely follows predictable laws with a small but frequent number of exceptions. In a one shot experiment, this uncertainty would accumulate exponentially making it seem unlikely that it would succeed.
I think clearly it can reduce the amount of experiments needed, but one shot seems like too high of a bar to hold through only generalizations that take less than 1 OOM of compute than the universe uses to “simulate” results as you experiment due to compounding errors.