The most realistic thing one could do is repeat this work, with the same settings, but using k-fold cross-validation, where test and training sets are never related (like what I did at Columbia).
This will show how well (or poorly, as the case may be) the method generalizes to unrelated proteins.
But do you need 3D structures to test alphafold? If alphafold makes a prediction about whether a given ligand binds to a protein, I would expect that testing whether or not that ligand binds to the protein is much cheaper.
He said that for the interactions of proteins with small (drug-like) molecules AlphaFold 3 showed 76% accuracy in a benchmark test (versus 52% for the next best tool).
Right. The benchmark (their test set) just compares 3D structures.
Side note: 52% also seems low for Vina, but I haven’t looked into this further. Maybe the benchmark is hard, or maybe the “search space” (user-specified) was too big.
On their other test (in their Extended Data), both Vina and AF3 do much better.
This makes sense, but isn’t alphafold available for use? Is it possible to verify this one way or another experimentally?
Determining 3D structures is expensive.
The most realistic thing one could do is repeat this work, with the same settings, but using k-fold cross-validation, where test and training sets are never related (like what I did at Columbia).
This will show how well (or poorly, as the case may be) the method generalizes to unrelated proteins.
I hope someone does it.
But do you need 3D structures to test alphafold? If alphafold makes a prediction about whether a given ligand binds to a protein, I would expect that testing whether or not that ligand binds to the protein is much cheaper.
Unlike Vina, AF3 only predicts 3D structures, I believe. It does not predict binding affinities.
AlphaFold 2 was only predicting 3D structures. From the abstract of the Alpha Fold 3 paper:
The FastCompany article says:
Right. The benchmark (their test set) just compares 3D structures.
Side note: 52% also seems low for Vina, but I haven’t looked into this further. Maybe the benchmark is hard, or maybe the “search space” (user-specified) was too big.
On their other test (in their Extended Data), both Vina and AF3 do much better.