Some people have short ai timelines based inner models that don’t communicate well. They might say “I think if company X trains according to new technique Y it should scale well and lead to AGI, and I expect them to use technique Y in the next few years”, and the reasons for why they think technique Y should work are some kind of deep understanding built from years of reading ml papers, that’s not particularly easy to transmit or debate.
In those cases, I want to avoid going into details and arguing directly, but would suggest that they use their deep knowledge of ML to predict existing recent results before looking at them. This would be easy to cheat, so I mostly suggest this for people to check themselves, or check people you trust to be honorable. Concretely, it’d be nice if when some new ml paper with a new technique comes out, someone compilés a list of questions answered by that paper (eg is technique A better than technique B for a particular result) and posts it to LW so people can track how well they understand ML, and thus (to some extent) short timelines.
For example a recent paper examinés how data affects performance on a bunch of benchmarks, and notably tested training either on an duplicated dataset (a bunch of common crawls), or deduplixated (the same except remove same documents that were shared between crawls). Do you expect deduplication in this case raises or lowers performance on benchmarks?
If we could have similar questions when new results come out it’s be nice.
Some people have short ai timelines based inner models that don’t communicate well. They might say “I think if company X trains according to new technique Y it should scale well and lead to AGI, and I expect them to use technique Y in the next few years”, and the reasons for why they think technique Y should work are some kind of deep understanding built from years of reading ml papers, that’s not particularly easy to transmit or debate.
In those cases, I want to avoid going into details and arguing directly, but would suggest that they use their deep knowledge of ML to predict existing recent results before looking at them. This would be easy to cheat, so I mostly suggest this for people to check themselves, or check people you trust to be honorable. Concretely, it’d be nice if when some new ml paper with a new technique comes out, someone compilés a list of questions answered by that paper (eg is technique A better than technique B for a particular result) and posts it to LW so people can track how well they understand ML, and thus (to some extent) short timelines.
For example a recent paper examinés how data affects performance on a bunch of benchmarks, and notably tested training either on an duplicated dataset (a bunch of common crawls), or deduplixated (the same except remove same documents that were shared between crawls). Do you expect deduplication in this case raises or lowers performance on benchmarks? If we could have similar questions when new results come out it’s be nice.