To be clear, I think I will lose, but I think this is weak evidence. The bet says that $1bn must be spent on a single training run, not a single supercomputer.
“OR hardware that would cost $1bn if purchased through competitive cloud computing vendors at the time on a training run to develop a single ML model”
Assuming the British government gets a fair price for the hardware, and actually has the machine running prior to the bet end date, does this satisfy the condition?
I don’t actually think it will be the one that ends the bet as I expect the British government to take a while to actually implement this, but possibly they finish before 2026.
Assuming the British government gets a fair price for the hardware, and actually has the machine running prior to the bet end date, does this satisfy the condition?
No.
That condition resolves on the basis of the cost of the training run, not the cost of the hardware. You can tell because we spelled out the full details of how to estimate costs, and it depends on the cost in FLOP for the training run.
But honestly at this point I’m considering conceding early and just paying out, because I don’t look forward to years of people declaring victory early, which seems to already be happening.
So MMLU is down:
Presumably MATH will be next—is Minerva still SOTA?
Did they reveal how GPT-4 did on every task in the MMLU? If not, it’s not clear whether the relevant condition here has been met yet.
So you may lose the bet imminently:
https://www.theguardian.com/technology/2023/mar/15/uk-to-invest-900m-in-supercomputer-in-bid-to-build-own-britgpt
900 million pounds is 1 billion USD
And for the other part, for MMLU your ‘doubt’ hinges on it doing <80% on a subtest while reaching 88% overall.
I know it’s just a bet over a small amount of money, but to lose in 1 year is something.
To be clear, I think I will lose, but I think this is weak evidence. The bet says that $1bn must be spent on a single training run, not a single supercomputer.
“OR hardware that would cost $1bn if purchased through competitive cloud computing vendors at the time on a training run to develop a single ML model”
Assuming the British government gets a fair price for the hardware, and actually has the machine running prior to the bet end date, does this satisfy the condition?
I don’t actually think it will be the one that ends the bet as I expect the British government to take a while to actually implement this, but possibly they finish before 2026.
No.
That condition resolves on the basis of the cost of the training run, not the cost of the hardware. You can tell because we spelled out the full details of how to estimate costs, and it depends on the cost in FLOP for the training run.
But honestly at this point I’m considering conceding early and just paying out, because I don’t look forward to years of people declaring victory early, which seems to already be happening.