I’m curious what you guys think about the opportunities to train llms to make forecasts. E.g. Google is rumored to have snapshots of the whole internet going back many years; they could train a big new model on pre-2014 data and then have it make forecasts of the next decade and then have those forecasts automatically scored and trained on. If you guys got acquired by Google and given 10e26 flop compute budget, how would you use it?
We’d probably try something along the lines you’re suggesting, but there are some interesting technical challenges to think through.
For example, we’d want to train the model to be good at predicting the future, not just knowing what happened. Under a naive implementation, weight updates would probably go partly towards better judgment and forecasting ability, but also partly towards knowing how the world played out after the initial training cutoff.
There are also questions around IR; it seems likely that models will need external retrieval mechanisms to forecast well for the next few years at least, and we’d want to train something that’s natively good at using retrieval tools to forecast, rather than relying purely on its crystalised knowledge.
Yeah I imagine it would partly just learn facts about what happened—but as long as it also partly learns general forecasting skills, that is important and measurable progress. Might be enough to be very useful.
Re:retrieval:yep I am imagining that being part of the setup and part of what it learns to be good at.
I’m curious what you guys think about the opportunities to train llms to make forecasts. E.g. Google is rumored to have snapshots of the whole internet going back many years; they could train a big new model on pre-2014 data and then have it make forecasts of the next decade and then have those forecasts automatically scored and trained on. If you guys got acquired by Google and given 10e26 flop compute budget, how would you use it?
We’d probably try something along the lines you’re suggesting, but there are some interesting technical challenges to think through.
For example, we’d want to train the model to be good at predicting the future, not just knowing what happened. Under a naive implementation, weight updates would probably go partly towards better judgment and forecasting ability, but also partly towards knowing how the world played out after the initial training cutoff.
There are also questions around IR; it seems likely that models will need external retrieval mechanisms to forecast well for the next few years at least, and we’d want to train something that’s natively good at using retrieval tools to forecast, rather than relying purely on its crystalised knowledge.
Yeah I imagine it would partly just learn facts about what happened—but as long as it also partly learns general forecasting skills, that is important and measurable progress. Might be enough to be very useful.
Re:retrieval:yep I am imagining that being part of the setup and part of what it learns to be good at.