We’d probably try something along the lines you’re suggesting, but there are some interesting technical challenges to think through.
For example, we’d want to train the model to be good at predicting the future, not just knowing what happened. Under a naive implementation, weight updates would probably go partly towards better judgment and forecasting ability, but also partly towards knowing how the world played out after the initial training cutoff.
There are also questions around IR; it seems likely that models will need external retrieval mechanisms to forecast well for the next few years at least, and we’d want to train something that’s natively good at using retrieval tools to forecast, rather than relying purely on its crystalised knowledge.
Yeah I imagine it would partly just learn facts about what happened—but as long as it also partly learns general forecasting skills, that is important and measurable progress. Might be enough to be very useful.
Re:retrieval:yep I am imagining that being part of the setup and part of what it learns to be good at.
We’d probably try something along the lines you’re suggesting, but there are some interesting technical challenges to think through.
For example, we’d want to train the model to be good at predicting the future, not just knowing what happened. Under a naive implementation, weight updates would probably go partly towards better judgment and forecasting ability, but also partly towards knowing how the world played out after the initial training cutoff.
There are also questions around IR; it seems likely that models will need external retrieval mechanisms to forecast well for the next few years at least, and we’d want to train something that’s natively good at using retrieval tools to forecast, rather than relying purely on its crystalised knowledge.
Yeah I imagine it would partly just learn facts about what happened—but as long as it also partly learns general forecasting skills, that is important and measurable progress. Might be enough to be very useful.
Re:retrieval:yep I am imagining that being part of the setup and part of what it learns to be good at.