Perhaps include the text of the question to allow more priming.
Upsides: It seems to me that making predictions is a huge part of intelligence, relatively easy to check and compare with humans
Downsides: Resolution will not be available for quite some time, and when the results are in, everybody will already be interested in the next AI project. Results only arrive “after the fact”.
Idea to for an approach how close GPT-3 is to “real intelligence”: generalist forecasting!
Give it the prompt: “Answer with a probability between 0 and 1. Will the UK’s Intelligence and Security committee publish the report into Russian interference by the end of July?”, repeat for a bunch of questions, grade it at resolution.
Similar things could be done for range questions: “Give a 50% confidence interval on values between −35 and 5. What will the US Q2 2020 GDP growth rate be, according to the US Bureau of Economic Analysis Advance Estimate?”.
Perhaps include the text of the question to allow more priming.
Upsides: It seems to me that making predictions is a huge part of intelligence, relatively easy to check and compare with humans
Downsides: Resolution will not be available for quite some time, and when the results are in, everybody will already be interested in the next AI project. Results only arrive “after the fact”.