Proposed explanation: o3 is very good at easy-to-check short horizon tasks that were put into the RL mix and worse at longer horizon tasks, tasks not put into its RL mix, or tasks which are hard/expensive to check.
I don’t think o3 is well described as superhuman—it is within the human range on all these benchmarks especially when considering the case where you give the human 8 hours to do the task.
(E.g., on frontier math, I think people who are quite good at competition style math probably can do better than o3 at least when given 8 hours per problem.)
Additionally, I’d say that some of the obstacles in outputing a good research paper could be resolved with some schlep, so I wouldn’t be surprised if we see some OK research papers being output (with some human assistance) next year.
Proposed explanation: o3 is very good at easy-to-check short horizon tasks that were put into the RL mix and worse at longer horizon tasks, tasks not put into its RL mix, or tasks which are hard/expensive to check.
I don’t think o3 is well described as superhuman—it is within the human range on all these benchmarks especially when considering the case where you give the human 8 hours to do the task.
(E.g., on frontier math, I think people who are quite good at competition style math probably can do better than o3 at least when given 8 hours per problem.)
Additionally, I’d say that some of the obstacles in outputing a good research paper could be resolved with some schlep, so I wouldn’t be surprised if we see some OK research papers being output (with some human assistance) next year.