“Can you clarify that a bit? When what project comes out? If you mean mine, I’m confused about why that would say something about the ability to derive special & general relativity.”
I mean your project. I’m hoping it can allow us to be more precise by ranking models abilities to characterize between well-known systems. Like a model can characterize Special Relativity given what Einstein knew at the time but not General Relativity. If you were to walk along some hypothetical road from SR to GR we might ballpark a model is 30% of the way there. Maybe this project could generate domains that are roughly some x% between SR and GR and validate our estimates.
”Agreed that each added step of mathematical complexity (in this case from linear to quadratic) will make it harder. I’m less convinced that acceleration being a second-order effect would make an additional difference, since that seems more like a conceptual framework we impose than like a direct property of the data.”
Right. The important point is that the equation it needs to find is quadratic instead of linear in the data.
Got it, thanks. We’re planning to try to avoid testing systems that are isomorphic to real-world examples, in the interest of making a crisp distinction between reasoning and knowledge. That said, if we come up with a principled way to characterize system complexity (especially the complexity of the underlying mathematical laws), and if (big if!) that turns out to match what LLMs find harder, then we could certainly compare results to the complexity of real-world laws. I hadn’t considered that, thanks for the idea!
“Can you clarify that a bit? When what project comes out? If you mean mine, I’m confused about why that would say something about the ability to derive special & general relativity.”
I mean your project. I’m hoping it can allow us to be more precise by ranking models abilities to characterize between well-known systems. Like a model can characterize Special Relativity given what Einstein knew at the time but not General Relativity. If you were to walk along some hypothetical road from SR to GR we might ballpark a model is 30% of the way there. Maybe this project could generate domains that are roughly some x% between SR and GR and validate our estimates.
”Agreed that each added step of mathematical complexity (in this case from linear to quadratic) will make it harder. I’m less convinced that acceleration being a second-order effect would make an additional difference, since that seems more like a conceptual framework we impose than like a direct property of the data.”
Right. The important point is that the equation it needs to find is quadratic instead of linear in the data.
Got it, thanks. We’re planning to try to avoid testing systems that are isomorphic to real-world examples, in the interest of making a crisp distinction between reasoning and knowledge. That said, if we come up with a principled way to characterize system complexity (especially the complexity of the underlying mathematical laws), and if (big if!) that turns out to match what LLMs find harder, then we could certainly compare results to the complexity of real-world laws. I hadn’t considered that, thanks for the idea!