Topics are selected based on cost/benefit; something that takes a long time to learn would need to be a lot more useful, or have major positive externalities to more people knowing it.
There’s more that’s going into a good test. A good test is not about whether a person learned a topic but about whether the person is likely perform well in whatever they want to do.
Dath Ilan is going to have many different tests for different jobs. There will be regular evaluations of how the test predicts performance in the job and the test will be adapted to optimize for predicting performance.
Instead of picking topic by putting in “a lot of thought” the main way will be picking topics empirically by looking at whether the knowledge predicts performance.
The only thing that does need thought is thinking about how to prevent goodharting.
What topics belong on it?
That completely depends on the job. It doesn’t make sense to have a uniform test that everyone is taking given that as a society you want diversity in skills in your population.
Partly agree with your criticism of the quoted claim, but there are two things I think you should consider.
First, evaluating tests for long-term outcomes is fundamentally hard. The extent to which a 5th grade civics or math test predicts performance in policy or engineering is negligible. In fact, I would expect that the feedback from test scores in determining what a child focuses on has a far larger impact on a child’s trajectory than the object level prediction allows.
Second, standardizing tests greatly reduces cost of development, and allows larger sample sizes for validation. For either reason alone, it makes sense to use standardized tests as much as possible.
I don’t believe that 5th grade civics or math tests are a good idea. At that age you want to encourage children to learn by following their curiosity and if you teach math in a structured way you likely want to have instant computer driven feedback and not the idea that children are supposed to have a certain level of math knowledge at a certain age for which they get tested.
That’s fine, but choosing the question set to give the self-motivated children on which you provide the instant computer driven feedback is the same type of question; what is it that we want the child interested in X to learn?
Concretely, my 8 year old son likes math. He’s fine with multiplication and division, but enjoys thinking about math. If I want him to be successful applying math later in life, should I start him on knot theory, pre-algebra equation solving, adding and subtracting unlike fractions, or coding in python? I see real advantages to each of these; proof-based thinking and abstraction from concrete to theoretical ideas, more abstract thinking about and manipulation of numbers, getting ahead of what he’ll need next to continue at the math he will need to learn, or giving him other tools that will expand his ability to think and apply ideas, respectively.
I’d love feedback about which of these (or which combination of these) is most likely to ensure he’s learning the things that are useful in helping him apply math in a decade, but I can’t get useful feedback without trying it on large samples over the course of decades. Or, since I don’t live in Dath Ilan, I can use my best judgement and ask others for feedback in an ad-hoc fashion.
There’s more that’s going into a good test. A good test is not about whether a person learned a topic but about whether the person is likely perform well in whatever they want to do.
Dath Ilan is going to have many different tests for different jobs. There will be regular evaluations of how the test predicts performance in the job and the test will be adapted to optimize for predicting performance.
Instead of picking topic by putting in “a lot of thought” the main way will be picking topics empirically by looking at whether the knowledge predicts performance.
The only thing that does need thought is thinking about how to prevent goodharting.
That completely depends on the job. It doesn’t make sense to have a uniform test that everyone is taking given that as a society you want diversity in skills in your population.
Partly agree with your criticism of the quoted claim, but there are two things I think you should consider.
First, evaluating tests for long-term outcomes is fundamentally hard. The extent to which a 5th grade civics or math test predicts performance in policy or engineering is negligible. In fact, I would expect that the feedback from test scores in determining what a child focuses on has a far larger impact on a child’s trajectory than the object level prediction allows.
Second, standardizing tests greatly reduces cost of development, and allows larger sample sizes for validation. For either reason alone, it makes sense to use standardized tests as much as possible.
I don’t believe that 5th grade civics or math tests are a good idea. At that age you want to encourage children to learn by following their curiosity and if you teach math in a structured way you likely want to have instant computer driven feedback and not the idea that children are supposed to have a certain level of math knowledge at a certain age for which they get tested.
That’s fine, but choosing the question set to give the self-motivated children on which you provide the instant computer driven feedback is the same type of question; what is it that we want the child interested in X to learn?
Concretely, my 8 year old son likes math. He’s fine with multiplication and division, but enjoys thinking about math. If I want him to be successful applying math later in life, should I start him on knot theory, pre-algebra equation solving, adding and subtracting unlike fractions, or coding in python? I see real advantages to each of these; proof-based thinking and abstraction from concrete to theoretical ideas, more abstract thinking about and manipulation of numbers, getting ahead of what he’ll need next to continue at the math he will need to learn, or giving him other tools that will expand his ability to think and apply ideas, respectively.
I’d love feedback about which of these (or which combination of these) is most likely to ensure he’s learning the things that are useful in helping him apply math in a decade, but I can’t get useful feedback without trying it on large samples over the course of decades. Or, since I don’t live in Dath Ilan, I can use my best judgement and ask others for feedback in an ad-hoc fashion.