The way that formalism would think about your problem is you have two “treatments” (type of test, that you can vary, and type of student), and an “outcome” (how a given student does on a given test, typically some sort of histogram that’s hopefully shaped like a bell).
Your goal is to efficiently vary “treatment” values to learn as much as possible about the causal relationship between how you structure a test, and student quality, and the outcome.
There’s reading you can do on this problem, it’s a classical problem in statistics. Both Jerzy Neyman and Ronald Fisher wrote a lot about this, the latter has a famous book.
In fact, in some sense this is the problem of statistics, in the sense that modern statistics could be said to have grown out of, and generalized from, this problem.
i do statistical consulting as part of my day job responsibilities, i’m afraid to say this is not how it works.
if you came to me with this question i would roll back to ask what exactly you are trying to achieve with the analyses, before getting into the additional constraints you want to include. unfortunately it’s far more challenging if the data owner comes to the statistician after the data are collected rather than before (when principles of experimental design as ilya mentioned can be considered to achieve ability to successfully answer those questions using statistical methods).
that said, temporarily ignoring the additional constraints you mentioned (e.g. whether and how to transform data; exponential decay and what that actually means with respect to student evaluation scores; magic word “bayes”) perhaps a useful search term would be “item response theory”.
You have an experimental design problem: https://en.wikipedia.org/wiki/Design_of_experiments.
The way that formalism would think about your problem is you have two “treatments” (type of test, that you can vary, and type of student), and an “outcome” (how a given student does on a given test, typically some sort of histogram that’s hopefully shaped like a bell).
Your goal is to efficiently vary “treatment” values to learn as much as possible about the causal relationship between how you structure a test, and student quality, and the outcome.
There’s reading you can do on this problem, it’s a classical problem in statistics. Both Jerzy Neyman and Ronald Fisher wrote a lot about this, the latter has a famous book.
In fact, in some sense this is the problem of statistics, in the sense that modern statistics could be said to have grown out of, and generalized from, this problem.
In your opinion what is a reasonable price to have a statistician write me a formula for this?
i do statistical consulting as part of my day job responsibilities, i’m afraid to say this is not how it works.
if you came to me with this question i would roll back to ask what exactly you are trying to achieve with the analyses, before getting into the additional constraints you want to include. unfortunately it’s far more challenging if the data owner comes to the statistician after the data are collected rather than before (when principles of experimental design as ilya mentioned can be considered to achieve ability to successfully answer those questions using statistical methods).
that said, temporarily ignoring the additional constraints you mentioned (e.g. whether and how to transform data; exponential decay and what that actually means with respect to student evaluation scores; magic word “bayes”) perhaps a useful search term would be “item response theory”.
good luck
Don’t know. Ask a statistician who knows about design.