I agree with Liam that amplifying ESCs with prediction markets would be a lot like John’s suggestion. I think an elegant approach would be something like setting up prediction markets, and then allowing users to set up their own data science pipelines as they see fit. My guess is that this would be essential if we wanted to predict a lot of books; say, 200 to 1 Million books.
If predictors did a decent job at this, then I’d be on the whole excited about it being known and for authors to try to perform better on it; because I believe it would reveal more signal than noise (well, as long as this prediction was done decently, for a vague definition of decent.)
My guess is that a strong analysis would only include “number of citations” as being a rather minor feature. If it became evident that authors were trying to actively munchkin[1] things, then predictors should pick up on that, and introduce features for things like “known munchkiner”, which would make this quite difficult. The timescales for authors to update and write books seem much longer than the timescales for predictors to recognize what’s going on.
[1] I realize that “’munchkining” is a pretty uncommon word, but I like it a lot, and it feels more relevant than powergaming. Please let me know if there’s a term you prefer. I think “Goodhart” is too generic, especially if things like “correlational Goodhart” count.
Just chiming in here;
I agree with Liam that amplifying ESCs with prediction markets would be a lot like John’s suggestion. I think an elegant approach would be something like setting up prediction markets, and then allowing users to set up their own data science pipelines as they see fit. My guess is that this would be essential if we wanted to predict a lot of books; say, 200 to 1 Million books.
If predictors did a decent job at this, then I’d be on the whole excited about it being known and for authors to try to perform better on it; because I believe it would reveal more signal than noise (well, as long as this prediction was done decently, for a vague definition of decent.)
My guess is that a strong analysis would only include “number of citations” as being a rather minor feature. If it became evident that authors were trying to actively munchkin[1] things, then predictors should pick up on that, and introduce features for things like “known munchkiner”, which would make this quite difficult. The timescales for authors to update and write books seem much longer than the timescales for predictors to recognize what’s going on.
[1] I realize that “’munchkining” is a pretty uncommon word, but I like it a lot, and it feels more relevant than powergaming. Please let me know if there’s a term you prefer. I think “Goodhart” is too generic, especially if things like “correlational Goodhart” count.