How can you be sure that in the historical scenario, the Byzantine Emperor actually did the “right thing”, i.e. he wouldn’t have done better by doing something else? It’s the teachers who have to decide that. Also, what if the Emperor got the “right answer” for the wrong reasons, and the student also got the “right answer” for the wrong reasons? It’s up to the teacher to decide that as well. The best thing you can do is have several groups of rationalists selecting the scenarios and verifying the students’ answers, but ultimately, when using either real life or fictional scenarios, you’re comparing the teachers to the students.
Same thing with measuring “success” of people in real life. They could’ve arrived at the correct answer for the wrong reasons, it’s up to the teachers to decide whether the reasons were right or wrong, i.e. whether they were actually rational or just lucky.
In order to assess the rationality of the students you need to use the sort of things/tests that convinced you that the teachers are rational in the first place. The same things that make the teacher’s tastes real can be matched against the student’s tastes.
How can you be sure that in the historical scenario, the Byzantine Emperor actually did the “right thing”, i.e. he wouldn’t have done better by doing something else? It’s the teachers who have to decide that. Also, what if the Emperor got the “right answer” for the wrong reasons, and the student also got the “right answer” for the wrong reasons? It’s up to the teacher to decide that as well. The best thing you can do is have several groups of rationalists selecting the scenarios and verifying the students’ answers, but ultimately, when using either real life or fictional scenarios, you’re comparing the teachers to the students.
Same thing with measuring “success” of people in real life. They could’ve arrived at the correct answer for the wrong reasons, it’s up to the teachers to decide whether the reasons were right or wrong, i.e. whether they were actually rational or just lucky.
In order to assess the rationality of the students you need to use the sort of things/tests that convinced you that the teachers are rational in the first place. The same things that make the teacher’s tastes real can be matched against the student’s tastes.