Yes, that’s a sneaky part of the scenario. In general, I think this is a realistic thing to occur: ‘other intelligent people optimizing around this data’ is one of the things that causes the most complicated things to happen in real-world data as well.
Christian Z R had a very good comment on this, where they mentioned looking at the subset of dungeons where Rooms 2 and 4 had the same encounter, or where Rooms 6 and 8 had the same encounter, to factor out the impact of intelligence and guarantee ‘they will encounter this specific thing’.
(Edited to add: actually, there are ~100 rows in the dataset where Room2=4, Room6=8, and Room3=5=7. This isn’t enough to get firm analysis on, but it could have served as a very strong sanity-check opportunity where you can look at a few dungeons where you know exactly what the route is.)
Yes, that’s a sneaky part of the scenario. In general, I think this is a realistic thing to occur: ‘other intelligent people optimizing around this data’ is one of the things that causes the most complicated things to happen in real-world data as well.
Christian Z R had a very good comment on this, where they mentioned looking at the subset of dungeons where Rooms 2 and 4 had the same encounter, or where Rooms 6 and 8 had the same encounter, to factor out the impact of intelligence and guarantee ‘they will encounter this specific thing’.
(Edited to add: actually, there are ~100 rows in the dataset where Room2=4, Room6=8, and Room3=5=7. This isn’t enough to get firm analysis on, but it could have served as a very strong sanity-check opportunity where you can look at a few dungeons where you know exactly what the route is.)