I feel like a big part of what tripped me up here was an inevitable part of the difficulty of the scenario that in retrospect should have been obvious. Specifically, if there is any variation in difficulty of an encounter that is known to the adventurers in advance, the score contribution of an encounter type in actual paths taken is less than the difficulty of the encounter as estimated by what best predicts the path taken (because the adventurer takes the path when it’s weak, but avoids when it’s strong).
So, I wound up with an epicycle saying hags and orcs were avoided more than their actual scores warranted, because that effect was most significant for them (goblins are chosen over most other encounters even if alerted, and Dragons mostly aren’t alerted).
This effect was made much worse by the fact that I was getting scores mainly from lower difficulty dungeons, with lots of “Nothing” rooms and low level encounters. But even once I estimated scores from the overall data with my best guesses for preference order, the issue still applied, just not quite so badly.
In the “what if” department, I had said:
> I’m also getting remarkably higher numbers for Hag compared with my earlier method. But I don’t immediately see a way to profitably exploit this.
The most obvious way to exploit this would have been the optimal solution. Why didn’t I do it? The answer is that, as indicated above, I was still underestimating the hag (whereas at this point I had mostly-accurate scores for the traps and orcs). With my underestimate for the hag’s score contribution, I didn’t think it was worth giving up an orc-boulder trap difference to get a hag-orc difference. I also didn’t realize I needed the hag to alert the dragon.
In general, I feel like I was pretty far along with discovering the mechanics despite some missteps. I correctly had the adventurers taking a 5-encounter path with right/down steps, the choice of next step being based on the encounters in the choices for the next room, with an alerting mechanism, and that the alerting mechanism didn’t apply to traps and golems.
On the other hand, I applied the alerting mechanism only to score and not to preference order, except for goblins and orcs (why didn’t I try to apply it to preference order for other encounters once I realized it applied to preference order for goblins and orcs and that some degree of alerting mechanism score effect applied to other encounters ?????) (I also got confused into thinking that the effect on orc preference order only applied if the current encounter was also orcs). I also didn’t realize that the alerting mechanism had different sensitivity for different encounters, and I had my mistaken belief about the preference order being different from expected score for some encounter types (hey, the text played up how unnerving the hag was, there was some plausibility there!).
I think if I had gotten to where I was in my last edit early on in the time frame for this scenario instead of near the end, and had posted it, and other people had read it and tried it out, collectively we would have had a good chance of solving the whole thing. I also would have been much more likely to get the optimal solution if I had paid more attention to what abstractapplic said, instead of only very briefly glancing over his comments after posting my very belated comment and going back to doing my own thing.
In my view, a fun, challenging and theoretically solvable scenario (even if actually not that close to being solved in practice), so I think it was quite good.
Yes, that’s a sneaky part of the scenario. In general, I think this is a realistic thing to occur: ‘other intelligent people optimizing around this data’ is one of the things that causes the most complicated things to happen in real-world data as well.
Christian Z R had a very good comment on this, where they mentioned looking at the subset of dungeons where Rooms 2 and 4 had the same encounter, or where Rooms 6 and 8 had the same encounter, to factor out the impact of intelligence and guarantee ‘they will encounter this specific thing’.
(Edited to add: actually, there are ~100 rows in the dataset where Room2=4, Room6=8, and Room3=5=7. This isn’t enough to get firm analysis on, but it could have served as a very strong sanity-check opportunity where you can look at a few dungeons where you know exactly what the route is.)
actually, there are ~100 rows in the dataset where Room2=4, Room6=8, and Room3=5=7.
I actually did look at that (at least some subset with that property) at some point, though I didn’t (think of/ get around to) re-looking at it with my later understanding.
In general, I think this is a realistic thing to occur: ‘other intelligent people optimizing around this data’ is one of the things that causes the most complicated things to happen in real-world data as well.
Indeed, I am not complaining! It was a good, fair difficulty to deal with.
That being said, there was one aspect I did feel was probably more complicated than ideal, and that was the combination of the tier-dependent alerting with the tiers not having any other relevance than this one aspect. That is, if the alerting had in each case been simply dependent on whether the adventurers were coming from an empty room or not, it would have been a lot simpler to work out. And if there was tier dependent alerting, but the tiers were more obvious in other ways*, it would still have been tricky but at least there would be a path to recognize the tiers and then try to figure out other ways that they might have relevance. The way it was it seemed to me you pretty much had to look at what were (ex ante) almost arbitrary combinations of (current encounter, next encounter) to figure that aspect out, unless you actually guessed the rationale of the alerting effect.
That might be me rationalizing my failure to figure it out though!
* e.g. perhaps the traps/golems could have had the same score as the same-tier nontrap encounter when alerted (or alternatively when not alerted)
Mostly fair, but tiers did have a slight other impact in that they were used to bias the final room: Clay Golem and Hag were equally more-likely to be in the final room, both less so than Dragon and Steel Golem but more so than Orcs and Boulder Trap.
I feel like a big part of what tripped me up here was an inevitable part of the difficulty of the scenario that in retrospect should have been obvious. Specifically, if there is any variation in difficulty of an encounter that is known to the adventurers in advance, the score contribution of an encounter type in actual paths taken is less than the difficulty of the encounter as estimated by what best predicts the path taken (because the adventurer takes the path when it’s weak, but avoids when it’s strong).
So, I wound up with an epicycle saying hags and orcs were avoided more than their actual scores warranted, because that effect was most significant for them (goblins are chosen over most other encounters even if alerted, and Dragons mostly aren’t alerted).
This effect was made much worse by the fact that I was getting scores mainly from lower difficulty dungeons, with lots of “Nothing” rooms and low level encounters. But even once I estimated scores from the overall data with my best guesses for preference order, the issue still applied, just not quite so badly.
In the “what if” department, I had said:
> I’m also getting remarkably higher numbers for Hag compared with my earlier method. But I don’t immediately see a way to profitably exploit this.
The most obvious way to exploit this would have been the optimal solution. Why didn’t I do it? The answer is that, as indicated above, I was still underestimating the hag (whereas at this point I had mostly-accurate scores for the traps and orcs). With my underestimate for the hag’s score contribution, I didn’t think it was worth giving up an orc-boulder trap difference to get a hag-orc difference. I also didn’t realize I needed the hag to alert the dragon.
In general, I feel like I was pretty far along with discovering the mechanics despite some missteps. I correctly had the adventurers taking a 5-encounter path with right/down steps, the choice of next step being based on the encounters in the choices for the next room, with an alerting mechanism, and that the alerting mechanism didn’t apply to traps and golems.
On the other hand, I applied the alerting mechanism only to score and not to preference order, except for goblins and orcs (why didn’t I try to apply it to preference order for other encounters once I realized it applied to preference order for goblins and orcs and that some degree of alerting mechanism score effect applied to other encounters ?????) (I also got confused into thinking that the effect on orc preference order only applied if the current encounter was also orcs). I also didn’t realize that the alerting mechanism had different sensitivity for different encounters, and I had my mistaken belief about the preference order being different from expected score for some encounter types (hey, the text played up how unnerving the hag was, there was some plausibility there!).
I think if I had gotten to where I was in my last edit early on in the time frame for this scenario instead of near the end, and had posted it, and other people had read it and tried it out, collectively we would have had a good chance of solving the whole thing. I also would have been much more likely to get the optimal solution if I had paid more attention to what abstractapplic said, instead of only very briefly glancing over his comments after posting my very belated comment and going back to doing my own thing.
In my view, a fun, challenging and theoretically solvable scenario (even if actually not that close to being solved in practice), so I think it was quite good.
Yes, that’s a sneaky part of the scenario. In general, I think this is a realistic thing to occur: ‘other intelligent people optimizing around this data’ is one of the things that causes the most complicated things to happen in real-world data as well.
Christian Z R had a very good comment on this, where they mentioned looking at the subset of dungeons where Rooms 2 and 4 had the same encounter, or where Rooms 6 and 8 had the same encounter, to factor out the impact of intelligence and guarantee ‘they will encounter this specific thing’.
(Edited to add: actually, there are ~100 rows in the dataset where Room2=4, Room6=8, and Room3=5=7. This isn’t enough to get firm analysis on, but it could have served as a very strong sanity-check opportunity where you can look at a few dungeons where you know exactly what the route is.)
I actually did look at that (at least some subset with that property) at some point, though I didn’t (think of/ get around to) re-looking at it with my later understanding.
Indeed, I am not complaining! It was a good, fair difficulty to deal with.
That being said, there was one aspect I did feel was probably more complicated than ideal, and that was the combination of the tier-dependent alerting with the tiers not having any other relevance than this one aspect. That is, if the alerting had in each case been simply dependent on whether the adventurers were coming from an empty room or not, it would have been a lot simpler to work out. And if there was tier dependent alerting, but the tiers were more obvious in other ways*, it would still have been tricky but at least there would be a path to recognize the tiers and then try to figure out other ways that they might have relevance. The way it was it seemed to me you pretty much had to look at what were (ex ante) almost arbitrary combinations of (current encounter, next encounter) to figure that aspect out, unless you actually guessed the rationale of the alerting effect.
That might be me rationalizing my failure to figure it out though!
* e.g. perhaps the traps/golems could have had the same score as the same-tier nontrap encounter when alerted (or alternatively when not alerted)
Mostly fair, but tiers did have a slight other impact in that they were used to bias the final room: Clay Golem and Hag were equally more-likely to be in the final room, both less so than Dragon and Steel Golem but more so than Orcs and Boulder Trap.