Post-mortem on my thinking about the sides being asymmetrical:
In order to determine whether there was symmetry, I applied the model to the following datasets and compared the validation scores:
The original data set
A flipped version of the data set
a “cut” version of the data set where some of the data points were flipped and others not (should contains all games in one form or the other, and not have any of the same games)
the complement of the above (everything flipped rather than 3
On finding that 1 and 2 had better validation scores than 3 and 4, and the gap between 1⁄2 and 3⁄4 was larger (but really not all that much larger!) than the gap between 1 and 2 or 3 and 4, I declared that there was asymmetry.
But, really this was totally invalid, because, 1 and 2 are isomorphic to each other under a column swap and bit flip (as pointed out by Maxwell) and while this transformation may affect the results it should not affect validation scores if the algorithm is unbiased, up to random variation if it has random elements (I don’t know if it does, but the validation scores were not actually identical). Likewise, 3 and 4 should have the same validation scores. On the other hand, 1⁄2 are not isomorphic to 3⁄4 up to such a transformation and so have no need to have the same validation scores. So there was a 50% chance of the observed result happening by chance.
Even if my method would have worked the way I had been thinking**, it would be a pretty weak* test. So why was I so willing to believe it? Well, in my previous analysis I had noticed differences in the sides, which might or might not be random, particularly the winrate for games with Greenery Giant on both sides. In such matchups, green wins 1192, (ironically) much less than blue’s 1308. This is not at all unlikely (less than 2 sigma (which I didn’t check), and many other possible hypotheses) but this plus the knowledge that League of Legend’s map is, while almost symmetrical, not perfectly so, led me to have a too-weak prior against asymmetry when going into poking Maxwell’s magic box.
Regardless of this mistake, I do think that my choice to create and use a merged data set including the original data and the flipped data was correct. Given that we either don’t care about asymmetries or don’t believe they exist, the ideal thing to do would be to add some kind of constraint to the learning algorithm to respect the assumed symmetry, but this is an easier alternative.
*Edit: in the sense of providing weak Bayesian evidence due to high false positive rate
**Edit: which I could have made happen by comparing results from disjoint subsets of the data in each of 1 and 2, etc.
Post-mortem on my thinking about the sides being asymmetrical:
In order to determine whether there was symmetry, I applied the model to the following datasets and compared the validation scores:
The original data set
A flipped version of the data set
a “cut” version of the data set where some of the data points were flipped and others not (should contains all games in one form or the other, and not have any of the same games)
the complement of the above (everything flipped rather than 3
On finding that 1 and 2 had better validation scores than 3 and 4, and the gap between 1⁄2 and 3⁄4 was larger (but really not all that much larger!) than the gap between 1 and 2 or 3 and 4, I declared that there was asymmetry.
But, really this was totally invalid, because, 1 and 2 are isomorphic to each other under a column swap and bit flip (as pointed out by Maxwell) and while this transformation may affect the results it should not affect validation scores if the algorithm is unbiased, up to random variation if it has random elements (I don’t know if it does, but the validation scores were not actually identical). Likewise, 3 and 4 should have the same validation scores. On the other hand, 1⁄2 are not isomorphic to 3⁄4 up to such a transformation and so have no need to have the same validation scores. So there was a 50% chance of the observed result happening by chance.
Even if my method would have worked the way I had been thinking**, it would be a pretty weak* test. So why was I so willing to believe it? Well, in my previous analysis I had noticed differences in the sides, which might or might not be random, particularly the winrate for games with Greenery Giant on both sides. In such matchups, green wins 1192, (ironically) much less than blue’s 1308. This is not at all unlikely (less than 2 sigma (which I didn’t check), and many other possible hypotheses) but this plus the knowledge that League of Legend’s map is, while almost symmetrical, not perfectly so, led me to have a too-weak prior against asymmetry when going into poking Maxwell’s magic box.
Regardless of this mistake, I do think that my choice to create and use a merged data set including the original data and the flipped data was correct. Given that we either don’t care about asymmetries or don’t believe they exist, the ideal thing to do would be to add some kind of constraint to the learning algorithm to respect the assumed symmetry, but this is an easier alternative.
*Edit: in the sense of providing weak Bayesian evidence due to high false positive rate
**Edit: which I could have made happen by comparing results from disjoint subsets of the data in each of 1 and 2, etc.