[An earlier version of this comment used teams that had accidentally been pessimized rather than optimized according to my model, as the result of a sign error. I think I have fixed it, though I make a lot of mistakes and I would be unsurprised if something were still broken, and the teams below might now make more sense. Thanks to Alumium for encouraging me to check for errors, though in fact I was already highly suspicious that I had made at least one.]
[After that, I thought I ought to check for overfitting, which I hadn’t done before because I’m lazy, and try a bit harder to avoid it. This resulted in a change to my regular team below. I was prompted to do this by the fact that Maxwell did something broadly similar to what I’d done, but more carefully, and got very different-looking results. I also tried something smarter for the PvP, driven mostly by realising that it would run quite efficiently using sklearn despite feeling expensive.]
Regular team:
ABGLP
PvP team:
BGNPT
I confess that here I just
made a random-forest model, tried all possible hands against DGPQT for the regular team, and ran some tournaments using that model to adjudicate for the PvP team. I haven’t made any attempt to use my brain to find any patterns, and it’s entirely possible that I’ve missed important things.
I hadn’t looked at anyone else’s comments before writing the first version of this. I did look at others’ comments between writing the first version of the above and correcting a sign-error bug, but I have made no other changes in my code. (In particular, the process that produced my PvP entry is in no way adapted to the entries others have already submitted.)
There is still an excellent chance, despite my having fixed one catastrophic bug, that I have made boneheaded errors that result in my teams actually being pessimized rather than optimized, having everyone off-by-one in the list of heroes, etc. :-)
My decision to do this fairly brainlessly wasn’t purely the result of laziness. OP is clearly intended to indicate that the game is super-complicated, so I wasn’t optimistic about the chances of spotting enough patterns to do better than a brainless approach. A few more technical details in case my entries turn out to do well enough or badly enough that anyone wants to learn from my success or failure:
The random forest model was built using the RandomForestClassifier class in the scikit-learn library for Python. It uses an ensemble of size 1000 and other parameters had their default values. It’s rather slow so for the tournament I used a cheaper random forest model. For PvP, what I did at first was to run 10 sub-tournaments with 30 players in each, initially selected at random. Each tournament is played round-robin and then three badly-performing hands are replaced by mutated versions of three well-performing hands. Scores in the tournament are not pure win/loss but are the total of the win probabilities according to the random forest model. After every 10 rounds in each tournament, I play another tournament using 3 randomly chosen players from each sub-tournament, again doing the replacing thing (so we get a little bit of mixing between sub-tournaments, but hopefully not so much that all diversity is lost too quickly). I ran a couple of hundred iterations of this and it seemed to have more or less converged. To choose my final PvP hand, I ran a single tournament on the best three hands from each subtournament using the more expensive and hopefully better random-forest model. -- But afterwards I thought better of this and decided instead to use the “multiplicative weights update” approach: just take all the possible teams and give each an initial “weight” of 1, then repeatedly play each against a random opponent and nudge its weight up or down according to the result, and choose the random opponents proportionally to the weights. Exactly what team comes out on top depends on exactly how you do the weight-nudging.
Man, it’s hard to say “check your work” without being rude.
I understood about half of what you said, so your analysis was almost certainly better than my very basic one. But according to my basic analysis you’ve submitted an elementally imbalanced team which manages to also be weak. Um, no offense. Sorry.
It’s just I’d want someone to correct me if I looked wrong.
It looks as if I did indeed have a sign error, so both my teams as currently constituted are pessimal rather than optimal according to my model. Working on fixing that now.
No offence taken. I did say there’s an excellent chance that I’ve made boneheaded errors, though, and I stand by that :-). I may or may not find time before the deadline to look over what I did and see if there are any errors boneheaded that even I can find them[1].
[1] I don’t mean that I’m exceptionally stupid; the point is that one’s own errors are usually extra-hard to spot.
[An earlier version of this comment used teams that had accidentally been pessimized rather than optimized according to my model, as the result of a sign error. I think I have fixed it, though I make a lot of mistakes and I would be unsurprised if something were still broken, and the teams below might now make more sense. Thanks to Alumium for encouraging me to check for errors, though in fact I was already highly suspicious that I had made at least one.]
[After that, I thought I ought to check for overfitting, which I hadn’t done before because I’m lazy, and try a bit harder to avoid it. This resulted in a change to my regular team below. I was prompted to do this by the fact that Maxwell did something broadly similar to what I’d done, but more carefully, and got very different-looking results. I also tried something smarter for the PvP, driven mostly by realising that it would run quite efficiently using sklearn despite feeling expensive.]
Regular team:
ABGLP
PvP team:
BGNPT
I confess that here I just
made a random-forest model, tried all possible hands against DGPQT for the regular team, and ran some tournaments using that model to adjudicate for the PvP team. I haven’t made any attempt to use my brain to find any patterns, and it’s entirely possible that I’ve missed important things.
I hadn’t looked at anyone else’s comments before writing the first version of this. I did look at others’ comments between writing the first version of the above and correcting a sign-error bug, but I have made no other changes in my code. (In particular, the process that produced my PvP entry is in no way adapted to the entries others have already submitted.)
There is still an excellent chance, despite my having fixed one catastrophic bug, that I have made boneheaded errors that result in my teams actually being pessimized rather than optimized, having everyone off-by-one in the list of heroes, etc. :-)
My decision to do this fairly brainlessly wasn’t purely the result of laziness. OP is clearly intended to indicate that the game is super-complicated, so I wasn’t optimistic about the chances of spotting enough patterns to do better than a brainless approach. A few more technical details in case my entries turn out to do well enough or badly enough that anyone wants to learn from my success or failure:
The random forest model was built using the RandomForestClassifier class in the scikit-learn library for Python. It uses an ensemble of size 1000 and other parameters had their default values. It’s rather slow so for the tournament I used a cheaper random forest model. For PvP, what I did at first was to run 10 sub-tournaments with 30 players in each, initially selected at random. Each tournament is played round-robin and then three badly-performing hands are replaced by mutated versions of three well-performing hands. Scores in the tournament are not pure win/loss but are the total of the win probabilities according to the random forest model. After every 10 rounds in each tournament, I play another tournament using 3 randomly chosen players from each sub-tournament, again doing the replacing thing (so we get a little bit of mixing between sub-tournaments, but hopefully not so much that all diversity is lost too quickly). I ran a couple of hundred iterations of this and it seemed to have more or less converged. To choose my final PvP hand, I ran a single tournament on the best three hands from each subtournament using the more expensive and hopefully better random-forest model. -- But afterwards I thought better of this and decided instead to use the “multiplicative weights update” approach: just take all the possible teams and give each an initial “weight” of 1, then repeatedly play each against a random opponent and nudge its weight up or down according to the result, and choose the random opponents proportionally to the weights. Exactly what team comes out on top depends on exactly how you do the weight-nudging.
Man, it’s hard to say “check your work” without being rude. I understood about half of what you said, so your analysis was almost certainly better than my very basic one. But according to my basic analysis you’ve submitted an elementally imbalanced team which manages to also be weak. Um, no offense. Sorry. It’s just I’d want someone to correct me if I looked wrong.
It looks as if I did indeed have a sign error, so both my teams as currently constituted are pessimal rather than optimal according to my model. Working on fixing that now.
No offence taken. I did say there’s an excellent chance that I’ve made boneheaded errors, though, and I stand by that :-). I may or may not find time before the deadline to look over what I did and see if there are any errors boneheaded that even I can find them[1].
[1] I don’t mean that I’m exceptionally stupid; the point is that one’s own errors are usually extra-hard to spot.