It seems like you’re talking about fairness in a way that isn’t responsive to Rekrul’s substantive point, which was about whether the test was unevenly favoring the AI on abilities like extremely high APM, unrelated to what we’d intuitively consider thinking, not about whether the test was generally uninformative.
Oh, I thought it was already plainly obvious to everyone that the victories were in part because of unfair AI advantages. We don’t need to discuss the APM cap for that, that much was already clear from the fact that the version of AlphaStar which had stricter vision limitations lost to MaNa.
That just seems like a relatively uninteresting point to me, since this looks like AlphaStar’s equivalent of the Fan Hui game. That is, it’s obvious that AlphaStar is still below the level of top human pros and wouldn’t yet beat them without unfair advantages, but if the history with AlphaGo is any guide, it’s only a matter of some additional tweaking and throwing more compute at it before it’s at the point where it will beat even the top players while having much stricter limitations in place.
Saying that its victories were *unrelated* to what we’d intuitively consider thinking seems too strong, though. I’m not terribly familiar with SC2, but a lot of the discussion that I’ve seen seems to tend towards AlphaStar’s macro being roughly on par with the pro level, and its superior micro being what ultimately carried the day. E.g. people focus a lot on that one game that MaNa arguably should have won but only lost due to AlphaStar having superhuman micro of several different groups of Stalkers, but I haven’t seen it suggested that all of its victories would have been attributable to that alone: I didn’t get that kind of a vibe from MaNa’s own post-game analysis of those matches, for instance. Nor from TLO’s equivalent analysis (lost the link, sorry) of his matches, where he IIRC only said something like “maybe they should look at the APM limits a bit [for future matches]”.
So it seems to me that even though its capability at what we’d intuitively consider thinking wouldn’t have been enough for winning all the matches, it would still have been good enough for winning several of the ones where people aren’t pointing to the superhuman micro as the sole reason of the victory.
This was definitely not initially obvious to everyone, and I expect many people still have the impression that the victories were not due to unfair AI advantages. I think you should double crux with Raemon on how many words people can be expected to read.
I mostly agree with this comment. My speculative best guess is that the main reason MaNa did better against the revised version of AlphaStar wasn’t due to the vision limitations, but rather some combination of:
MaNa had more time to come up with a good strategy and analyze previous games.
MaNa had more time to warm up, and was generally in a better headspace.
The previous version of AlphaStar was unusually good, and the new version was an entirely new system, so the new version regressed to the mean a bit. (On the dimension “can beat human pros”, even though it was superior on the dimension “can beat other AlphaStar strategies”.)
It seems like you’re talking about fairness in a way that isn’t responsive to Rekrul’s substantive point, which was about whether the test was unevenly favoring the AI on abilities like extremely high APM, unrelated to what we’d intuitively consider thinking, not about whether the test was generally uninformative.
Oh, I thought it was already plainly obvious to everyone that the victories were in part because of unfair AI advantages. We don’t need to discuss the APM cap for that, that much was already clear from the fact that the version of AlphaStar which had stricter vision limitations lost to MaNa.
That just seems like a relatively uninteresting point to me, since this looks like AlphaStar’s equivalent of the Fan Hui game. That is, it’s obvious that AlphaStar is still below the level of top human pros and wouldn’t yet beat them without unfair advantages, but if the history with AlphaGo is any guide, it’s only a matter of some additional tweaking and throwing more compute at it before it’s at the point where it will beat even the top players while having much stricter limitations in place.
Saying that its victories were *unrelated* to what we’d intuitively consider thinking seems too strong, though. I’m not terribly familiar with SC2, but a lot of the discussion that I’ve seen seems to tend towards AlphaStar’s macro being roughly on par with the pro level, and its superior micro being what ultimately carried the day. E.g. people focus a lot on that one game that MaNa arguably should have won but only lost due to AlphaStar having superhuman micro of several different groups of Stalkers, but I haven’t seen it suggested that all of its victories would have been attributable to that alone: I didn’t get that kind of a vibe from MaNa’s own post-game analysis of those matches, for instance. Nor from TLO’s equivalent analysis (lost the link, sorry) of his matches, where he IIRC only said something like “maybe they should look at the APM limits a bit [for future matches]”.
So it seems to me that even though its capability at what we’d intuitively consider thinking wouldn’t have been enough for winning all the matches, it would still have been good enough for winning several of the ones where people aren’t pointing to the superhuman micro as the sole reason of the victory.
This was definitely not initially obvious to everyone, and I expect many people still have the impression that the victories were not due to unfair AI advantages. I think you should double crux with Raemon on how many words people can be expected to read.
I mostly agree with this comment. My speculative best guess is that the main reason MaNa did better against the revised version of AlphaStar wasn’t due to the vision limitations, but rather some combination of:
MaNa had more time to come up with a good strategy and analyze previous games.
MaNa had more time to warm up, and was generally in a better headspace.
The previous version of AlphaStar was unusually good, and the new version was an entirely new system, so the new version regressed to the mean a bit. (On the dimension “can beat human pros”, even though it was superior on the dimension “can beat other AlphaStar strategies”.)