It might be that for such tasks solutions using handcrafted rewards/heuristics would be viable, but wouldn’t scale to more complex tasks.
I agree that’s possible. Tbc, we did spend some time thinking about how we might use handcrafted rewards / heuristics to solve the tasks, and eliminated a couple based on this, so I think it probably won’t be true here.
Suppose the designer trains a neural network to recognize trees in minecraft by getting the minecraft engine to generate lots of images of trees. The resulting network is then used as a hardcoded part of the agent architecture. Is that allowed?
No.
If not, how well can you enforce it (I imagine something of the sort can be done in subtler ways)?
For the competition, there’s a ban on pretrained models that weren’t publicly available prior to competition start. We look at participants’ training code to ensure compliance. It is still possible to violate this rule in a way that we may not catch (e.g. maybe you use internal simulator details to do hyperparameter tuning, and then hardcode the hyperparameters in your training code), but it seems quite challenging and not worth the effort even if you are willing to cheat.
For the benchmark (which is what I’m more excited about in the longer run), we’re relying on researchers to follow the rules. Science already relies on researchers honestly reporting their results—it’s pretty hard to catch cases where you just make up numbers for your experimental results.
(Also in the benchmark version, people are unlikely to write a paper about how they solved the task using special-case heuristics; that would be an embarrassing paper.)
I agree that’s possible. Tbc, we did spend some time thinking about how we might use handcrafted rewards / heuristics to solve the tasks, and eliminated a couple based on this, so I think it probably won’t be true here.
No.
For the competition, there’s a ban on pretrained models that weren’t publicly available prior to competition start. We look at participants’ training code to ensure compliance. It is still possible to violate this rule in a way that we may not catch (e.g. maybe you use internal simulator details to do hyperparameter tuning, and then hardcode the hyperparameters in your training code), but it seems quite challenging and not worth the effort even if you are willing to cheat.
For the benchmark (which is what I’m more excited about in the longer run), we’re relying on researchers to follow the rules. Science already relies on researchers honestly reporting their results—it’s pretty hard to catch cases where you just make up numbers for your experimental results.
(Also in the benchmark version, people are unlikely to write a paper about how they solved the task using special-case heuristics; that would be an embarrassing paper.)