Sure, but coming up with what to try, which hyperparameters to adjust, which heuristics to apply, etc. is not something for which we have a meaningful programme. You can’t brute force taste!
(Even if we eventually CAN, we’ve got a long way to go before we can disregard the intentions of the researchers entirely. You can’t Goodhart taste either!)
Everything benefits from scaling compute, because with more compute you can indiscriminately try more of everything, apply more meta-learning, etc.
Sure, but coming up with what to try, which hyperparameters to adjust, which heuristics to apply, etc. is not something for which we have a meaningful programme. You can’t brute force taste!
(Even if we eventually CAN, we’ve got a long way to go before we can disregard the intentions of the researchers entirely. You can’t Goodhart taste either!)