Yes, that’s right. In some sense they’re evaluating different capabilities—both “can a model find a way to do this task” and “can a model do what humans do on this task” are separate capabilities, and which one you’re more interested in might vary depending on why you care about the capability evaluation. In many cases, “can a model do this task the way humans do it” might be more useful, since e.g. you might care a lot if the model is capable enough to replicate complex human labor, but not really care at all if the model can find some weird hack.
Yes, that’s right. In some sense they’re evaluating different capabilities—both “can a model find a way to do this task” and “can a model do what humans do on this task” are separate capabilities, and which one you’re more interested in might vary depending on why you care about the capability evaluation. In many cases, “can a model do this task the way humans do it” might be more useful, since e.g. you might care a lot if the model is capable enough to replicate complex human labor, but not really care at all if the model can find some weird hack.