Bailey: GPTs are Predictors, not Imitators (nor Simulators).
Motte: The training task for GPTs is a prediction task.
The title and the concluding sentence both plainly advocate for (1), but it’s not really touched by the overall post, and I think it’s up for debate (related: reward is not the optimization target). Instead there is an argument for (2). Perhaps the intention of the final sentence was to oppose Simulators? If that’s the case, cite it, be explicit. This could be a really easy thing for an editor to fix.
Does this look like a motte-and-bailey to you?
Bailey: The task that GPTs are being trained on is … harder than the task of being a human.
Motte: Being an actual human is not enough to solve GPT’s task.
As I read it, (1) is false, the task of being a human doesn’t cap out at human intelligence. More intelligent humans are better at minimizing prediction error, achieving goals, inclusive genetic fitness, whatever you might think defines “the task of being a human”. In the comments, Yudkowsky retreats to (2), which is true. But then how should I understand this whole paragraph from the post?
And since the task that GPTs are being trained on is different from and harder than the task of being a human, it would be surprising—even leaving aside all the ways that gradient descent differs from natural selection—if GPTs ended up thinking the way humans do, in order to solve that problem.
If we’re talking about how natural selection trained my genome, why are we talking about how well humans perform the human task? Evolution is optimizing over generations. My human task is optimizing over my lifetime. Also, if we’re just arguing for different thinking, surely it mostly matters whether the training task is different, not whether it is harder?
Overall I think “Is GPT-N bounded by human capabilities? No.” is a better post on the mottes and avoids staking out unsupported baileys. This entire topic is becoming less relevant because AIs are getting all sorts of synthetic data and RLHF and other training techniques thrown at them. The 2022 question of the capabilities of a hypothetical GPT-N that was only trained on the task of predicting human text is academic in 2024. On the other hand, it’s valuable for people to practice on this simpler question before moving on to harder ones.
Does this look like a motte-and-bailey to you?
Bailey: GPTs are Predictors, not Imitators (nor Simulators).
Motte: The training task for GPTs is a prediction task.
The title and the concluding sentence both plainly advocate for (1), but it’s not really touched by the overall post, and I think it’s up for debate (related: reward is not the optimization target). Instead there is an argument for (2). Perhaps the intention of the final sentence was to oppose Simulators? If that’s the case, cite it, be explicit. This could be a really easy thing for an editor to fix.
Does this look like a motte-and-bailey to you?
Bailey: The task that GPTs are being trained on is … harder than the task of being a human.
Motte: Being an actual human is not enough to solve GPT’s task.
As I read it, (1) is false, the task of being a human doesn’t cap out at human intelligence. More intelligent humans are better at minimizing prediction error, achieving goals, inclusive genetic fitness, whatever you might think defines “the task of being a human”. In the comments, Yudkowsky retreats to (2), which is true. But then how should I understand this whole paragraph from the post?
If we’re talking about how natural selection trained my genome, why are we talking about how well humans perform the human task? Evolution is optimizing over generations. My human task is optimizing over my lifetime. Also, if we’re just arguing for different thinking, surely it mostly matters whether the training task is different, not whether it is harder?
Overall I think “Is GPT-N bounded by human capabilities? No.” is a better post on the mottes and avoids staking out unsupported baileys. This entire topic is becoming less relevant because AIs are getting all sorts of synthetic data and RLHF and other training techniques thrown at them. The 2022 question of the capabilities of a hypothetical GPT-N that was only trained on the task of predicting human text is academic in 2024. On the other hand, it’s valuable for people to practice on this simpler question before moving on to harder ones.