I would estimate that the difference between “hire some mechanical turkers and have them think for like a few seconds” and the actual data collection process accounts for around 1⁄3 of the effort that went into WebGPT, rising to around 2⁄3 if you include model assistance in the form of citations. So I think that what you wrote gives a misleading impression of the aims and priorities of RLHF work in practice.
I think it’s best to err on the side of not saying things that are false in a literal sense when the distinction is important to other people, even when the distinction isn’t important to you—although I can see why you might not have realized the importance of the distinction to others from reading papers alone, and “a few minutes” is definitely less inaccurate.
Sorry, yeah, I definitely just messed up in my comment here in the sense that I do think that after looking at the research, I definitely should have said “spent a few minutes on each datapoint”, instead of “a few seconds” (and indeed I noticed myself forgetting that I had said “seconds” instead of “minutes” in the middle of this conversation, which also indicates I am a bit triggered and doing an amount of rhetorical thinking and weaseling that I think is pretty harmful, and I apologize for kind of sliding between seconds and minutes in my last two comments).
I think the two orders of magnitude of time spent evaluating here is important, and though I don’t think it changes my overall answer very much, I do agree with you that it’s quite important to not give literal falsehoods especially when I am aware that other people care about the details here.
I do think the distinction between Mechanical Turkers and Scale AI/Upwork is pretty minimal, and I think what I said in that respect is fine. I don’t think the people you used were much better educated than the average mechanical turker, though I do think one update most people should make here is towards “most mechanical turkers are actually well-educated americans”, and I do think there is something slightly rhetorically tricky going on when I just say “random mechanical turkers” which I think people might misclassify as being less educated and smart than they actually are.
I do think a revised summary sentence “most RLHF as currently practiced is mostly just Mechanical Turkers with like half an hour of training and a reward button thinking about each datapoint for a few minutes” seems accurate to me, and feels like an important thing to understand when thinking about the question of “why doesn’t RLHF just solve AI Alignment?”.
I would estimate that the difference between “hire some mechanical turkers and have them think for like a few seconds” and the actual data collection process accounts for around 1⁄3 of the effort that went into WebGPT, rising to around 2⁄3 if you include model assistance in the form of citations. So I think that what you wrote gives a misleading impression of the aims and priorities of RLHF work in practice.
I think it’s best to err on the side of not saying things that are false in a literal sense when the distinction is important to other people, even when the distinction isn’t important to you—although I can see why you might not have realized the importance of the distinction to others from reading papers alone, and “a few minutes” is definitely less inaccurate.
Sorry, yeah, I definitely just messed up in my comment here in the sense that I do think that after looking at the research, I definitely should have said “spent a few minutes on each datapoint”, instead of “a few seconds” (and indeed I noticed myself forgetting that I had said “seconds” instead of “minutes” in the middle of this conversation, which also indicates I am a bit triggered and doing an amount of rhetorical thinking and weaseling that I think is pretty harmful, and I apologize for kind of sliding between seconds and minutes in my last two comments).
I think the two orders of magnitude of time spent evaluating here is important, and though I don’t think it changes my overall answer very much, I do agree with you that it’s quite important to not give literal falsehoods especially when I am aware that other people care about the details here.
I do think the distinction between Mechanical Turkers and Scale AI/Upwork is pretty minimal, and I think what I said in that respect is fine. I don’t think the people you used were much better educated than the average mechanical turker, though I do think one update most people should make here is towards “most mechanical turkers are actually well-educated americans”, and I do think there is something slightly rhetorically tricky going on when I just say “random mechanical turkers” which I think people might misclassify as being less educated and smart than they actually are.
I do think a revised summary sentence “most RLHF as currently practiced is mostly just Mechanical Turkers with like half an hour of training and a reward button thinking about each datapoint for a few minutes” seems accurate to me, and feels like an important thing to understand when thinking about the question of “why doesn’t RLHF just solve AI Alignment?”.