I think it’s important to understand that the two explanations I gave in the post can work together. After more than a year, I would state my current beliefs as something closer to the following thesis:
Given equal background and motivation, there is a lot less inequality in the rates human learn new tasks, compared to the inequality in how humans perform learned tasks. By “less inequality” I don’t mean “roughly equal” as your prediction-specifications would indicate; the reason is because human learning rates are still highly unequal, despite the fact that nearly all humans have similar neural architectures. As I explained in section two of the post, a similar architecture does not imply similar performance. A machine with a broken part is nearly structurally identical to a machine with no broken parts, yet it does not work.
Given equal background and motivation, there is a lot less inequality in the rates human learn new tasks, compared to the inequality in how humans perform learned tasks.
Huh, my guess is the opposite. That is, all expert plumbers are similarly competent at performing tasks, and the thing that separate a bright plumber from a dull plumber is how quickly they become expert.
Quite possibly we’re looking at different tasks? I’d be interested in examples of domains where this sort of thing has been quantized and you see the hypothesized relationship (where variation in learning speed is substantially smaller than variation in performance). Most of the examples I can think of that seem plausible are exercise-related, where you might imagine people learn proper technique with a tighter distribution than the underlying strength distribution, but this is cheating by using intellectual and physical faculties as separate sources of variation.
Huh, my guess is the opposite. That is, all expert plumbers are similarly competent at performing tasks, and the thing that separate a bright plumber from a dull plumber is how quickly they become expert.
I don’t know much about plumbing, but a possible explanation here is that there’s a low skill ceiling in plumbing. If we were talking about chess, or Go, or other tasks with high skill ceilings, it definitely seems incorrect that the brightest players won’t be much better than the dullest players.
This also strikes me as backwards, and the literature seems to back this up. Learning rates seem to differ a lot between different people, and also be heavily g-loaded.
Do you have thoughts on what I said here? I agree that learning rates differ between people, but I don’t think that contradicts what I said? Perhaps you disagree. From my perspective, what matters more is whether learning rates have more or less variance than the variance of skill on learned tasks (especially those with high skill-ceilings).
Formulations are basically just lifted from the post verbatim, so the response might be some evidence that it would be good to rework the post a bit before people vote on it.
I thought a bit about how to turn Katja’s core claim into a poll question, but didn’t come up with any great ideas. Suggestions welcome.
As for whether the claims are true or not --
The “broken parts” argument is one counter-argument.
But another is that it matters a lot what learning algorithm you use. Someone doing deliberate practice (in a field where that’s possible) will vastly outperform someone who just does “guessing and checking”, or who Goodharts very hard on short-term metrics.
Maybe you’d class that under “background knowledge”? Or maybe the claim is that, modulo broken parts, motivation, and background knowledge, different people can meta-learn the same effective learning strategies?
Formulations are basically just lifted from the post verbatim, so the response might be some evidence that it would be good to rework the post a bit before people vote on it.
But I think I already addressed the fundamental reply at the beginning of the section 2. The theses themselves are lifted from the post verbatim, however, I state that they are incomplete.
Maybe you’d class that under “background knowledge”? Or maybe the claim is that, modulo broken parts, motivation, and background knowledge, different people can meta-learn the same effective learning strategies?
I would really rather avoid making strict claims about learning rates being “roughly equal” and would prefer to talk about how, given the same learning environment (say, a lecture) and backgrounds, human learning rates are closer to equal than human performance in learned tasks.
I think it’s important to understand that the two explanations I gave in the post can work together. After more than a year, I would state my current beliefs as something closer to the following thesis:
Given equal background and motivation, there is a lot less inequality in the rates human learn new tasks, compared to the inequality in how humans perform learned tasks. By “less inequality” I don’t mean “roughly equal” as your prediction-specifications would indicate; the reason is because human learning rates are still highly unequal, despite the fact that nearly all humans have similar neural architectures. As I explained in section two of the post, a similar architecture does not imply similar performance. A machine with a broken part is nearly structurally identical to a machine with no broken parts, yet it does not work.
Huh, my guess is the opposite. That is, all expert plumbers are similarly competent at performing tasks, and the thing that separate a bright plumber from a dull plumber is how quickly they become expert.
Quite possibly we’re looking at different tasks? I’d be interested in examples of domains where this sort of thing has been quantized and you see the hypothesized relationship (where variation in learning speed is substantially smaller than variation in performance). Most of the examples I can think of that seem plausible are exercise-related, where you might imagine people learn proper technique with a tighter distribution than the underlying strength distribution, but this is cheating by using intellectual and physical faculties as separate sources of variation.
I don’t know much about plumbing, but a possible explanation here is that there’s a low skill ceiling in plumbing. If we were talking about chess, or Go, or other tasks with high skill ceilings, it definitely seems incorrect that the brightest players won’t be much better than the dullest players.
This also strikes me as backwards, and the literature seems to back this up. Learning rates seem to differ a lot between different people, and also be heavily g-loaded.
Do you have thoughts on what I said here? I agree that learning rates differ between people, but I don’t think that contradicts what I said? Perhaps you disagree. From my perspective, what matters more is whether learning rates have more or less variance than the variance of skill on learned tasks (especially those with high skill-ceilings).
Formulations are basically just lifted from the post verbatim, so the response might be some evidence that it would be good to rework the post a bit before people vote on it.
I thought a bit about how to turn Katja’s core claim into a poll question, but didn’t come up with any great ideas. Suggestions welcome.
As for whether the claims are true or not --
The “broken parts” argument is one counter-argument.
But another is that it matters a lot what learning algorithm you use. Someone doing deliberate practice (in a field where that’s possible) will vastly outperform someone who just does “guessing and checking”, or who Goodharts very hard on short-term metrics.
Maybe you’d class that under “background knowledge”? Or maybe the claim is that, modulo broken parts, motivation, and background knowledge, different people can meta-learn the same effective learning strategies?
But I think I already addressed the fundamental reply at the beginning of the section 2. The theses themselves are lifted from the post verbatim, however, I state that they are incomplete.
I would really rather avoid making strict claims about learning rates being “roughly equal” and would prefer to talk about how, given the same learning environment (say, a lecture) and backgrounds, human learning rates are closer to equal than human performance in learned tasks.