One (soft) takeaway from the discussion here is that if training “real-life” modern LLMs involves reasoning in the same reference class as parity, then it is likely that the algorithm they learn is not globally optimal (in a Bayesian sense).
I think this is a crux for me. I don’t have a good guess how common this phenomenon is. The parity problem feels pathological in some sense, but I wouldn’t surprised if there are other classes of problems that would fall into the same category + are represented in some training data.
Yes I agree this is an important crux. I’m not sure which way I lean here. On the one hand, most specific things we can discover about human thinking are highly parallel. On the other hand, it seems very plausible that there are some complicated sequential things going on in the brain that don’t return partial outputs, which are in the same reference class as parity; if this is the case then, insofar as an LLM is reconstructing human brain function, it would need Bayesian-suboptimal “training wheels” to capture these processes.
I think this is a crux for me. I don’t have a good guess how common this phenomenon is. The parity problem feels pathological in some sense, but I wouldn’t surprised if there are other classes of problems that would fall into the same category + are represented in some training data.
Yes I agree this is an important crux. I’m not sure which way I lean here. On the one hand, most specific things we can discover about human thinking are highly parallel. On the other hand, it seems very plausible that there are some complicated sequential things going on in the brain that don’t return partial outputs, which are in the same reference class as parity; if this is the case then, insofar as an LLM is reconstructing human brain function, it would need Bayesian-suboptimal “training wheels” to capture these processes.