But we have no evidence that this homunculus exists inside GPT-4, or any LLM. More pointedly, as LLMs have made remarkable strides toward human-level general intelligence, we have not observed a parallel trend toward becoming “more homuncular,” more like a generally capable agent being pressed into service for next-token prediction.
Uncovering mesa-optimization algorithms in transformers seems to indicate otherwise, though perhaps you could reasonably object to this claim that “optimization is not agency; agency is when optimization is directed at a referent outside the optimizer”. Other than that objection, I think it’s pretty reasonable to conclude that transformers significantly learn optimization algorithms.
Uncovering mesa-optimization algorithms in transformers seems to indicate otherwise, though perhaps you could reasonably object to this claim that “optimization is not agency; agency is when optimization is directed at a referent outside the optimizer”. Other than that objection, I think it’s pretty reasonable to conclude that transformers significantly learn optimization algorithms.
See my comment here, about the predecessor to the paper you linked—the same point applies to the newer paper as well.