Re (2) it may also be recomputed if the LLM reads that same text later. Or systems operating in the real world might just keep a long context in memory. But I’ll drop this, because maintaining state or not seems somewhat irrelevant.
(1) Yep, current LLM systems are pretty good. I’m not very convinced about generalization. It’s hard to test LLMs on outside distribution problems because currently they tend to just give dumb answers that aren’t that interesting.
(Thinking of some guy who was recently hyped about asking o1 for the solution to quantum gravity—it gave the user some gibberish that he thought looked exciting, which would have been a good move in the RL training environment where the user has a reward button, but is just totally disconnected from how you need to interact with the real world.)
But in a sense that’s my point (well, plus some other errors like sycophancy) - the reasons a present-day LLM uses a word can often be shown to generalize in some dumb way when you challenge it with a situation that the model isn’t well-suited for. This can be true at the same time it’s true that the model is pretty good at morality on the distribution it is competent over. This is still sufficient to show that present systems generalize in some amoral ways, and if we probably disagree about future ststems, this likely comes down to classic AI safetyist arguments about RL incentivizing deceiving of the user as the world-model gets better.
Re (2) it may also be recomputed if the LLM reads that same text later. Or systems operating in the real world might just keep a long context in memory. But I’ll drop this, because maintaining state or not seems somewhat irrelevant.
(1) Yep, current LLM systems are pretty good. I’m not very convinced about generalization. It’s hard to test LLMs on outside distribution problems because currently they tend to just give dumb answers that aren’t that interesting.
(Thinking of some guy who was recently hyped about asking o1 for the solution to quantum gravity—it gave the user some gibberish that he thought looked exciting, which would have been a good move in the RL training environment where the user has a reward button, but is just totally disconnected from how you need to interact with the real world.)
But in a sense that’s my point (well, plus some other errors like sycophancy) - the reasons a present-day LLM uses a word can often be shown to generalize in some dumb way when you challenge it with a situation that the model isn’t well-suited for. This can be true at the same time it’s true that the model is pretty good at morality on the distribution it is competent over. This is still sufficient to show that present systems generalize in some amoral ways, and if we probably disagree about future ststems, this likely comes down to classic AI safetyist arguments about RL incentivizing deceiving of the user as the world-model gets better.
yes, but this is pretty typical for what a human would generate.