Running Lightcone Infrastructure, which runs LessWrong. You can reach me at habryka@lesswrong.com
habryka(Oliver Habryka)
Karma: 31,861
You can! Just go to the all-posts page, sort by year, and the highest-rated shortform posts for each year will be in the Quick Takes section:
2024:
2023:
2022:
Promoted to curated: Formalizing what it means for transformers to learn “the underlying world model” when engaging in next-token prediction tasks seems pretty useful, in that it’s an abstraction that I see used all the time when discussing risks from models where the vast majority of the compute was spent in pre-training, where the details usually get handwaived. It seems useful to understand what exactly we mean by that in more detail.
I have not done a thorough review of this kind of work, but it seems to me that also others thought the basic ideas in the work hold up, and I thought reading this post gave me crisper abstractions to talk about this kind of stuff in the future.
Don’t really think this makes sense as a tag page. Too subjective.
Three is a bit much. I am honestly not sure what’s better. My guess is putting them all into one. (Context, I am one of the LTFF fund managers)
Yeah, I feel kind of excited about having some strong-downvote and strong-upvote UI which gives you one of a standard set of options for explaining your vote, or allows you to leave it unexplained, all anonymous.
I edited the top-comment to do that.
We’ll send out location details to anyone who buys a ticket (and also feel free to ping us and we’ll tell you).
I’ve had some experience with people trying to disrupt events, and trivial inconveniences of figuring out the address makes a non negligible difference in people doing stuff like that.
I tried setting up an account, but it just told me it had sent me an email to confirm my account that never arrived.
GDPR is a giant mess, so it’s pretty unclear what it requires us to implement. My current understanding is that it just requires us to tell you that we are collecting analytics data if you are from the EU.
And the kind of stuff we are sending over to Recombee would be covered by it being data necessary to provide site functionality, not just analytics, so wouldn’t be covered by that (if you want to avoid data being sent to Google Analytics in-particular, you can do that by just blocking the GA script in uBlock origin or whatever other adblocker you use, which it should do by default).
I am pretty excited about doing something more in-house, but it’s much easier to get data about how promising this direction is by using some third-party services that already have all the infrastructure.
If it turns out to be a core part of LW, it makes more sense to in-house it. It’s also really valuable to have an relatively validated baseline to compare things to.
There are a bunch of third-party services we couldn’t really replace that we send user data to. Hex.tech as our analytics dashboard service. Google Analytics for basic user behavior and patterns. A bunch of AWS services. Implementing the functionality of all of that ourselves, or putting a bunch of effort into anonymizing the data is not impossible, but seems pretty hard, and Recombee seems about par for the degree to which I trust them to not do anything with that data themselves.
Mod note: I clarified the opening note a bit more, to make the start and nature of the essay more clear.
If you have recommendations, post them! I doubt the author tried to filter the subjects very much by “book subjects” it’s just what people seem to have found good ones so far.
This probably should be made more transparent, but the reason why these aren’t in the library is because they don’t have images for the sequence-item. We display all sequences that people create that have proper images on the library (otherwise we just show it on user’s profiles).
I think this just doesn’t work very well, because it incentivizes the model to output a token which makes subsequent tokens easier to predict, as long as the benefit in predictability of the subsequent token(s) outweighs the cost of the first token.
Hmm, this doesn’t sound right. The ground truth data would still be the same, so if you were to predict “aaaaaa” you would get the answer wrong. In the above example, you are presumably querying the log props of the model that was trained on 1-token prediction, which of course would think it’s quite likely that conditional on the last 10 characters being “a” the next one will be “a”, but I am saying “what is the probability of the full completion ‘a a a a a...’ given the prefix ‘Once upon a time, there was a’”, which doesn’t seem very high.
The only thing I am saying here is “force the model to predict more than one token at a time, conditioning on its past responses, then evaluate the model on performance of the whole set of tokens”. I didn’t think super hard about what the best loss function here is, and whether you would have to whip out PPO for this. Seems plausible.
Yeah, I was indeed confused, sorry. I edited out the relevant section of the dialogue and replaced it with the correct relevant point (the aside here didn’t matter because a somewhat stronger condition is true, which is that during training we always just condition on the right answer instead of conditioning on the output for the next token in the training set).
In autoregressive transformers an order is imposed by masking, but all later tokens attend to all earlier tokens in the same way.
Yeah, the masking is what threw me off. I was trying to think about whether any information would flow from the internal representations used to predict the second token to predicting the third token, and indeed, if you were to backpropagate the error after each specific token prediction, then there would be some information from predicting the second token available to predicting the third token (via the the updated weights).
However, batch-sizes make this also inapplicable (I think you would basically never do a backpropagation after each token, that would kind of get rid of the whole benefit of parallel training), and even without that, the amount of relevant information flowing this way would be very miniscule and there wouldn’t be any learning going for how this information flows.
I reference this in this section:
I do think saying “the system is just predicting one token at a time” is wrong, but I guess the way the work a transformer puts into token N gets rewarded or punished when it predicts token N + M feels really weird and confusing to me and still like it can be summarized much more as “it’s taking one token at a time” than “it’s doing reasoning across the whole context
IIRC at least for a standard transformer (which maybe had been modified with the recent context length extension) the gradients only flow through a subset of the weights (for a token halfway through the context, the gradients flow through half the weights that were responsible for the first token, IIRC).
I think you are talking about a different probability distribution here.
You are right that this allows you to sample non-greedily from the learned distribution over text, but I was talking about the inductive biases on the model.
My claim was that the way LLMs are trained, the way the inductive biases shake out is that the LLM won’t be incentivized to output tokens that predictably have low probability, but make it easier to predict future tokens (by, for example, in the process of trying to predict a proof, reminding itself of all the of the things its knows before those things leave its context window, or when doing an addition that it can’t handle in a single forward pass, outputting a token that’s optimized to give itself enough serial depth to perform the full addition of two long n-digit digit numbers, which would then allow it to get the next n tokens right and so overall achieve lower joint loss).
Goal oriented cognition in “a single forward pass”
Yeah, I am also not seeing anything. Maybe it was something temporary, but I thought we had set it up to leave a trace if any automatic rate limits got applied in the past.
Curious what symptom Nora observed (GreaterWrong has been having some problems with rate-limit warnings that I’ve been confused by, so I can imagine that looking like a rate-limit from our side).
@johnswentworth I think this paper basically does the thing I was talking about (with pretty impressive results), though I haven’t read it in a ton of detail: https://news.ycombinator.com/item?id=40220851