If you have any questions for me or just want to talk, feel free to reach out by sending a private message on this site or by sending an e-mail to egeerdil96@gmail.com.
You can also find me on Metaculus at https://www.metaculus.com/accounts/profile/116023/, or on Discord with the username starfall7651.
Yeah, that’s right. Fixed.
If people vote as if their individual vote determines the vote of a non-negligible fraction of the voter pool, then you only need (averaged over the whole population, so the value of the entire population is instead of , which seems much more realistic.
So voting blue can make sense for a sufficiently large coalition of “ordinary altruists” with who are able to pre-commit to their vote and think people outside the coalition might vote blue by mistake etc. rather than the “extraordinary altruists” we need in the original situation with . Ditto if you’re using a decision theory where it makes sense to suppose such a commitment already exists when making your decision.
That would be questioning the assumption that your cost function as an altruist should be linear in the number of lives lost. I’m not sure why you would question this assumption, though; it seems rather unnatural to make this a concave function, which is what you would need for your logic to work.
I’m surprised by how much this post is getting upvoted. It gives us essentially zero information about any question of importance, for reasons that have already been properly explained by other commenters:
Chess is not like the real world in important respects. What the threshold is for material advantage such that a 1200 elo player could beat Stockfish at chess tells us basically nothing about what the threshold is for humans, either individually or collectively, to beat an AGI in some real-world confrontation. This point is so trivial that I feel somewhat embarrassed to be making it, but I have to think that people are just not getting the message here.
Even focusing only on chess, the argument here is remarkably weak because Stockfish is not a system trained to beat weaker opponents with piece odds. There are Go AIs that have been trained for this kind of thing, e.g. KataGo can play reasonably well in positions with a handicap if you tell it that its opponent is much weaker than itself. In my experience, KataGo running on consumer hardware can give the best players in the world 3-4 stones and have an even game.
If someone could try to convince me that this experiment was not pointless and actually worth running for some reason, I would be interested to hear their arguments. Note that I’m more sympathetic to “this kind of experiment could be valuable if ran in the right environment”, and my skepticism is specifically about running it for chess.
Are neural networks trained using reinforcement learning from human feedback in a sufficiently complex environment biased towards learning the human simulator or the direct translator, in the sense of the ELK report?
I think there are arguments in both directions and it’s not obvious which solution a neural network would prefer if trained in a sufficiently complex environment. I also think the question is central to how difficult we should expect aligning powerful systems trained in the current paradigm to be.
I’m curious if these rate limits were introduced as a consequence of some recent developments. Has the website been having more problems with spam and low-quality content lately, or has the marginal benefit of making these changes gone up in some other way?
It could also be that you had this idea only recently and in retrospect it had been a good idea for a long time, of course.
Yes, in practice having a model of what is actually driving the correlations can help you do better than these estimates. A causal model would be helpful for that.
The product estimate for the expected correlation is only useful in a setting where nothing else is known about the relationship between the three variables than the two correlations, but in practice you often have some beliefs about what drives the correlations you observe, and if you’re a good Bayesian you should of course also condition on all of that.
That’s a reasonable picture to have in expectation, yeah.
As an aside, I’ve tried to work out what the optimal learning rate for a large language model should be based on the theory in the post, and if I’m doing the calculations correctly (which is a pretty big if) it doesn’t match actual practice very well, suggesting there is actually something important missing from this picture.
Essentially, the coefficient should be where is the variance of the per-parameter noise in SGD. If you have a learning rate , you scale the objective you’re optimizing by a factor and the noise variance by a factor . Likewise, a bigger batch size lowers the noise variance by a linear factor. So the equilibrium distribution ends up proportional to
where is the per-token average loss and should be equal to the mean square of the partial derivative of the per-token loss function with respect to one of the neural network parameters. If the network is using some decent batch or layer normalization this should probably be where is the model size.
We want what’s inside the exponential to just be , because we want the learning to be equivalent to doing a Bayesian update over the whole data. This suggests we should pick
which is a pretty bad prediction. So there’s probably something important that’s being left out of this model. I’m guessing that a smaller learning rate just means you end up conditioning on minimum loss and that’s all you need to in practice, and larger learning rates cause problems with convergence.
That’s useful to know, thanks. Is anything else known about the properties of the noise covariance beyond “it’s not constant”?
Some comments on the paper itself: if the problem is that SGD with homoskedastic Gaussian noise fails to converge to a stationary distribution, why don’t they define SGD over a torus instead? Seems like it would fix the problem they are talking about, and if it doesn’t change the behavior it means their explanation of what’s going on is incorrect.
If the only problem is that with homoskedastic Gaussian noise convergence to a stationary distribution is slow (when a stationary distribution does exist), I could believe that. Similar algorithms such as Metropolis-Hastings also have pretty abysmal convergence rates in practice when applied to any kind of complicated problem. It’s possible that SGD with batch noise has better regularization properties and therefore converges faster, but I don’t think that changes the basic qualitative picture I present in the post.
Check the Wikipedia section for the stationary distribution of the overdamped Langevin equation.
I should probably clarify that it’s difficult to have a rigorous derivation of this claim in the context of SGD in particular, because it’s difficult to show absence of heteroskedasticity in SGD residuals. Still, I believe that this is probably negligible in practice, and in principle this is something that can be tested by experiment.
Sure, I agree that I didn’t put this information into the post. However, why do you need to know which is more likely to know anything about e.g. how neural networks generalize?
I understand that SLT has some additional content beyond what is in the post, and I’ve tried to explain how you could make that fit in this framework. I just don’t understand why that additional content is relevant, which is why I left it out.
As an additional note, I wasn’t really talking about floating point precision being the important variable here. I’m just saying that if you want -complexity to match the notion of real log canonical threshold, you have to discretize SLT in a way that might not be obvious at first glance, and in a way where some conclusions end up being scale-dependent. This is why if you’re interested in studying this question of the relative contribution of singular points to the partition function, SLT is a better setting to be doing it in. At the risk of repeating myself, I just don’t know why you would try to do that.
You need to discretize the function before taking preimages. If you just take preimages in the continuous setting, of course you’re not going to see any of the interesting behavior SLT is capturing.
In your case, let’s say that we discretize the function space by choosing which one of the functions you’re closest to for some . In addition, we also discretize the codomain of by looking at the lattice for some . Now, you’ll notice that there’s a radius disk around the origin which contains only functions mapping to the zero function, and as our lattice has fundamental area this means the “relative weight” of the singularity at the origin is like .
In contrast, all other points mapping to the zero function only get a relative weight of where is the absolute value of their nonzero coordinate. Cutting off the domain somewhere to make it compact and summing over all to exclude the disk at the origin gives for the total contribution of all the other points in the minimum loss set. So in the limit the singularity at the origin accounts for almost everything in the preimage of . The origin is privileged in my picture just as it is in the SLT picture.
I think your mistake is that you’re trying to translate between these two models too literally, when you should be thinking of my model as a discretization of the SLT model. Because it’s a discretization at a particular scale, it doesn’t capture what happens as the scale is changing. That’s the main shortcoming relative to SLT, but it’s not clear to me how important capturing this thermodynamic-like limit is to begin with.
Again, maybe I’m misrepresenting the actual content of SLT here, but it’s not clear to me what SLT says aside from this, so...
Attribution of causality is tricky with this event, but I would agree if you said the fall coincided with a major slowdown in European economic growth.
I think a problem re: China is that a lot of population decline estimates for China are based on the official census, and as far as I know China didn’t have a formal census before the Xin dynasty, and certainly not before unification in the 3rd century BC. So the fact that we don’t see comparable population declines reported may just be an artifact of that measurement issue. We certainly see plenty of them in the second millennium.
I haven’t seen estimates of this that put it anywhere near the Mongol conquests, so I would assume not particularly bloody relative to what was to come later. I would also guess that the Islamic world probably saw significant population growth around that time.
Yeah, it’s possible that this is the explanation, but if so it’s rather hard to know because there’s no principled way to compare events like these to analogs in other time periods.