gwern
In that brief moment of uncertainty, anything could have happened. If one person had just packed up and left, everyone might have followed suit. But nobody reacted. Perhaps what kept the room still was the fear of being perceived as scared. Or the belief that surely, bad things could not happen to them. Or maybe they’d heard enough false alarms in their lives. I’m not sure.
One of the most depressing things about the Replication Crisis in especially social psychology is that many results from the 1950s and 1960s failed to replicate at all… except the Asch conformity experiments. Those seem to replicate just fine.
At first glance, your linked document seems to match this. The herald who calls the printer “pig-headed” does so in direct connection with calling him “dull”, which at least in modern terms would be considered a way of calling him stupid?
Not necessarily. ‘Dull’ can mean, in 1621 just as well as 2025, plenty of other things: eg “Causing depression or ennui; tedious, uninteresting, uneventful; the reverse of exhilarating or enlivening.” (OED example closest in time: “Are my discourses dull? Barren my wit?”—Jonson’s good friend & fellow playwright, William Shakespeare, Comedy of Errors (1623)); or, “Of persons, or their mood: Having the natural vivacity or cheerfulness blunted; having the spirits somewhat depressed; listless; in a state approaching gloom, melancholy, or sadness: the opposite of lively or cheerful.” (Shakespeare again: “Sweet recreation barr’d, what doth ensue / But moodie and dull melancholly?”) Which in the context of a ‘dull’ tradesman who refuses to hear the exciting news being brought by no less than 2 heralds before he knows ‘the price’, is sensible enough.
not reading your entire document?
That would certainly help, because if you read the rest of the Printer’s rather cynical comments, constantly undermining the heralds, he doesn’t sound in the slightest bit like he is supposed to be stupid or retarded—as opposed to a curmodgeonly critic constantly—obstinately, even—throwing water on a good time by sardonically remarking that he makes money by changing the dates on newspaper plates to print the old news as new news or mocking their talk of moonlight by noting that his telescope-maker has brought him moonshine before. (Not that printers, like Benjamin Franklin, were an occupation associated with low intelligence to begin with.)
OP’s example is correct and you are wrong. ‘Pigheaded’ is neither a proposed root cause analysis nor does it mean ‘are dumb’; perhaps you should check a dictionary before correcting others’ usage. It means stubborn, strong-willed, obstinate, often to the point of foolishness or taking very harmful actions, or to quote the OED: “Having a head like that of a pig. Chiefly figurative: stupidly obstinate, perverse, or set in one’s ways.” Note: it is “stupidly obstinate”, and not “stupid”. This is because pigs are notoriously smart but stubborn: very strong, heavy, often hungry, whose mind can’t easily be changed by an unfortunate swineherd or passerby in their way. (And this usage has been consistent since the start: the OED will give you the first attestation of it to Ben Jonson, where it describes a small-minded* printer who thinks that high-quality news has to be paid for, because that’s how he operates; Jonson then mocks some other tradesmen for their own kinds of narrowmindedness, but not for any of them being low-IQ.) Hence, the Russell conjugation is correct: “pigheaded” is the highly insulting figurative term which intensifies the negative “obstinate” which is the bad version of the positive “firm”. Just as ‘firm’ does not principally mean ‘dumb’, ‘pigheaded’ doesn’t principally mean it either.
* note, by the way, that ‘small-minded’ doesn’t mean, ‘has a low cranial volume and thus lower than average intelligence’, nor is it a root-cause analysis that their low intelligence is caused by inadequate neural tissue.
But the caveat there is that this is inherently a backwards-looking result:
We consider GPT-4o (OpenAI, 2024), Claude-3.5-Sonnet (Anthropic, 2024), Grok-2 (xAI, 2024), Gemini-1.5-Pro (Google, 2024), and DeepSeek-V3 (DeepSeek-AI, 2024).
So one way to put it would be that people & classifiers are good at detecting mid-2024-era chatbot prose. Unfortunately, somewhere after then, at least OpenAI and Google apparently began to target the problem of ChatGPTese (possibly for different reasons: Altman’s push into consumer companion-bots/personalization/social-networking, and Google just mostly ignoring RLHF in favor of capabilities), and the chatbot style seems to have improved substantially. Even the current GPT-4o doesn’t sound nearly as 4o-like as it did just back in November 2024. Since mode-collapse/ChatGPTese stuff was never a capabilities problem per se (just look at GPT-3!), but mostly just neglect/apathy on part of the foundation labs (as I’ve been pointing out since the beginning), it’s not a surprise that it could improve rapidly once they put (possibly literally) any effort into fixing it.
Between the continued rapid increase in capabilities and paying some attention to esthetics & prose style and attackers slowly improving their infrastructure in the obvious ways, I expect over the course of 2025 that detecting prose from a SOTA model is going to get much more difficult. (And this excludes the cumulative effect on humans increasingly writing like ChatGPT.)
EDIT: today on HN, a post was on the front page for several hours with +70 upvotes, despite being blatantly new-4o-written (and impressively vapid). Is this the highest-upvoted LLM text on HN to date? I suspect that if it is, we’ll soon see higher...
I’m not sure this is a big problem. How much net attrition do you really expect over a decade, say? By which point who really cares? You will have so much more AI progress, and accumulated data (particularly if you’ve been gradually replacing the lower-level employees and you have an ‘automation wave’ moving through the organization where employees increasingly train their automated replacements or their job is simply reorganizing the jobs to enable automation).
It seems like to the extent there’s much attrition at high levels, it is reduced in considerable part by these very dynamics: as returns to high-level human labor go up, presumably, there is less attrition from voluntary retirement or leisure consumption (and if the returns go down, then that implies that there is no ‘shortage’ of people for such high-level positions and so no problem); and also as the remaining human work becomes more ‘white-collar’ and based on difficult-for-AI things like reputation or experience or ownership or creativity, aging or opportunity costs begin to matter less, reducing another source of attrition.
(Even if AI or robotics is unable to do the ‘core’ of a job, they can help deal with various obstacles which might prevent a human from doing the job. An elderly manager who might decide to retire in part because they are low-key becoming worried about safely driving to/from the office will no longer think about that when they have a self-driving car or remote working becomes ever more feasible; older managers who might be slipping in their grasp of details or who have ‘senior moments’ will be able to rely on AI secretaries to catch those or just pause stuff for a while until they’re back to normal; elite women might invest more in careers if they have Claude-bot as a trustworthy nanny and chauffeur, etc. One is reminded of President Biden: his staffers were able to work around his issues by doing things like rescheduling or canceling events to avoid exposing him publicly when he was bad; it was only an event that even the POTUS can’t arbitrarily schedule, a presidential debate, that punctured the carefully-constructed illusion. Few of those staffers were qualified to be President of the United States, and yet, you don’t have to be a good president to observe “sounds like Joe’s having a bad day today” and quietly cancel his evening appointments for him so he can get to bed early.)
Also notable: the big OpenAI reveal today was some sort of better personalization. Instead of the crude ‘saved facts’ personalization ChatGPT has had for a long time and which has never made much of a difference, they’re doing… something. Unclear if it’s merely RAG or if they are also doing something interesting like lightweight finetuning. But the GPTs definitely seem to have much better access to your other sessions in the web interface, and as far as I know, few other interfaces with frontier models have tried to do much personalization, so this will be an interesting real-world test at scale about how much simple personalization can help with LLMs (similar to Midjourney’s relatively new personalization feature, which I get a lot out of).
I don’t think this is true at all. How do you translate, say, rotating multiple shapes in parallel into text?
At least for multimodal LLMs in the pure-token approach like Gato or DALL-E 1 (and probably GPT-4o and Gemini, although few details have been published), you would be able to do that by generating the tokens which embody an encoded image (or video!) of several shapes, well, rotating in parallel. Then you just look at them.
Pursuit of novelty is not vnm-incoherent. Furthermore, it is an instrumentally convergent drive; power-seeking agents will seek novelty as well, because learning increases power in expectation (see: value of information).
Or to put it another way: any argument which convincingly proves that ‘incoherent search processes ultimately outcompete coherent search processes’ is also an argument which convinces a VNM agent to harness the superior incoherent search processes instead of the inferior coherent ones.
It’s good someone else did it, but it has the same problems as the paper: not updated since May 2024, and limited to open source base models. So it needs to be started back up and add in approximate estimators for the API/chatbot models too before it can start providing a good universal capability benchmark in near-realtime.
One of the most robust benchmarks of generalized ability, which is extremely easy to update (unlike benchmarks like Humanity’s Last Exam), would just be to estimate the pretraining loss (ie. the compression ratio).
No, it would probably be a mix of “all of the above”. FB is buying data from the same places everyone else does, like Scale (which we know from anecdotes like when Scale delivered FB a bunch of blatantly-ChatGPT-written ‘human rating data’ and FB was displeased), and was using datasets like
books3that are reasonable quality. The reported hardware efficiency numbers have never been impressive, they haven’t really innovated in architecture or training method (even the co-distillation for Llama-4 is not new, eg. ERNIE was doing that like 3 years ago), and insider rumors/gossip don’t indicate good things about the quality of the research culture. (It’s a stark contrast to things like Jeff Dean overseeing a big overhaul to ensure bit-identical reproducibility of runs and Google apparently getting multi-datacenter training working by emphasizing TPU interconnect.) So my guess is that if it’s bad, it’s not any one single thing like ‘we trained for too few tokens’ or ‘some of our purchased data was shite’: it’s just everything in the pipeline being a bit mediocre and it multiplying out to a bad end-product which is less than the sum of its parts.Remember Karpathy’s warning: “neural nets want to work”. You can screw things up and the neural nets will still work, they will just be 1% worse than they should be. If you don’t have a research culture which is rigorous about methodology or where people just have good enough taste/intuition to always do the right thing, you’ll settle for whatever seems to work… (Especially if you are not going above and beyond to ensure your metrics aren’t fooling yourself.) Now have a 1% penalty on everything, from architecture to compute throughput to data quality to hyperparameters to debugging implementation issues, and you wind up with a model which is already obsolete on release with no place on the Pareto frontier and so gets 0% use.
That would be tricky because you are comparing apples and oranges. Consider that for the USA, there are only 11 cardinals (of 252 worldwide), while there are 10x more federal senators at any moment (I don’t know if there would be more or less total: senators tend to be much younger but cardinals also tend to be long-lived), and I can’t even guess how many ‘Fortune 500 C-level employees’ there might be given corporate turnover and the size of many ‘C-suites’ - tens of thousands, maybe? So your suggestions span ~1-3 orders of magnitude less selectivity than cardinals do.
whomever makes it into the college of cardinals.
I would be surprised if that was the primary homosexuality-enriching step, given that reporting has always been that quite a lot of low-level parish-level priests are also gay. (Note, for example, how many of the sexual abuse scandal victims were boys/men.) I would guess that it operates fairly steadily at all levels, starting from simply which young boys opt for the priesthood (known to be a demand and difficult occupation even if the celibacy requirement is, for you, not so onerous) and operating from there; if I had to guess where the biggest enrichment is, it’d be at the ‘leaving your country for the Vatican’ step, given how notoriously gay the Vatican is. So going there suggests either that you are gay (and so the buggery isn’t a bug, it’s a feature) or you are highly ambitious and don’t mind it (or are willing to exploit it and again, not a bug but a feature).
We have not yet tried 4.5 as it’s so expensive that we would not be able to deploy it, even for limited sections.
Still seems like potentially valuable information to know: how much does small-model smell cost you? What happens if you ablate reasoning? If it is factual knowledge and GPT-4.5 performs much better, then that tells you things like ‘maybe finetuning is more useful than we think’, etc. If you are already set up to benchmark all these OA models, then a datapoint from GPT-4.5 should be quite easy and just a matter of a small amount of chump change in comparison to the insight, like a few hundred bucks.
22.3 percent of cardinals are reported as eldest children. That compares to 21.6 percent which are youngest children. Eldest children are still favored.
I kept waiting for you to discuss this point: in considering analysis of cardinals (as opposed to ordinary random people), what about the other relevant birth-order effects? Like the… first-born eldest birth order effect, where first-borns are smarter, more extraverted, stabler, higher-SES etc. All of which sounds exactly like the sort of thing you need to rise through an extreme hierarchy to the top.
After all, surely homosexuality is not the only (or even primary) trait the Catholic Church hierarchy is trying to select for?
The failure of the compute-rich Llama models to compete with the compute poorer but talent and drive rich Alibaba and DeepSeek
This seems like it’s exaggerating the Llama failure. Maybe the small Llama-4s just released yesterday are a bit of a disappointment because they don’t convincingly beat all the rivals; but how big a gap is that absolutely? When it comes to DL models, there’s generally little reason to use #2; but that doesn’t mean #2 was all that much worse and ‘a failure’ - it might only have been weeks behind #1. (Indeed, a model might’ve been the best when it was trained, and release just took a while. Would it be reasonable to call such a model a ‘failure’? I wouldn’t. It might be a failure of a business model or a corporate strategy, but that model qua model is a good model, Bront.) #2 just means it’s #2, lesser by any amount. How far back would we have to go for the small Llama-4s to have been on the Pareto frontier? It’s still early, but I’m getting the impression so far that you wouldn’t have to go that far back. Certainly not ‘years’ (it couldn’t perform that well on LMArena in its ‘special chatbot configuration’ even sloptimized if it was years behind), unless the wilder rumors turn out to be true (like deliberately training on the test sets—in which case, Zuckerberg may have to burn FB AI with fire and reboot the entire AI org because the culture is irretrievably rotten—but of course such rumors usually do not, so I mention this mostly to indicate that right now Llama Internet commentary is high on heat and low on light).
The failure of the compute-rich Llama models to compete with the compute poorer but talent and drive rich Alibaba and DeepSeek shows that even a substantial compute lead can be squandered. Given that there is a lot of room for algorithmic improvements (as proven by the efficiency of the human brain), this means that determined engineering plus willingness to experiment rather than doubling-down on currently working tech.
I’m not really following your argument here. Even if LLaMA-4 is disappointing compared to what DeepSeek could’ve done with the same compute because they’d get 40% MFU instead of FB’s 20% or whatever, and are 2x as good in effective-compute, that doesn’t close the lead when FB finishes its new Manhattan-sized datacenter, say, and has 100x DS’s compute. Or are you arguing for the possibility of someone making an asymptotic scaling law breakthrough with a better exponent, so that even with 1/100th the compute, they can beat one of the giants?
The human microbiome is irrelevant to this topic. The microbiome is highly heritable (usual twin studies & SNP heritabilities), and it is caused by genes and the environment, as well as unstable; its direct causal effects in normal humans are minimal. We know that it is supremely irrelevant because environmental changes like antibiotics or new food or global travel which produce large changes in personal (and offspring) microbiomes do not produce large changes in intelligence (of oneself or offspring); and most dramatically, germ-free humans exist and are of normal or even above-average intelligence, eg the fascinating mistakes and delusions of David despite his high intelligence. (Amusingly, germ-free mice apparently even live longer.) Microbiome research is, in general, very low quality and can’t be taken seriously—look at your link:
Examples of how important the gut microbiome, and the parents’ health, are for human development: https://humanmicrobiome.info/maternity/
Most of this page is meaningless mouse studies (infamous for not replicating and getting whatever result the experimenter wants and the animal model literature having huge systemic biases), and the handful of actual human studies I see here are all garbage—things like cross-sectional studies with large known familial confounding, or heavy reliance on things like breastfeeding where the beneficial effects disappear when controlling for just some confounds. This also goes for much-touted correlations like autism. There’s not a single result on this page that provides a shred of evidence for your implied thesis that microbiome interventions could, even in theory, possibly matter to ‘how to make superbabies’. It doesn’t.
In 2021 a geneticist insisted to me that the microbiome was just a fad.
He was right. BTW, you remember what happened in 2021, right?
EDIT: If anyone cares, I’m not bothering to respond to Harrop’s comment in depth because I think his Gish-gallopy response manages to exemplify many of the criticisms I already made, where they are not outright non-responsive (eg. his ‘disagreement’ about the reasons why germ-free mice live longer is obviously irrelevant to my point that they do), and I’m not worried anyone is going to waste any time on ‘the microbiome’ for these purposes when this is the best case he can make. You can see he has no case for ‘superbabies’ having anything to do with known microbiome stuff, and I do not care enough about the microbiome per se to prosecute it further.
I don’t really understand how a local copy of the weights gives the terrorists more practical control over the software’s alignment. I don’t think it’s easy to manually tweak weights for so specific a purpose. Maybe they just mean the API is doing a good job of blocking sketchy requests?
You can finetune models for any specific purpose: just provide a few datapoints and train. The more specific the purpose, the easier tweaking the weights is, not harder. (Surely, if nothing else, you’ve seen all of the LoRAs and other things for finetuning image generation model to generate a specific character?) There is an extensive literature at this point on how it is trivial to strip away all of the friendly chatbot persona from released checkpoints, such as LLaMA, if you are able to access and modify the model slow weights and fast weights directly.
Yes, in dire straits. But it’s usually called ‘hyperinflation’ when you try to make seignorage equivalent to >10% of GDP and fund the government through deliberately creating high inflation (which is on top of any regular inflation, of course). And because inflation is about expectations in considerable part, you can’t stop it either. Not to mention what happens when you start hyperinflation.
(FWIW, this is a perfectly reasonable question to ask a LLM first. eg Gemini-2.5-pro will give you a thorough and sensible answer as to why this would be extraordinarily destructive and distortionary, and far worse than the estimated burden of tax return filing, and it would likely satisfy your curiosity on this thought-experiment with a much higher quality answer than anyone on LW2, including me, is ever likely to provide.)
I think it’s a little more concerning that Dwarkesh has invested in this startup:
And I do not see any disclosure of this in either the Youtube description or the Substack transcript at present.
EDIT: a disclosure has been added to both