A simple idea here would be to just pay machinists and other blue collar workers to wear something like a Go-Pro (and possibly even other sensors along the body that track movement) all throughout the day. You then use the same architectures that currently predict video to predict what the blue collar worker does during their day.
SorenJ
Karma: 87
Thank you very much, this is so helpful! I want to know if I am understanding things correctly again, so please correct me if I am wrong on any of the following:
By “used for inference,” this just means basically letting people use the model? Like when I go to the chatgpt website, I am using the datacenter campus computers that were previously used for training? (Again, please forgive my noobie questions.)
For 2025, Abilene is building a 100,000-chip campus. This is plausibly around the same number of chips that were used to train the~3e26 FLOPs GPT4.5 at the Goodyear campus. However, the Goodyear campus was using H100 chips, but Abilene will be using Blackwell NVL72 chips. These improved chips means that for the same number of chips we can now train a 1e27 FLOPs model instead of just a 3e26 model. The chips can be built by summer 2025, and a new model trained by around end of year 2025.
1.5 years after the Blackwell chips, the new Rubin chip will arrive. The time is now currently ~2027.5.
Now a few things need to happen:The revenue growth rate from 2024 to 2025 of 3x/year continues to hold. In that case, after 1.5 years, we can expect $60bn in revenue by 2027.5.
The ‘raised money’ : ‘revenue’ ratio of $30bn : $12bn in 2025 holds again. In that case we have $60bn x 2.5 = $150bn.
The decision would need to be made to purchase the $150 bn worth of Rubin chips (and Nvidia would need to be able to supply this.)
More realistically, assuming (1) and (2) hold, it makes more sense to wait until the Rubin Ultra comes out to spend the $150bn on.
Or, some type of mixed buildout would occur, some of that $150bn in 2027.5 would use the Rubin non-Ultra to train a 2e28 FLOPs model, and the remainder would be used to build an even bigger model in 2028 that uses Rubin Ultra.
Do I have the high level takeaways here correct? Forgive my use of the phrase “Training size,” but I know very little about diferent chips, so I am trying to distill it down to simple numbers.
2024:
a) OpenAI revenue: $3.7 billion.
b) Training size: 3e26 to 1e27 FLOPs.
c) Training cost: $4-5 billion.
2025 Projections:a) OpenAI revenue: $12 billion.
b) Training size: 5e27 FLOPs.
b) Training cost: $25-30 billion.
2026 Projections:
a) OpenAI revenue: ~$36 billion to $60 billlion.
At this point I am confused: why you are saying Rubin arriving after Blackwell would make the revenue more like $60 billion? Again, I know very little about chips. Wouldn’t the arrival of a different chip also change OpenAIs cost?
b) Training size: 5e28 FLOPs.
c) Training cost: $150 billion.
Assuming investors are willing to take the same ratio of revenue : training cost as before, this would predict $70 billion to $150 billion. In other words, to get to the $150 billion mark requires that Rubin arrives after Blackwell, openAI makes revenue $60 billion in revenue, and investors take a 2.5 multiplier for $60 x 2.5 = $150 billion.
Is there anything else that I missed?
For what it is worth, the later parts of the book discuss the things you might be more intersted in, like meditative/path models. The scientific research is quite interesting, in particular, I find the brain scans of monks to be incredible.
I agree with a lot of what you have said, and I am largely on board with the thrust of your message. The later parts of the book discuss some of the things more relevant to what we have been talking about, like meditation and “awakening,” and these are also the more interesting bits, in my opinion. It also shouldn’t be surprising that the Pali Canon contains 2,500 year old texts that we find odd! -- but if you enjoy history/mythology/sociology then it can be quite interesting.
I think what is missing is that a proper takeaway, for you, should be to update from the secular models as well. You say that the creators of these models have more reliable epistemics, but I do not like the comparison: the creators of the secular models have poor epistemics. As you put it:
“the impression I get from the book is that it really is a religion complete with all the supernaturalness and superstition … I now think the people saying that you can really just read the stuff metaphorically are cherry-picking the bits that happen to fit the framework they’re in favor of.”
and I completely agree with this! But this makes me trust the modern authors less, not more. “Religious Buddhism” may be too hard to swallow, but “Metaphorical Buddhism” is dishonest and does not make sense when evaluated on its own terms. Unfortunately, proponents of Metaphorical Buddhism launder the quality of their ideas with the reputation of “The Buddha.” I think if you finish the book and then go back and read something like MCTB you won’t be able to look at it in the same way.
Let me try to give a rough summary of my own overall view on the issue: There is now a panoply of things calling themselves Buddhist meditation, or Buddhist meditation inspired, that are present in contemporary life. The latest example I am aware of is jhourney.io. These range the gamut from completely secular to very traditional with a lot of stuff in between. Programs like Mindfulness Based Stress Reduction (MBSR) have clinical evidence that a moderate amount of (MBSR’s version of) mindfulness meditation leads to demonstrable changes in psychological traits that most of us would find desirable. I find the evidence for this to be quite good, though perhaps not excellent, and the cost to benefit ratio of adding this type of meditation to a secular life is probably worthwhile.
In addition, I imagine that for many people going to something like an American Zen center weekly would enhance their quality of life, and I would bet you could easily demonstrate this at the clinical level.
But I do not trust a lot of the modern authors who venture beyond this territory and into “higher stages of the path,” like enlightenment or deep stages of meditation. The claims they make about “being enlightened” make little sense from a traditional perspective, are poorly defined within individual authors and contradictory between authors, and are often cherry picked, distorted, and dishonest presentations. For example, Scott Alexander reviewed MCTB and in his review at one point asked, “if this is enlightenment, then why would you even want it?”—which was absolutely the correct question to ask given Ingram’s description of enlightenment!
I think that the authors should just stop calling themselves Buddhist and acknowledge that they are running their own religious experiments. There is nothing wrong with this, per se, but the mixing of the two is what bothers me. Earlier I said,
“If you’re interested in learning about Buddhism, I recommend starting with the Pali Canon, the collection of Early Buddhist Texts that offer the best historical record of the teachings of Siddhartha Gautama, the Buddha.”
and you replied,
“I think the book is, if anything, dissuading me from the idea that modern Western practitioners would benefit from spending time familiarizing themselves with the Pali Canon.”
I think I need to be more precise: if you are intersted in learning about Buddhism, then start with the Pali Canon, or an anthology like “In the Words of the Buddha.”
On the other hand, if you are fairly confident already that you don’t want religious baggage, and do not believe in things like other realms, rebirth, etc., but you are still interested in some type of ideal like enlightenment and radical change through meditation, then go ahead with more modern secular authors. But know that if those modern authors say they are Buddhist inspired, their link to historical Buddhism and to the historical Buddha’s ideas is a lot more tenuous than they are implying and would have you believe. Try to evaluate their ideas on their own terms, and see if you like what you see. Furthermore, know that the track record of modern people (and certainly not just modern people) running religious experiments and claiming to be enlightened is not necessarily amazing.
Anyway, do let me know if you finish the book, and then go back and read something like MCTB. I would be very curious to know your thoughts!
I don’t say, “I just zoned out,” but I do often say, “sorry, could you repeat that?” I think people are less likely to take offense to this.
Right, I don’t blame you for referencing those books to communicate, because they are what a lot of readers on a forum like this would be familiar with. TMI is also important in my personal chronology, but I wouldn’t recommend it either. What I would want to popularize among crowds like this is the recent scholarly study and practice of “Early Buddhism.”
I started typing out more of a reply, but I think I should maybe just make a post.
By the way, I started going to a local Zendo within the past 12 months, and it is actually in the lineage of Philip Kapleau Roshi, so one of the books they recommend is The Three Pillars of Zen: Teaching, Practice, and Enlightenment as well (I haven’t taken a look at it yet). The Visuddhimagga is debatably in the “Pali Canon.” It is a commentary written by the monk Buddhaghosha some 1200 years after the compilation of the Suttas. The book I recommended earlier by Bhikkhu Bodhi is just from the Suttas.
The books you mention, The Mind Illuminated (TMI), and Mastering the Core Teachings of the Buddha (MCTB), are quite popular among Western atheists, secular people, rationalists, and technically-minded intelligent individuals who spend a lot of time reading on the internet. I don’t mean that pejoratively—I count myself among these groups. However, if you have read these books, there is still much more to learn (and unlearn).
They are appealing because they promise to offer Buddhist meditation, insights, and strategies while jettisoning what they consider unnecessary religious scaffolding. Unfortunately, perhaps because of this, they contain numerous errors and often miseducate their readers about meditation and Buddhism in general. For example, Scott Alexander reviewed MCTB positively, saying it made him finally “get” Buddhism. This highlights an understandable bias: monks in robes teaching Buddhism, who accept religious and supernatural elements, are dismissed as religious men. On the other hand, figures like Daniel Ingram (author of MCTB) and John Yates, a neuroscientist and author of TMI, are seen as offering something more accessible to the secular mindset. There is also just the fact that Ingram and Yates were better at marketing and packaging their books in an enticing format than monks who aren’t allowed to touch money (literally).
However, in some circles, mentioning these books is considered a red flag. It’s well-known that readers of MCTB are particularly prone to psychotic breaks and highly unstable practice. From a traditional perspective, this is seen as a clear sign that something is amiss. Bhikkhu Analayo, for instance, has critically reviewed Ingram’s book, and it’s worth noting that John Yates was involved in a sex scandal.
If you’re interested in learning about Buddhism, I recommend starting with the Pali Canon, the collection of Early Buddhist Texts that offer the best historical record of the teachings of Siddhartha Gautama, the Buddha. This is akin to suggesting that someone interested in Christianity should read the Bible or an anthology of it before diving into modern interpretations that might strip away key religious elements.
The Pali Canon is extensive, so a curated selection is a good starting point. In the Words of the Buddha, an excellent text by scholar-monk Bhikkhu Bodhi, is a great introduction. Once you have a foundational understanding, you can better decide which aspects resonate with you and which might be products of folk religious beliefs.
Eh, maybe I should just make this a post.
Did people seem to find shit getting on everything deep? Did serious writings explore the experiences of all manner of shit, and the nuances of the feces and piss involved?
It doesn’t seem like it? Maybe read Gargantua and Pantagruel for ideas…
Yeah, that’s the question. Saying that means that . So if doubles, then it’s required that is at least cut in half. I doubt there is a proof of this per se, but in a situation as strange as this it seems reasonable to me that if you claim you can do 10 times as much of something, then that is at least 10 times less likely.
I guess the main point I wanted to make is that in the usual phrasings of Pascal’s Mugging the choice of is oftentimes taken after the choice of . But should be a function of . So the mugger at least has to include this in his argument, and (some of) the burden of proof is on him.
They’re pretty bad, but they seem about GPT-2 level bad? So plausibly in a couple of years they will be GPT-4 level good, if things go the same way?
This does seem pretty difficult. The only idea I have is having humans wear special gloves with sensors on them, and maybe explain their thoughts aloud as they work, and then collecting all of this data.
Before you go to RL you need to train on prediction with a large amount of data first. We don’t have this yet for blue collar work. Then once you have the prediction model, robots, and rudimentary agents, you try to get the robots to do simple tasks in isolated environments. If they succeed they get rewarded. This feels quite a bit more than 3 years away...
In general, I think the ideas is that you first get a superhuman coder, then you get a superhuman AI researcher, then you get a any-task superhuman researcher, and then you use this superhuman researcher to solve all of the problems we have been discussing in lightning fast time.