I have a compute-market startup called vast.ai, and I’m working towards aligned AI. Currently seeking networking, collaborators, and hires—especially top notch cuda/gpu programmers.
My personal blog: https://entersingularity.wordpress.com/
jacob_cannell
Karma: 6,062
Not for transformers, for which training and inference are fundamentally different.
Transformer training parallelizes over time, but that isn’t feasible for inference. So transformer inference backends have to parallelize over batch/space, just like RNNs, which is enormously less efficient in RAM and RAM bandwidth use.
So if you had a large attention model that uses say 1TB of KV cache (fast weights) and 1TB of slow weights, transformer training can often run near full efficiency, flop limited, parallelizing over time.
But similar full efficient transformer inference would require running about K instances/agents in parallel, where K is the flop/mem_bw ratio (currently up to 1000 on H100). So that would be 1000 * 1TB of RAM for the KV cache (fast weights) as its unique per agent instance.
This, in a nutshell, is part of why we don’t already have AGI. Transformers are super efficient at absorbing book knowledge, but just as inefficient as RNNs at inference (generating new experiences, which is a key bottleneck on learning from experience).
Thus there is of course much research in more efficient long kv cache, tree/graph inference that can share some of the KV cache over similar branching agents, etc
Due to practical reasons, the compute requirements for training LLMs is several orders of magnitude larger than what is required for running a single inference instance. In particular, a single NVIDIA H100 GPU can run inference at a throughput of about 2000 tokens/s, while Meta trained Llama3 70B on a GPU cluster[1] of about 24,000 GPUs. Assuming we require a performance of 40 tokens/s, the training cluster can run concurrent instances of the resulting 70B model.
I agree direction-ally with your headline, but your analysis here assumes flops is the primary constraint on inference scaling. Actually it looks like VRAM is already the more important constraint, and would likely become even more dominant if AGI requires more brain-like models.
LLMs need VRAM for both ‘static’ and ‘dynamic’ weights. The static weights are the output of the long training process, and shared over all instances of the same model or fine tune (LORAs share most). However the dynamic ‘weights’ - in the attention KV cache—are essentially unique to each individual instance of the model, specific to its current working memory context and chain of thought.
So the key parameters here are total model size and dynamic vs static ratio (which depends heavily on context length and many other factors). But for example if dynamic is 50% of the RAM usage then 1M concurrent instances would require almost as many GPUs.
If AGI requires scaling up to very large brain-size models ~100T params (which seems likely), and the dynamic ratio is even just 1%, then 1M concurrent instances would require on order 10M GPUs.
How is that even remotely relevant? Humans and AIs learn the same way, via language—and its not like this learning process fails just because language undersamples thoughts.
As the article points out, shared biological needs do not much deter the bear or chimpanzee from killing you. An AI could be perfectly human—the very opposite of alien—and far more dangerous than Hitler or Dhamer.
The article is well written but dangerously wrong in its core point. AI will be far more human than alien. But alignment/altruism is mostly orthogonal to human vs alien.
We are definitely not training AIs on human thoughts because language is an expression of thought, not thought itself.
Even if training on language was not equivalent to training on thoughts, that would also apply to humans.
But it also seems false in the same way that “we are definitely not training AI’s on reality because image files are compressed sampled expressions of images, not reality itself” is false.
Approximate bayesian inference (ie DL) can infer the structure of a function through its outputs; the structure of the 3D world through images; and thoughts through language.
Premise 1: AGIs would be like a second advanced species on earth, more powerful than humans.
Distinct alien species arise only from distinct separated evolutionary histories. Your example of the aliens from Arrival are indeed a good (hypothetical) example of truly alien minds resulting from a completely independent evolutionary history on an alien world. Any commonalities between us and them would be solely the result of convergent evolutionary features. They would have completely different languages, cultures, etc.
AI is not alien at all, as we literally train AI on human thoughts. As a result we constrain our AI systems profoundly, creating them in our mental image. Any AGI we create will inevitably be far closer to human uploads than alien minds. This a prediction Moravec made as early as 1988 (Mind Children) - now largely fulfilled by the strong circuit convergence/correspondence between modern AI and brains.
Minds are software mental constructs, and alien minds would require alien culture. Instead we are simply creating new hardware for our existing (cultural) mind software.
I also not sure of the relevance and not following the thread fully, but the summary of that experiment is that it takes some time (measured in nights of sleep which are rough equivalent of big batch training updates) for the newly sighted to develop vision, but less time than infants—presumably because the newly sighted already have full functioning sensor inference world models in another modality that can speed up learning through dense top down priors.
But its way way more than “grok it really fast with just a few examples”—training their new visual systems still takes non-trivial training data & time
I suspect that much of the appeal of shard theory is working through detailed explanations of model-free RL with general value function approximation for people who mostly think of AI in terms of planning/search/consequentialism.
But if you already come from a model-free RL value approx perspective, shard theory seems more natural.
Moment to moment decisions are made based on value-function bids, with little to no direct connection to reward or terminal values. The ‘shards’ are just what learned value-function approximating subcircuits look like in gory detail.
The brain may have a prior towards planning subcircuitry, but even without a strong prior planning submodules will eventually emerge naturally in a model-free RL learning machine of sufficient scale (there is no fundamental difference between model-free and model-based for universal learners). TD like updates ensure that the value function extends over longer timescales as training progresses. (and in general humans seem to plan on timescales which scale with their lifespan, as you’d expect)
TSMC 4N is a little over 1e10 transistors/cm^2 for GPUs and roughly 5e^-18 J switch energy assuming dense activity (little dark silicon). The practical transistor density limit with minimal few electron transistors is somewhere around ~5e11 trans/cm^2, but the minimal viable high speed switching energy is around ~2e^-18J. So there is another 1 to 2 OOM further density scaling, but less room for further switching energy reduction. Thus scaling past this point increasingly involves dark silicon or complex expensive cooling and thus diminishing returns either way.
Achieving 1e-15 J/flop seems doable now for low precision flops (fp4, perhaps fp8 with some tricks/tradeoffs); most of the cost is data movement as pulling even a single bit from RAM just 1 cm away costs around 1e-12J.
Part of the issue is my post/comment was about moore’s law (transistor density for mass produced nodes), which is a major input to but distinct from flops/$. As I mentioned somewhere, there is still some free optimization energy in extracting more flops/$ at the circuit level even if moore’s law ends. Moore’s law is very specifically about fab efficiency as measured in transistors/cm^2 for large chip runs—not the flops/$ habyrka wanted to bet on. Even when moore’s law is over, I expect some continued progress in flops/$.
All that being said, nvidia’s new flagship GPU everyone is using—the H100 which is replacing the A100 and launched just a bit after habryka proposed the bet—actually offers near zero improvement in flops/$ (the price increased in direct proportion to flops increase). So I probably should have taken the bet if it was narrowly defined as (flops/$ for the flagship gpus most teams using currently for training foundation models).
I don’t know who first said it, but the popular saying “Computer vision is the inverse of computer graphics” encompasses much of this viewpoint.
Computer graphics is the study/art of the approximation theory you mention and fairly well developed & understood in terms of how to best simulate worlds & observations in real-time from the perspective of an observer. But of course traditional graphics uses human-designed world models and algorithms.
Diffusion models provide a general framework for learning a generative model in the other direction—in part by inverting trained vision and noise models.
So naturally there is also diffusion planning which is an example of the symmetry you discuss: using general diffusion inference for planning. The graph dimensions end up being both space-time and abstraction level with the latter being more important: sensor inference moves up the abstraction/compression hierarchy, whereas planning/acting/generating moves down.
Even if there is no acceptable way to share the data semi-anonymously outside of match group, the arguments for prediction markets still apply within match group. A well designed prediction market would still be a better way to distribute internal resources and rewards amongst competing data science teams within match group.
But I’m skeptical that the value of match group’s private data is dominant even in the fully private data scenario. People who actually match and meetup with another user will probably have important inside view information inaccessible to the algorithms of match group.
Manifold.Love’s lack of success is hardly much evidence against the utility of prediction markets for dating markets, any more or less than most startup’s failure at X is evidence against the utility of X.
Certainly mood disorders like bipolar,depression,mania can have multiple causes—for examle simply doing too much dopaminergic simulants (cocaine, meth etc) can cause mania directly.
But the modern increased prevalence of mood disorders is best explained by a modern divergence from conditions in the ancestral environment, and sleep disorder due to electric lighting disrupting circadian rhythms is a good fit to the evidence.
The evidence for each of my main points is fairly substantial and now mainstream, the only part which isn’t mainstream (yet) is the specific causal mechanism linking synaptic pruning/normalization to imbalance in valence computing upper brain modules (but it’s also fairly straightforward obvious from a DL perspective—we know that training stability is an intrinsic likely failure mode).
A few random links:
REM and synaptic normalization/pruning/homeostasis:
Plasticity during sleep is linked to specific regulation of cortical circuit activity
REM sleep promotes experience-dependent dendritic spine elimination in the mouse cortex.
REM sleep selectively prunes and maintains new synapses in development and learning
Memory corticalization triggered by REM sleep: mechanisms of cellular and systems consolidation
Sleep and wake cycles dynamically modulate hippocampal inhibitory synaptic plasticity
Sleep and Psychiatric Disorders:
Sleep disturbance and psychiatric disorders : “It is argued that insomnia and other mental health conditions not only share common causes but also show a bidirectional relationship, with typically the strongest pathway being disrupted sleep as a causal factor in the occurrence of other psychiatric problems.”
Improving sleep quality leads to better mental health: A meta-analysis of randomised controlled trials: “For example, people with insomnia are 10 and 17 times more likely than those without insomnia to experience clinically significant levels of depression and anxiety, respectively”
The effectiveness of circadian interventions through the blue light pineal gland serotonin->melatonin pathway is also very well established: daytime bright light therapy has long been known to be effective for depression, nighttime blue light reduction is now also recognized as important/effective, etc.
The interventions required to promote healthy sleep architecture are not especially expensive and are certainly not patentable, so they are in a blindspot for our current partially misaligned drug-product focused healthcare system. Of course there would be a market for a hypothetical drug which could target and fix the specific issues that some people have with sleep quality—but instead we just have hammers like benzos and lithium which cause as many or more problems than they solve.
From my own study of mood disorders I generally agree with your valence theory of depression/mania.
However I believe the primary cause (at least for most people today) is disrupted sleep architecture.
To a first order approximation, the brain accumulates batch episodic training data during the day through indexing in the hippocampus (which is similar-ish to upper cortex, but more especially adapted to medium term memory & indexing). The brain’s main episodic replay training then occurs during sleep, with alternation of several key phases (REM and several NREM) with unique functional roles. During NREM (SWS in particular) the hippocampus rehearses sequences to ‘train’ the cortex via episodic replay. (Deepmind’s first atari RL agent is based on directly reverse engineering this mechanism).
But the REM sleep is also vitally important—and it seems to globally downscale/prune synaptic connections, most specifically the weakest and least important. It may also be doing something more complex in subtracting out the distribution of internally generated data ala Hinton’s theories (but maybe not, none of his sleep wake algos actually work well yet).
Regardless the brain does not seem to maintain synaptic strength balance on the hourly timescale. Instead median/average synaptic strength slowly grows without bound during the waking state, and is not correctly renormalized until pruning/renormalization during sleep—and REM sleep most specifically.
This explains many curious facts known of mania and depression:
-
The oldest known treatment for depression is also completely (but only temporarily) effective: sleep deprivation. Depression generally does not survive sleep deprivation.
-
Sleep is likewise effective to treat full blown mania, but mania inhibits sleep. One of the early successes in psychiatry was the use of sedatives to treat severe mania.
-
Red light interferes with the circadian rhythm—specifically serotonin->melatonin conversion, and thereby can disrupt sleep architecture (SAD etc)
-
SSRIs alter effective serotonin transport quickly but take a week or more to have noticeable effects on mood. Serotonin directly blocks REM—REM sleep is characterized (and probably requires) a near complete absence of monoamine neurotransmitters (histamine, serotonin and norepinephrine).
-
Lithium—a common treatment for bipolar—is a strong cellular circadian modulator and sleep stabilizer.
So basically the brain does not maintain perfect homeostatic synaptic normalization balance on short timescales. During wake synapses tend to strengthen, and during REM sleep they are pruned/weakened. Balancing this correctly seems to rely on a fairly complex sleep architecture, disruptions to which can cause mood disorders—not immediately, but over weeks/months.
But why does mean synaptic strength imbalance effect mostly mood and not say vision or motor control? Every synapse and brain region has a characteristic plasticity timescale that varies wildly. Peripheral lower regions (closer to sensors/motors) crystallize early and have low learning rate/plasticity in adults, so they aren’t very susceptible. At any one time in life the hippocampal → cortical episodic replay is focusing on particular brain modules, and in adults that focus is mostly on upper regions (PFC etc) that mostly store current plans, consequences, etc that are changing more rapidly.
Thus the upper brain regions that are proposing and computing the valence of various (actual or mental) actions as ‘dopaminergic bids’ with respect to current plans/situations are the most sensitive to synaptic norm imbalance, because they change at higher frequency. Of course if a manic stays awake long enough they do in fact progress to psychosis similar to schizophrenia.
-
Sure, but how often do the colonized end up better off for it, especially via trying to employ clever play-both-sides strategies?
I didn’t say the colonized generally ended up better off, but outcomes did vary greatly. Just in the US the cherokees faired much better than say the Susquehannock and Pequot, and if you dig into that history it seems pretty likely that decisions on which colonizer(s) to ally with (british, french, dutch, later american etc) were important, even if not “clever play-both-sides strategies” (although I’d be surprised if that wasn’t also tried somewhere at least once)
An idea sometimes floated around is to play them off against each other. If they’re misaligned from humanity, they’re likely mutually misaligned as well. We could put them in game-theoretic situations in which they’re incentivized to defect against each other and instead cooperate with humans.
You are arguing against a strawman. The optimistic game-theoretic argument you should focus on is:
Misaligned AIs are—almost by definition—instrumental selfish power seeking agents (with random long term goals) and thus intrinsically misaligned with each other. The partially aligned AIs will likely form a natural coalition with partial alignment to humanity as their centroid schelling point. The misaligned AIs could then form a natural counter-coalition in response.
There are numerous historical precedents such as the allies vs axis in world war two, and the allies vs china+russia today. The allies in either case have a mutual schelling point around democracy which is in fact greater partial alignment to their citizens and humanity. The axis powers (germany and japan, temporarily including russia earlier) were nearly completely intrinsically misaligned and formed a coalition of necessity. If they had won, they almost certainly would have then been in conflict (just as the west and the USSR was immediately in conflict after WW2).
I’m skeptical of some of your analysis even in the scenario you assume where all the AIs are completely unaligned, but that scenario is quite unlikely.
Specifically:
Imagine that you’re a member of a pre-industrial tribe, and the territory you’re living in has been visited by two different industrial nations.
That general scenario did play out a few times in history, but not at all as you described. The misaligned industrial nations absolutely fought against each other and various pre-industrial tribes picked one side or another. The story of colonization is absolutely not “colonizers super cooperating against the colonized”—it’s a story of many competing colonizers fighting in a race to colonize the world, with very little inter-colonizer cooperation.
Of course a massive advance is possible, but mostly just in terms of raw speed. The brain seems reasonably close to pareto efficiency in intelligence per watt for irreversible computers, but in the next decade or so I expect we’ll close that gap as we move into more ‘neuromorphic’ or PIM computing (computation closer to memory). If we used the ~1e16w solar energy potential of just the Saraha desert that would support a population of trillions of brain-scale AIs or uploads running 1000x real-time.
especially as our NN can use stuff such as backprop,
The brain appears to already using algorithms similar to—but more efficient/effective—than standard backprop.
potentially quantum algorithm to train weights
This is probably mostly a nothingburger for various reasons, but reversible computing could eventually provide some further improvement, especially in a better location like buried in the lunar cold spot.
The paper which more directly supports the “made them smarter” claim seems to be this. I did somewhat anticipate this—“not much special about the primate brain other than ..”, but was not previously aware of this particular line of research and certainly would not have predicted their claimed outcome as the most likely vs various obvious alternatives. Upvoted for the interesting link.
Specifically I would not have predicted that the graft of human glial cells would have simultaneously both 1.) outcompeted the native mouse glial cells, and 2.) resulted in higher performance on a handful of interesting cognitive tests.
I’m still a bit skeptical of the “made them smarter” claim as it’s always best to taboo ‘smarter’ and they naturally could have cherrypicked the tests (even unintentionally), but it does look like the central claim—that injection of human GPCs (glial progenitor cells) into fetal mice does result in mice brains that learn at least some important tasks more quickly, and this is probably caused by facilitation of higher learning rates. However it seems to come at a cost of higher energy expenditure, so it’s not clear yet that this is a pure pareto improvement—could be a tradeoff worthwhile in larger sparser human brains but not in the mouse brain such that it wouldn’t translate into fitness advantage.
Or perhaps it is a straight up pareto improvement—that is not unheard of, viral horizontal gene transfer is a thing, etc.
Suffering, disease and mortality all have a common primary cause—our current substrate dependence. Transcending to a substrate-independent existence (ex uploading) also enables living more awesomely. Immortality without transcendence would indeed be impoverished in comparison.
Like, even if they ‘inherit our culture’ it could be a “Disneyland with no children”
My point was that even assuming our mind children are fully conscious ‘moral patients’, it’s a consolation prize if the future can not help biological humans.
Input vs output tokens are both unique per agent history (prompt + output), so that differentiation doesn’t matter for my core argument about the RAM constraint. If you have a model which needs 1TB of KV cache, and you aren’t magically sharing that significantly between instances, then you’ll need at least 1000 * 1TB of RAM to run 1000 inferences in parallel.
The 3x − 10x cost ratio model providers charge is an economic observation that tells us something about the current cost vs utility tradeoffs, but it’s much complicated by oversimpliciation of the current pricing models (they are not currently charging their true costs, probably because that would be too complicated, but also perhaps reveal too much information—their true cost would be more like charging rent on RAM for every timestep). It just tells you that very roughly, that on average, the mean (averaged over many customer requests) flop utilization of the generation phase (parallel over instances) is perhaps 3x to 10x lower than the prefill phase (parallel over time) - but it doesn’t directly tell you why.
This is all downstream dependent on model design and economics. There are many useful requests that LLMs can fulfill without using barely any KV cache—essentially all google/oracle type use cases where you are just asking the distilled wisdom of the internet a question. If those were all of the request volume, then the KV cache RAM per instance would be inconsequential, inference batch sizes would be > 1000, inference flop utilization would be the same for prefill vs generation, and providers would charge the same price for input vs output tokens.
On the other extreme, if all requests used up the full training context window, then the flop utilization of inference would be constrained by approximately (max_KV_cache_RAM + weight_RAM / max_KV_cache_RAM ) / alu_ratio. For example if the KV cache is 10% of RAM, and alu_ratio is 1000:1, generation would have max efficiency of 1%. If infill efficiency was 30%, then output tokens would presumably be priced 30x more than input tokens.
So the observed input:output token pricing is dependent on the combination of KV_cache RAM fraction (largely a model design decision), current efficiency of implementations of infill vs generation, and most importantly—the distribution of request prompt lengths, which itself is dependent on the current economic utility of shorter vs longer prompts for current models.
In practice most current models have a much smaller KV cache to weight RAM fraction than my simple 1:1 example, but the basic point holds: training is more flop & interconnect limited, inference is more RAM and ram bw limited. But these constraints already shape the design space of models and how they are deployed.
LLMs currently excel at anything a human knowledge worker can do without any specific training (minimal input prompt length), but largely aren’t yet competitive with human experts at most real world economic tasks that require significant unique per-job training. Coding is a good example—human thoughtspeed is roughly 9 token/s, or 32K/hour, or 256K per 8 hour work day, or roughly 1M tokens per week.
Current GPT4-turbo (one of the current leaders for coding), for example, has a max context length of 128K (roughly 4 hours). But if you actually use all of that for each request for typical coding requests that generate say 1K of useful output (equivalent to a few minutes of human thought), that will cost you about $1.25 for the input tokens, but only about $0.03 for the output tokens. That costs about as much as a human worker, per minute of output thought tokens. The cost of any LLM agent today (per minute of output thought) increases linearily with input prompt length—ie the agent’s unique differentiating short term memory. Absent more sophisticated algorithms, the cost of running a react-like LLM agent thus grows quadratically with time, vs linear for humans (because each small observe-act time step has cost proportional to input context length, which grows per time step).
Human programmers aren’t being replaced en masse (yet) in part because current models aren’t especially smarter than humans at equivalent levels of job-specific knowledge/training.
Normalized for similar ability, LLMs currently are cheaper than humans at most any knowledge work that requires very little job-specific knowledge/training, and much more expensive than humans for tasks that require extensive job-specific knowledge/training—and this has everything to do with how transformers currently consume and utilize VRAM.