(...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties.
- Saharon Shelah
As a true-born Dutchman I endorse Crocker’s rules.
For my most of my writing see my short-forms (new shortform, old shortform)
Twitter: @FellowHominid
Personal website: https://sites.google.com/view/afdago/home
I am not sure what ‘it’ refers to in ‘it is bad’.
People read more into this shortform than I intended. It is not a cryptic reaction, criticism, or reply to/of another post.
I don’t know what you mean by intelligent [pejorative] but it sounds sarcarcastic.
To be clear, the low predictive efficiency is not a dig at archeology. It seems I have triggered something here.
Whether a question/domain has low or high (marginal) predictive effiency is not a value judgement, just an observation.
I don’t dispute these facts.
A lot of discussion of intelligence considers it as a scalar value that measures a general capability to solve a wide range of tasks. In this conception of intelligence it is primarily a question of having a ′ good Map’ . This is a simplistic picture since it’s missing the intrinsic limits imposed on prediction by the Territory. Not all tasks or domains have the same marginal returns to intelligence—these can vary wildly.
Let me tell you about a ‘predictive efficiency’ framework that I find compelling & deep and that will hopefully give you some mathematical flesh to these intuitions. I initially learned about these ideas in the context of Computational Mechanics, but I realized that there underlying ideas are much more general.
Let be a predictor variable that we’d like to use to predict a target variable under a joint distribution . For instance could be the contex window and could be the next hundred tokens, or could be the past market data and is the future market data.
In any prediction task there are three fundamental and independently varying quantities that you need to think of:
quantifies the reducible uncertainty or the amount of predictable information contained in .
For the third quantity, let us introduce the notion of causal states or minimally sufficient statistics. We define an equivalence relation on by declaring
The resulting equivalence classes, denoted as , yield a minimal sufficient statistic for predicting . This construction is ``minimal″ because it groups together all those that lead to the same predictive distribution , and it is ``sufficient″ because, given the equivalence class , no further refinement of can improve our prediction of .
From this, we define the forecasting complexity (or statistical complexity) as
which measures the amount of information—the cost in bits—to specify the causal state of . Finally, the \emph{predictive efficiency} is defined by the ratio
which tells us how much of the complexity actually contributes to reducing uncertainty in . In many real-world domains, even if substantial information is stored (high ), the gain in predictability () might be modest. This situation is often encountered in fields where, despite high skill ceilings (i.e. very high forecasting complexity), the net effect of additional expertise is limited because the predictive information is a small fraction of the complexity.
Example of low efficiency.
Let be the outcome of 100 independent fair coin flips, so each has bits.
Define as a single coin flip whose bias is determined by the proportion of heads in . That is, if has heads then:
Total information in : \\
When averaged over all possible , the mean bias is 0.5 so that is marginally a fair coin. Hence,
Conditional Entropy or irreducible uncertainty : \\
Given , the outcome is drawn from a Bernoulli distribution whose entropy depends on the number of heads in . For typical (around 50 heads), bit; however, averaging over all yields a slightly lower value. Numerically, one finds:
Predictable Information : \\
With the above numbers, the mutual information is
Forecasting Complexity : \\
The causal state construction groups together all sequences with the same number of heads. Since , there are 101 equivalence classes. The entropy of these classes is given by the entropy of the binomial distribution . Using an approximation:
Predictive Efficiency :
In this example, a vast amount of internal structural information (the cost to specify the causal state) is required to extract just a tiny bit of predictability. In practical terms, this means that even if one possesses great expertise—analogous to having high forecasting complexity —the net benefit is modest because the inherent (predictive efficiency) is low. Such scenarios are common in fields like archaeology or long-term political forecasting, where obtaining a single predictive bit of information may demand enormous expertise, data, and computational resources.
Please read carefully what I wrote—I am talking about energy consumption worldwide not electricity consumption in the EU. Electricity in the EU accounts only for a small percentage of carbon emissions.
See
As you can see, solar energy is still a tiny percentage of total energy sources. I don’t think it is an accident that the electricity split graph in the EU has been cited in this discussion because it is a proxy that is much more rose-colored.
Energy and electricity are often conflated in discussions around climate change, perhaps not coincidentally because the latter seems much more tractable to generate renewably than total energy production.
No, it is not confused. Be careful with reading precisely what I wrote. I said total energy production worldwide, not electricity production in the european union.
As you can see Solar is still a tiny percentage of energy consumption. That is not to say that things will not change—I certainly hope so! I give it significant probability. But if we are to be honest with ourselves than it is currently yet to be seen whether solar energy will prove to be the solution.
Moreover, in the case that solar energy does take over and ′ solve’ climate change that still does not prove the thesis—that solar energy solving climate change being majorly the result of deliberate policy instead of the result of market forces / ceteris paribus technological development.
I somewhat agree but
The correlation is not THAT strong
The correlation differs by field
And finally there is a difference between skill ceilings for domains with high versus low predictive efficiency. In the latter much more intelligence will still yield returns but rapidly diminishing
(See my other comment for more details on predictive effiency)
One aspect I didnt speak about that may be relevant here is the distinction between
irreducible uncertainty h (noise, entropy)
reducible uncertainty E (‘excess entropy’)
and forecasting complexity C (‘stochastic complexity’).
All three can independently vary in general.
Domains can be more or less noisy (more entropy h)- both inherently and because of limited observations
Some domains allow for a lot of prediction (there is a lot of reducible uncertainty E) while others allow for only limited prediction (eg political forecasting over longer time horizons)
And said prediction can be very costly to predict (high forecasting complexity C). Archeology is a good example: to predict one bit about the far past correctly might require an enormous amount of expertise, data and information. In other words it s really about the ratio between the reducible uncertainty and the forecasting complexity: E/C.
Some fields have very high skill ceiling but because of a low E/C ratio the net effect of more intelligence is modest. Some domains arent predictable at all, i.e. E is low. Other domains have a more favorable E/C ratio and C is high. This is typically a domain where there is a high skill ceiling and the leverage effect of addiitonal intelligence is very large.
[For a more precise mathematical toy model of h, E,C take a look at computational mechanics]
Wow! I like the idea of persuasion as acting on the lack of a fully coherent preference! Something to ponder 🤔
Is Superhuman Persuasion a thing?
Sometimes I see discussions of AI superintelligence developping superhuman persuasion and extraordinary political talent.
Here’s some reasons to be skeptical of the existence of ‘superhuman persuasion’.
We don’t have definite examples of extraordinary political talent.
Famous politicians rose to power only once or twice. We don’t have good examples of an individual succeeding repeatedly in different political environments.
Examples of very charismatic politicans can be better explained by ′ the right person at the right time or place’.
Neither do we have strong examples of extraordinary persuasion.
>> For instance hypnosis is mostly explained by people wanting to be persuaded by the hypnotist. If you don’t want to be persuaded it’s very hard to change your mind. There is some skill in persuasion required for sales, and sales people are explicitly trained to do so but beyond a fairly low bar the biggest predictors for salesperson success is finding the correct audience and making a lot of attempts.
Another reason has to do with the ′ intrinsic skill ceiling of a domain’ .
For an agent A to have a very high skill in a given domain is not just a question of the intelligence of A or the resources they have at their disposal; it also a question of how high the skill ceiling of that domain is.
Domains differ in how high their skill ceilings go. For instance, the skill ceiling of tic-tac-toe is very low. [1] Domains like medicine and law have moderately high skill ceiling: it takes a long time to become a doctor, and most people don’t have the ability to become a good doctor.
Domains like mathematics or chess have very high skill ceilings where a tiny group of individuals dominate everybody else. We can measure this fairly explicitly in games like Chess through an ELO rating system.
The domain of ′ becoming rich’ is mixed: the richest people are founders—becoming a wildly succesful founder requires a lot of skill but it is also very luck based.
Political forecasting is a measureable domain close to political talent. It seems to be very mixed bag whether this domain allows for a high skill ceiling. Most ′ political experts’ are not experts as shown by Tetlock et al. But even superforecaster only outperform for quite limited time horizons.
Domains with high skill ceilings are quite rare. Typically they operate in formal systems with clear rules and objective metrics for success and low noise. By contrast, persuasion and political talent likely have lower natural ceilings because they function in noisy, high-entropy social environments.
What we call political genius often reflects the right personality at the right moment rather than superhuman capability. While we can identify clear examples of superhuman technical ability (even in today’s AI systems), the concept of “superhuman persuasion” may be fundamentally limited by the unpredictable, context-dependent, and adaptive & adversarial [people resist hostile persuasion] nature of human social response.
Most persuasive domains may cap out at relatively modest skill ceilings because the environment is too chaotic and subjective to allow for the kind of systematic skill development possible in more structured domains.
I’m amused that many frontier models still struggle with tic-tac-toe; though likely for not so good reasons.
Ryan Greenblatt on steering the AI paradigm:
I’m skeptical of strategies which look like “steer the paradigm away from AI agents + modern generative AI paradigm to something else which is safer”. Seems really hard to make this competitive enough and I have other hopes that seem to help a bunch while being more likely to be doable.
(This isn’t to say I expect that the powerful AI systems will necessarily be trained with the most basic extrapolation of the current paradigm, just that I think steering this ultimate paradigm to be something which is quite different and safer is very difficult.)
I agree and I would add I am generally suspicious of a general pattern here. Let’s call it the ′ Steering Towards Safe Paradigms’-strategy.
It goes something like this:
>>> There is a technology X causing public harm Z. The problem is that X is very competitive—individual actors benefit greatly from using X.
Instead of working to
(i) [develop technologies to] mitigate the harms of technology X,
(ii) put in existing safeguards or develop new safeguards to prevent harm Z,
(iii) or ban/govern technology X
we should instead purposely and deliberately develop a technology Y that is fundamentally different than X that does the same role as X without harm Z but slightly less competitively. >>>
A good example could be the development of renewable energy to replace fossil fuels to prevent climate change.
I am generally skeptical of strategies that look like ′ steer the paradigm’ as I think it overstates in general how much influence ′ we’ (individuals, governments, human society as a whole etc) have and falls prey to a social desirability bias. In so far as it is based on uncalibrated hopes and beliefs about the actual action space it can be net harmful.
Note that this is distinct from humanity automatically developing a technology Y’ that is more competitive than X without harm Z because Y’ is actually just straight up the next step in the tech tree that is more competitive for mundane market reasons.
An example could be the benefit of plastic billard balls instead of ivory billard balls saving the elephant. Although saving the elephant was iirc part of the reason to develop plastic billard balls the reason it worked was more mundane: plastic billard balls are far cheaper than ivory billard balls. In other words, standard market forces were likely the prime cause, not deliberate and purposeful policy to develop a less competitive but safer technology.
Let’s zoom in to the renewable energy example. Are these examples where the ′ Steering towards Safe Paradigms’-strategy was succesful?
The new tech (fusion, fission, solar, wind) is based on fundamental principles than the old tech (oil and gas).
Fusion would be an example but perpetually thirty years away. Fission works but wasnt purposely develloped to fight climate change. Wind is deliberatily and purposefully developed to combat climate change but unfortunately it is not competitive without large subsidies and most likely never will [mostly because of low energy density and off-peak load balance concerns].
Solar is at least somewhat competitive with fossil fuels [except because of load balancing it may not be able to replace fossil fuels completely] and purposely developped out of environmental concerns and would be the best example.
I think my main question marks here is: solar energy is still a promise. It hasnt even begun to make a dent in total energy consumption ( a quick perplexity search reveals only 2 percent of global energy is solar-generated). Despite the hype it is not clear climate change will be solved by solar energy.
Moreover, the real question is to what degree the development of competitive solar energy was the result of a purposeful policy. People like to believe that tech development subsidies have a large counterfactual but imho this needs to be explicitly proved and my prior is that the effect is probably small compared to overall general development of technology & economic incentives that are not downstream of subsidies / government policy.
Let me contrast this with two different approaches to solving a problem Z (climate change).
Deploy existing competitive technology (fission)
Mitigate the harm directly (geo-engineering)
It seems to me that in general the latter two approaches have a far better track record of counterfactually Actually Solving the Problem. And perhaps more cynically, the hopeful promise of ‘Steering towards Safe Paradigms’ - strategy in combatting climate change is taking away from the above more mundane strategies.
My comment was a little ambiguous. What I meant was human society purposely differentially researching and developing technology X instead of Y where Y has a public (global) harm Z but private benefit and X is based on a different design principle than Y but slightly less competitive but still able to replace Y.
A good example would be the development of renewable energy to replace fossil fuels to prevent climate change.
The new tech (fusion, fission, solar, wind) is based on fundamental principles than the old tech (oil and gas).
Lets zoom in:
Fusion would be an example but perpetually thirty years away. Fission works but wasnt purposely develloped to fight climate change. Wind is not competitive without large subsidies and most likely never will.
Solar is at least lomited competitive with fossil fuels [except because of load balancing it may not be able to replace fossil fuels completely] , purposely developped out of environmental concerns and would be the best example.
I think my main question marks here is: solar energy is still a promise. It hasnt even begun to make a dent in total energy consumption ( a quick perplexity search reveals only 2 percent of global energy is solar-generated). Despite the hype it is not clear climate change will be solved by solar energy.
Moreover, the real question is to what degree the development of competitive solar energy was the result of a purposeful policy. People like to believe that tech development subsidies have a large counterfactual but imho this needs to be explicitly proved and my prior is that the effect is probably small compared to overall general development of technology & economic incentives that are not downstream of subsidies / government policy.
Let me contrast this with two different approaches to solving a problem Z (climate change).
Deploy existing competitive technology (fission)
Solve the problem directly (geo-engineering)
It seems to me that in general the latter two approaches have a far better track record of counterfactually Actually Solving the Problem.
Yes—this is specifically staying within the framework of hidden markov chains.
Even if you go outside though it seems you agree there is a generative predictive gap—you’re just saying it’s not infinite.
Eggsyntax below gives the canonical example of hash function where prediction is harder than generation which hold for general computable processes.
A suggestive analogy of AI takeover might be the early-modern European colonial takeovers of the New World, India under Pizarro, Cortez, Clives.
Colonialism took several hundred years but the major disempowerement events were quite relatively quick, involving explicit hostile action using superior technology [and many native allies].
Are you imagining AI takeover to be slower or similar to Pizarro/Cortez/Clives campaigns?
A potential counterargument to gradual disempowerement is that one of the decisive advantages that AIs have over humanity & its decisionmaking processing are its speed of thinking and it’s speed and coherency of decision-making (not now but for future agentic systems). Most decisionmaking at a national and international level is slow and mostly concerned with symbolics because of the inherent limitations of decisionmaking at the scale of large human societies [even authoritarian states struggle with good and fast decisionmaking]. AIs could circumvent this and act much faster and decisively.
Another argument is that humans aren’t stupid. The public doesn’t trust AI. When the power of AI is clearly visible it seems likely the public will react given time. Being publicly semi-hostile will incur a response. One shouldn’t fall into the sleepwalker bias. Faking alignment and then suddenly striking seems to be a much more rational move than gradual disempowerment.
Congratulations on this paper. It seems like a major result.
Any chance of more exposition for those of us less cognitively-inclined? =)
the steelman is that quants do the version of technical analysis that works—they disprove the EMH proportional to quant salaries.
Couldn’t agree more. Variants of this strategy get proposed often.
If you are a proponent of this strategy—I’m curious whether you know of any examples in history where humanity purposefully and succesfully steered towards a significantly less competitive [economically, militarily,...] technology that was nonetheless safer.
Threefold duality and the Eightfold path of Option Trading
It is a truth universally acknowledged that the mere whiff of duality is catnip to the mathematician.
Given any asset X, like a stock, sold for a price P means there is a duality between buying and selling: one party buys X for P while another sells X for P.
Implicitly there is another duality: instead of interchanging the buy and sell actions, one can interchange the asset X and the price P, treating money as an asset and the asset as a medium of exchange.
A European call option at strike price S gives one the option to buy the underlying asset X at price S on the expiration date. [1]
There is a dual option—called a (European) put option that gives one the right [option] to sell the underlying on the expiration date.
Optionality is beautiful. Optionality is brilliant. The pursuit of optionality writ large is the great purpose of the higher limbic system. An option is optionality incarnate, offering limited downside yet unlimited upside potential.
There are much more classic financial instruments, but options represent a powerful abstraction layer on top of direct asset ownership.
Rk. Any security’s payoff profile can be theoretically replicated using only put and call options. This is known as the Options Replication Theorem or sometimes the Fundamental Theorem of Asset Pricing in its broader form.
Options themselves can be bought and sold.
There is now a fourfold of options:
Buy a Call option
Sell a Call option
Buy a Put option
Sell a Put option
A single transaction might offer all four in some combination at the same time.
There is a third hidden duality in option trading—that between owning and creating an option. So far we have regarded as the existence of options as given—options are bought and sold and they can be exercised (extinguished) but where do options come from? But options have to be created.
Creating an option is called a ‘writing an option’. Writing an option is an inherent dangerous activity, only done by the largest, strongest, most experienced economical actors. Writing an option exposes its scribe to unlimited downside yet limited upside.
Options are closely related to infraBayesianism and Imprecise Probability
Duality is always interesting.
There has been some speculation about a Logic of Capitalism—intrinstic Grammatica of Cash. Perhaps the eightfold duality of option trading is the skeleton key to its formulation.
^ There is a minor difference between so-called ‘American’ style and ‘European’ style options. The former allows exercising the options up to the expiry date while the latter only allows exercising at the date. I’m taking European options by default, mostly because the Black-Scholes equation[2] is more easily expressed for European options—they admit a closed form solutions while there is no closed form solution for American options in general.
^ Note that the Black-Scholes equation is a special case of the Hamilton-Jacobi-Bellman equation, the central equation in both physics and reinforcement learning & control theory.
From marginal revolution.
What does this crowd think? These effects are surprisingly small. Do we believe these effects? Anecdotally the effect of LLMs has been enormous for my own workflow and colleagues. How can this be squared with the supposedly tiny labor market effect?
Are we that selected of a demographic?