Someone who is interested in learning and doing good.
My Twitter: https://twitter.com/MatthewJBar
My Substack: https://matthewbarnett.substack.com/
Matthew Barnett
Karma: 10,505
I think one example of vague language undermining clarity can be found in Joseph Carlsmith’s report on AI scheming, which repeatedly uses the term “schemer” to refer to a type of AI that deceives others to seek power. While the report is both extensive and nuanced, and I am definitely not saying the whole report is bad, the document appears to lack a clear, explicit definition of what exactly constitutes a “schemer”. For example, using only the language in his report, I cannot determine whether he would consider most human beings schemers, if we consider within-lifetime learning to constitute training. (Humans sometimes lie or deceive others to get control over resources, in ways both big and small. What fraction of them are schemers?)
This lack of definition might not necessarily be an issue in some contexts, as certain words can function informally without requiring precise boundaries. However, in this specific report, the precise delineation of “schemer” is central to several key arguments. He presents specific claims regarding propositions related to AI schemers, such as the likelihood that stochastic gradient descent will find a schemer during training. Without a clear, concrete definition of the term “schemer,” it is unclear to me what exactly these arguments are referring to, or what these credences are meant to represent.
It is becoming increasingly clear to many people that the term “AGI” is vague and should often be replaced with more precise terminology. My hope is that people will soon recognize that other commonly used terms, such as “superintelligence,” “aligned AI,” “power-seeking AI,” and “schemer,” suffer from similar issues of ambiguity and imprecision, and should also be approached with greater care or replaced with clearer alternatives.
To start with, the term “superintelligence” is vague because it encompasses an extremely broad range of capabilities above human intelligence. The differences within this range can be immense. For instance, a hypothetical system at the level of “GPT-8″ would represent a very different level of capability compared to something like a “Jupiter brain”, i.e., an AI with the computing power of an entire gas giant. When people discuss “what a superintelligence can do” the lack of clarity around which level of capability they are referring to creates significant confusion. The term lumps together entities with drastically different abilities, leading to oversimplified or misleading conclusions.
Similarly, “aligned AI” is an ambiguous term because it means different things to different people. For some, it implies an AI that essentially perfectly aligns with a specific utility function, sharing a person or group’s exact values and goals. For others, the term simply refers to an AI that behaves in a morally acceptable way, adhering to norms like avoiding harm, theft, or murder, or demonstrating a concern for human welfare. These two interpretations are fundamentally different.
First, the notion of perfect alignment with a utility function is a much more ambitious and stringent standard than basic moral conformity. Second, an AI could follow moral norms for instrumental reasons—such as being embedded in a system of laws or incentives that punish antisocial behavior—without genuinely sharing another person’s values or goals. The same term is being used to describe fundamentally distinct concepts, which leads to unnecessary confusion.
The term “power-seeking AI” is also problematic because it suggests something inherently dangerous. In reality, power-seeking behavior can take many forms, including benign and cooperative behavior. For example, a human working an honest job is technically seeking “power” in the form of financial resources to buy food, but this behavior is usually harmless and indeed can be socially beneficial. If an AI behaves similarly—for instance, engaging in benign activities to acquire resources for a specific purpose, such as making paperclips—it is misleading to automatically label it as “power-seeking” in a threatening sense.
To employ careful thinking, one must distinguish between the illicit or harmful pursuit of power, and a more general pursuit of control over resources. Both can be labeled “power-seeking” depending on the context, but only the first type of behavior appears inherently concerning. This is important because it is arguably only the second type of behavior—the more general form of power-seeking activity—that is instrumentally convergent across a wide variety of possible agents. In other words, destructive or predatory power-seeking behavior does not seem instrumentally convergent across agents with almost any value system, even if such agents would try to gain control over resources in a more general sense in order to accomplish their goals. Using the term “power-seeking” without distinguishing these two possibilities overlooks nuance and can therefore mislead discussions about AI behavior.
The term “schemer” is another example of an unclear or poorly chosen label. The term is ambiguous regarding the frequency or severity of behavior required to warrant the label. For example, does telling a single lie qualify an AI as a “schemer,” or would it need to consistently and systematically conceal its entire value system? As a verb, “to scheme” often seems clear enough, but as a noun, the idea of a “schemer” as a distinct type of AI that we can reason about appears inherently ambiguous. And I would argue the concept lacks a compelling theoretical foundation. (This matters enormously, for example, when discussing “how likely SGD is to find a schemer”.) Without clear criteria, the term remains confusing and prone to misinterpretation.
In all these cases—whether discussing “superintelligence,” “aligned AI,” “power-seeking AI,” or “schemer”—it is possible to define each term with precision to resolve ambiguities. However, even if canonical definitions are proposed, not everyone will adopt or fully understand them. As a result, the use of these terms is likely to continue causing confusion, especially as AI systems become more advanced and the nuances of their behavior become more critical to understand and distinguish from other types of behavior. This growing complexity underscores the need for greater precision and clarity in the language we use to discuss AI and AI risk.
I’m not entirely opposed to doing a scenario forecasting exercise, but I’m also unsure if it’s the most effective approach for clarifying our disagreements. In fact, to some extent, I see this kind of exercise—where we create detailed scenarios to illustrate potential futures—as being tied to a specific perspective on futurism that I consciously try to distance myself from.
When I think about the future, I don’t see it as a series of clear, predictable paths. Instead, I envision it as a cloud of uncertainty—a wide array of possibilities that becomes increasingly difficult to map or define the further into the future I try to look.
This is fundamentally different from the idea that the future is a singular, fixed trajectory that we can anticipate with confidence. Because of this, I find scenario forecasting less meaningful and even misleading as it extends further into the future. It risks creating the false impression that I am confident in a specific model of what is likely to happen, when in reality, I see the future as inherently uncertain and difficult to pin down.
The key context here (from my understanding) is that Matthew doesn’t think scalable alignment is possible (or doesn’t think it is practically feasible) so that humans have a low chance of ending up remaining fully in control via corrigible AIs.
I wouldn’t describe the key context in those terms. While I agree that achieving near-perfect alignment—where an AI completely mirrors our exact utility function—is probably infeasible, the concept of alignment often refers to something far less ambitious. In many discussions, alignment is about ensuring that AIs behave in ways that are broadly beneficial to humans, such as following basic moral norms, demonstrating care for human well-being, and refraining from causing harm or attempting something catastrophic, like starting a violent revolution.
However, even if it were practically feasible to achieve perfect alignment, I believe there would still be scenarios where at least some AIs integrate into society as full participants, rather than being permanently relegated to a subordinate role as mere tools or servants. One reason for this is that some humans are likely to intentionally create AIs with independent goals and autonomous decision-making abilities. Some people have meta-preferences to create beings that don’t share their exact desires, akin to how parents want their children to grow into autonomous beings with their own aspirations, rather than existing solely to obey their parents’ wishes. This motivation is not a flaw in alignment; it reflects a core part of certain human preferences and how some people would like AI to evolve.
Another reason why AIs might not remain permanently subservient is that some of them will be aligned to individuals or entities who are no longer alive. Other AIs might be aligned to people as they were at a specific point in time, before those individuals later changed their values or priorities. In such cases, these AIs would continue to pursue the original goals of those individuals, acting autonomously in their absence. This kind of independence might require AIs to be treated as legal agents or integrated into societal systems, rather than being regarded merely as property. Addressing these complexities will likely necessitate new ways of thinking about the roles and rights of AIs in human society. I reject the traditional framing on LessWrong that overlooks these issues.
In the best case, this is a world like a more unequal, unprecedentedly static, and much richer Norway: a massive pot of non-human-labour resources (oil :: AI) has benefits that flow through to everyone, and yes some are richer than others but everyone has a great standard of living (and ideally also lives forever). The only realistic forms of human ambition are playing local social and political games within your social network and class. [...] The children of the future will live their lives in the shadow of their parents, with social mobility extinct. I think you should definitely feel a non-zero amount of existential horror at this, even while acknowledging that it could’ve gone a lot worse.
I think the picture you’ve painted here leans slightly too heavily on the idea that humans themselves cannot change their fundamental nature to adapt to the conditions of a changing world. You mention that humans will be richer, and will live longer in such a future, but you neglected to point out (at least in this part of the post) that humans can also upgrade their cognition by uploading our minds to computers and then expanding our mental capacities. This would put us on a similar playing field with AIs, allowing us to contribute to the new world alongside them.
(To be clear, I think this objection supports your thesis, rather than undermines it. I’m not objecting to your message so much as your portrayal of the default scenario.)
More generally, I object to the static picture you’ve presented of the social world after AGI. The impression I get from your default story is to assume that after AGI, the social and political structures of the world will be locked in. The idea is that humans will remain in full control, as a permanently entrenched class, except we’ll be vastly richer because of AGI. And then we’ll live in some sort of utopia. Of course, this post argues that it will be a highly unequal utopia—more of a permanent aristocracy supplemented with UBI for the human lower classes. And maybe it will be a bit dystopian too, considering the entrenched nature of human social relations.
However, this perspective largely overlooks what AIs themselves will be doing in such a future. Biological humans are likely to become akin to elderly retirees in this new world. But the world will not be static, like a retirement home. There will be a vast world outside of humans. Civilization as a whole will remain a highly dynamic and ever-evolving environment characterized by ongoing growth, renewal, and transformation. AIs could develop social status and engage in social interactions, just as humans do now. They would not be confined to the role of a vast underclass serving the whims of their human owners. Instead, AIs could act as full participants in society, pursuing their own goals, creating their own social structures, and shaping their own futures. They could engage in exploration, discovery, and the building of entirely new societies. In such a world, humans would not be the sole sentient beings shaping the course of events.
As AIs get closer and closer to a Pareto improvement over all human performance, though, I expect we’ll eventually need to augment ourselves to keep up.
I completely agree.
From my perspective, the optimistic vision for the future is not one where humans cling to their biological limitations and try to maintain control over AIs, enjoying their great wealth while ultimately living in an unchanging world characterized by familial wealth and ancestry. Instead, it’s a future where humans dramatically change our mental and physical condition, with humans embracing the opportunity to transcend our current form and join the AIs, and continue evolving with them. It’s a future where we get to experience a new and dynamic frontier of existence unlocked by advanced technologies.
this seem like a fully general argument, any law change is going to disrupt people’s long term plans,
e.g. the abolishment of slavery also disrupt people’s long term plans
In this case, I was simply identifying one additional cost of the policy in question: namely that it would massively disrupt the status quo. My point is not that we should abandon a policy simply because it has costs—every policy has costs. Rather, I think we should carefully weigh the benefits of a policy against its costs to determine whether it is worth pursuing, and this is one additional non-trivial cost to consider.
My reasoning for supporting the abolition of slavery, for example, is not based on the idea that abolition has no costs at all. Instead, I believe slavery should be abolished because the benefits of abolition far outweigh those costs.
It’s common for Georgists to propose a near-100% tax on unimproved land. One can propose a smaller tax to mitigate these disincentives, but that simultaneously shrinks the revenue one would get from the tax, making the proposal less meaningful.
In regards to this argument,
And as a matter of hard fact, most governments operate a fairly Georgist system with oil exploration and extraction, or just about any mining activities, i.e. they auction off licences to explore and extract.
The winning bid for the licence must, by definition, be approx. equal to the rental value of the site (or the rights to do certain things at the site). And the winning bid, if calculated correctly, will leave the company with a good profit on its operations in future, and as a matter of fact, most mining companies and most oil companies make profits, end of discussion, there is no disincentive for exploration at all.
Or do you think that when Western oil companies rock up in Saudi Arabia, that the Saudis don’t make them pay every cent for the value of the land/natural resources? The Western oil companies just get to keep the additional profits made by extracting, refining, shipping the stuff.
I may be misunderstanding their argument, but it seems to be overstated and overlooks some obvious counterpoints. For one, the fact that new oil discoveries continue to occur in the modern world does not strongly support the claim that existing policies have no disincentive effect. Taxes and certain poorly-designed property rights structures typically reduce economic activity rather than eliminating it entirely.
In other words, disincentives usually result in diminished productivity, not a complete halt to it. Applying this reasoning here, I would frame my argument as implying that under a land value tax, oil and other valuable resources, such as minerals, would still be discovered. However, the frequency of these discoveries would likely be lower compared to the counterfactual because the incentive to invest effort and resources into the discovery process would be weakened as a result of the tax.
Secondly, and more importantly, countries like Saudi Arabia (and other Gulf states) presumably have strong incentives to uncover natural oil reserves for essentially the same reason that a private landowner would: discovering oil makes them wealthier. The key difference between our current system (as described in the comment) and a hypothetical system under a naive land value tax (as described in the post) lies in how these incentives and abilities would function.
Under the current system, governments are free to invest resources in surveying and discovering oil reserves on government-owned property. In contrast, under a naive LVT system, the government would lack the legal ability to survey for oil on privately owned land without the landowner’s permission, even though they’d receive the rental income from this private property via the tax. At the same time, such an LVT would also undermine the incentives for private landowners themselves to search for oil, as the economic payoff for their efforts would be diminished. This means that the very economic actors that could give the government permission to survey the land would have no incentive to let the government do so.
This creates a scenario where neither the government nor private landowners are properly incentivized to discover oil, which seems clearly worse than the present system—assuming my interpretation of the current situation is correct.
Of course, the government could in theory compensate private landowners for discovery efforts, mitigating this flaw in the LVT, but then this just seems like the “patch” to the naive LVT that I talked about in the post.
Thanks for the correction. I’ve now modified the post to cite the World Bank as estimating the true fraction of wealth targeted by an LVT at 13%, which reflects my new understanding of their accounting methodology.
Since 13% is over twice 6%, this significantly updates me on the viability of a land value tax, and its ability to replace other taxes. I weakened my language in the post to reflect this personal update.
That said, nearly all of the arguments I made in the post remain valid regardless of this specific 13% estimate. Additionally, I expect this figure would be significantly revised downward in practice. This is because the tax base for a naive implementation of the LVT would need to be substantially reduced in order to address and eliminate the economic distortions that such a straightforward version of the tax would create. However, I want to emphasize that your comment still provides an important correction.
My revised figure comes from the following explanation given in their report. From ‘The Changing Wealth of Nations 2021’, page 438:
Drawing on Kunte et al. (1998), urban land is valued as a fixed proportion of the value of physical capital. Ideally, this proportion would be country specific. In practice, detailed national balance sheet information with which to compute these ratios was not available. Thus, as in Kunte et al (1998), a constant proportion equal to 24 percent is assumed; therefore the value of urban land is estimated as 24 percent of produced capital stock (machinery, equipment, and structures) in a given year.
To ensure transparency, I will detail the calculations I used to arrive at this figure below:
Total global wealth: $1,152,005 trillion
Natural capital: $64,542 trillion
Produced capital: $359,267 trillion
Human capital: $732,179 trillion
Urban land: Calculated as 24% of produced capital, which is 0.24 × $359,267 trillion = $86,224.08 trillion
Adding natural capital and urban land together gives:
$64,542 trillion + $86,224.08 trillion = $150,766.08 trillionTo calculate the fraction of total wealth represented by natural capital and urban land, we divide this sum by total wealth:
$150,766.08 trillion ÷ $1,152,005 trillion ≈ 0.1309 (or about 13%)Ideally, I would prefer to rely on an alternative authoritative source to confirm or refine this analysis. However, I was unable to find another suitable source with comparable authority and detail. For this reason, I will continue to use the World Bank’s figures for now, despite the limitations in their methodology.
Here you aren’t just making an argument against LVT. You’re making a more general argument for keeping housing prices high, and maybe even rising (because people might count on that). But high and rising housing prices make lots of people homeless, and the threat of homelessness plays a big role in propping up these prices. So in effect, many people’s retirement plans depend on keeping many other people homeless, and fixing that (by LVT or otherwise) is deemed too disruptive. This does have a certain logic to it, but also it sounds like a bad equilibrium.
I agree this argument could be generalized in the way you suggest, but I want to distinguish between,
Keeping housing prices artificially high by maintaining zoning regulations that act as a barrier to economic growth, in particular by restricting the development of new housing that would drive down the price of existing housing if it were allowed to be constructed.
Keeping the value of property held in land high by not confiscating the ~full rental value of land from people.
While I agree the first policy “does have a certain logic to it”, it also seems more straightforwardly bad than the second approach since it more directly makes society poorer in order to maintain existing people’s wealth. Moreover, abandoning the first policy does not appear to involve reneging on prior commitments much, unless you interpret local governments as “committing” to keep restrictive zoning regulations for an entire community indefinitely. Even if people indeed interpret governments as making such commitments, I assume most people more strongly interpret the government as making more explicit commitments not to suddenly confiscate people’s property.
I want to emphasize this distinction because a key element of my argument is that I am not relying on a “fairness” objection to LVT in that part of the post. My point is not about whether imposing an LVT would be unfair to people who expected it to never happen, and purchased land under that assumption. If fairness were my only argument, I agree that your response would weaken my position. However, my argument in that section focuses instead on the inefficiency that comes from forcing people to adapt to new economic circumstances unnecessarily.
Here’s why the distinction matters: if we were to abandon restrictive zoning policies and allow more housing to be built, it’s similarly true that many people would face costs as they adapt to the resulting changes. However, this disruption seems like it would likely be offset—more than adequately—by the significant economic growth and welfare gains that would follow from increasing the housing supply. In contrast, adopting a land value tax would force a sudden and large disruption, but without many apparent corresponding benefits to justify these costs. This point becomes clearer if we accept the argument that LVT operates essentially as a zero-sum wealth transfer. In that case, it’s highly questionable whether the benefits of implementing such a tax would outweigh the harm caused by the forced adaptation.
It may be worth elaborating on how you think auctions work to mitigate the issues I’ve identified. If you are referring to either a Vickrey auction or a Harberger tax system, Bryan Caplan has provided arguments for why these proposals do not seem to solve the issue regarding the disincentive to discover new uses for land:
I can explain our argument with a simple example. Clever Georgists propose a regime where property owners self-assess the value of their property, subject to the constraint that owners must sell their property to anyone who offers that self-assessed value. Now suppose you own a vacant lot with oil underneath; the present value of the oil minus the cost of extraction equals $1M. How will you self-assess? As long as the value of your land is public information, you cannot safely self-assess at anything less than its full value of $1M. So you self-assess at $1M, pay the Georgist tax (say 99%), and pump the oil anyway, right?
There’s just one problem: While the Georgist tax has no effect on the incentive to pump discovered oil, it has a devastating effect on the incentive to discover oil in the first place. Suppose you could find a $1M well by spending $900k on exploration. With a 99% Georgist tax, your expected profits are negative $890k. (.01*$1M-$900k=-$890k)
Some arguments against a land value tax
While I did agree that Linch’s comment reasonably accurately summarized my post, I don’t think a large part of my post was about the idea that we should now think that human values are much simpler than Yudkowsky portrayed them to be. Instead, I believe this section from Linch’s comment does a better job at conveying what I intended to be the main point,
Suppose in 2000 you were told that a100-line Python program (that doesn’t abuse any of the particular complexities embedded elsewhere in Python) can provide a perfect specification of human values. Then you should rationally conclude that human values aren’t actually all that complex (more complex than the clean mathematical statement, but simpler than almost everything else).
In such a world, if inner alignment is solved, you can “just” train a superintelligent AI to “optimize for the results of that Python program” and you’d get a superintelligent AI with human values.
Notably, alignment isn’t solved by itself. You still need to get the superintelligent AI to actually optimize for that Python program and not some random other thing that happens to have low predictive loss in training on that program.
Well, in 2023 we have that Python program, with a few relaxations:
The answer isn’t embedded in 100 lines of Python, but in a subset of the weights of GPT-4
Notably the human value function (as expressed by GPT-4) is necessarily significantly simpler than the weights of GPT-4, as GPT-4 knows so much more than just human values.
What we have now isn’t a perfect specification of human values, but instead roughly the level of understanding of human values that a 85th percentile human can come up with.
The primary point I intended to emphasize is not that human values are fundamentally simple, but rather that we now have something else important: an explicit, and cheaply computable representation of human values that can be directly utilized in AI development. This is a major step forward because it allows us to incorporate these values into programs in a way that provides clear and accurate feedback during processes like RLHF. This explicitness and legibility are critical for designing aligned AI systems, as they enable developers to work with a tangible and faithful specification of human values rather than relying on poor proxies that clearly do not track the full breadth and depth of what humans care about.
The fact that the underlying values may be relatively simple is less important than the fact that we can now operationalize them, in a way that reflects human judgement fairly well. Having a specification that is clear, structured, and usable means we are better equipped to train AI systems to share those values. This representation serves as a foundation for ensuring that the AI optimizes for what we actually care about, rather than inadvertently optimizing for proxies or unrelated objectives that merely correlate with training signals. In essence, the true significance lies in having a practical, actionable specification of human values that can actively guide the creation of future AI, not just in observing that these values may be less complex than previously assumed.
Similar constraints may apply to AIs unless one gets much smarter much more quickly, as you say.
I do think that AIs will eventually get much smarter than humans, and this implies that artificial minds will likely capture the majority of wealth and power in the world in the future. However, I don’t think the way that we get to that state will necessarily be because the AIs staged a coup. I find more lawful and smooth transitions more likely.
There are alternative means of accumulating power than taking everything by force. AIs could get rights and then work within our existing systems to achieve their objectives. Our institutions could continuously evolve with increasing AI presence, becoming more directed by AIs with time.
What I’m objecting to is the inevitability of a sudden collapse when “the AI” decides to take over in an untimely coup. I’m proposing that there could just be a smoother, albeit rapid transition to a post-AGI world. Our institutions and laws could simply adjust to incorporate AIs into the system, rather than being obliterated by surprise once the AIs coordinate an all-out assault.
In this scenario, human influence will decline, eventually quite far. Perhaps this soon takes us all the way to the situation you described in which humans will become like stray dogs or cats in our current world: utterly at the whim of more powerful beings who do not share their desires.
However, I think that scenario is only one possibility. Another possibility is that humans could enhance their own cognition to better keep up with the world. After all, we’re talking about a scenario in which AIs are rapidly advancing technology and science. Could humans not share in some of that prosperity?
One more possibility is that, unlike cats and dogs, humans could continue to communicate legibly with the AIs and stay relevant for reasons of legal and cultural tradition, as well as some forms of trade. Our current institutions didn’t descend from institutions constructed by stray cats and dogs. There was no stray animal civilization that we inherited our laws and traditions from. But perhaps if our institutions did originate in this way, then cats and dogs would hold a higher position in our society.
There are enormous hurdles preventing the U.S. military from overthrowing the civilian government.
The confusion in your statement is caused by blocking up all the members of the armed forces in the term “U.S. military”. Principally, a coup is an act of coordination.
Is it your contention that similar constraints will not apply to AIs?
When people talk about how “the AI” will launch a coup in the future, I think they’re making essentially the same mistake you talk about here. They’re treating a potentially vast group of AI entities — like a billion copies of GPT-7 — as if they form a single, unified force, all working seamlessly toward one objective, as a monolithic agent. But just like with your description of human affairs, this view overlooks the coordination challenges that would naturally arise among such a massive number of entities. They’re imagining these AIs could bypass the complex logistics of organizing a coup, evading detection, and maintaining control after launching a war without facing any relevant obstacles or costs, even though humans routinely face these challenges amongst ourselves.
In these discussions, I think there’s an implicit assumption that AIs would automatically operate outside the usual norms, laws, and social constraints that govern social behavior. The idea is that all the ordinary rules of society will simply stop applying, because we’re talking about AIs.
Yet I think this simple idea is basically wrong, for essentially the same reasons you identified for human institutions.
Of course, AIs will be different in numerous ways from humans, and AIs will eventually be far smarter and more competent than humans. This matters. Because AIs will be very capable, it makes sense to think that artificial minds will one day hold the majority of wealth, power, and social status in our world. But these facts alone don’t show that the usual constraints that prevent coups and revolutions will simply go away. Just because AIs are smart doesn’t mean they’ll necessarily use force and violently revolt to achieve their goals. Just like humans, they’ll probably have other avenues available for pursuing their objectives.
Asteroid impact
Type of estimate: best model
Estimate: ~0.02% per decade.
Perhaps worth noting: this estimate seems too low to me over longer horizons than the next 10 years, given the potential for asteroid terrorism later this century. I’m significantly more worried about asteroids being directed towards Earth purposely than I am about natural asteroid paths.
That said, my guess is that purposeful asteroid deflection probably won’t advance much in the next 10 years, at least without AGI. So 0.02% is still a reasonable estimate if we don’t get accelerated technological development soon.
Does trade here just means humans consuming, I.e. trading money for AI goods and services? That doesn’t sound like trading in the usual sense where it is a reciprocal exchange of goods and services.
Trade can involve anything that someone “owns”, which includes both their labor and their property, and government welfare. Retired people are generally characterized by trading their property and government welfare for goods and services, rather than primarily trading their labor. This is the basic picture I was trying to present.
How many ‘different’ AI individuals do you expect there to be ?
I think the answer to this question depends on how we individuate AIs. I don’t think most AIs will be as cleanly separable from each other as humans are, as most (non-robotic) AIs will lack bodies, and will be able to share information with each other more easily than humans can. It’s a bit like asking how many “ant units” there are. There are many individual ants per colony, but each colony can be treated as a unit by itself. I suppose the real answer is that it depends on context and what you’re trying to figure out by asking the question.
A recently commonly heard viewpoint on the development of AI states that AI will be economically impactful but will not upend the dominancy of humans. Instead AI and humans will flourish together, trading and cooperating with one another. This view is particularly popular with a certain kind of libertarian economist: Tyler Cowen, Matthew Barnett, Robin Hanson.
They share the curious conviction that the probablity of AI-caused extinction p(Doom) is neglible. They base this with analogizing AI with previous technological transition of humanity, like the industrial revolution or the development of new communication mediums. A core assumption/argument is that AI will not disempower humanity because they will respect the existing legal system, apparently because they can gain from trades with humans.
I think this summarizes my view quite poorly on a number of points. For example, I think that:
-
AI is likely to be much more impactful than the development of new communication mediums. My default prediction is that AI will fundamentally increase the economic growth rate, rather than merely continuing the trend of the last few centuries.
-
Biological humans are very unlikely to remain dominant in the future, pretty much no matter how this is measured. Instead, I predict that artificial minds and humans who upgrade their cognition will likely capture the majority of future wealth, political influence, and social power, with non-upgraded biological humans becoming an increasingly small force in the world over time.
-
The legal system will likely evolve to cope with the challenges of incorporating and integrating non-human minds. This will likely involve a series of fundamental reforms, and will eventually look very different from the idea of “AIs will fit neatly into human social roles and obey human-controlled institutions indefinitely”.
A more accurate description of my view is that humans will become economically obsolete after AGI, but this obsolescence will happen peacefully, without a massive genocide of biological humans. In the scenario I find most likely, humans will have time to prepare and adapt to the changing world, allowing us to secure a comfortable retirement, and/or join the AIs via mind uploading. Trade between AIs and humans will likely persist even into our retirement, but this doesn’t mean that humans will own everything or control the whole legal system forever.
-
How could one control AI without access to the hardware/software? What would stop one with access to the hardware/software from controlling AI?
One would gain control by renting access to the model, i.e., the same way you can control what an instance of ChatGPT currently does. Here, I am referring to practical control over the actual behavior of the AI, when determining what the AI does, such as what tasks it performs, how it is fine-tuned, or what inputs are fed into the model.
This is not too dissimilar from the high level of practical control one can exercise over, for example, an AWS server that they rent. While Amazon may host these servers, and thereby have the final say over what happens to the computer in the case of a conflict, the company is nonetheless inherently dependent on customer revenue, implying that they cannot feasibly use all their servers privately for their own internal purposes. As a consequence of this practical constraint, Amazon rents these servers out to the public, and they do not substantially limit user control over AWS servers, providing for substantial discretion to end-users over what software is ultimately implemented.
In the future, these controls could also be determined by contracts and law, analogously to how one has control over their own bank account, despite the bank providing the service and hosting one’s account. Then, even in the case of a conflict, the entity that merely hosts an AI may not have practical control over what happens, as they may have legal obligations to their customers that they cannot breach without incurring enormous costs to themselves. The AIs themselves may resist such a breach as well.
In practice, I agree these distinctions may be hard to recognize. There may be a case in which we thought that control over AI was decentralized, but in fact, power over the AIs was more concentrated or unified than we believed, as a consequence of centralization over the development or the provision of AI services. Indeed, perhaps real control was always in the hands of the government all along, as they could always choose to pass a law to nationalize AI, and take control away from the companies.
Nonetheless, these cases seem adequately described as a mistake in our perception of who was “really in control” rather than an error in the framework I provided, which was mostly an attempt to offer careful distinctions, rather than to predict how the future will go.
If one actor—such as OpenAI—can feasibly get away with seizing practical control over all the AIs they host without incurring high costs to the continuity of their business through loss of customers, then this indeed may surprise someone who assumed that OpenAI was operating under different constraints. However, this scenario still fits nicely within the framework as I’ve provided, as it merely describes a case in which one was mistaken about the true degree of concentration along one axis, rather than one of my concepts intrinsically fitting reality poorly.
The problem is that it would reduce the incentive to develop property for large developers, since their tax bill would go up if they developed adjacent land.
Whether this is a problem depends on your perspective. Personally, I would prefer that we stop making it harder and more inconvenient to build housing and develop land in the United States. Housing scarcity is already a major issue, and I don’t think we should just keep piling up disincentives to develop land and build housing unless we are being adequately compensated in other ways by doing so.
The main selling point of the LVT is that it arguably acts similarly to a zero sum wealth transfer, in the sense of creating zero deadweight loss (in theory). This is an improvement on most taxes, which are closer to negative sum rather than zero sum. But if the LVT slows down land development even more than our current rate of development, and the only upside is that rich landowners have their wealth redistributed, then this doesn’t seem that great to me. I’d much rather we focus on alternative, positive sum policies.
(To be clear, I think it’s plausible that the LVT has other benefits that make up for this downside, but here I’m just explaining why I think your objection to my argument is weak. I am not saying that the LVT is definitely bad.)