(edit 3: i’m not sure, but this text might be net-harmful to discourse)
i continue to feel so confused at what continuity led to some users of this forum asking questions like, “what effect will superintelligence have on the economy?” or otherwise expecting an economic ecosystem of superintelligences (e.g. 1[1], 2 (edit 2: I misinterpreted this question)).
it actually reminds me of this short story by davidad, in which one researcher on an alignment team has been offline for 3 months, and comes back to find the others on the team saying things like “[Coherent Extrapolated Volition?] Yeah, exactly! Our latest model is constantly talking about how coherent he is. And how coherent his volitions are!”, in that it’s something i thought this forum would have seen as ‘confused about the basics’ just a year ago, and i don’t yet understand what led to it.
(edit: i’m feeling conflicted about this shortform after seeing it upvoted this much. the above paragraph would be unsubstantive/bad discourse if read as an argument by analogy, which i’m worried it was (?). i was mainly trying to express confusion.)
I keep trying to explain to people that the archetype of intelligence is not Dustin Hoffman in Rain Man. It is a human being, period. It is squishy things that explode in a vacuum, leaving footprints on their moon. Within that gray wet lump is the power to search paths through the great web of causality, and find a road to the seemingly impossible—the power sometimes called creativity.
People—venture capitalists in particular—sometimes ask how, if the Machine Intelligence Research Institute successfully builds a true AI, the results will be commercialized. This is what we call a framing problem. [...]
a value-aligned superintelligence directly creates utopia. an “intent-aligned” or otherwise non-agentic truthful superintelligence, if that were to happen, is most usefully used to directly tell you how to create a value-aligned agentic superintelligence. if the thing in question cannot do one of these things it is not superintelligence, but something else.
Another issue is the Eternal September issue where LW membership has grown a ton due to the AI boom (see the LW site metrics in the recent fundraiser post), so as one might expect, most new users haven’t read the old stuff on the site. There are various ways in which the LW team tries to encourage them to read those, but nevertheless.
As far as I know, my post started the recent trend you complain about.
Several commenters on this thread (e.g. @Lucius Bushnaqhere and @MondSemmelhere) mention LessWrong’s growth and the resulting influx of uninformed new users as the likely cause. Any such new users may benefit from reading my recently-curated review of Planecrash, the bulk of which is about summarising Yudkowsky’s worldview.
i continue to feel so confused at what continuity led to some users of this forum asking questions like, “what effect will superintelligence have on the economy?” or otherwise expecting an economic ecosystem of superintelligences
If there’s decision-making about scarce resources, you will have an economy. Even superintelligence does not necessarily imply infinite abundance of everything, starting with the reason that our universe only has so many atoms. Multipolar outcomes seem plausible under continuous takeoff, which the consensus view in AI safety (as I understand it) sees as more likely than fast takeoff. I admit that there are strong reasons for thinking that the aggregate of a bunch of sufficiently smart things is agentic, but this isn’t directly relevant for the concerns about humans within the system in my post.
a value-aligned superintelligence directly creates utopia
[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.
Peter Thiel might call this “indefinite optimism”: delay all planning or visualisation because there’s some later point where it’s trusted things will all sort themselves out. Now, if you think that takeoff will definitely be extremely hard and the resulting superintelligence will effortlessly take over the world, then obviously it makes sense to focus on what that superintelligence will want to do. But what if takeoff lasts months or years or decades? (Note that there can be lots of change even within months if the stakes look extreme to powerful actors!) Aren’t you curious about what an aligned superintelligence will end up deciding about society and humans? Are you so sure about the transition period being so short and the superintelligence being so unitary and multipolar outcomes being so unlikely that we’ll never have to worry about problems downstream of the incentive issues and competitive pressures that I discuss (which Beren recently had an excellent post on)? Are you so sure that there is not a single interesting, a priori deducible fact about the superintelligent economy beyond “a singleton is in charge and everything is utopia”?
The default outcome is an unaligned superintelligence singleton destroying the world and not caring about human concepts like property rights. Whereas an aligned superintelligence can create a far more utopian future than a human could come up with, and cares about capitalism and property rights only to the extent that that’s what it was designed to care about.
So I indeed don’t get your perspective. Why are humans still appearing as agents or decision-makers in your post-superintelligence scenario at all? If the superintelligence for some unlikely reason wants a human to stick around and to do something, then it doesn’t need to pay them. And if a superintelligence wants a resource, it can just take it, no need to pay for anything.
@L Rudolf L can talk on his own, but for me, a crux probably is I don’t expect either unaligned superintelligence singleton or a value aligned superintelligence creating utopia as the space of likely outcomes within the next few decades.
For the unaligned superintelligence point, my basic reasons is I now believe the alignment problem got significantly easier compared to 15 years ago, I’ve become more bullish on AI control working out since o3, and I’ve come to think instrumental convergence is probably correct for some AIs we build in practice, but that instrumental drives are more constrainable on the likely paths to AGI and ASI.
For the alignment point, a big reason for this is I now think a lot of what makes an AI aligned is primarily data, compared to inductive biases, and one of my biggest divergences with the LW community comes down to me thinking that inductive bias is way less necessary for alignment than people usually think, especially compared to 15 years ago.
For AI control, one update I’ve made for o3 is that I believe OpenAI managed to get the RL loop working in domains where outcomes are easily verifiable, but not in domains where verifying is hard, and programming/mathematics are such domains where verifying is easy, but the tie-in is that capabilities will be more spikey/narrow than you may think, and this matters since I believe narrow/tool AI has a relevant role to play in an intelligence explosion, so you can actually affect the outcome by building narrow capabilities AI for a few years, and the fact that AI capabilities are spikey in domains where we can easily verify outcomes is good for eliciting AI capabilities, which is a part of AI control.
For the singleton point, it’s probably because I believe takeoff is both slow and somewhat distributed enough such that multiple superintelligent AIs can arise.
For the value-aligned superintelligence creating a utopia for everyone, my basic reason for why I don’t really believe in this is because I believe value conflicts are effectively irresolvable due to moral subjectivism, which forces the utopia to be a utopia for some people, and I expect the set of people that are in an individual utopia to be small in practice (because value conflicts become more relevant for AIs that can create nation-states all by themselves.)
For why humans are decision makers, this is probably because AI is either controlled or certain companies have chosen to make AIs follow instruction-following drives, and that actually succeeding.
And why must alignment be binary? (aligned, or misaligned, where misaligned necessarily means it destroys the world and does not care about property rights)
Why can you not have an a superintelligence that is only misaligned when it comes to issues of wealth distribution?
I guess we could in theory fail and only achieve partial alignment, but that seems like a weird scenario to imagine. Like shooting for a 1 in big_number target (= an aligned mind design in the space of all potential mind designs) and then only grazing it. How would that happen in practice?
And what does it even mean for a superintelligence to be “only misaligned when it comes to issues of wealth distribution”? Can’t you then just ask your pretty-much-perfectly-aligned entity to align itself on that remaining question?
I guess we could in theory fail and only achieve partial alignment, but that seems like a weird scenario to imagine. Like shooting for a 1 in big_number target (= an aligned mind design in the space of all potential mind designs) and then only grazing it. How would that happen in practice?
Are you saying that the 1 aligned mind design in the space of all potential mind designs is an easier target than the subspace composed of mind designs that does not destroy the world? If so, why? is it a bigger target? is it more stable?
Can’t you then just ask your pretty-much-perfectly-aligned entity to align itself on that remaining question?
No, because the you who can ask (the persons in power) is themselves misaligned with the 1 alignment target that perfectly captures all our preferences.
Are you so sure that there is not a single interesting, a priori deducible fact about the superintelligent economy beyond “a singleton is in charge and everything is utopia”?
End points are easier to infer than trajectories, so sure, I think there’s some reasonable guesses you can try to make about how the world might look after aligned superintelligence, should we get it somehow.
For example, I think it’s a decent bet that basically all minds would exist solely as uploads almost all of the time, because living directly in physical reality is astronomically wasteful and incredibly inconvenient. Turning on a physical lamp every time you want things to be brighter means wiggling about vast numbers of particles and wasting an ungodly amount of negentropy just for the sake of the teeny tiny number of bits about these vast numbers of particles that actually make it to your eyeballs, and the even smaller number of bits that actually end up influencing your mind state and making any difference to your perception of the world. All of the particles[1] in the lamp in my bedroom, the air its light shines through, and the walls it bounces off, could be so much more useful arranged in an ordered dance of logic gates where every single movement and spin flip is actually doing something of value. If we’re not being so incredibly wasteful about it, maybe we can run whole civilisations for aeons on the energy and negentropy that currently make up my bedroom. What we’re doing right now is like building an abacus out of supercomputers. I can’t imagine any mature civilisation would stick with this.
It’s not that I refuse to speculate about how a world post aligned superintelligence might look. I just didn’t think that your guess was very plausible. I don’t think pre-existing property rights or state structures would matter very much in such a world, even if we don’t get what is effectively a singleton, which I doubt. If a group of superintelligent AGIs is effectively much more powerful and productive than the entire pre-existing economy, your legal share of that pre-existing economy is not a very relevant factor in your ability to steer the future and get what you want. The same goes for pre-existing military or legal power.
Something like a crux here is I believe the trajectories non-trivially matter for which end-points we get, and I don’t think it’s like entropy where we can easily determine the end-point without considering the intermediate trajectory, because I do genuinely think some path-dependentness is present in history, which is why even if I were way more charitable towards communism I don’t think this was ever defensible:
[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.
The incentive problem still remains, such that it’s more effective to use the price system than to use a command economy to deal with incentive issues:
Related to this, perhaps the outer loss of the markets isn’t nearly as dispensable as a lot of people on LW believe, and contact with reality is a necessary part of all future AIs.
A potentially large crux is I don’t really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied, even charitably assuming a restriction on the utopia concept to be relative to someone else’s values.
The incentive problem still remains, such that it’s more effective to use the price system than to use a command economy to deal with incentive issues:
going by the linked tweet, does “incentive problem” mean “needing to incentivize individuals to share information about their preferences in some way, which is currently done through their economic behavior, in order for their preferences to be fulfilled”? and contrasted with a “command economy”, where everything is planned out long in advance, and possibly on less information about the preferences of individual moral patients?
if so, those sound like abstractions which were relevant to the world so far, but can you not imagine any better way a superintelligence could elicit this information? it does not need to use prices or trade. some examples:
it could have many copies of itself talk to them
it could let beings enter whatever they want into a computer in real time, or really let beings convey their preferences in whatever medium they prefer, and fulfill them[1]
it could mind-scan those who are okay with this.
(these are just examples selected for clarity; i personally would expect something more complex and less thing-oriented, around moral patients who are okay with/desire it, where superintelligence imbues itself as computation throughout the lowest level of physics upon which this is possible, and so it is as if physics itself is contextually aware and benevolent)
(i think these also sufficiently address your point 2, about SI needing ‘contact with reality’)
there is also a second (but non-cruxy) assumption here, that preference information would need to be dispersed across some production ecosystem, which would not be true given general-purpose superintelligent nanofactories. this though is not a crux as long as whatever is required for production can fit on, e.g., a planet (which the information derived in, e.g., one of those listed ways, can be communicated across at light-speed, as we partially do now).
A potentially large crux is I don’t really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied
i interpret this to mean “some entities’ values will want to use as much matter as they can for things, so not all values can be unboundedly fulfilled”. this is true and not a crux. if a moral patient who wants to make unboundedly much of something actually making unboundedly much of it would be less good than other ways the world could be, then an (altruistically-)aligned agent would choose one of the other ways.
superintelligence is context-aware in this way, it is not {a rigid system which fails to outliers it doesn’t expect (e.g.: “tries to create utopia, but instead gives all the lightcone to whichever maximizer requests it all first”), and so which needs a somewhat less rigid but not-superintelligent system (an economy) to avoid this}. i suspect this (superintelligence being context-aware) is effectively the crux here.
i interpret this to mean “some entities’ values will want to use as much matter as they can for things, so not all values can be unboundedly fulfilled”. this is true and not a crux. if a moral patient who wants to make unboundedly much of something actually making unboundedly much of it would be less good than other ways the world could be, then an (altruistically-)aligned agent would choose one of the other ways.
superintelligence is context-aware in this way, it is not {a rigid system which fails to outliers it doesn’t expect (e.g.: “tries to create utopia, but instead gives all the lightcone to whichever maximizer requests it all first”), and so which needs a somewhat less rigid but not-superintelligent system (an economy) to avoid this}. i suspect this (superintelligence being context-aware) is effectively the crux here.
The other issue is value conflicts, which I expect to be mostly irresolvable in a satisfying way by default due to moral subjectivism combined with me believing that lots of value conflicts today are mostly suppressed because people can’t make their own nation-states, but with AI, they can, and superintelligence makes the problem worse.
My guess is that it’s just an effect of field growth. A lot of people coming in now weren’t around when the consensus formed and don’t agree with it or don’t even know much about it.
Also, the consensus wasn’t exactly uncontroversial on LW even way back in the day. Hanson’s Ems inhabit a somewhat more recognisable world and economy that doesn’t have superintelligence in it, and lots of skeptics used to be skeptical in the sense of thinking all of this AI stuff was way too speculative and wouldn’t happen for hundreds of years if ever, so they made critiques of that form or just didn’t engage in AI discussions at all. LW wasn’t anywhere near this AI-centric when I started reading it around 2010.
My question specifically asks about the transition to ASI, which, while I think it’s really hard to predict, seems likely to take years, during which time we have intelligences just a bit above human level, before they’re truly world-changingly superintelligent. I understand this isn’t everyone’s model, and it’s not necessarily mine, but I think it is plausible.
Asking “how could someone ask such a dumb question?” is a great way to ensure they leave the community. (Maybe you think that’s a good thing?)
I’m fine. Don’t worry to much about this. It just made me think, what am I doing here? For someone to single out my question and say “it’s dumb to even ask such a thing” (and the community apparently agrees)… I just think I’ll be better off not spending time here.
I’d guess that most just skimmed what was visible from the hoverover, while under the impression it was what my text said. The engagement on your post itself is probably more representative.
I guess part of the issue is that in any discussion, people don’t use the same terms in the same way. Some people call present-day AI capabilities by terms like “superintelligent” in a specific domain. Which is not how I understand the term, but I understand where the idea to call it that comes from. But of course such mismatched definitions make discussions really hard. Seeing stuff like that makes it very understandable why Yudkowsky wrote the LW Sequences...
Anyway, here is an example of a recent shortform post which grapples with the same issue that vague terms are confusing.
I feel like this is a bit incorrect. There are imaginable things that are smarter than humans at some tasks, smart as average humans at others, thus overall superhuman, yet controllable and therefore possible to integrate in an economy without immediately exploding into an utopian (or dystopian) singularity. The question is whether we are liable to build such things before we build the exploding singularity kind, or if the latter is in some sense easier to build and thus stumble upon first. Most AI optimists think these limited and controllable intelligences are the default natural outcome of our current trajectory and thus expect mere boosts in productivity.
There are imaginable things that are smarter than humans at some tasks, smart as average humans at others, thus overall superhuman, yet controllable and therefore possible to integrate in an economy
sure, e.g. i think (<- i may be wrong about what the average human can do) that GPT-4 meets this definition (far superhuman at predicting author characteristics, above-average-human at most other abstract things). that’s a totally different meaning.
Most AI optimists think these limited and controllable intelligences are the default natural outcome of our current trajectory and thus expect mere boosts in productivity.
do you mean they believe superintelligence (the singularity-creating kind) is impossible, and so don’t also expect it to come after? it’s not sufficient for less capable AIs to defaultly come before superintelligence.
I think some believe it’s downright impossible and others that we’ll just never create it because we have no use for something so smart it overrides our orders and wishes. That at most we’ll make a sort of magical genie still bound by us expressing our wishes.
(edit 3: i’m not sure, but this text might be net-harmful to discourse)
i continue to feel so confused at what continuity led to some users of this forum asking questions like, “what effect will superintelligence have on the economy?” or otherwise expecting an economic ecosystem of superintelligences (e.g. 1[1],
2(edit 2: I misinterpreted this question)).it actually reminds me of this short story by davidad, in which one researcher on an alignment team has been offline for 3 months, and comes back to find the others on the team saying things like “[Coherent Extrapolated Volition?] Yeah, exactly! Our latest model is constantly talking about how coherent he is. And how coherent his volitions are!”, in that it’s something i thought this forum would have seen as ‘confused about the basics’ just a year ago, and i don’t yet understand what led to it.
(edit: i’m feeling conflicted about this shortform after seeing it upvoted this much. the above paragraph would be unsubstantive/bad discourse if read as an argument by analogy, which i’m worried it was (?). i was mainly trying to express confusion.)
from the power of intelligence (actually, i want to quote the entire post, it’s short):
a value-aligned superintelligence directly creates utopia. an “intent-aligned” or otherwise non-agentic truthful superintelligence, if that were to happen, is most usefully used to directly tell you how to create a value-aligned agentic superintelligence. if the thing in question cannot do one of these things it is not superintelligence, but something else.
comment thread between me and the post’s author
People are confused about the basics because the basics are insufficiently justified.
Another issue is the Eternal September issue where LW membership has grown a ton due to the AI boom (see the LW site metrics in the recent fundraiser post), so as one might expect, most new users haven’t read the old stuff on the site. There are various ways in which the LW team tries to encourage them to read those, but nevertheless.
As far as I know, my post started the recent trend you complain about.
Several commenters on this thread (e.g. @Lucius Bushnaq here and @MondSemmel here) mention LessWrong’s growth and the resulting influx of uninformed new users as the likely cause. Any such new users may benefit from reading my recently-curated review of Planecrash, the bulk of which is about summarising Yudkowsky’s worldview.
If there’s decision-making about scarce resources, you will have an economy. Even superintelligence does not necessarily imply infinite abundance of everything, starting with the reason that our universe only has so many atoms. Multipolar outcomes seem plausible under continuous takeoff, which the consensus view in AI safety (as I understand it) sees as more likely than fast takeoff. I admit that there are strong reasons for thinking that the aggregate of a bunch of sufficiently smart things is agentic, but this isn’t directly relevant for the concerns about humans within the system in my post.
In his review of Peter Singer’s commentary on Marx, Scott Alexander writes:
Peter Thiel might call this “indefinite optimism”: delay all planning or visualisation because there’s some later point where it’s trusted things will all sort themselves out. Now, if you think that takeoff will definitely be extremely hard and the resulting superintelligence will effortlessly take over the world, then obviously it makes sense to focus on what that superintelligence will want to do. But what if takeoff lasts months or years or decades? (Note that there can be lots of change even within months if the stakes look extreme to powerful actors!) Aren’t you curious about what an aligned superintelligence will end up deciding about society and humans? Are you so sure about the transition period being so short and the superintelligence being so unitary and multipolar outcomes being so unlikely that we’ll never have to worry about problems downstream of the incentive issues and competitive pressures that I discuss (which Beren recently had an excellent post on)? Are you so sure that there is not a single interesting, a priori deducible fact about the superintelligent economy beyond “a singleton is in charge and everything is utopia”?
The default outcome is an unaligned superintelligence singleton destroying the world and not caring about human concepts like property rights. Whereas an aligned superintelligence can create a far more utopian future than a human could come up with, and cares about capitalism and property rights only to the extent that that’s what it was designed to care about.
So I indeed don’t get your perspective. Why are humans still appearing as agents or decision-makers in your post-superintelligence scenario at all? If the superintelligence for some unlikely reason wants a human to stick around and to do something, then it doesn’t need to pay them. And if a superintelligence wants a resource, it can just take it, no need to pay for anything.
@L Rudolf L can talk on his own, but for me, a crux probably is I don’t expect either unaligned superintelligence singleton or a value aligned superintelligence creating utopia as the space of likely outcomes within the next few decades.
For the unaligned superintelligence point, my basic reasons is I now believe the alignment problem got significantly easier compared to 15 years ago, I’ve become more bullish on AI control working out since o3, and I’ve come to think instrumental convergence is probably correct for some AIs we build in practice, but that instrumental drives are more constrainable on the likely paths to AGI and ASI.
For the alignment point, a big reason for this is I now think a lot of what makes an AI aligned is primarily data, compared to inductive biases, and one of my biggest divergences with the LW community comes down to me thinking that inductive bias is way less necessary for alignment than people usually think, especially compared to 15 years ago.
For AI control, one update I’ve made for o3 is that I believe OpenAI managed to get the RL loop working in domains where outcomes are easily verifiable, but not in domains where verifying is hard, and programming/mathematics are such domains where verifying is easy, but the tie-in is that capabilities will be more spikey/narrow than you may think, and this matters since I believe narrow/tool AI has a relevant role to play in an intelligence explosion, so you can actually affect the outcome by building narrow capabilities AI for a few years, and the fact that AI capabilities are spikey in domains where we can easily verify outcomes is good for eliciting AI capabilities, which is a part of AI control.
For the singleton point, it’s probably because I believe takeoff is both slow and somewhat distributed enough such that multiple superintelligent AIs can arise.
For the value-aligned superintelligence creating a utopia for everyone, my basic reason for why I don’t really believe in this is because I believe value conflicts are effectively irresolvable due to moral subjectivism, which forces the utopia to be a utopia for some people, and I expect the set of people that are in an individual utopia to be small in practice (because value conflicts become more relevant for AIs that can create nation-states all by themselves.)
For why humans are decision makers, this is probably because AI is either controlled or certain companies have chosen to make AIs follow instruction-following drives, and that actually succeeding.
And why must alignment be binary? (aligned, or misaligned, where misaligned necessarily means it destroys the world and does not care about property rights)
Why can you not have an a superintelligence that is only misaligned when it comes to issues of wealth distribution?
Relatedly, are we sure that CEV is computable?
I guess we could in theory fail and only achieve partial alignment, but that seems like a weird scenario to imagine. Like shooting for a 1 in big_number target (= an aligned mind design in the space of all potential mind designs) and then only grazing it. How would that happen in practice?
And what does it even mean for a superintelligence to be “only misaligned when it comes to issues of wealth distribution”? Can’t you then just ask your pretty-much-perfectly-aligned entity to align itself on that remaining question?
Are you saying that the 1 aligned mind design in the space of all potential mind designs is an easier target than the subspace composed of mind designs that does not destroy the world? If so, why? is it a bigger target? is it more stable?
No, because the you who can ask (the persons in power) is themselves misaligned with the 1 alignment target that perfectly captures all our preferences.
End points are easier to infer than trajectories, so sure, I think there’s some reasonable guesses you can try to make about how the world might look after aligned superintelligence, should we get it somehow.
For example, I think it’s a decent bet that basically all minds would exist solely as uploads almost all of the time, because living directly in physical reality is astronomically wasteful and incredibly inconvenient. Turning on a physical lamp every time you want things to be brighter means wiggling about vast numbers of particles and wasting an ungodly amount of negentropy just for the sake of the teeny tiny number of bits about these vast numbers of particles that actually make it to your eyeballs, and the even smaller number of bits that actually end up influencing your mind state and making any difference to your perception of the world. All of the particles[1] in the lamp in my bedroom, the air its light shines through, and the walls it bounces off, could be so much more useful arranged in an ordered dance of logic gates where every single movement and spin flip is actually doing something of value. If we’re not being so incredibly wasteful about it, maybe we can run whole civilisations for aeons on the energy and negentropy that currently make up my bedroom. What we’re doing right now is like building an abacus out of supercomputers. I can’t imagine any mature civilisation would stick with this.
It’s not that I refuse to speculate about how a world post aligned superintelligence might look. I just didn’t think that your guess was very plausible. I don’t think pre-existing property rights or state structures would matter very much in such a world, even if we don’t get what is effectively a singleton, which I doubt. If a group of superintelligent AGIs is effectively much more powerful and productive than the entire pre-existing economy, your legal share of that pre-existing economy is not a very relevant factor in your ability to steer the future and get what you want. The same goes for pre-existing military or legal power.
Well, the conserved quantum numbers of my room, really.
Assuming that which end point you get to doesn’t depend on the intermediate trajectories at least.
Something like a crux here is I believe the trajectories non-trivially matter for which end-points we get, and I don’t think it’s like entropy where we can easily determine the end-point without considering the intermediate trajectory, because I do genuinely think some path-dependentness is present in history, which is why even if I were way more charitable towards communism I don’t think this was ever defensible:
The basic answer is the following:
The incentive problem still remains, such that it’s more effective to use the price system than to use a command economy to deal with incentive issues:
https://x.com/MatthewJBar/status/1871640396583030806
Related to this, perhaps the outer loss of the markets isn’t nearly as dispensable as a lot of people on LW believe, and contact with reality is a necessary part of all future AIs.
More here:
https://gwern.net/backstop
A potentially large crux is I don’t really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied, even charitably assuming a restriction on the utopia concept to be relative to someone else’s values.
going by the linked tweet, does “incentive problem” mean “needing to incentivize individuals to share information about their preferences in some way, which is currently done through their economic behavior, in order for their preferences to be fulfilled”? and contrasted with a “command economy”, where everything is planned out long in advance, and possibly on less information about the preferences of individual moral patients?
if so, those sound like abstractions which were relevant to the world so far, but can you not imagine any better way a superintelligence could elicit this information? it does not need to use prices or trade. some examples:
it could have many copies of itself talk to them
it could let beings enter whatever they want into a computer in real time, or really let beings convey their preferences in whatever medium they prefer, and fulfill them[1]
it could mind-scan those who are okay with this.
(these are just examples selected for clarity; i personally would expect something more complex and less thing-oriented, around moral patients who are okay with/desire it, where superintelligence imbues itself as computation throughout the lowest level of physics upon which this is possible, and so it is as if physics itself is contextually aware and benevolent)
(i think these also sufficiently address your point 2, about SI needing ‘contact with reality’)
there is also a second (but non-cruxy) assumption here, that preference information would need to be dispersed across some production ecosystem, which would not be true given general-purpose superintelligent nanofactories. this though is not a crux as long as whatever is required for production can fit on, e.g., a planet (which the information derived in, e.g., one of those listed ways, can be communicated across at light-speed, as we partially do now).
i interpret this to mean “some entities’ values will want to use as much matter as they can for things, so not all values can be unboundedly fulfilled”. this is true and not a crux. if a moral patient who wants to make unboundedly much of something actually making unboundedly much of it would be less good than other ways the world could be, then an (altruistically-)aligned agent would choose one of the other ways.
superintelligence is context-aware in this way, it is not {a rigid system which fails to outliers it doesn’t expect (e.g.: “tries to create utopia, but instead gives all the lightcone to whichever maximizer requests it all first”), and so which needs a somewhat less rigid but not-superintelligent system (an economy) to avoid this}. i suspect this (superintelligence being context-aware) is effectively the crux here.
(if morally acceptable, e.g. no creating hells)
The other issue is value conflicts, which I expect to be mostly irresolvable in a satisfying way by default due to moral subjectivism combined with me believing that lots of value conflicts today are mostly suppressed because people can’t make their own nation-states, but with AI, they can, and superintelligence makes the problem worse.
That’s why you can’t have utopia for everyone.
My guess is that it’s just an effect of field growth. A lot of people coming in now weren’t around when the consensus formed and don’t agree with it or don’t even know much about it.
Also, the consensus wasn’t exactly uncontroversial on LW even way back in the day. Hanson’s Ems inhabit a somewhat more recognisable world and economy that doesn’t have superintelligence in it, and lots of skeptics used to be skeptical in the sense of thinking all of this AI stuff was way too speculative and wouldn’t happen for hundreds of years if ever, so they made critiques of that form or just didn’t engage in AI discussions at all. LW wasn’t anywhere near this AI-centric when I started reading it around 2010.
My question specifically asks about the transition to ASI, which, while I think it’s really hard to predict, seems likely to take years, during which time we have intelligences just a bit above human level, before they’re truly world-changingly superintelligent. I understand this isn’t everyone’s model, and it’s not necessarily mine, but I think it is plausible.
Asking “how could someone ask such a dumb question?” is a great way to ensure they leave the community. (Maybe you think that’s a good thing?)
I don’t, sorry. (I’d encourage you not to leave just because of this, if it was just this. maybe LW mods can reactivate your account? @Habryka)
Yeah looks like I misinterpreted it. I agree that time period will be important.
I’ll try to be more careful.
Fwiw, I wasn’t expecting this shortform to get much engagement, but given it did it probably feels like public shaming, if I imagine what it’s like.
(Happy to reactivate your account, though I think you can also do it yourself)
I hope you’re okay btw
I’m fine. Don’t worry to much about this. It just made me think, what am I doing here? For someone to single out my question and say “it’s dumb to even ask such a thing” (and the community apparently agrees)… I just think I’ll be better off not spending time here.
I’d guess that most just skimmed what was visible from the hoverover, while under the impression it was what my text said. The engagement on your post itself is probably more representative.
Did not mean to do that.
I guess part of the issue is that in any discussion, people don’t use the same terms in the same way. Some people call present-day AI capabilities by terms like “superintelligent” in a specific domain. Which is not how I understand the term, but I understand where the idea to call it that comes from. But of course such mismatched definitions make discussions really hard. Seeing stuff like that makes it very understandable why Yudkowsky wrote the LW Sequences...
Anyway, here is an example of a recent shortform post which grapples with the same issue that vague terms are confusing.
I feel like this is a bit incorrect. There are imaginable things that are smarter than humans at some tasks, smart as average humans at others, thus overall superhuman, yet controllable and therefore possible to integrate in an economy without immediately exploding into an utopian (or dystopian) singularity. The question is whether we are liable to build such things before we build the exploding singularity kind, or if the latter is in some sense easier to build and thus stumble upon first. Most AI optimists think these limited and controllable intelligences are the default natural outcome of our current trajectory and thus expect mere boosts in productivity.
sure, e.g. i think (<- i may be wrong about what the average human can do) that GPT-4 meets this definition (far superhuman at predicting author characteristics, above-average-human at most other abstract things). that’s a totally different meaning.
do you mean they believe superintelligence (the singularity-creating kind) is impossible, and so don’t also expect it to come after? it’s not sufficient for less capable AIs to defaultly come before superintelligence.
I think some believe it’s downright impossible and others that we’ll just never create it because we have no use for something so smart it overrides our orders and wishes. That at most we’ll make a sort of magical genie still bound by us expressing our wishes.