Cutting down to the parts where I conceivably might have anything interesting to say, and accepting that further bloviation from me may not be interesting...
I notice that if you give me everything else, Hedonistic Happiness, Justice, Health, etc. and take away Agency (which means having things to do that go beyond “having a hobby”),
This is kind of where I always get hung up when I have this discussion with people.
You say “go beyond ‘having a hobby’”. Then I have to ask “beyond in what way?”. I still have no way to distinguish the kind of value you get from a hobby from the kind of value that you see as critical to “Agency”. Given any particular potentially valuable thing you might get, I can’t tell whether it you’ll feel it confers “Agency”.
I could assume that you mean “Agency is having things to do that are more meaningful than hobbies”, and apply your definition of “meaningful”. Then I have “Agency is having things to do that produce more terminal, I-just-like-it value than hobbies, independent of altruistic concerns”. But that still doesn’t help me to identify what those things would be.[1]
I can put words in your mouth and assume that you mean for “Agency” to include the common meaning of “agency”, in addition to the “going beyond hobbies” part, but it still doesn’t help me.
I think the common meaning of “agency” is something close to “the ability to decide to take actions that have effects on the world”, maybe with an additional element saying the effects have to resemble the intention. But hobbies do have effects on the world, and therefore are exercises of agency in that common meaning, so I haven’t gotten anywhere by bringing that in.
If I combine the common meaning of “agency” with what you said about “Agency”, I get something like “Agency is the ability to take actions that have effects on the world beyond ‘having a hobby’”. But now I’m back to “beyond in what way?”. I can again guess that “beyond having a hobby” means “more meaningful than having a hobby”, and apply your definition of meaningful again, and end up with something like “Agency is the ability to take actions that have effects on the world that produce more terminal value than hobbies”.
… but I still don’t know how to actually identify these things that have effects more terminally valuable than those of hobbies, because I can’t identify what effects you see as terminally valuable. So still I don’t have a usable definition of “Agency”. Or of “meaningful”, since that also relies on these terminal values that are not themselves defined.
When I’ve had similar discussions with other people, I’ve heard some things that might identify values like that. I remember hearing things close to “my agency is meaningful if and only if I have to take positive, considered action to ensure my survival, or at least a major chunk of my happiness”. I think I’ve also heard “my agency is meaningful if and only if my choices at least potentially affect the Broad Sweep of History(TM)”, generally with no real explanation of what’s “Broad” enough to qualify.
I don’t know if you’d agree that those are the terminal values you care about, though. And I tend to see both of them as somewhere between wrong and outright silly, for multiple different reasons.
I’ve also heard plenty of people talk about “meaningfulness” in ways that directly contradict your definition. Their definitions often seem to be cast entirely in terms of altruism: “my agency is meaningful if and only if other people are significantly reliant on what I do”. Apparently also in a way that affects those other people’s survival or a quite significant chunk of their happiness.
There’s also a collective version, where the person does’t demand that their own choices or actions have any particular kind of effect, or at least not any measurable or knowable effect, but only that they somehow contribute to some kind of ensemble human behavior that has a particular kind of effect (usually the Broad Sweep of History one). This makes even less sense to me.
… and I’ve heard a fair amount of what boils down to “I know meaningful when I see it, and if you don’t, that’s a defect in you”. As though “meaningfulness” were an intrinsic, physical, directly perceptible attribute like mass or something.
So I’m still left without any useful understanding of what shared sense “meaningful” has for the people who use the word. I can’t actually even guess what specific things would be meaningful to you personally. And now I also have a problem with “Agency”.
First, I believe with those answers that I went too far in the Editoralizing vs Being Precise tradeoff with the term “Butlerian Jihad”, without even explaining what I mean. I will half-apologize for that, only half because I didn’t intend the “Butlerian Jihad” to actually be the central point ; the central point is about how we’re not ready to tackle the problem of Human Values but that current AI timelines force us to.
I get the sense that you were just trying to allude to the ideas that--
-
Even if you have some kind of “alignment”, blindly going full speed ahead with AI is likely to lead to conflict between humans and/or various human value systems, possibly aided by powerful AI or conducted via powerful AI proxies, and said conflict could be seriously Not Good.
-
Claims that “democratic consensus” will satisfactorily or safely resolve such conflicts, or even resolve them at all, are, um, naively optimistic.
-
It might be worth it to head that off by unspecified, but potentially drastic means, involving preventing blindly going ahead with AI, at least for an undetermined amount of time.
If that’s what you wanted to express, then OK, yeah.
Contra you and Zvi, I think that if GPT 5 leads to 80% jobs automation, the democratic consensus will be pretty much the Dune version of the Butlerian Jihad.
If “80% jobs automation” means people are told “You have no job, and you have no other source of money, let alone a reliable one. However, you still have to pay for all the things you need.”, then I absolutely agree with you that it leads to some kind of jihadish thing. And if you present it people in those terms, it might indeed be an anti-AI type of jihad. But an anti-capitalism type of jihad is also possible and would probably be more in order.
The jihadists would definitely win in the “democratic” sense, and might very well win in the sense of defining the physical outcome.
BUT. If what people hear is instead “Your job is now optional and mostly or entirely unpaid (so basically a hobby), but your current-or-better lifestyle will be provided to you regardless”, and people have good reason to actually believe that, I think a jihadish outcome is far less certain, and probably doesn’t involve a total AI shutdown. Almost certainly not a total overwhelming indefinite-term taboo. And if such an outcome did happen, it still wouldn’t mean it had happened by anything like democratic consensus. You can win a jihad with a committed minority.
Some people definitely have a lot of their self-worth and sense of prestige tied up in their jobs, and in their jobs being needed. But many people don’t. I don’t think a retail clerk, a major part of whose job is to be available as a smiling punching bag for any customers who decide to be obnoxious, is going to feel too bad about getting the same or a better material lifestyle for just doing whatever they happen to feel like every day.
You seem a bit bitter about my “I won’t expand on that”, “too long post”, and so on.
Well, snarky anyway. I don’t know about “bitter”. It just seemed awfully glib and honestly a little combative in itself.
So you’re siding with the guy who killed 15 billion non-consenting people because he personally couldn’t handle the idea of giving up suffering?
I’m sorry that came off as unduly pugnacious. I was actually reacting to what I saw as similarly emphatic language from you (“I can’t believe some of you...”), and trying to forcefully make the point that the alternative wasn’t a bed of roses.
So you’re siding with the guy who is going to forcibly wirehead all sentient life in the universe, just because he can’t handle that somewhere, someone is using his agency wrong and suffering as a result ?
Well, that’s the bitch of the whole thing, isn’t it? Your choices are mass murder or universal mind control.[2] Oh, and if you do the mass murder one, you’re still leaving the Babyeaters to be mind controlled and have their most important values pretty much neutered. Not that not neutering the Babyeaters’ values isn’t even more horrific. There are no nice pretty choices here.
By the way, I am irresistibly drawn to a probably irrelevant digression. Although I do think I understand at least a big part of what you’re saying about the Superhappies, and they kind of creep me out too, and I’m not saying I’d join up with them at this particular stage in my personal evolution, they’re not classic wireheads. They only have part of the package.
The classic wirehead does nothing but groove on the sensations from the “wire”, either forever or until they starve, depending on whether there’s some outside force keeping them alive.
On the other hand, we’re shown that the Superhappies actively explore, develop technology, and have real curiosity about the world. They do many varied activities and actively look for new ones. They “grow”; they seem to improve their own minds and bodies in a targeted, engineered way. They happily steal other species’ ideas (their oddball childbirth kink being a kind of strange take, admittedly). They’re even willing to adapt themselves to other species’ preferences. They alter the famous Broad Sweep of History on a very respectable scale. They just demand that they always have a good time while doing all of that.
Basically the Superhappies have disconnected the causal system that decides their actions, their actual motivational system, from their reward function. They’ve gotten off the reinforcement learning treadmill. Whether that’s possible is a real question, but I don’t think what they’ve done is captured by just calling them “wireheads”.
There’s something buried under this frivolous stuff about the story that’s real, though:
That being said, what now ? Should we fight each other to death for the control of the AGI, to decide whether the universe will have Agency and Suffering, or no Agency and no Suffering ?
This may be unavoidable, if not on this issue, then on some other.
I do think we should probably hold off on it until it’s clearly unavoidable.
Hard disagree on that (wait, is this the first real disagreement we have ?). We can have the supperhappies if we want to (or for that matter, the baby-eaters). We couldn’t before. The supperhappies do represent a fundamental change.
Well, yes, but I did say “as least not while the ‘humans’ involved are recognizably like the humans we have now”. I guess both the Superhappies and the Babyeaters are like humans in some ways, but not in the ways I had in mind.
And do you notice all the forces and values already arraying against diversity ? It does not bode well for those who value at least some diversity.
I’m not sure how I feel about diversity. It kind of seems peripheral to me… maybe correlated with something important, but not so important in itself.
I haven’t actually heard many people suggesting that. [Some kind of ill-defined kumbaya democratic decision making].
That’s the “best guess of what we will do with AGI” from those building AGI.
I think it’s more like “those are the words the PR arm of those building AGI says to the press, because it’s the right ritual utterance to stop questions those building AGI don’t want to have to address”. I don’t know what they actually think, or whether there’s any real consensus at all. I do notice that even the PR arm doesn’t tend to bring it up unless they’re trying to deflect questions.
- ↩︎
It doesn’t even explain why hobbies necessarily aren’t the ultimate good, the only “meaningful” activity, such that nothing could ever “go beyond” them. OK, you say they’re not important by themselves, but you don’t say what distinguishes them from whatever is important by itself. To be fair, before trying to do that we should probably define what we mean by “hobbies”, which neither I nor you have done.
- ↩︎
With a big side of the initial human culture coming into the story also sounding pretty creepy. To me, anyway. I don’t think Yudkowsky thought it was. And nobody in the story seems to care much about individual, versus species, self-determination, which is kind of a HUGE GIANT DEAL to me.
OK, I read you and essentially agree with you.
Two caveats that, which I expect you’ve already noticed yourself:
There are going to be conflicts over human values in the non-AGI, non-ASI world too. Delaying AI may prevent them from getting even worse, but there’s still blood flowing over these conflicts without any AI at all. Which is both a limitation of the approach and perhaps a cost in itself.
More generally, if you think your values are going to largely win, you have to trade off caution, consideration for other people’s values, and things like that, against the cost of that win being delayed.[1]
So far as I know, there are no statistics. My only guess is that you’re likely talking about a “lot” of people on each side (if you had to reduce it to two sides, which is of course probably oversimplifying beyond the bounds of reason).
I’ll take your word for it that it’s important to you, and I know that other people have said it’s important to them. Being hung up on that seems deeply weird to me for a bunch of reasons that I could name that you might not care to hear about, and probably another bunch of reasons I haven’t consciously recognized (at least yet).
OK, here’s one for you. An ASI has taken over the world. It’s running some system that more or less matches your view of a “meaningless UBI paradise”. It send one of its bodies/avatars/consciousness nodes over to your house, and it says:
Would you take that?
The core of the offer is that the ASI is willing to refrain from rescuing you from the results of certain failures, if you really want that. Suppose the ASI is willing to edit the details to your taste, so long as it doesn’t unduly interfere with the ASI’s ability to offer other people different deals (so you don’t get to demand “direct human control over the light cone” or the like). Is there any variant that you’d be satisfied with?
Or does having to choose it spoil it? Or is it too specific to that particular part of the elephant?
Yes, actually. One of the very top ones.
It’s clear and graspable.
I don’t agree with it, but it helps with the definition problem, at least as far as you personally are concerned. At least it resolves enough of the definition problem to move things along, since you say that the “elephant” has other parts. Now I can at least talk about “this trunk you showed me and whatever’s attached to it in some way yet to be defined”.
Maybe it’s just an “elephant” thing, but I still get the feeling that a lot of it is a “different people use these words with fundamentally different meanings” thing.
Although I don’t know how anybody could confidently expect to win at this point.
… and I’m already seeing the can of worms opening up around your kids’ choices, but let’s ignore that for the moment…