cousin_it
Good post. But I thought about this a fair bit and I think I disagree with the main point.
Let’s say we talk about two AIs merging. Then the tuple of their expected utilities from the merge had better be on the Pareto frontier, no? Otherwise they’d just do a better merge that gets them onto the frontier. Which specific point on the frontier is a matter of bargaining, but the fact that they want to hit the frontier isn’t, it’s a win-win. And the merges that get them to the frontier are exactly those that output a EUM agent, maximizing some linear combination of their utilities. If the point they want to hit is in a flat region of the frontier, the merge will involve coinflips to choose which EUM agent to become; and if it’s curvy at that point, the merge will be deterministic. For realistic agents who have more complex preferences than just linearly caring about one cake, I expect the frontier will be curvy, so deterministic merge into a EUM agent will be the best choice.
“Apparatchik” in the USSR was some middle-aged Ivan Ivanovich who’d yell at you in his stuffy office for stepping out of line. His power came from the party apparatus. While the power of Western activists is the opposite: it comes from civil society, people freely associating with each other.
This rhetorical move, calling a Western thing by an obscure and poorly fitting Soviet name, is a favorite of Yarvin: “Let’s talk about Google, my friends, but let’s call it Gosplan for a moment. Humor me.” In general I’d advise people to stay away from his nonsense, it’s done enough harm already.
The objection I’m most interested in right now is the one about induced demand (that’s not the right term but let’s roll with it). Like, let’s say we build many cheap apartments in Manhattan. Then the first bidders for them will be rich people—from all over the world! - who would love to get a Manhattan apartment for a bargain price. The priced-out locals will stay just as priced out, shuffled to the back of the line, because there’s quite many rich people in the world who are willing to outbid them. Maybe if we build very many apartments, and not just in Manhattan but everywhere, the effect will eventually run out; but it’ll take very many indeed.
The obvious fix is to put a thumb on the scale somehow, for example sell these cheap apartments only as primary residences. But then we lose the theoretical beauty of “just build more”, and we really should figure out what mix of “just build more” and “put a thumb on the scale” is the most cost-efficient for achieving what we want. Maybe some thumb on the scale will even give us what we want without building more, since there’s a lot of empty housing and non-primary housing.
Maybe you’re pushing your proposal a bit much, but anyway as creative writing it’s interesting to think about such scenarios. I had a sketch for a weird utopia story where just before the singularity, time stretches out for humans because they’re being run at increasing clock speed, and the Earth’s surface also becomes much larger and growing. So humanity becomes this huge, fast-running civilization living inside an AI (I called it “Quetzalcoatl”, not sure why) and advising it how it should act in the external world.
My wife used to have a talking doll that said one phrase in a really annoying voice. Well, at some point the doll short-circuited or something, and started turning on at random times. In the middle of the night for example it would yell out its phrase and wake everyone up. So eventually my wife took the doll to the garbage dump. And on the way back she couldn’t stop thinking about the doll sitting there in the garbage, occasionally yelling out its phrase: “Let’s go home! I’m already hungry!” This isn’t creative writing btw, this actually happened.
The thread about Tolkien reminded me of Andrew Hussie’s writing process. Start by writing cool scenes, including any elements you like. A talking tree? Okay. Then worry about connecting it with the story. The talking tree comes from an ancient forest and so on. And if you’re good, the finished story will feel like it always needed a talking tree.
I’d be really interested in a similar breakdown of JK Rowling’s writing process, because she’s another author with a limitless “toybox”.
I think something like the Culture, with aligned superintelligent “ships” keeping humans as basically pets, wouldn’t be too bad. The ships would try to have thriving human societies, but that doesn’t mean granting all wishes—you don’t grant all wishes of your cat after all. Also it would be nice if there was an option to increase intelligence, conditioned on increasing alignment at the same time, so you’d be able to move up the spectrum from human to ship.
Maybe tangential, but this reminded me of a fun fact about Hong Kong’s metro: it’s funded by land value. They put a station and get some land development rights near it. Well, building the station obviously makes land around it more valuable. So they end up putting stations where they’d be most useful, and fares can be cheap because the metro company makes plenty of money from land. So the end result is cheap, well-planned public transport which is profitable and doesn’t take government money.
Not to pick on you specifically, but just as a general comment, I’m getting a bit worried about the rationalist book review pipeline. It seems it usually goes like this: someone writes a book with an interesting idea → a rationalist (like Scott) writes a review of it, maybe not knowing much about the topic but being intrigued by the idea → lots of other rationalists get the idea cached in their minds. So maybe it’d be better if book reviews were written by people who know a lot about the topic, and can evaluate the book in context.
Like, a while ago someone on LW asked people to recommend textbooks on various topics, but you couldn’t recommend a textbook if it was the only one you’d read on the topic, you had to read at least two and then recommend one. That seems on the right track to me, and requiring more knowledge of the topic would be better still.
I think I can destroy this philosophy in two kicks.
Kick 1: pleasure is not one-dimensional. There are different parts of your brain that experience different pleasures, with no built-in way to compare between them.
When you retreat from kick 1 by saying “my decision-making provides a way to compare, the better pleasure is the one I’ll choose when asked”, here comes kick 2: your decision-making won’t work for that. There are compulsive behaviors that people want to do but don’t get much pleasure from them. And in every decision there’s a possible component of that, however small.
You could say “I’ll compare decisions based on how much pleasure they bring, excluding compulsiveness”, but you can’t do that due to kick 1 again. So the philosophy just collapses.
Good point. But I think the real game changer will be self-modification tech, not longevity tech. In that case we won’t have a “slow adaptation” problem, but we’ll have a “fast adaptation in weird directions” problem which is probably worse.
In Copenhagen every street has wide sidewalks and bike lanes in both directions, and there’s lots of public transport too. It’s good.
I don’t understand Eliezer’s explanation. Imagine Alice is hard-working and Bob is lazy. Then Alice can make goods and sell them to Bob. Half the money she’ll spend on having fun, the other half she’ll save. In this situation she’s rich and has a trade surplus, but the other parts of the explanation—different productivity between different parts of Alice (?) and inability to judge her own work fairly (?) - don’t seem to be present.
No. Committing a crime inflicts damage. But interacting with a person who committed a crime in the past doesn’t inflict any damage on you.
Because the smaller measure should (on my hypothesis) be enough to prevent crime, and inflicting more damage than necessary for that is evil.
Because otherwise everyone will gleefully discriminate against them in every way they possibly can.
I think the US has too much punishment as it is, with very high incarceration rate and prison conditions sometimes approaching torture (prison rape, supermax isolation).
I’d rather give serial criminals some kind of surveillance collars that would detect reoffending and notify the police. I think a lot of such people can be “cured” by high certainty of being caught, not by severity of punishment. There’d need to be laws to prevent discrimination against people with collars, though.
Yeah, I stumbled on this idea a long time ago as well. I never drink sugary drinks, my laptop is permanently in grayscale mode and so on. And it doesn’t feel like missing out on fun; on the contrary, it allows me to not miss out. When I “mute” some big, addictive, one-dimensional thing, I start noticing all the smaller things that were being drowned out by it. Like, as you say, noticing the deliciousness of baked potatoes when you’re not eating sugar every day, or noticing all the colors in my home and neighborhood when my screen is on grayscale.
I suppose the superassistants could form coalitions and end up as a kind of “society” without too much aggression. But this all seems moot, because superassistants will anyway get outcompeted by AIs that focus on growth. That’s the real danger.
I agree feedback is a big part of it. For example, the times in my life when I’ve been most motivated to play musical instruments were when I had regular opportunities to play in front of people. Whenever that disappeared, the interest went away too.
But also I think some of it is sticky, or due to personality factors. We could even say it’s not about willpower at all, but about value differences. Some people are just more okay with homeostasis, staying at a certain level (which can be lower or higher for different people) and using only as much effort as needed for that. While others keep climbing and applying effort without ever reaching a level that lets them relax. Many billionaires seem to be of that second type. I’m more of the first type, with many of my active periods being prompted by external changes, threats to homeostasis. It’s clear that type 2 achieves more than type 1, but it’s not clear which type is happier and whether one should want to switch types.