I score this as “Good enough that I debated not bothering to correct anything.”
I think some corrections might be helpful though:
The internet is causing rapid memetic evolution…
While I think that’s true, that’s not really central to what I’m saying. I think these forces have been the main players for way, way longer than we’ve had an internet. The internet — like every other advance in communication — just increased evolutionary pressure at the memetic level by bringing more of these hypercreatures into contact with one another and with resources they could compete for.
These memes push people who host them (all of us, to be clear) towards behaviors which are not in the best interests of humanity, because Orthogonality Thesis
Yes. I’d just want to add that not all of them do. It’s just that the ones that tend to dominate tend to be unFriendly.
Two counterexamples:
Science. Not as an establishment, but as a kind of clarifying intelligence. This strikes me as a Friendly hypercreature. (The ossified practices of science, like “RCTs are the gold standard” and “Here’s the Scientific Method!”, tend to pull toward stupidity via Goodhart. A lot of LW is an attempt to reclaim the clarifying influence of this hypercreature’s intelligence.)
Jokes. These are sort of like innocuous memetic insects. As long as they don’t create problems for more powerful hypercreatures, they can undergo memetic evolution and spread. They aren’t particularly Friendly or unFriendly for the most part. Some of them add a little value via humor, although that’s not what they’re optimizing for. (The evolutionary pressure on jokes is “How effectively does hearing this joke cause the listener to faithfully repeat it?”). But if a joke were to somehow evolve into a more coherent behavior-controlling egregore, by default it’ll be an unFriendly one.
Before you can work effectively on AI stuff, you have to clear out the misaligned memes stuck in your head.
Almost. I think it’s more important that you have installed a system for noticing and weeding out these influences.
Like how John Vervaeke argues that the Buddha’s Eightfold Noble Path is a kind of virtual engine for creating relevant insight. The important part isn’t the insight but is instead the engine. Because the same processes that create insight also create delusion, so you need a systematic way of course-correcting.
This can get you the clarity/agency necessary, and make sure that (if successful) you actually produce AGI aligned with “you”, not some meme
No correction here. I just wanted to say, this is a delightfully clear way of saying what I meant.
…we shouldn’t try and just solve x-risk…
While I agree (both with the claim and with the fact that this is what I said), when I read you saying it I worry about an important nuance getting lost.
The emphasis here should be on “solve”, not “x-risk”. Solving xrisk is superhuman. So is xrisk itself for that matter. “God scale.”
However! Friendly hypercreatures need our minds in order to think. In order for a memetic strategy to result in solving AI risk, we need to understand the problem. We need to see its components clearly.
So I do think it helps to model xrisk. See its factors. See its Gears. See the landscape it’s embedded in.
Sort of like, a healthy marriage is more likely to emerge if both people make an effort to understand themselves, each other, and their dynamic within a context of togetherness and mutual care. But neither person is actually responsible for creating a healthy marriage. It sort of emerges organically from mutual open willingness plus compatibility.
…we should focus on rationality, cultivating our internal meme garden, and favoring memes which will push the world in the direction we want it to go
FWIW, this part sounds redundant to me. A “rationality” that is something like a magical completion of the Art would, as far as I can tell, consist almost entirely of consciously cultivating one’s internal memetic garden, which is nearly the same thing as favoring Friendly memes.
But after reading and replying to Scott’s comment, I’d adjust a little bit in the OP. For basically artistic reasons I mentioned “rationality for its own sake, period.” But I now think that’s distracting. What I’m actually in favor of is memetic literacy by whatever name. I think there’s an important art here whose absence causes people to focus on AI risk in unhelpful and often anti-helpful ways.
Also, on this part:
…which will push the world in the direction we want it to go
I want to emphasize that best as I can figure, we don’t have control over that. That’s more god-scale stuff. What each of us can do is notice what seems clarifying and kind to ourselves and to lean that way. I think there’s some delightful game theory that suggests that doing this supports Friendly hypercreatures.
I like this, thank you.
I score this as “Good enough that I debated not bothering to correct anything.”
I think some corrections might be helpful though:
While I think that’s true, that’s not really central to what I’m saying. I think these forces have been the main players for way, way longer than we’ve had an internet. The internet — like every other advance in communication — just increased evolutionary pressure at the memetic level by bringing more of these hypercreatures into contact with one another and with resources they could compete for.
Yes. I’d just want to add that not all of them do. It’s just that the ones that tend to dominate tend to be unFriendly.
Two counterexamples:
Science. Not as an establishment, but as a kind of clarifying intelligence. This strikes me as a Friendly hypercreature. (The ossified practices of science, like “RCTs are the gold standard” and “Here’s the Scientific Method!”, tend to pull toward stupidity via Goodhart. A lot of LW is an attempt to reclaim the clarifying influence of this hypercreature’s intelligence.)
Jokes. These are sort of like innocuous memetic insects. As long as they don’t create problems for more powerful hypercreatures, they can undergo memetic evolution and spread. They aren’t particularly Friendly or unFriendly for the most part. Some of them add a little value via humor, although that’s not what they’re optimizing for. (The evolutionary pressure on jokes is “How effectively does hearing this joke cause the listener to faithfully repeat it?”). But if a joke were to somehow evolve into a more coherent behavior-controlling egregore, by default it’ll be an unFriendly one.
Almost. I think it’s more important that you have installed a system for noticing and weeding out these influences.
Like how John Vervaeke argues that the Buddha’s Eightfold Noble Path is a kind of virtual engine for creating relevant insight. The important part isn’t the insight but is instead the engine. Because the same processes that create insight also create delusion, so you need a systematic way of course-correcting.
No correction here. I just wanted to say, this is a delightfully clear way of saying what I meant.
While I agree (both with the claim and with the fact that this is what I said), when I read you saying it I worry about an important nuance getting lost.
The emphasis here should be on “solve”, not “x-risk”. Solving xrisk is superhuman. So is xrisk itself for that matter. “God scale.”
However! Friendly hypercreatures need our minds in order to think. In order for a memetic strategy to result in solving AI risk, we need to understand the problem. We need to see its components clearly.
So I do think it helps to model xrisk. See its factors. See its Gears. See the landscape it’s embedded in.
Sort of like, a healthy marriage is more likely to emerge if both people make an effort to understand themselves, each other, and their dynamic within a context of togetherness and mutual care. But neither person is actually responsible for creating a healthy marriage. It sort of emerges organically from mutual open willingness plus compatibility.
FWIW, this part sounds redundant to me. A “rationality” that is something like a magical completion of the Art would, as far as I can tell, consist almost entirely of consciously cultivating one’s internal memetic garden, which is nearly the same thing as favoring Friendly memes.
But after reading and replying to Scott’s comment, I’d adjust a little bit in the OP. For basically artistic reasons I mentioned “rationality for its own sake, period.” But I now think that’s distracting. What I’m actually in favor of is memetic literacy by whatever name. I think there’s an important art here whose absence causes people to focus on AI risk in unhelpful and often anti-helpful ways.
Also, on this part:
I want to emphasize that best as I can figure, we don’t have control over that. That’s more god-scale stuff. What each of us can do is notice what seems clarifying and kind to ourselves and to lean that way. I think there’s some delightful game theory that suggests that doing this supports Friendly hypercreatures.
And if not, I think we’re just fucked.