Also, I am excited about people trying to follow paths to all of their long-term goals/flourishing, including their romantic and reproductive goals, and I am actively not excited about people deciding to shelve that because they think AI risk demands it.
I’m going to reply in a pedantic way, then say why I did so.
Pedantic response: Trying to solve AI risk is one way to increase the odds you’ll achieve goals like that. (E.g., if you want to support your kids going to college, you should consider putting aside money for them and keeping an eye out for ways to reduce the risk you and your kids die from rogue AI disassembling the inner solar system before they’re of college age.)
Explanation:
I make this point not to argue against finding love or starting a family, but to argue against a mindset that treats AGI and daily life as more or less two different magisteria.
I think there’s a pendulum-swing-y thing going on: people trying to counter burnout, freakouts, etc. heavily emphasize mundane, non-sci-fi-sounding reality, so as to push back a pendulum they see as having swung too far toward ‘my mental universe is entirely about Dyson swarms, and not at all about dental appointments’.
I find this mostly doesn’t work for me personally. I come away feeling like half the blog posts I read are disassociated from ordinary day-to-day life, and the other half are disassociated from ‘the portion of my life that will occur during and after the invention of this wild technology’, such that I end up wary of both kinds of post.
Your overall post is a whole lot better than most on this dimension. It doesn’t just treat ‘be psychologically happy and healthy this year’ as obviously lexicographically more important than preventing your and everyone else’s deaths in twenty-five years (or whatever). It gives specific arguments for why the happy-and-healthy stuff is super important even if you’re confident your personal CEV’s terminal values assign overwhelmingly more weight to x-risk stuff than to your happiness over the next fifty years. It acknowledges the “‘Emergencies’ as wake-up calls” thing.
But it still feels to me like it’s a post trying to push the pendulum in a particular direction, rather than trying to fully and openly embody the optimal-by-your-lights Balancing Point.
(Where de-prioritizing pendulum pushing indeed risks worsening the current problem we face, insofar as EAs today do over-extend more than they under-extend. Maybe I just selfishly prefer reading posts like that, even if they’re not optimal for the community.)
It still doesn’t feel to me like it’s fully speaking as though the two worlds as one world, or fully embracing that a lot of people probably need to hear something closer to the opposite advice. (‘You’re asking too little of yourself; there are great things you could achieve if you were more strategic and purposeful with your time, rather than just following the nearest hedonic gradient; you are not fragile, or a ticking time bomb that will surely flame out if you start doing qualitatively harder things; you can output good-by-maxipok-lights things without waiting to first resolve all your personal issues or get your life fully in order; the alignment problem may sound crazy and intense to you from a distance, and it still possibly be true that you could do good work without sacrificing a flourishing, well-rounded pre-AGI life, and perhaps even flourish more pre-AGI as a consequence.’)
That might just be because I’m not the target audience and this style works great for others, but I wanted to mention it.
I make this point not to argue against finding love or starting a family, but to argue against a mindset that treats AGI and daily life as more or less two different magisteria….
It still doesn’t feel to me like it’s fully speaking as though the two worlds as one world
The situation is tricky, IMO. There is, of course, at the end of the day only one world. If we want to have kids who can grow up to adulthood, and who can have progeny of their own, this will require that there be a piece of universe hospitable to human life where they can do that growing up.
At the same time:
a) IMO, there is a fair amount of “belief in belief” about AI safety and adjacent things. In particular, I think many people believe they ought to believe that various “safety” efforts help, without really anticipating-as-if this sort of thing can help.
(The argument for “be worried about the future” is IMO simpler, more obvious, and more likely to make it to the animal, that particular beliefs that particular strategies have much shot. I’m not sure how much this is or isn’t about AI; many of my Uber drivers seem weirdly worried about the future.)
a2) Also, IMO, a fair number of peoples’ beliefs (or “belief in beliefs”) about AI safety are partly downstream of others’ political goals, e.g. of others’ social incentives that those people believe in particular narratives about AI safety and about how working at place X can help with AI safety. This can accent the “belief in belief” thing.
b) Also, even where people have honest/deep/authentic verbal-level beliefs about a thing, it often doesn’t percolate all the way down into the animal. For example, a friend reports having interviewed a number of people about some sex details, and coming out believing that some people do and some people don’t have a visceral animal-level understanding that birth control prevents pregnancy, and reports furthermore that such animal-level beliefs often backpropagate to peoples’ desires or lack of desires for different kinds of sex. I believe my friend here, although this falls under “hard to justify personal beliefs.”
c) As I mentioned in the OP, I am worried that when a person “gives up on” their romantic and/or reproductive goals (or other goals that are as deeply felt and that close to the center of a person, noting that the details here vary by individual AFAICT), this can mess up their access to caring and consciousness in general (in Divia’s words, can risk “some super central sign error deep in their psychology”).
a-c, in combination, leave me nervous/skeptical about people saying that their plan is to pursue romance/children “after the Singularity,” especially if they’re already nearing the end of their biological window. And especially if it’s prompted by that being a standard social script in some local circle. I am worried that people may say this, intend it with some portion of themselves, but have their animal hear “I’m going to abandon these goals and live in belief-in-beliefs.”
I have a personal thing up for me about this one. Back in ~2016, I really wanted to try to have a kid, but thought that short timelines plus my own ability to contribute to AI safety efforts meant I probably shouldn’t. I dialoged with my system 1 using all the tools in the book. I consulted all the people near me who seemed like they might have insight into the relevant psychology. My system 1 / animal-level orientation, after dialog, seemed like it favored waiting, hoping for a kid on the other side of the singularity. I mostly passed other peoples’ sanity checks, both at the time and a few years later when I was like “hey, we were worried about this messing with my psyche, but it seems like it basically worked, right? what is your perception?” And even so, IMO, I landed in a weird no man’s land of a sort of bleached out depression and difficulty caring about anything after awhile, that was kinda downstream of this but very hard for me to directly perceive, but made it harder to really mean anything and easier to follow the motions of looking like I was trying.
The take-away I’m recommending from this is something like: “be careful about planning on paths toward your deepest, animal-level goals that your animal doesn’t buy. And note that it’s can be hard to robustly know what your animal is or isn’t buying. ” Also, while there’s only one magesterium, if people are animal-level and speech-level reasoning as though there’s several, that’s a real and confusing-to-me piece of context to how we’re living here.
That makes sense to me, and it updates me toward your view on the kid-having thing. (Which wasn’t the focus of my comment, but is a thing I was less convinced of before.) I feel sad about that having happened. :( And curious about whether I (or other people I know) are making a similar mistake.
(My personal state re kids is that it feels a bit odd/uncanny when I imagine myself having them, and I don’t currently viscerally feel like I’m giving something up by not reproducing. Though if I lived for centuries, I suspect I’d want kids eventually in the same way I’d want to have a lot of other cool experiences.)
I feel kinda confused about how “political” my AGI-beliefs are. The idea of dying to AGI feels very sensorily-real to me — I feel like my brain puts it in the same reference class as ‘dying from a gunshot wound’, which is something I worry about at least a little in my day-to-day life (even though I live in a pretty safe area by US megalopolis standards), have bad dreams about, semi-regularly idly imagine experiencing, etc. I don’t know how that relates to the “is this belief political?” question, or how to assess that.
Regardless, I like this:
I am worried that people may say this, intend it with some portion of themselves, but have their animal hear “I’m going to abandon these goals and live in belief-in-beliefs.”
I’d also assume by default that ‘the animal discounts more heavily than the philosopher does’ is a factor here...? And/or ‘the animal is better modeled on this particular question as an adaptation-executer rather than a utility-maximizer, such that there’s no trade you can make that will satisfy the animal if it involves trading away having-kids-before-age-45’?
It could be wise to have kids, for the sake of harmony between your parts/subgoals, even if you judge that having kids isn’t directly x-risk-useful and your animal seems to have a pretty visceral appreciation for x-risk—just because that is in fact what an important part of you wants/needs/expects/etc.
I want to have a dialog about what’s true, at the level of piece-by-piece reasoning and piece-by-piece causes. I appreciate that you Rob are trying to do this; “pedantry” as you put it is great, and seems to me to be a huge chunk of why LW is a better place to sort some things out than is most of the internet.
I’m a bit confused that you call it “pedantry”, and that you talk of my post as trying to push the pendulum in a particular way, and “people trying to counter burnout,” and whether this style of post “works” for others. The guess I’m forming, as I read your (Rob’s) comment and to a lesser extent the other comments, is that a bunch of people took my post as a general rallying cry against burnout, and felt it necessary to upvote my post, or to endorse my post, because they personally wish to take a stand against burnout. Does something like that seem right/wrong to anyone? (I want to know.)
I… don’t want that, although I may have done things in my post to encourage it anyhow, without consciously paying attention. But if we have rallying cries, we won’t have the kind of shared unfiltered reasoning that someone wanting truth can actually update on.
I’m in general pretty interested in strategies anyone has for having honest, gritty, mechanism-by-mechanism discussion near a Sacred Value. “Don’t burn people out” is arguably a Sacred Value, such that it’ll be hard to have open conversation near it in which all the pedantry is shared in all the directions. I’d love thoughts on how to do it anyhow.
I want to have a dialog about what’s true, at the level of piece-by-piece reasoning and piece-by-piece causes. I appreciate that you Rob are trying to do this; “pedantry” as you put it is great, and seems to me to be a huge chunk of why LW is a better place to sort some things out than is most of the internet.
Yay! I basically agree. The reason I called it “pedantry” was because I said it even though (a) I thought you already believed it (and were just speaking imprecisely / momentarily focusing on other things), (b) it’s an obvious observation that a lot of LWers already have cached, and (c) it felt tangential to the point you were making. So I wanted to flag it as a change of topic inspired by your word choice, rather than as ordinary engagement with the argument I took you to be making.
and that you talk of my post as trying to push the pendulum in a particular way, and “people trying to counter burnout,” and whether this style of post “works” for others.
I think I came to the post with a long-term narrative (which may have nothing to do with the post):
There are a bunch of (Berkeley-ish?) memes in the water related to radical self-acceptance, being kind to yourself, being very wary of working too hard, staying grounded in ordinary day-to-day life, being super skeptical and cautious around Things Claiming To Be Really Important and around moralizing, etc.
I think these are extremely important and valuable memes that LW would do well to explore, discuss, and absorb much more than it already has. I’ve found them personally extremely valuable, and a lot of my favorite blog posts to send to new EAs/rats make points like ‘be cautious around things that make big moralistic demands of you’, etc.
But I also think that these kinds of posts are often presented in ways that compete against the high value I (genuinely, thoughtfully) place on the long-term future, and the high probability I (genuinely, thoughtfully) place on AI killing me and my loved ones, as though I need to choose between the “chill grounded happy self-loving unworried” aesthetic or the “working really hard to try to solve x-risk” aesthetic.
This makes me very wary, especially insofar as it isn’t making an explicit argument against x-risk stuff, but is just sort of vaguely associating not-worrying-so-much-about-human-extinction with nice-sounding words like ‘healthy’, ‘grounded’, ‘relaxed’, etc. If these posts spent more time explicitly arguing for their preferred virtues and for why those virtues imply policy X versus policy Y, rather than relying on connotation and implicature to give their arguments force, my current objection would basically go away.
If more of the “self-acceptance, be kind to yourself, be vary wary of working too hard, etc.” posts were more explicit about making space for possibilities like ‘OK, but my best self really does care overwhelmingly more about x-risk stuff than everything else’ and/or ‘OK, but making huge life-changes to try to prevent human extinction really is the psychologically healthiest option for me’, I would feel less suspicious that some of these posts are doing the dance wrong, losing sight of the fact that both magisteria are real, are part of human life.
I may have been primed to interpret this post in those terms too much, because I perceived it to be a reaction to Eliezer’s recent doomy-sounding blog posts (and people worrying about AI more than usual recently because of that, plus ML news, plus various complicated social dynamics), trying to prevent the community from ‘going too far’ in certain directions.
I think the post is basically good and successful at achieving that goal, and I think it’s a very good goal. I expect to link to the OP post a lot in the coming months. But it sounds like I may be imposing context on the post that isn’t the way you were thinking about it while writing it.
I may have been primed to interpret this post in those terms too much, because I perceived it to be a reaction to Eliezer’s recent doomy-sounding blog posts (and people worrying about AI more than usual recently because of that, plus ML news, plus various complicated social dynamics), trying to prevent the community from ‘going too far’ in certain directions. … But it sounds like I may be imposing context on the post that isn’t the way you were thinking about it while writing it.
Oh, yeah, maybe. I was not consciously responding to that. I was consciously responding to a thing that’s been bothering me quite a bit about EA for ~5 or more years, which is that there’s not enough serious hobbies around here IMO, and also people often report losing the ability to enjoy hanging out with friends, especially friends who aren’t in these circles, and just enjoying one anothers’ company while doing nothing, e.g. at the beach with some beers on a Saturday. (Lots of people tell me they try allocating days to this, it isn’t about the time, it’s about an acquired inability to enter certain modes.)
Thanks for clarifying this though, that makes sense.
I have some other almost-written blog posts that’re also about trying to restore access to “hanging out with friends enjoying people” mode and “serious hobbies” mode, that I hope to maybe post in the next couple weeks.
Back in ~2008, I sat around with some others trying to figure out: if we’re successful in getting a lot of people involved in AI safety—what can we hope to see at different times? And now it’s 2022. In terms of “there’ll be a lot of dollars people are up for spending on safety”, we’re basically “hitting my highest 2008 hopes”. In terms of “there’ll be a lot of people who care”, we’re… less good than I was hoping for, but certainly better than I was expecting. “Hitting my 2008 ‘pretty good’ level.” In terms of “and those people who care will be broad and varied and trying their hands at making movies and doing varied kinds of science and engineering research and learning all about the world while keeping their eyes open for clues about the AI risk conundrum, and being ready to act when a hopeful possibility comes up” we’re doing less well compared to my 2008 hopes. I want to know why and how to unblock it.
In terms of “and those people who care will be broad and varied and trying their hands at making movies and doing varied kinds of science and engineering research and learning all about the world while keeping their eyes open for clues about the AI risk conundrum, and being ready to act when a hopeful possibility comes up” we’re doing less well compared to my 2008 hopes. I want to know why and how to unblock it.
I think to the extent that people are failing to be interesting in all the ways you’d hoped they would be, it’s because being interesting in those ways seems to them to have greater costs than benefits. If you want people to see the benefits of being interesting as outweighing the costs, you should make arguments to help them improve their causal models of the costs, and to improve their causal models of the benefits, and to compare the latter to the former. (E.g., what’s the causal pathway by which an hour of thinking about Egyptology or repairing motorcycles or writing fanfic ends up having, not just positive expected usefulness, but higher expected usefulness at the margin than an hour of thinking about AI risk?) But you haven’t seemed very interested in explicitly building out this kind of argument, and I don’t understand why that isn’t at the top of your list of strategies to try.
I think of this in terms of personal vs. civilization-scale value loci distinction. Personal-scale values, applying to individual modern human minds, speaking of those minds, might hold status quo anchoring sacred and dislike presence of excessive awareness of disruptive possible changes. While civilization-scale values, even as they are facilitated by individuals, do care about accurate understanding of reality regardless of what it says.
People shouldn’t move too far towards becoming decision theoretic agents, even if they could, other than for channeling civilization. The latter is currently a necessity (that’s very dangerous to neglect), but it’s not fundamentally a necessity. What people should move towards is a more complicated question with some different answer (which does probably include more clarity in thinking than is currently the norm or physiologically possible, but still). People are vessels of value, civilization is its custodian. These different roles call for different shapes of cognition.
In this model, it’s appropriate / morally-healthy / intrinsically-valuable for people to live more fictional lives (as they prefer) while civilization as a whole is awake, and both personal-scale values and civilization-scale values agree on this point.
But it still feels to me like it’s a post trying to push the pendulum in a particular direction, rather than trying to fully and openly embody the optimal-by-your-lights Balancing Point.
AFAICT, I am trying to fully and openly embody the way of reasoning that actually makes sense to me in this domain, which… isn’t really a “balancing point.” It’s more like the anarchist saying “the means are the ends.” Or it’s more like Szilard’s “ten commandments,” (which I highly recommend reading for anyone who hasn’t; they’re short). Or more like the quote from the novel The Dispossessed: “To reassert its validity and strength, he thought, one need only act, without fear of punishment and without hope of reward: act from the center of one’s soul. “
I don’t have the right concepts or articulation here. This is an example of the “hard to justify personal opinions” I warned about in my “epistemic status.” But IMO, thinking about tradeoffs and balancing points can be good when your map is good enough; at other times, it’s more like I want to try to hone in on priors, on deep patternness, on where reasoning is before it’s reasoning. This is where the power of leisure comes from, where the possibility of hobbies that end up giving you glimpses of new bits of the universe come from. And it’s a thing I’m trying to show here. Not a particular balance-point between depleting your long-term resources and “being nice to yourself” by eating chocolates and cartoons. (Neither of those help with getting to the tao, usually, AFAICT.)
We can be empirical about trying to see which actions, which mindsets, add to our and others’ long-term robust abilities.
In short-term crises for which you have decent-quality maps, balance-points and trading things off with local consequentialist reasoning makes sense to me. But not the rest of everywhere.
I agree many people underestimate their own capacities, and too seldom try hard or scary things. I think this is often many of the same people who burn themselves out.
Sorry this reply, and my other one, are somewhat incoherent. I’m having trouble mapping both where you’re coming from, and why/where I disagree.
Yeah, that makes sense to me. I’m complaining about a larger class of posts, so maybe this one isn’t really an example and I’m just pattern-matching. I do still wish there existed more posts that were very obviously examples of the ‘both-and’ things I was pointing at. (Both dentist appointments and Dyson spheres; both embrace slack and embrace maxipok; etc.)
It might be that if my thinking were clearer here, I’d be able to recognize more posts as doing ‘both-and’ even if they don’t explicitly dwell on it as much as I want.
I think I feel a similar mix of love and frustration for your comment as I read your comment expressing with the post.
Let me be a bit theoretical for a moment. It makes sense for me to think of utilities as a sum U=aUa+bUb where Ua is the utility of things after singularity/superintelligence/etc and Ub the utility for things before then (assuming both are scaled to have similar magnitudes so the relative importance is given by the scaling factors). There’s no arguing about the shape of these or what factors people chose because there’s no arguing about utility functions (although people can be really bad at actually visualizing Ua).
Separately form this we have actions that look like optimizing for Ua (e.g. AI Safety research and raising awareness), and those that look like optimizing for Ub (e.g. having kids and investing in/for their education). The post argues that some things that look like optimizing for Ub are actually very useful for optimizing Ua (as I understand, it mostly because AI timelines are long enough and the optimization space muddled enough that most people contribute more in expectation from maintaining and improving their general capabilities in a sustainable way at the moment).
Your comment (the pedantic response part) talks about how optimizing for Ua is actually very useful for optimizing Ub. I’m much more sceptical of this claim. The reason is due to expected impact per unit of effort. Let’s consider the sending your kids to college. It looks like top US colleges cost around $50k more per year than state schools, adding up to $200k for a four year programme. This is maybe not several times better as the price tags suggests, but if your child is interested and able to get in to such a school it’s probably at least 10% better (to be quite conservative). A lot of people would be extremely excited for an opportunity to lower the existential risk from AI by 10% for $200k. Sure, sending your kids to college isn’t everything there is to Ub, but it looks like the sign remains the same for a couple of orders of magnitude.
Your talk of a pendulum makes it sound like you want to create a social environment that incentivizes things that look like optimizing for Ua regardless of whether they’re actually in anyone’s best interest. I’m sceptical of trying to get anyone to act against their interests. Rather than make everyone signal that a≫b it makes more sense to have space for people with a≈b or even a<b to optimize for their values and extract gains from trade. A successful AI Safety project probably looks a lot more like a network of very different people figuring out how to collaborate for mutual benefit than a cadre of self-sacrificing idealists.
I chose the college example because it’s especially jarring / especially disrespectful of trying to separate the world into two “pre-AGI versus post-AGI” magisteria.
A more obvious way to see that x-risk matters for ordinary day-to-day goals is that parents want their kids to have long, happy lives (and nearly all of the variance in length and happiness is, in real life, dependent on whether the AGI transition goes well or poorly). It’s not a separate goal; it’s the same goal, optimized without treating ‘AGI kills my kids’ as though it’s somehow better than ‘my kids die in a car accident’.
Your talk of a pendulum makes it sound like you want to create a social environment that incentivizes things that look like optimizing for Ua regardless of whether they’re actually in anyone’s best interest.
I and my kids not being killed by AGI is in my best interest!
A successful AI Safety project probably looks a lot more like a network of very different people figuring out how to collaborate for mutual benefit than a cadre of self-sacrificing idealists.
Not letting AGI kill me and everyone I love isn’t the “self-sacrificing” option! Allowing AGI to kill me is the “self-sacrificing” option — it is literally allowing myself to be sacrificed, albeit for ~zero gain. (Which is even worse than sacrificing yourself for a benefit!)
I’m not advocating for people to pretend they’re more altruistic than they are, and I don’t see myself as advocating against any of the concrete advice in the OP. I’m advocating for people to stop talking/thinking as though post-AGI life is a different magisterium from pre-AGI life, or as though AGI has no effect on their ability to realize the totally ordinary goals of their current life.
I think this would help with shrugging-at-xrisk psychological factors that aren’t ‘people aren’t altruistic enough’, but rather ‘people are more myopic than they wish they were’, ‘people don’t properly emotionally appreciate risks and opportunities that are novel and weird’, etc.
I’m advocating for people to stop talking/thinking as though post-AGI life is a different magisterium from pre-AGI life
Seems undignified to pretend that it isn’t? The balance of forces that make up our world isn’t stable. One way or the other, it’s not going to last. It would certainly be nice, if someone knew how, to arrange for there to be something of human value on the other side. But it’s not a coincidence that the college example is about delaying the phase transition to the other magisterium, rather than expecting as a matter of course that people in technologically mature civilizations will be going to college, even conditional on the somewhat dubious premise that technologically mature civilizations have “people” in them.
The physical world has phase transitions, but it doesn’t have magisteria. ‘Non-overlapping magisteria’, as I’m using the term, is a question about literary genres; about which inferences are allowed to propagate or transfer; about whether a thing feels near-mode or far-mode; etc.
The idea of “going to college” post-AGI sounds silly for two distinct reasons:
The post-singularity world will genuinely be very different from today’s world, and institutions like college are likely to be erased or wildly transformed on relatively short timescales.
The post-singularity world feels like an inherently “far-mode world” where everything that happens is fantastic and large-scale; none of the humdrum minutiae of a single person’s life, ambitions, day-to-day routine, etc. This includes ‘personal goals are near, altruistic goals are far’.
1 is reasonable, but 2 is not.
The original example was about “romantic and reproductive goals”. If the AGI transition goes well, it’s true that romance and reproduction may work radically differently post-AGI, or may be replaced with something wild and weird and new.
But it doesn’t follow from this that we should think of post-AGI-ish goals as a separate magisterium from romantic and reproductive goals. Making the transition to AGI go well is still a good way to ensure romantic and reproductive success (especially qua “long-term goals/flourishing”, as described in the OP), or success on goals that end up mattering even more to you than those things, if circumstances change in such a way that there’s now some crazy, even better posthuman opportunity that you prefer even more.
(I’m assuming here that we shouldn’t optimize goals like “kids get to go to college if they want” in totally qualitatively different ways than we optimize “kids get to go to college if they want, modulo the fact that circumstances might change in ways that bring other values to the fore instead”. I’m deliberately choosing an adorably circa-2022 goal that seems especially unlikely to carry over to a crazy post-AGI world, “college”, because I think the best way to reason about a goal like that is similar to the best way to reason about other goals where it’s more uncertain whether the goal will transfer over to the new phase.)
I love this post a lot.
I’m going to reply in a pedantic way, then say why I did so.
Pedantic response: Trying to solve AI risk is one way to increase the odds you’ll achieve goals like that. (E.g., if you want to support your kids going to college, you should consider putting aside money for them and keeping an eye out for ways to reduce the risk you and your kids die from rogue AI disassembling the inner solar system before they’re of college age.)
Explanation:
I make this point not to argue against finding love or starting a family, but to argue against a mindset that treats AGI and daily life as more or less two different magisteria.
I think there’s a pendulum-swing-y thing going on: people trying to counter burnout, freakouts, etc. heavily emphasize mundane, non-sci-fi-sounding reality, so as to push back a pendulum they see as having swung too far toward ‘my mental universe is entirely about Dyson swarms, and not at all about dental appointments’.
I find this mostly doesn’t work for me personally. I come away feeling like half the blog posts I read are disassociated from ordinary day-to-day life, and the other half are disassociated from ‘the portion of my life that will occur during and after the invention of this wild technology’, such that I end up wary of both kinds of post.
Your overall post is a whole lot better than most on this dimension. It doesn’t just treat ‘be psychologically happy and healthy this year’ as obviously lexicographically more important than preventing your and everyone else’s deaths in twenty-five years (or whatever). It gives specific arguments for why the happy-and-healthy stuff is super important even if you’re confident your personal CEV’s terminal values assign overwhelmingly more weight to x-risk stuff than to your happiness over the next fifty years. It acknowledges the “‘Emergencies’ as wake-up calls” thing.
But it still feels to me like it’s a post trying to push the pendulum in a particular direction, rather than trying to fully and openly embody the optimal-by-your-lights Balancing Point.
(Where de-prioritizing pendulum pushing indeed risks worsening the current problem we face, insofar as EAs today do over-extend more than they under-extend. Maybe I just selfishly prefer reading posts like that, even if they’re not optimal for the community.)
It still doesn’t feel to me like it’s fully speaking as though the two worlds as one world, or fully embracing that a lot of people probably need to hear something closer to the opposite advice. (‘You’re asking too little of yourself; there are great things you could achieve if you were more strategic and purposeful with your time, rather than just following the nearest hedonic gradient; you are not fragile, or a ticking time bomb that will surely flame out if you start doing qualitatively harder things; you can output good-by-maxipok-lights things without waiting to first resolve all your personal issues or get your life fully in order; the alignment problem may sound crazy and intense to you from a distance, and it still possibly be true that you could do good work without sacrificing a flourishing, well-rounded pre-AGI life, and perhaps even flourish more pre-AGI as a consequence.’)
That might just be because I’m not the target audience and this style works great for others, but I wanted to mention it.
The situation is tricky, IMO. There is, of course, at the end of the day only one world. If we want to have kids who can grow up to adulthood, and who can have progeny of their own, this will require that there be a piece of universe hospitable to human life where they can do that growing up.
At the same time:
a) IMO, there is a fair amount of “belief in belief” about AI safety and adjacent things. In particular, I think many people believe they ought to believe that various “safety” efforts help, without really anticipating-as-if this sort of thing can help.
(The argument for “be worried about the future” is IMO simpler, more obvious, and more likely to make it to the animal, that particular beliefs that particular strategies have much shot. I’m not sure how much this is or isn’t about AI; many of my Uber drivers seem weirdly worried about the future.)
a2) Also, IMO, a fair number of peoples’ beliefs (or “belief in beliefs”) about AI safety are partly downstream of others’ political goals, e.g. of others’ social incentives that those people believe in particular narratives about AI safety and about how working at place X can help with AI safety. This can accent the “belief in belief” thing.
b) Also, even where people have honest/deep/authentic verbal-level beliefs about a thing, it often doesn’t percolate all the way down into the animal. For example, a friend reports having interviewed a number of people about some sex details, and coming out believing that some people do and some people don’t have a visceral animal-level understanding that birth control prevents pregnancy, and reports furthermore that such animal-level beliefs often backpropagate to peoples’ desires or lack of desires for different kinds of sex. I believe my friend here, although this falls under “hard to justify personal beliefs.”
c) As I mentioned in the OP, I am worried that when a person “gives up on” their romantic and/or reproductive goals (or other goals that are as deeply felt and that close to the center of a person, noting that the details here vary by individual AFAICT), this can mess up their access to caring and consciousness in general (in Divia’s words, can risk “some super central sign error deep in their psychology”).
a-c, in combination, leave me nervous/skeptical about people saying that their plan is to pursue romance/children “after the Singularity,” especially if they’re already nearing the end of their biological window. And especially if it’s prompted by that being a standard social script in some local circle. I am worried that people may say this, intend it with some portion of themselves, but have their animal hear “I’m going to abandon these goals and live in belief-in-beliefs.”
I have a personal thing up for me about this one. Back in ~2016, I really wanted to try to have a kid, but thought that short timelines plus my own ability to contribute to AI safety efforts meant I probably shouldn’t. I dialoged with my system 1 using all the tools in the book. I consulted all the people near me who seemed like they might have insight into the relevant psychology. My system 1 / animal-level orientation, after dialog, seemed like it favored waiting, hoping for a kid on the other side of the singularity. I mostly passed other peoples’ sanity checks, both at the time and a few years later when I was like “hey, we were worried about this messing with my psyche, but it seems like it basically worked, right? what is your perception?” And even so, IMO, I landed in a weird no man’s land of a sort of bleached out depression and difficulty caring about anything after awhile, that was kinda downstream of this but very hard for me to directly perceive, but made it harder to really mean anything and easier to follow the motions of looking like I was trying.
The take-away I’m recommending from this is something like: “be careful about planning on paths toward your deepest, animal-level goals that your animal doesn’t buy. And note that it’s can be hard to robustly know what your animal is or isn’t buying. ” Also, while there’s only one magesterium, if people are animal-level and speech-level reasoning as though there’s several, that’s a real and confusing-to-me piece of context to how we’re living here.
❤
That makes sense to me, and it updates me toward your view on the kid-having thing. (Which wasn’t the focus of my comment, but is a thing I was less convinced of before.) I feel sad about that having happened. :( And curious about whether I (or other people I know) are making a similar mistake.
(My personal state re kids is that it feels a bit odd/uncanny when I imagine myself having them, and I don’t currently viscerally feel like I’m giving something up by not reproducing. Though if I lived for centuries, I suspect I’d want kids eventually in the same way I’d want to have a lot of other cool experiences.)
I feel kinda confused about how “political” my AGI-beliefs are. The idea of dying to AGI feels very sensorily-real to me — I feel like my brain puts it in the same reference class as ‘dying from a gunshot wound’, which is something I worry about at least a little in my day-to-day life (even though I live in a pretty safe area by US megalopolis standards), have bad dreams about, semi-regularly idly imagine experiencing, etc. I don’t know how that relates to the “is this belief political?” question, or how to assess that.
Regardless, I like this:
I’d also assume by default that ‘the animal discounts more heavily than the philosopher does’ is a factor here...? And/or ‘the animal is better modeled on this particular question as an adaptation-executer rather than a utility-maximizer, such that there’s no trade you can make that will satisfy the animal if it involves trading away having-kids-before-age-45’?
It could be wise to have kids, for the sake of harmony between your parts/subgoals, even if you judge that having kids isn’t directly x-risk-useful and your animal seems to have a pretty visceral appreciation for x-risk—just because that is in fact what an important part of you wants/needs/expects/etc.
I want to have a dialog about what’s true, at the level of piece-by-piece reasoning and piece-by-piece causes. I appreciate that you Rob are trying to do this; “pedantry” as you put it is great, and seems to me to be a huge chunk of why LW is a better place to sort some things out than is most of the internet.
I’m a bit confused that you call it “pedantry”, and that you talk of my post as trying to push the pendulum in a particular way, and “people trying to counter burnout,” and whether this style of post “works” for others. The guess I’m forming, as I read your (Rob’s) comment and to a lesser extent the other comments, is that a bunch of people took my post as a general rallying cry against burnout, and felt it necessary to upvote my post, or to endorse my post, because they personally wish to take a stand against burnout. Does something like that seem right/wrong to anyone? (I want to know.)
I… don’t want that, although I may have done things in my post to encourage it anyhow, without consciously paying attention. But if we have rallying cries, we won’t have the kind of shared unfiltered reasoning that someone wanting truth can actually update on.
I’m in general pretty interested in strategies anyone has for having honest, gritty, mechanism-by-mechanism discussion near a Sacred Value. “Don’t burn people out” is arguably a Sacred Value, such that it’ll be hard to have open conversation near it in which all the pedantry is shared in all the directions. I’d love thoughts on how to do it anyhow.
Yay! I basically agree. The reason I called it “pedantry” was because I said it even though (a) I thought you already believed it (and were just speaking imprecisely / momentarily focusing on other things), (b) it’s an obvious observation that a lot of LWers already have cached, and (c) it felt tangential to the point you were making. So I wanted to flag it as a change of topic inspired by your word choice, rather than as ordinary engagement with the argument I took you to be making.
I think I came to the post with a long-term narrative (which may have nothing to do with the post):
There are a bunch of (Berkeley-ish?) memes in the water related to radical self-acceptance, being kind to yourself, being very wary of working too hard, staying grounded in ordinary day-to-day life, being super skeptical and cautious around Things Claiming To Be Really Important and around moralizing, etc.
I think these are extremely important and valuable memes that LW would do well to explore, discuss, and absorb much more than it already has. I’ve found them personally extremely valuable, and a lot of my favorite blog posts to send to new EAs/rats make points like ‘be cautious around things that make big moralistic demands of you’, etc.
But I also think that these kinds of posts are often presented in ways that compete against the high value I (genuinely, thoughtfully) place on the long-term future, and the high probability I (genuinely, thoughtfully) place on AI killing me and my loved ones, as though I need to choose between the “chill grounded happy self-loving unworried” aesthetic or the “working really hard to try to solve x-risk” aesthetic.
This makes me very wary, especially insofar as it isn’t making an explicit argument against x-risk stuff, but is just sort of vaguely associating not-worrying-so-much-about-human-extinction with nice-sounding words like ‘healthy’, ‘grounded’, ‘relaxed’, etc. If these posts spent more time explicitly arguing for their preferred virtues and for why those virtues imply policy X versus policy Y, rather than relying on connotation and implicature to give their arguments force, my current objection would basically go away.
If more of the “self-acceptance, be kind to yourself, be vary wary of working too hard, etc.” posts were more explicit about making space for possibilities like ‘OK, but my best self really does care overwhelmingly more about x-risk stuff than everything else’ and/or ‘OK, but making huge life-changes to try to prevent human extinction really is the psychologically healthiest option for me’, I would feel less suspicious that some of these posts are doing the dance wrong, losing sight of the fact that both magisteria are real, are part of human life.
I may have been primed to interpret this post in those terms too much, because I perceived it to be a reaction to Eliezer’s recent doomy-sounding blog posts (and people worrying about AI more than usual recently because of that, plus ML news, plus various complicated social dynamics), trying to prevent the community from ‘going too far’ in certain directions.
I think the post is basically good and successful at achieving that goal, and I think it’s a very good goal. I expect to link to the OP post a lot in the coming months. But it sounds like I may be imposing context on the post that isn’t the way you were thinking about it while writing it.
Oh, yeah, maybe. I was not consciously responding to that. I was consciously responding to a thing that’s been bothering me quite a bit about EA for ~5 or more years, which is that there’s not enough serious hobbies around here IMO, and also people often report losing the ability to enjoy hanging out with friends, especially friends who aren’t in these circles, and just enjoying one anothers’ company while doing nothing, e.g. at the beach with some beers on a Saturday. (Lots of people tell me they try allocating days to this, it isn’t about the time, it’s about an acquired inability to enter certain modes.)
Thanks for clarifying this though, that makes sense.
I have some other almost-written blog posts that’re also about trying to restore access to “hanging out with friends enjoying people” mode and “serious hobbies” mode, that I hope to maybe post in the next couple weeks.
Back in ~2008, I sat around with some others trying to figure out: if we’re successful in getting a lot of people involved in AI safety—what can we hope to see at different times? And now it’s 2022. In terms of “there’ll be a lot of dollars people are up for spending on safety”, we’re basically “hitting my highest 2008 hopes”. In terms of “there’ll be a lot of people who care”, we’re… less good than I was hoping for, but certainly better than I was expecting. “Hitting my 2008 ‘pretty good’ level.” In terms of “and those people who care will be broad and varied and trying their hands at making movies and doing varied kinds of science and engineering research and learning all about the world while keeping their eyes open for clues about the AI risk conundrum, and being ready to act when a hopeful possibility comes up” we’re doing less well compared to my 2008 hopes. I want to know why and how to unblock it.
I think to the extent that people are failing to be interesting in all the ways you’d hoped they would be, it’s because being interesting in those ways seems to them to have greater costs than benefits. If you want people to see the benefits of being interesting as outweighing the costs, you should make arguments to help them improve their causal models of the costs, and to improve their causal models of the benefits, and to compare the latter to the former. (E.g., what’s the causal pathway by which an hour of thinking about Egyptology or repairing motorcycles or writing fanfic ends up having, not just positive expected usefulness, but higher expected usefulness at the margin than an hour of thinking about AI risk?) But you haven’t seemed very interested in explicitly building out this kind of argument, and I don’t understand why that isn’t at the top of your list of strategies to try.
I think of this in terms of personal vs. civilization-scale value loci distinction. Personal-scale values, applying to individual modern human minds, speaking of those minds, might hold status quo anchoring sacred and dislike presence of excessive awareness of disruptive possible changes. While civilization-scale values, even as they are facilitated by individuals, do care about accurate understanding of reality regardless of what it says.
People shouldn’t move too far towards becoming decision theoretic agents, even if they could, other than for channeling civilization. The latter is currently a necessity (that’s very dangerous to neglect), but it’s not fundamentally a necessity. What people should move towards is a more complicated question with some different answer (which does probably include more clarity in thinking than is currently the norm or physiologically possible, but still). People are vessels of value, civilization is its custodian. These different roles call for different shapes of cognition.
In this model, it’s appropriate / morally-healthy / intrinsically-valuable for people to live more fictional lives (as they prefer) while civilization as a whole is awake, and both personal-scale values and civilization-scale values agree on this point.
AFAICT, I am trying to fully and openly embody the way of reasoning that actually makes sense to me in this domain, which… isn’t really a “balancing point.” It’s more like the anarchist saying “the means are the ends.” Or it’s more like Szilard’s “ten commandments,” (which I highly recommend reading for anyone who hasn’t; they’re short). Or more like the quote from the novel The Dispossessed: “To reassert its validity and strength, he thought, one need only act, without fear of punishment and without hope of reward: act from the center of one’s soul. “
I don’t have the right concepts or articulation here. This is an example of the “hard to justify personal opinions” I warned about in my “epistemic status.” But IMO, thinking about tradeoffs and balancing points can be good when your map is good enough; at other times, it’s more like I want to try to hone in on priors, on deep patternness, on where reasoning is before it’s reasoning. This is where the power of leisure comes from, where the possibility of hobbies that end up giving you glimpses of new bits of the universe come from. And it’s a thing I’m trying to show here. Not a particular balance-point between depleting your long-term resources and “being nice to yourself” by eating chocolates and cartoons. (Neither of those help with getting to the tao, usually, AFAICT.)
We can be empirical about trying to see which actions, which mindsets, add to our and others’ long-term robust abilities.
In short-term crises for which you have decent-quality maps, balance-points and trading things off with local consequentialist reasoning makes sense to me. But not the rest of everywhere.
I agree many people underestimate their own capacities, and too seldom try hard or scary things. I think this is often many of the same people who burn themselves out.
Sorry this reply, and my other one, are somewhat incoherent. I’m having trouble mapping both where you’re coming from, and why/where I disagree.
Yeah, that makes sense to me. I’m complaining about a larger class of posts, so maybe this one isn’t really an example and I’m just pattern-matching. I do still wish there existed more posts that were very obviously examples of the ‘both-and’ things I was pointing at. (Both dentist appointments and Dyson spheres; both embrace slack and embrace maxipok; etc.)
It might be that if my thinking were clearer here, I’d be able to recognize more posts as doing ‘both-and’ even if they don’t explicitly dwell on it as much as I want.
I think I feel a similar mix of love and frustration for your comment as I read your comment expressing with the post.
Let me be a bit theoretical for a moment. It makes sense for me to think of utilities as a sum U=aUa+bUb where Ua is the utility of things after singularity/superintelligence/etc and Ub the utility for things before then (assuming both are scaled to have similar magnitudes so the relative importance is given by the scaling factors). There’s no arguing about the shape of these or what factors people chose because there’s no arguing about utility functions (although people can be really bad at actually visualizing Ua).
Separately form this we have actions that look like optimizing for Ua (e.g. AI Safety research and raising awareness), and those that look like optimizing for Ub (e.g. having kids and investing in/for their education). The post argues that some things that look like optimizing for Ub are actually very useful for optimizing Ua (as I understand, it mostly because AI timelines are long enough and the optimization space muddled enough that most people contribute more in expectation from maintaining and improving their general capabilities in a sustainable way at the moment).
Your comment (the pedantic response part) talks about how optimizing for Ua is actually very useful for optimizing Ub. I’m much more sceptical of this claim. The reason is due to expected impact per unit of effort. Let’s consider the sending your kids to college. It looks like top US colleges cost around $50k more per year than state schools, adding up to $200k for a four year programme. This is maybe not several times better as the price tags suggests, but if your child is interested and able to get in to such a school it’s probably at least 10% better (to be quite conservative). A lot of people would be extremely excited for an opportunity to lower the existential risk from AI by 10% for $200k. Sure, sending your kids to college isn’t everything there is to Ub, but it looks like the sign remains the same for a couple of orders of magnitude.
Your talk of a pendulum makes it sound like you want to create a social environment that incentivizes things that look like optimizing for Ua regardless of whether they’re actually in anyone’s best interest. I’m sceptical of trying to get anyone to act against their interests. Rather than make everyone signal that a≫b it makes more sense to have space for people with a≈b or even a<b to optimize for their values and extract gains from trade. A successful AI Safety project probably looks a lot more like a network of very different people figuring out how to collaborate for mutual benefit than a cadre of self-sacrificing idealists.
I chose the college example because it’s especially jarring / especially disrespectful of trying to separate the world into two “pre-AGI versus post-AGI” magisteria.
A more obvious way to see that x-risk matters for ordinary day-to-day goals is that parents want their kids to have long, happy lives (and nearly all of the variance in length and happiness is, in real life, dependent on whether the AGI transition goes well or poorly). It’s not a separate goal; it’s the same goal, optimized without treating ‘AGI kills my kids’ as though it’s somehow better than ‘my kids die in a car accident’.
I and my kids not being killed by AGI is in my best interest!
Not letting AGI kill me and everyone I love isn’t the “self-sacrificing” option! Allowing AGI to kill me is the “self-sacrificing” option — it is literally allowing myself to be sacrificed, albeit for ~zero gain. (Which is even worse than sacrificing yourself for a benefit!)
I’m not advocating for people to pretend they’re more altruistic than they are, and I don’t see myself as advocating against any of the concrete advice in the OP. I’m advocating for people to stop talking/thinking as though post-AGI life is a different magisterium from pre-AGI life, or as though AGI has no effect on their ability to realize the totally ordinary goals of their current life.
I think this would help with shrugging-at-xrisk psychological factors that aren’t ‘people aren’t altruistic enough’, but rather ‘people are more myopic than they wish they were’, ‘people don’t properly emotionally appreciate risks and opportunities that are novel and weird’, etc.
Seems undignified to pretend that it isn’t? The balance of forces that make up our world isn’t stable. One way or the other, it’s not going to last. It would certainly be nice, if someone knew how, to arrange for there to be something of human value on the other side. But it’s not a coincidence that the college example is about delaying the phase transition to the other magisterium, rather than expecting as a matter of course that people in technologically mature civilizations will be going to college, even conditional on the somewhat dubious premise that technologically mature civilizations have “people” in them.
The physical world has phase transitions, but it doesn’t have magisteria. ‘Non-overlapping magisteria’, as I’m using the term, is a question about literary genres; about which inferences are allowed to propagate or transfer; about whether a thing feels near-mode or far-mode; etc.
The idea of “going to college” post-AGI sounds silly for two distinct reasons:
The post-singularity world will genuinely be very different from today’s world, and institutions like college are likely to be erased or wildly transformed on relatively short timescales.
The post-singularity world feels like an inherently “far-mode world” where everything that happens is fantastic and large-scale; none of the humdrum minutiae of a single person’s life, ambitions, day-to-day routine, etc. This includes ‘personal goals are near, altruistic goals are far’.
1 is reasonable, but 2 is not.
The original example was about “romantic and reproductive goals”. If the AGI transition goes well, it’s true that romance and reproduction may work radically differently post-AGI, or may be replaced with something wild and weird and new.
But it doesn’t follow from this that we should think of post-AGI-ish goals as a separate magisterium from romantic and reproductive goals. Making the transition to AGI go well is still a good way to ensure romantic and reproductive success (especially qua “long-term goals/flourishing”, as described in the OP), or success on goals that end up mattering even more to you than those things, if circumstances change in such a way that there’s now some crazy, even better posthuman opportunity that you prefer even more.
(I’m assuming here that we shouldn’t optimize goals like “kids get to go to college if they want” in totally qualitatively different ways than we optimize “kids get to go to college if they want, modulo the fact that circumstances might change in ways that bring other values to the fore instead”. I’m deliberately choosing an adorably circa-2022 goal that seems especially unlikely to carry over to a crazy post-AGI world, “college”, because I think the best way to reason about a goal like that is similar to the best way to reason about other goals where it’s more uncertain whether the goal will transfer over to the new phase.)