Thank you for the post, it was quite a nostalgia trip back to 2015 for me because of all the Wait But Why references. However, my impression is that the Kurzweilian Accelerationism school of thought has largely fallen out of favor in transhumanist circles since that time, with prominent figures like Peter Thiel and Scott Alexander arguing that not only are we not accelerating, we can barely even keep up with 19th century humanity in terms of growth rate. Life expectancy in the US has actually gone down in recent years for the first time.
An important consideration that was left out is temporal discounting. Since you assumed linear scaling of value with post-Singularity QALYs, your result is extremely sensitive to your choice of post-Singularity life expectancy. I felt like it was moot to go into such detailed analysis of the other factors when this one alone could easily vary by ten orders of magnitude. By choosing a sufficiently large yet physically plausible number (such as 100 trillion years), you could justify almost any measure to reduce your risk of dying before Singularity and unambiguously resolve e.g. the question of driving risk.
But I doubt that’s a good representation of your actual values. I think you’re much more likely to do exponential discounting of future value, such that the integral of value over time remains finite even in the limit of infinite time. This should lead to much more stable results.
I predict that a lot of people will interpret the claim of “you should expect to live for 10k years” as wacky, and not take it seriously.
Thank you for the post, it was quite a nostalgia trip back to 2015 for me because of all the Wait But Why references.
Sure thing! Yeah I felt similar nostalgia. I love and miss Wait But Why.
However, my impression is that the Kurzweilian Accelerationism school of thought has largely fallen out of favor in transhumanist circles since that time, with prominent figures like Peter Thiel and Scott Alexander arguing that not only are we not accelerating, we can barely even keep up with 19th century humanity in terms of growth rate. Life expectancy in the US has actually gone down in recent years for the first time.
To continue the nostalgia:
The trajectory of very recent history often tells a distorted story. First, even a steep exponential curve seems linear when you only look at a tiny slice of it, the same way if you look at a little segment of a huge circle up close, it looks almost like a straight line. Second, exponential growth isn’t totally smooth and uniform. Kurzweil explains that progress happens in “S-curves”:
An S is created by the wave of progress when a new paradigm sweeps the world. The curve goes through three phases:
Slow growth (the early phase of exponential growth)
Rapid growth (the late, explosive phase of exponential growth)
A leveling off as the particular paradigm matures3
If you look only at very recent history, the part of the S-curve you’re on at the moment can obscure your perception of how fast things are advancing. The chunk of time between 1995 and 2007 saw the explosion of the internet, the introduction of Microsoft, Google, and Facebook into the public consciousness, the birth of social networking, and the introduction of cell phones and then smart phones. That was Phase 2: the growth spurt part of the S. But 2008 to 2015 has been less groundbreaking, at least on the technological front. Someone thinking about the future today might examine the last few years to gauge the current rate of advancement, but that’s missing the bigger picture. In fact, a new, huge Phase 2 growth spurt might be brewing right now.
Maybe you’re right though. I don’t have a good enough understanding to say. Thanks for contributing this thought.
I felt like it was moot to go into such detailed analysis of the other factors when this one alone could easily vary by ten orders of magnitude.
Yeah, in retrospect that does make sense. I think I got lost in the weeds.
An important consideration that was left out is temporal discounting.
...
By choosing a sufficiently yet physically plausible number (such as 100 trillion years), you could justify almost any measure to reduce your risk of dying before Singularity and unambiguously resolve e.g. the question of driving risk.
But I doubt that’s a good representation of your actual values. I think you’re much more likely to do exponential discounting of future value, making the integral of value over time finite even in the limit of infinite time. This should lead to much more stable results.
Sigh. Yeah, this was a crucial thing that I left out. It might even be the crux. My mistake for not talking about it in the OP.
Here are my thoughts. Descriptively, I see that temporal discounting is something that people do. But prescriptively, I don’t see why it’s something that we should do. Maybe I am just different, but when I think about, say, 100 year old me vs current 28 year old me, I don’t feel like I should prioritize that version less. Like everyone else, there is a big part of me that thinks “Ugh, let me just eat the pizza instead of the salad, forget about future me”. But when I think about what I should do, and how I should prioritize future me vs present me, I don’t really feel like there should be discounting.
That was for 100 year old me. What about 1k? 10k? 100k? Eh, maybe. The feeling of “screw it, forget that future self” is stronger. But I still question whether that is how I should weigh things. After all, 100k year old me is still just as conscious and real as 28 year old me.
Regret minimization seems relevant, although I’ve only ever taken a cursory look at the concept. If I didn’t weigh 100k year old me enough, it could be something I end up regretting.
Another perspective is the one I take at the end in the Assume The Hypothetical section. If you were going to live to 100k years old, how would you feel? For me it makes me feel like I should really value that future self.
However, once we start talking about stuff like a trillions years and 3^^^3, I start to think, “This is getting crazy, maybe we should be temporally discounting.” Why at that point and not at 100k? Idk. Because that’s what my intuition is? I guess I don’t have a great grasp on this question.
Really? This is LessWrong after all^^
Yeah. I have a lot of respect for LessWrongers in general but I do think that there are some common failure modes, and not Taking Ideas Seriously enough is one of them. (I mean that criticism constructively not contentiously, of course.)
And I think that the reception this post has gotten is decently strong evidence that I was correct (not perfect of course; there are alternative explanations). I see this as a very important topic that doesn’t have obvious conclusions, and thus should receive more attention. Even if you are correct about temporal discounting, it is not at all obvious to me how much we should discount and how that affects the final conclusion regarding how much we should value life.
To be sure, I don’t actually think whether Accelerationism is right has any effect on the validity of your points. Indeed, there is no telling whether the AI experts from the surveys even believe in Accelerationism. A fast-takeoff model where the world experiences zero growth from now to Singularity, followed by an explosion of productivity would yield essentially the same conclusions as long as the date is the same, and so does any model in between. But I’d still like to take apart the arguments from Wait But Why just for fun:
First, exponential curves are continuous, they don’t produce singularities. This is what always confused me about Ray Kurzweil as he likes to point to the smooth exponential improvement in computing yet in the next breath predict the Singularity in 2029. You only get discontinuities when your model predicts superexponential growth, and Moore’s law is no evidence for that.
Second, while temporary deviations from the curve can be explained by noise for exponential growth, the same can’t be said so easily for superexponential growth. Here, doubling time scales with the countdown to Singularity, and what can be considered “temporary” is highly dependent on how long we have left to go. If we were in 10,000 BC, a slowing growth rate over half a century could indeed be seen as noise. But if we posit the Singularity at 2060, then we have less than 40 years left. As per Scott Alexander, world GDP doubling time has been increasing since 1960. However you look at it, the trend has been deviating from the ideal curve for far, far too long to be a mere fluke.
The most prominent example of many small S-curves adding up to an overall exponential trend line is, again, Moore’s law. From the inside view, proponents argue that doomsayers are short sighted because they only see the limits of current techniques, but such limits have appeared many times before since the dawn of computing and each time it was overcome by the introduction of a new technique. For instance, most recently, chip manufacturers have been using increasingly complex photolithography masks to print ever smaller features onto microchips using the same wavelengths of UV light, which isn’t sustainable. Then came the crucial breakthrough last year with the introduction of EUV, a novel technique that uses shorter wavelengths and allows the printing of even smaller features with a simple mask, and the refining process can start all over again.
But from the outside view, Moore’s law has buckled (notice the past tense). One by one, the trend lines have flattened out, starting with processor frequency in 2006, and most recently with transistors per dollar (Kurzweil’s favorite metric) in 2018. Proponents of Moore’s law’s validity had to keep switching metrics for 1.5 decades, and they have a few left—transistor density for instance, or TOP500 performance. But the noose is tightening, and some truly fundamental limitations such as the Landauer limit are on the horizon. As I often like to say, when straight lines run into physical limitations, physics wins.
Keep in mind that as far as Moore’s law goes, this is what death looks like. A trend line never halts abruptly, it’s always going to peter out gradually at the end.
By the way, the reason I keep heckling Moore’s law is because Moore’s law itself is the last remnant of the age of accelerating technological progress. Outside the computing industry, things are looking much more dire.
Here are my thoughts. Descriptively, I see that temporal discounting is something that people do. But prescriptively, I don’t see why it’s something that we should do. Maybe I am just different, but when I think about, say, 100 year old me vs current 28 year old me, I don’t feel like I should prioritize that version less. Like everyone else, there is a big part of me that thinks “Ugh, let me just eat the pizza instead of the salad, forget about future me”. But when I think about what I should do, and how I should prioritize future me vs present me, I don’t really feel like there should be discounting.
I’m not sure the prescriptive context is meaningful with regard to values. It’s like having a preference over preferences. You want whatever you want, and what you should want doesn’t matter because you don’t actually want that, wherever that should came from. A useful framework to think about this problem is to model your future self as other people and reduce it to the classic egoism-altruism balance. Would you say perfect altruism is the correct position to adopt? Are you therefore a perfect altruist?
You could make up philosophical thought experiments and such to discover how much you actually care about others, but I bet you can’t just decide to become a perfect altruist no matter how loudly a philosophy professor might scream at you. Similarly, whether you believe temporal discounting to be the right call or not in the abstract, you can’t actually stop doing it; you’re not a perfect altruist with respect to your future selves and to dismiss it would only lead to confusion in my opinion.
To be sure, I don’t actually think whether Accelerationism is right has any effect on the validity of your points.
Yeah I agree. Seems like it just gives a small bump in the likelihood of living to the singularity, but that has a very small impact on the larger question, because the larger question is a lot more sensitive to other variables like how long a lifespan you’d expect post-singularity, and how much if any temporal discounting should be done.
To the rest of your points about exponential growth, I unfortunately don’t understand it enough to really be able to respond, sorry.
I’m not sure the prescriptive context is meaningful with regard to values. It’s like having a preference over preferences. You want whatever you want, and what you should want doesn’t matter because you don’t actually want that, wherever that should came from.
First of all, thank you very much for engaging with me here. This is exactly the sort of thing I was hoping to get in the comments. A good critique that I hadn’t thought (enough) about, and one that hits my central point hard, rather than just hitting tangential points that don’t have much influence on the central point (although I appreciate those too). I also think you’re expressing yourself very clearly, which makes it pleasant to engage with.
The more I think about it, the more I think I should apply some temporal discounting. However, I still lean towards not doing it too much.
In some theoretical sense, I agree that rationality can’t tell you what to value, only how to achieve your values (as well as how to figure out what is true). But in a more practical sense, I think that often times you can examine your values and realize that, well, I shouldn’t say “there is good reason to change them”, but I guess I could say “you find yourself inspired to change them” or “they’ve just been changed”. Like you mention, thought experiments can be a great tool, but I think it’s more than just that they help you discover things. I think they can inspire you to change your values. I do agree that it isn’t really something that you can just decide to change though.
As an example, consider an immature teenager who doesn’t care at all about his future self and just wants to have fun right now. Would you say, “Well, he values what he values.”? Probably not.
So then, I think this question of temporal discounting is really one that needs to be explored. It’s not enough to just say, “10k years from now? I don’t care about that.” Maybe we’re being immature teenagers.
I think they can inspire you to change your values.
Taken at face value, this statement doesn’t make much sense because it immediately begs the question of change according to what, and in what sense isn’t that change part of your value already. My guess here is that your mental model says something like “there’s a set of primal drives inside my head like eating pizza that I call ‘values’, and then there are my ‘true’ values like a healthy lifestyle which my conscious, rational mind posits, and I should change my primal drives to match my ‘true’ values” (pardon for straw-manning your position, but I need it to make my point).
A much better model in my opinion would be that all these values belong to the same exact category. These “values” or “drives” then duke it out amongst each other, and your conscious mind merely observes and makes up a plausible-sounding socially-acceptable story about your motivations (this is, after all, the evolutionary function of human intelligence in the first place as far as I know), like a press secretary sitting silently in the corner while generals are having a heated debate.
At best, your conscious mind might act as a mediator between these generals, coming up with clever ideas that pushes the Pareto boundary of these competing values so that they can all be satisfied to a greater degree at the same time. Things like “let’s try e-cigarettes instead of regular tobacco—maybe it satisfies both our craving for nicotine and our long-term health!”.
Even high-falutin values like altruism or long-term health are induced by basic drives like empathy and social status. They are no different to, say, food cravings, not even in terms of inferential distance. Compare for instance “I ate pizza, it was tasty and I felt good” with “I was chastised for eating unhealthily, it felt bad”. Is there really any important difference here?
You could of course deny this categorization and insist that only a part of this value set represents your true values. The danger here isn’t that you’ll end up optimizing for the wrong set of values since who’s to tell you what “wrong” is, it’s that you’ll be perpetually confused about why you keeping failing to act upon your declared “true” values—why your revealed preferences through behavior keep diverging from the stated preferences, and you end up making bad decisions. Decisions that are suboptimal even when judged only against your “true” values, because you have not been leveraging your conscious, rational mind properly by giving it bad epistemics.
As an example, consider an immature teenager who doesn’t care at all about his future self and just wants to have fun right now. Would you say, “Well, he values what he values.”?
Haha, unfortunately you posed the question to the one guy out of 100 who would gladly answer “Absolutely”, followed by “What’s wrong with being an immature teenager?”
On a more serious note, it is true that our values often shift over time, but it’s unclear to me why that makes regret minimization the correct heuristic. Regret can occur in two ways: One is that we have better information later in life, along the lines of “Oh I should have picked these numbers in last week’s lottery instead of the numbers I actually picked”. But this is just hindsight and useless to your current self because you don’t have access to that knowledge.
The other is through value shift, along the lines of “I just ate a whole pizza and now that my food-craving brain-subassembly has shut up my value function consists mostly of concerns for my long-term health”. Even setting temporal discounting aside, I fail to see why your post-dinner-values should take precedence over your pre-dinner-values, or for that matter why deathbed-values should take precedence over teenage-values. They are both equally real moments of conscious experience.
But, since we only ever live and make decisions in the present moment, if you happen to have just finished a pizza, you now have the opportunity to manipulate your future values to match your current values by taking actions that makes the salad option more available the next time pizza-craving comes around by e.g. shopping for ingredients. In AI lingo, you’ve just made yourself subagent-stable.
My personal anecdote is that as a teenager I did listen to the “mature adults” to study more and spend less time having fun. It was a bad decision according to both my current values and teenage-values, made out of ignorance about how the world operates.
As a final thought, I would give the meta-advice of not trying to think too deeply about normative ethics. Take AlphaGo as a cautionary tale: after 2000 years of pondering, the deepest truths of Go are revealed to be just a linear combination of a bunch of feature vectors. Quite poetic, if you ask me.
I think I may have lead us down the wrong path here. The ultimate question is the one of temporal discounting, and that question depends on how much we do/should value those post-singularity life years. If values can’t shift, then there isn’t really anything to talk about; you just ask yourself how much you value those years, and then move on. But if they can shift, and you acknowledge that they can, then we can discuss some thought experiments and stuff. It doesn’t seem important to discuss whether those shifts are due to discovering more about your pre-existing values, or due to actually changing those pre-existing values.
Haha, unfortunately you posed the question to the one guy out of 100 who would gladly answer “Absolutely”, followed by “What’s wrong with being an immature teenager?”
Ah, I see. You and I probably just have very different intuitions regarding what to value then, and I sense that thought experiments won’t bring us much closer.
Actually, I wonder what you think of this. Are you someone who sees death as a wildlyterriblething (I am)? If so, isn’t it because you place a correspondingly high value on the years of life you’d be losing?
The other is through value shift, along the lines of “I just ate a whole pizza and now that my food-craving brain-subassembly has shut up my value function consists mostly of concerns for my long-term health”. Even setting temporal discounting aside, I fail to see why your post-dinner-values should take precedence over your pre-dinner-values, or for that matter why deathbed-values should take precedence over teenage-values. They are both equally real moments of conscious experience.
In the pizza example, I think the value shift would moreso be along the lines of “I was prioritizing my current self too much relative to my future selves”. Presumably, post-dinner-values would be incorporating pre-dinner-self. Eg. it wouldn’t just say, “Screw my past self, my values are only about the present moment and onwards.” So I see your current set of values as being the most “accurate”, in which case regret minimization seems like it makes sense.
The ultimate question is the one of temporal discounting, and that question depends on how much we do/should value those post-singularity life years. If values can’t shift, then there isn’t really anything to talk about; you just ask yourself how much you value those years, and then move on. But if they can shift, and you acknowledge that they can, then we can discuss some thought experiments and stuff.
I think we’re getting closer to agreement as I’m starting to see what you’re getting at. My comment here would be that yes, your values can shift, and they have shifted after thinking hard about what post-Singularity life will be like and getting all excited. But the shift it has caused is a larger multiplier in front of the temporal discounted integral, not the disabling of temporal discounting altogether.
Actually, I wonder what you think of this. Are you someone who sees death as a wildlyterriblething (I am)?
Yes, but I don’t think there is any layer of reasoning beneath that preference. Evading death is just something that is very much hard-coded into us by evolution.
In the pizza example, I think the value shift would moreso be along the lines of “I was prioritizing my current self too much relative to my future selves”. Presumably, post-dinner-values would be incorporating pre-dinner-self.
I don’t think that’s true. Crucially, there is no knowledge being gained over the course of dinner, only value shift. It’s not like you didn’t know beforehand that pizza was unhealthy, or that you will regret your decision. And if post-dinner self does not take explicit steps to manipulate future value, the situation will repeat itself the next day, and the day after, and so on for hundreds of times.
I think we’re getting closer to agreement as I’m starting to see what you’re getting at. My comment here would be that yes, your values can shift, and they have shifted after thinking hard about what post-Singularity life will be like and getting all excited. But the shift it has caused is a larger multiplier in front of the temporal discounted integral, not the disabling of temporal discounting altogether.
I’m in agreement here! Some follow up questions: what are your thoughts on how much discounting should be done? Relatedly, what are your thoughts on how much we should value life? Is it obvious that past eg. 500 years, it’s far enough into the future that it becomes negligible? If not, why aren’t these things that are discussed? Also, do you share my impression that people (on LW) largely assume that life expectancy is something like 80 years and life is valued at something like $10M?
Yes, but I don’t think there is any layer of reasoning beneath that preference. Evading death is just something that is very much hard-coded into us by evolution.
Regardless of whether it stems from a layer of reasoning or whether it is hard-coded, doesn’t it imply that you aren’t doing too much temporal discounting? If you did a lot of temporal discounting and didn’t value the years beyond eg. 250 years old very much, then death wouldn’t be that bad, right?
Thank you for the post, it was quite a nostalgia trip back to 2015 for me because of all the Wait But Why references. However, my impression is that the Kurzweilian Accelerationism school of thought has largely fallen out of favor in transhumanist circles since that time, with prominent figures like Peter Thiel and Scott Alexander arguing that not only are we not accelerating, we can barely even keep up with 19th century humanity in terms of growth rate. Life expectancy in the US has actually gone down in recent years for the first time.
An important consideration that was left out is temporal discounting. Since you assumed linear scaling of value with post-Singularity QALYs, your result is extremely sensitive to your choice of post-Singularity life expectancy. I felt like it was moot to go into such detailed analysis of the other factors when this one alone could easily vary by ten orders of magnitude. By choosing a sufficiently large yet physically plausible number (such as 100 trillion years), you could justify almost any measure to reduce your risk of dying before Singularity and unambiguously resolve e.g. the question of driving risk.
But I doubt that’s a good representation of your actual values. I think you’re much more likely to do exponential discounting of future value, such that the integral of value over time remains finite even in the limit of infinite time. This should lead to much more stable results.
Really? This is LessWrong after all^^
Sure thing! Yeah I felt similar nostalgia. I love and miss Wait But Why.
To continue the nostalgia:
Maybe you’re right though. I don’t have a good enough understanding to say. Thanks for contributing this thought.
Yeah, in retrospect that does make sense. I think I got lost in the weeds.
Sigh. Yeah, this was a crucial thing that I left out. It might even be the crux. My mistake for not talking about it in the OP.
Here are my thoughts. Descriptively, I see that temporal discounting is something that people do. But prescriptively, I don’t see why it’s something that we should do. Maybe I am just different, but when I think about, say, 100 year old me vs current 28 year old me, I don’t feel like I should prioritize that version less. Like everyone else, there is a big part of me that thinks “Ugh, let me just eat the pizza instead of the salad, forget about future me”. But when I think about what I should do, and how I should prioritize future me vs present me, I don’t really feel like there should be discounting.
That was for 100 year old me. What about 1k? 10k? 100k? Eh, maybe. The feeling of “screw it, forget that future self” is stronger. But I still question whether that is how I should weigh things. After all, 100k year old me is still just as conscious and real as 28 year old me.
Regret minimization seems relevant, although I’ve only ever taken a cursory look at the concept. If I didn’t weigh 100k year old me enough, it could be something I end up regretting.
Another perspective is the one I take at the end in the Assume The Hypothetical section. If you were going to live to 100k years old, how would you feel? For me it makes me feel like I should really value that future self.
However, once we start talking about stuff like a trillions years and 3^^^3, I start to think, “This is getting crazy, maybe we should be temporally discounting.” Why at that point and not at 100k? Idk. Because that’s what my intuition is? I guess I don’t have a great grasp on this question.
Yeah. I have a lot of respect for LessWrongers in general but I do think that there are some common failure modes, and not Taking Ideas Seriously enough is one of them. (I mean that criticism constructively not contentiously, of course.)
And I think that the reception this post has gotten is decently strong evidence that I was correct (not perfect of course; there are alternative explanations). I see this as a very important topic that doesn’t have obvious conclusions, and thus should receive more attention. Even if you are correct about temporal discounting, it is not at all obvious to me how much we should discount and how that affects the final conclusion regarding how much we should value life.
To be sure, I don’t actually think whether Accelerationism is right has any effect on the validity of your points. Indeed, there is no telling whether the AI experts from the surveys even believe in Accelerationism. A fast-takeoff model where the world experiences zero growth from now to Singularity, followed by an explosion of productivity would yield essentially the same conclusions as long as the date is the same, and so does any model in between. But I’d still like to take apart the arguments from Wait But Why just for fun:
First, exponential curves are continuous, they don’t produce singularities. This is what always confused me about Ray Kurzweil as he likes to point to the smooth exponential improvement in computing yet in the next breath predict the Singularity in 2029. You only get discontinuities when your model predicts superexponential growth, and Moore’s law is no evidence for that.
Second, while temporary deviations from the curve can be explained by noise for exponential growth, the same can’t be said so easily for superexponential growth. Here, doubling time scales with the countdown to Singularity, and what can be considered “temporary” is highly dependent on how long we have left to go. If we were in 10,000 BC, a slowing growth rate over half a century could indeed be seen as noise. But if we posit the Singularity at 2060, then we have less than 40 years left. As per Scott Alexander, world GDP doubling time has been increasing since 1960. However you look at it, the trend has been deviating from the ideal curve for far, far too long to be a mere fluke.
The most prominent example of many small S-curves adding up to an overall exponential trend line is, again, Moore’s law. From the inside view, proponents argue that doomsayers are short sighted because they only see the limits of current techniques, but such limits have appeared many times before since the dawn of computing and each time it was overcome by the introduction of a new technique. For instance, most recently, chip manufacturers have been using increasingly complex photolithography masks to print ever smaller features onto microchips using the same wavelengths of UV light, which isn’t sustainable. Then came the crucial breakthrough last year with the introduction of EUV, a novel technique that uses shorter wavelengths and allows the printing of even smaller features with a simple mask, and the refining process can start all over again.
But from the outside view, Moore’s law has buckled (notice the past tense). One by one, the trend lines have flattened out, starting with processor frequency in 2006, and most recently with transistors per dollar (Kurzweil’s favorite metric) in 2018. Proponents of Moore’s law’s validity had to keep switching metrics for 1.5 decades, and they have a few left—transistor density for instance, or TOP500 performance. But the noose is tightening, and some truly fundamental limitations such as the Landauer limit are on the horizon. As I often like to say, when straight lines run into physical limitations, physics wins.
Keep in mind that as far as Moore’s law goes, this is what death looks like. A trend line never halts abruptly, it’s always going to peter out gradually at the end.
By the way, the reason I keep heckling Moore’s law is because Moore’s law itself is the last remnant of the age of accelerating technological progress. Outside the computing industry, things are looking much more dire.
I’m not sure the prescriptive context is meaningful with regard to values. It’s like having a preference over preferences. You want whatever you want, and what you should want doesn’t matter because you don’t actually want that, wherever that should came from. A useful framework to think about this problem is to model your future self as other people and reduce it to the classic egoism-altruism balance. Would you say perfect altruism is the correct position to adopt? Are you therefore a perfect altruist?
You could make up philosophical thought experiments and such to discover how much you actually care about others, but I bet you can’t just decide to become a perfect altruist no matter how loudly a philosophy professor might scream at you. Similarly, whether you believe temporal discounting to be the right call or not in the abstract, you can’t actually stop doing it; you’re not a perfect altruist with respect to your future selves and to dismiss it would only lead to confusion in my opinion.
Yeah I agree. Seems like it just gives a small bump in the likelihood of living to the singularity, but that has a very small impact on the larger question, because the larger question is a lot more sensitive to other variables like how long a lifespan you’d expect post-singularity, and how much if any temporal discounting should be done.
To the rest of your points about exponential growth, I unfortunately don’t understand it enough to really be able to respond, sorry.
First of all, thank you very much for engaging with me here. This is exactly the sort of thing I was hoping to get in the comments. A good critique that I hadn’t thought (enough) about, and one that hits my central point hard, rather than just hitting tangential points that don’t have much influence on the central point (although I appreciate those too). I also think you’re expressing yourself very clearly, which makes it pleasant to engage with.
The more I think about it, the more I think I should apply some temporal discounting. However, I still lean towards not doing it too much.
In some theoretical sense, I agree that rationality can’t tell you what to value, only how to achieve your values (as well as how to figure out what is true). But in a more practical sense, I think that often times you can examine your values and realize that, well, I shouldn’t say “there is good reason to change them”, but I guess I could say “you find yourself inspired to change them” or “they’ve just been changed”. Like you mention, thought experiments can be a great tool, but I think it’s more than just that they help you discover things. I think they can inspire you to change your values. I do agree that it isn’t really something that you can just decide to change though.
As an example, consider an immature teenager who doesn’t care at all about his future self and just wants to have fun right now. Would you say, “Well, he values what he values.”? Probably not.
So then, I think this question of temporal discounting is really one that needs to be explored. It’s not enough to just say, “10k years from now? I don’t care about that.” Maybe we’re being immature teenagers.
Taken at face value, this statement doesn’t make much sense because it immediately begs the question of change according to what, and in what sense isn’t that change part of your value already. My guess here is that your mental model says something like “there’s a set of primal drives inside my head like eating pizza that I call ‘values’, and then there are my ‘true’ values like a healthy lifestyle which my conscious, rational mind posits, and I should change my primal drives to match my ‘true’ values” (pardon for straw-manning your position, but I need it to make my point).
A much better model in my opinion would be that all these values belong to the same exact category. These “values” or “drives” then duke it out amongst each other, and your conscious mind merely observes and makes up a plausible-sounding socially-acceptable story about your motivations (this is, after all, the evolutionary function of human intelligence in the first place as far as I know), like a press secretary sitting silently in the corner while generals are having a heated debate.
At best, your conscious mind might act as a mediator between these generals, coming up with clever ideas that pushes the Pareto boundary of these competing values so that they can all be satisfied to a greater degree at the same time. Things like “let’s try e-cigarettes instead of regular tobacco—maybe it satisfies both our craving for nicotine and our long-term health!”.
Even high-falutin values like altruism or long-term health are induced by basic drives like empathy and social status. They are no different to, say, food cravings, not even in terms of inferential distance. Compare for instance “I ate pizza, it was tasty and I felt good” with “I was chastised for eating unhealthily, it felt bad”. Is there really any important difference here?
You could of course deny this categorization and insist that only a part of this value set represents your true values. The danger here isn’t that you’ll end up optimizing for the wrong set of values since who’s to tell you what “wrong” is, it’s that you’ll be perpetually confused about why you keeping failing to act upon your declared “true” values—why your revealed preferences through behavior keep diverging from the stated preferences, and you end up making bad decisions. Decisions that are suboptimal even when judged only against your “true” values, because you have not been leveraging your conscious, rational mind properly by giving it bad epistemics.
Haha, unfortunately you posed the question to the one guy out of 100 who would gladly answer “Absolutely”, followed by “What’s wrong with being an immature teenager?”
On a more serious note, it is true that our values often shift over time, but it’s unclear to me why that makes regret minimization the correct heuristic. Regret can occur in two ways: One is that we have better information later in life, along the lines of “Oh I should have picked these numbers in last week’s lottery instead of the numbers I actually picked”. But this is just hindsight and useless to your current self because you don’t have access to that knowledge.
The other is through value shift, along the lines of “I just ate a whole pizza and now that my food-craving brain-subassembly has shut up my value function consists mostly of concerns for my long-term health”. Even setting temporal discounting aside, I fail to see why your post-dinner-values should take precedence over your pre-dinner-values, or for that matter why deathbed-values should take precedence over teenage-values. They are both equally real moments of conscious experience.
But, since we only ever live and make decisions in the present moment, if you happen to have just finished a pizza, you now have the opportunity to manipulate your future values to match your current values by taking actions that makes the salad option more available the next time pizza-craving comes around by e.g. shopping for ingredients. In AI lingo, you’ve just made yourself subagent-stable.
My personal anecdote is that as a teenager I did listen to the “mature adults” to study more and spend less time having fun. It was a bad decision according to both my current values and teenage-values, made out of ignorance about how the world operates.
As a final thought, I would give the meta-advice of not trying to think too deeply about normative ethics. Take AlphaGo as a cautionary tale: after 2000 years of pondering, the deepest truths of Go are revealed to be just a linear combination of a bunch of feature vectors. Quite poetic, if you ask me.
I think I may have lead us down the wrong path here. The ultimate question is the one of temporal discounting, and that question depends on how much we do/should value those post-singularity life years. If values can’t shift, then there isn’t really anything to talk about; you just ask yourself how much you value those years, and then move on. But if they can shift, and you acknowledge that they can, then we can discuss some thought experiments and stuff. It doesn’t seem important to discuss whether those shifts are due to discovering more about your pre-existing values, or due to actually changing those pre-existing values.
Ah, I see. You and I probably just have very different intuitions regarding what to value then, and I sense that thought experiments won’t bring us much closer.
Actually, I wonder what you think of this. Are you someone who sees death as a wildly terrible thing (I am)? If so, isn’t it because you place a correspondingly high value on the years of life you’d be losing?
In the pizza example, I think the value shift would moreso be along the lines of “I was prioritizing my current self too much relative to my future selves”. Presumably, post-dinner-values would be incorporating pre-dinner-self. Eg. it wouldn’t just say, “Screw my past self, my values are only about the present moment and onwards.” So I see your current set of values as being the most “accurate”, in which case regret minimization seems like it makes sense.
I think we’re getting closer to agreement as I’m starting to see what you’re getting at. My comment here would be that yes, your values can shift, and they have shifted after thinking hard about what post-Singularity life will be like and getting all excited. But the shift it has caused is a larger multiplier in front of the temporal discounted integral, not the disabling of temporal discounting altogether.
Yes, but I don’t think there is any layer of reasoning beneath that preference. Evading death is just something that is very much hard-coded into us by evolution.
I don’t think that’s true. Crucially, there is no knowledge being gained over the course of dinner, only value shift. It’s not like you didn’t know beforehand that pizza was unhealthy, or that you will regret your decision. And if post-dinner self does not take explicit steps to manipulate future value, the situation will repeat itself the next day, and the day after, and so on for hundreds of times.
I’m in agreement here! Some follow up questions: what are your thoughts on how much discounting should be done? Relatedly, what are your thoughts on how much we should value life? Is it obvious that past eg. 500 years, it’s far enough into the future that it becomes negligible? If not, why aren’t these things that are discussed? Also, do you share my impression that people (on LW) largely assume that life expectancy is something like 80 years and life is valued at something like $10M?
Regardless of whether it stems from a layer of reasoning or whether it is hard-coded, doesn’t it imply that you aren’t doing too much temporal discounting? If you did a lot of temporal discounting and didn’t value the years beyond eg. 250 years old very much, then death wouldn’t be that bad, right?