To be sure, I don’t actually think whether Accelerationism is right has any effect on the validity of your points. Indeed, there is no telling whether the AI experts from the surveys even believe in Accelerationism. A fast-takeoff model where the world experiences zero growth from now to Singularity, followed by an explosion of productivity would yield essentially the same conclusions as long as the date is the same, and so does any model in between. But I’d still like to take apart the arguments from Wait But Why just for fun:
First, exponential curves are continuous, they don’t produce singularities. This is what always confused me about Ray Kurzweil as he likes to point to the smooth exponential improvement in computing yet in the next breath predict the Singularity in 2029. You only get discontinuities when your model predicts superexponential growth, and Moore’s law is no evidence for that.
Second, while temporary deviations from the curve can be explained by noise for exponential growth, the same can’t be said so easily for superexponential growth. Here, doubling time scales with the countdown to Singularity, and what can be considered “temporary” is highly dependent on how long we have left to go. If we were in 10,000 BC, a slowing growth rate over half a century could indeed be seen as noise. But if we posit the Singularity at 2060, then we have less than 40 years left. As per Scott Alexander, world GDP doubling time has been increasing since 1960. However you look at it, the trend has been deviating from the ideal curve for far, far too long to be a mere fluke.
The most prominent example of many small S-curves adding up to an overall exponential trend line is, again, Moore’s law. From the inside view, proponents argue that doomsayers are short sighted because they only see the limits of current techniques, but such limits have appeared many times before since the dawn of computing and each time it was overcome by the introduction of a new technique. For instance, most recently, chip manufacturers have been using increasingly complex photolithography masks to print ever smaller features onto microchips using the same wavelengths of UV light, which isn’t sustainable. Then came the crucial breakthrough last year with the introduction of EUV, a novel technique that uses shorter wavelengths and allows the printing of even smaller features with a simple mask, and the refining process can start all over again.
But from the outside view, Moore’s law has buckled (notice the past tense). One by one, the trend lines have flattened out, starting with processor frequency in 2006, and most recently with transistors per dollar (Kurzweil’s favorite metric) in 2018. Proponents of Moore’s law’s validity had to keep switching metrics for 1.5 decades, and they have a few left—transistor density for instance, or TOP500 performance. But the noose is tightening, and some truly fundamental limitations such as the Landauer limit are on the horizon. As I often like to say, when straight lines run into physical limitations, physics wins.
Keep in mind that as far as Moore’s law goes, this is what death looks like. A trend line never halts abruptly, it’s always going to peter out gradually at the end.
By the way, the reason I keep heckling Moore’s law is because Moore’s law itself is the last remnant of the age of accelerating technological progress. Outside the computing industry, things are looking much more dire.
Here are my thoughts. Descriptively, I see that temporal discounting is something that people do. But prescriptively, I don’t see why it’s something that we should do. Maybe I am just different, but when I think about, say, 100 year old me vs current 28 year old me, I don’t feel like I should prioritize that version less. Like everyone else, there is a big part of me that thinks “Ugh, let me just eat the pizza instead of the salad, forget about future me”. But when I think about what I should do, and how I should prioritize future me vs present me, I don’t really feel like there should be discounting.
I’m not sure the prescriptive context is meaningful with regard to values. It’s like having a preference over preferences. You want whatever you want, and what you should want doesn’t matter because you don’t actually want that, wherever that should came from. A useful framework to think about this problem is to model your future self as other people and reduce it to the classic egoism-altruism balance. Would you say perfect altruism is the correct position to adopt? Are you therefore a perfect altruist?
You could make up philosophical thought experiments and such to discover how much you actually care about others, but I bet you can’t just decide to become a perfect altruist no matter how loudly a philosophy professor might scream at you. Similarly, whether you believe temporal discounting to be the right call or not in the abstract, you can’t actually stop doing it; you’re not a perfect altruist with respect to your future selves and to dismiss it would only lead to confusion in my opinion.
To be sure, I don’t actually think whether Accelerationism is right has any effect on the validity of your points.
Yeah I agree. Seems like it just gives a small bump in the likelihood of living to the singularity, but that has a very small impact on the larger question, because the larger question is a lot more sensitive to other variables like how long a lifespan you’d expect post-singularity, and how much if any temporal discounting should be done.
To the rest of your points about exponential growth, I unfortunately don’t understand it enough to really be able to respond, sorry.
I’m not sure the prescriptive context is meaningful with regard to values. It’s like having a preference over preferences. You want whatever you want, and what you should want doesn’t matter because you don’t actually want that, wherever that should came from.
First of all, thank you very much for engaging with me here. This is exactly the sort of thing I was hoping to get in the comments. A good critique that I hadn’t thought (enough) about, and one that hits my central point hard, rather than just hitting tangential points that don’t have much influence on the central point (although I appreciate those too). I also think you’re expressing yourself very clearly, which makes it pleasant to engage with.
The more I think about it, the more I think I should apply some temporal discounting. However, I still lean towards not doing it too much.
In some theoretical sense, I agree that rationality can’t tell you what to value, only how to achieve your values (as well as how to figure out what is true). But in a more practical sense, I think that often times you can examine your values and realize that, well, I shouldn’t say “there is good reason to change them”, but I guess I could say “you find yourself inspired to change them” or “they’ve just been changed”. Like you mention, thought experiments can be a great tool, but I think it’s more than just that they help you discover things. I think they can inspire you to change your values. I do agree that it isn’t really something that you can just decide to change though.
As an example, consider an immature teenager who doesn’t care at all about his future self and just wants to have fun right now. Would you say, “Well, he values what he values.”? Probably not.
So then, I think this question of temporal discounting is really one that needs to be explored. It’s not enough to just say, “10k years from now? I don’t care about that.” Maybe we’re being immature teenagers.
I think they can inspire you to change your values.
Taken at face value, this statement doesn’t make much sense because it immediately begs the question of change according to what, and in what sense isn’t that change part of your value already. My guess here is that your mental model says something like “there’s a set of primal drives inside my head like eating pizza that I call ‘values’, and then there are my ‘true’ values like a healthy lifestyle which my conscious, rational mind posits, and I should change my primal drives to match my ‘true’ values” (pardon for straw-manning your position, but I need it to make my point).
A much better model in my opinion would be that all these values belong to the same exact category. These “values” or “drives” then duke it out amongst each other, and your conscious mind merely observes and makes up a plausible-sounding socially-acceptable story about your motivations (this is, after all, the evolutionary function of human intelligence in the first place as far as I know), like a press secretary sitting silently in the corner while generals are having a heated debate.
At best, your conscious mind might act as a mediator between these generals, coming up with clever ideas that pushes the Pareto boundary of these competing values so that they can all be satisfied to a greater degree at the same time. Things like “let’s try e-cigarettes instead of regular tobacco—maybe it satisfies both our craving for nicotine and our long-term health!”.
Even high-falutin values like altruism or long-term health are induced by basic drives like empathy and social status. They are no different to, say, food cravings, not even in terms of inferential distance. Compare for instance “I ate pizza, it was tasty and I felt good” with “I was chastised for eating unhealthily, it felt bad”. Is there really any important difference here?
You could of course deny this categorization and insist that only a part of this value set represents your true values. The danger here isn’t that you’ll end up optimizing for the wrong set of values since who’s to tell you what “wrong” is, it’s that you’ll be perpetually confused about why you keeping failing to act upon your declared “true” values—why your revealed preferences through behavior keep diverging from the stated preferences, and you end up making bad decisions. Decisions that are suboptimal even when judged only against your “true” values, because you have not been leveraging your conscious, rational mind properly by giving it bad epistemics.
As an example, consider an immature teenager who doesn’t care at all about his future self and just wants to have fun right now. Would you say, “Well, he values what he values.”?
Haha, unfortunately you posed the question to the one guy out of 100 who would gladly answer “Absolutely”, followed by “What’s wrong with being an immature teenager?”
On a more serious note, it is true that our values often shift over time, but it’s unclear to me why that makes regret minimization the correct heuristic. Regret can occur in two ways: One is that we have better information later in life, along the lines of “Oh I should have picked these numbers in last week’s lottery instead of the numbers I actually picked”. But this is just hindsight and useless to your current self because you don’t have access to that knowledge.
The other is through value shift, along the lines of “I just ate a whole pizza and now that my food-craving brain-subassembly has shut up my value function consists mostly of concerns for my long-term health”. Even setting temporal discounting aside, I fail to see why your post-dinner-values should take precedence over your pre-dinner-values, or for that matter why deathbed-values should take precedence over teenage-values. They are both equally real moments of conscious experience.
But, since we only ever live and make decisions in the present moment, if you happen to have just finished a pizza, you now have the opportunity to manipulate your future values to match your current values by taking actions that makes the salad option more available the next time pizza-craving comes around by e.g. shopping for ingredients. In AI lingo, you’ve just made yourself subagent-stable.
My personal anecdote is that as a teenager I did listen to the “mature adults” to study more and spend less time having fun. It was a bad decision according to both my current values and teenage-values, made out of ignorance about how the world operates.
As a final thought, I would give the meta-advice of not trying to think too deeply about normative ethics. Take AlphaGo as a cautionary tale: after 2000 years of pondering, the deepest truths of Go are revealed to be just a linear combination of a bunch of feature vectors. Quite poetic, if you ask me.
I think I may have lead us down the wrong path here. The ultimate question is the one of temporal discounting, and that question depends on how much we do/should value those post-singularity life years. If values can’t shift, then there isn’t really anything to talk about; you just ask yourself how much you value those years, and then move on. But if they can shift, and you acknowledge that they can, then we can discuss some thought experiments and stuff. It doesn’t seem important to discuss whether those shifts are due to discovering more about your pre-existing values, or due to actually changing those pre-existing values.
Haha, unfortunately you posed the question to the one guy out of 100 who would gladly answer “Absolutely”, followed by “What’s wrong with being an immature teenager?”
Ah, I see. You and I probably just have very different intuitions regarding what to value then, and I sense that thought experiments won’t bring us much closer.
Actually, I wonder what you think of this. Are you someone who sees death as a wildlyterriblething (I am)? If so, isn’t it because you place a correspondingly high value on the years of life you’d be losing?
The other is through value shift, along the lines of “I just ate a whole pizza and now that my food-craving brain-subassembly has shut up my value function consists mostly of concerns for my long-term health”. Even setting temporal discounting aside, I fail to see why your post-dinner-values should take precedence over your pre-dinner-values, or for that matter why deathbed-values should take precedence over teenage-values. They are both equally real moments of conscious experience.
In the pizza example, I think the value shift would moreso be along the lines of “I was prioritizing my current self too much relative to my future selves”. Presumably, post-dinner-values would be incorporating pre-dinner-self. Eg. it wouldn’t just say, “Screw my past self, my values are only about the present moment and onwards.” So I see your current set of values as being the most “accurate”, in which case regret minimization seems like it makes sense.
The ultimate question is the one of temporal discounting, and that question depends on how much we do/should value those post-singularity life years. If values can’t shift, then there isn’t really anything to talk about; you just ask yourself how much you value those years, and then move on. But if they can shift, and you acknowledge that they can, then we can discuss some thought experiments and stuff.
I think we’re getting closer to agreement as I’m starting to see what you’re getting at. My comment here would be that yes, your values can shift, and they have shifted after thinking hard about what post-Singularity life will be like and getting all excited. But the shift it has caused is a larger multiplier in front of the temporal discounted integral, not the disabling of temporal discounting altogether.
Actually, I wonder what you think of this. Are you someone who sees death as a wildlyterriblething (I am)?
Yes, but I don’t think there is any layer of reasoning beneath that preference. Evading death is just something that is very much hard-coded into us by evolution.
In the pizza example, I think the value shift would moreso be along the lines of “I was prioritizing my current self too much relative to my future selves”. Presumably, post-dinner-values would be incorporating pre-dinner-self.
I don’t think that’s true. Crucially, there is no knowledge being gained over the course of dinner, only value shift. It’s not like you didn’t know beforehand that pizza was unhealthy, or that you will regret your decision. And if post-dinner self does not take explicit steps to manipulate future value, the situation will repeat itself the next day, and the day after, and so on for hundreds of times.
I think we’re getting closer to agreement as I’m starting to see what you’re getting at. My comment here would be that yes, your values can shift, and they have shifted after thinking hard about what post-Singularity life will be like and getting all excited. But the shift it has caused is a larger multiplier in front of the temporal discounted integral, not the disabling of temporal discounting altogether.
I’m in agreement here! Some follow up questions: what are your thoughts on how much discounting should be done? Relatedly, what are your thoughts on how much we should value life? Is it obvious that past eg. 500 years, it’s far enough into the future that it becomes negligible? If not, why aren’t these things that are discussed? Also, do you share my impression that people (on LW) largely assume that life expectancy is something like 80 years and life is valued at something like $10M?
Yes, but I don’t think there is any layer of reasoning beneath that preference. Evading death is just something that is very much hard-coded into us by evolution.
Regardless of whether it stems from a layer of reasoning or whether it is hard-coded, doesn’t it imply that you aren’t doing too much temporal discounting? If you did a lot of temporal discounting and didn’t value the years beyond eg. 250 years old very much, then death wouldn’t be that bad, right?
To be sure, I don’t actually think whether Accelerationism is right has any effect on the validity of your points. Indeed, there is no telling whether the AI experts from the surveys even believe in Accelerationism. A fast-takeoff model where the world experiences zero growth from now to Singularity, followed by an explosion of productivity would yield essentially the same conclusions as long as the date is the same, and so does any model in between. But I’d still like to take apart the arguments from Wait But Why just for fun:
First, exponential curves are continuous, they don’t produce singularities. This is what always confused me about Ray Kurzweil as he likes to point to the smooth exponential improvement in computing yet in the next breath predict the Singularity in 2029. You only get discontinuities when your model predicts superexponential growth, and Moore’s law is no evidence for that.
Second, while temporary deviations from the curve can be explained by noise for exponential growth, the same can’t be said so easily for superexponential growth. Here, doubling time scales with the countdown to Singularity, and what can be considered “temporary” is highly dependent on how long we have left to go. If we were in 10,000 BC, a slowing growth rate over half a century could indeed be seen as noise. But if we posit the Singularity at 2060, then we have less than 40 years left. As per Scott Alexander, world GDP doubling time has been increasing since 1960. However you look at it, the trend has been deviating from the ideal curve for far, far too long to be a mere fluke.
The most prominent example of many small S-curves adding up to an overall exponential trend line is, again, Moore’s law. From the inside view, proponents argue that doomsayers are short sighted because they only see the limits of current techniques, but such limits have appeared many times before since the dawn of computing and each time it was overcome by the introduction of a new technique. For instance, most recently, chip manufacturers have been using increasingly complex photolithography masks to print ever smaller features onto microchips using the same wavelengths of UV light, which isn’t sustainable. Then came the crucial breakthrough last year with the introduction of EUV, a novel technique that uses shorter wavelengths and allows the printing of even smaller features with a simple mask, and the refining process can start all over again.
But from the outside view, Moore’s law has buckled (notice the past tense). One by one, the trend lines have flattened out, starting with processor frequency in 2006, and most recently with transistors per dollar (Kurzweil’s favorite metric) in 2018. Proponents of Moore’s law’s validity had to keep switching metrics for 1.5 decades, and they have a few left—transistor density for instance, or TOP500 performance. But the noose is tightening, and some truly fundamental limitations such as the Landauer limit are on the horizon. As I often like to say, when straight lines run into physical limitations, physics wins.
Keep in mind that as far as Moore’s law goes, this is what death looks like. A trend line never halts abruptly, it’s always going to peter out gradually at the end.
By the way, the reason I keep heckling Moore’s law is because Moore’s law itself is the last remnant of the age of accelerating technological progress. Outside the computing industry, things are looking much more dire.
I’m not sure the prescriptive context is meaningful with regard to values. It’s like having a preference over preferences. You want whatever you want, and what you should want doesn’t matter because you don’t actually want that, wherever that should came from. A useful framework to think about this problem is to model your future self as other people and reduce it to the classic egoism-altruism balance. Would you say perfect altruism is the correct position to adopt? Are you therefore a perfect altruist?
You could make up philosophical thought experiments and such to discover how much you actually care about others, but I bet you can’t just decide to become a perfect altruist no matter how loudly a philosophy professor might scream at you. Similarly, whether you believe temporal discounting to be the right call or not in the abstract, you can’t actually stop doing it; you’re not a perfect altruist with respect to your future selves and to dismiss it would only lead to confusion in my opinion.
Yeah I agree. Seems like it just gives a small bump in the likelihood of living to the singularity, but that has a very small impact on the larger question, because the larger question is a lot more sensitive to other variables like how long a lifespan you’d expect post-singularity, and how much if any temporal discounting should be done.
To the rest of your points about exponential growth, I unfortunately don’t understand it enough to really be able to respond, sorry.
First of all, thank you very much for engaging with me here. This is exactly the sort of thing I was hoping to get in the comments. A good critique that I hadn’t thought (enough) about, and one that hits my central point hard, rather than just hitting tangential points that don’t have much influence on the central point (although I appreciate those too). I also think you’re expressing yourself very clearly, which makes it pleasant to engage with.
The more I think about it, the more I think I should apply some temporal discounting. However, I still lean towards not doing it too much.
In some theoretical sense, I agree that rationality can’t tell you what to value, only how to achieve your values (as well as how to figure out what is true). But in a more practical sense, I think that often times you can examine your values and realize that, well, I shouldn’t say “there is good reason to change them”, but I guess I could say “you find yourself inspired to change them” or “they’ve just been changed”. Like you mention, thought experiments can be a great tool, but I think it’s more than just that they help you discover things. I think they can inspire you to change your values. I do agree that it isn’t really something that you can just decide to change though.
As an example, consider an immature teenager who doesn’t care at all about his future self and just wants to have fun right now. Would you say, “Well, he values what he values.”? Probably not.
So then, I think this question of temporal discounting is really one that needs to be explored. It’s not enough to just say, “10k years from now? I don’t care about that.” Maybe we’re being immature teenagers.
Taken at face value, this statement doesn’t make much sense because it immediately begs the question of change according to what, and in what sense isn’t that change part of your value already. My guess here is that your mental model says something like “there’s a set of primal drives inside my head like eating pizza that I call ‘values’, and then there are my ‘true’ values like a healthy lifestyle which my conscious, rational mind posits, and I should change my primal drives to match my ‘true’ values” (pardon for straw-manning your position, but I need it to make my point).
A much better model in my opinion would be that all these values belong to the same exact category. These “values” or “drives” then duke it out amongst each other, and your conscious mind merely observes and makes up a plausible-sounding socially-acceptable story about your motivations (this is, after all, the evolutionary function of human intelligence in the first place as far as I know), like a press secretary sitting silently in the corner while generals are having a heated debate.
At best, your conscious mind might act as a mediator between these generals, coming up with clever ideas that pushes the Pareto boundary of these competing values so that they can all be satisfied to a greater degree at the same time. Things like “let’s try e-cigarettes instead of regular tobacco—maybe it satisfies both our craving for nicotine and our long-term health!”.
Even high-falutin values like altruism or long-term health are induced by basic drives like empathy and social status. They are no different to, say, food cravings, not even in terms of inferential distance. Compare for instance “I ate pizza, it was tasty and I felt good” with “I was chastised for eating unhealthily, it felt bad”. Is there really any important difference here?
You could of course deny this categorization and insist that only a part of this value set represents your true values. The danger here isn’t that you’ll end up optimizing for the wrong set of values since who’s to tell you what “wrong” is, it’s that you’ll be perpetually confused about why you keeping failing to act upon your declared “true” values—why your revealed preferences through behavior keep diverging from the stated preferences, and you end up making bad decisions. Decisions that are suboptimal even when judged only against your “true” values, because you have not been leveraging your conscious, rational mind properly by giving it bad epistemics.
Haha, unfortunately you posed the question to the one guy out of 100 who would gladly answer “Absolutely”, followed by “What’s wrong with being an immature teenager?”
On a more serious note, it is true that our values often shift over time, but it’s unclear to me why that makes regret minimization the correct heuristic. Regret can occur in two ways: One is that we have better information later in life, along the lines of “Oh I should have picked these numbers in last week’s lottery instead of the numbers I actually picked”. But this is just hindsight and useless to your current self because you don’t have access to that knowledge.
The other is through value shift, along the lines of “I just ate a whole pizza and now that my food-craving brain-subassembly has shut up my value function consists mostly of concerns for my long-term health”. Even setting temporal discounting aside, I fail to see why your post-dinner-values should take precedence over your pre-dinner-values, or for that matter why deathbed-values should take precedence over teenage-values. They are both equally real moments of conscious experience.
But, since we only ever live and make decisions in the present moment, if you happen to have just finished a pizza, you now have the opportunity to manipulate your future values to match your current values by taking actions that makes the salad option more available the next time pizza-craving comes around by e.g. shopping for ingredients. In AI lingo, you’ve just made yourself subagent-stable.
My personal anecdote is that as a teenager I did listen to the “mature adults” to study more and spend less time having fun. It was a bad decision according to both my current values and teenage-values, made out of ignorance about how the world operates.
As a final thought, I would give the meta-advice of not trying to think too deeply about normative ethics. Take AlphaGo as a cautionary tale: after 2000 years of pondering, the deepest truths of Go are revealed to be just a linear combination of a bunch of feature vectors. Quite poetic, if you ask me.
I think I may have lead us down the wrong path here. The ultimate question is the one of temporal discounting, and that question depends on how much we do/should value those post-singularity life years. If values can’t shift, then there isn’t really anything to talk about; you just ask yourself how much you value those years, and then move on. But if they can shift, and you acknowledge that they can, then we can discuss some thought experiments and stuff. It doesn’t seem important to discuss whether those shifts are due to discovering more about your pre-existing values, or due to actually changing those pre-existing values.
Ah, I see. You and I probably just have very different intuitions regarding what to value then, and I sense that thought experiments won’t bring us much closer.
Actually, I wonder what you think of this. Are you someone who sees death as a wildly terrible thing (I am)? If so, isn’t it because you place a correspondingly high value on the years of life you’d be losing?
In the pizza example, I think the value shift would moreso be along the lines of “I was prioritizing my current self too much relative to my future selves”. Presumably, post-dinner-values would be incorporating pre-dinner-self. Eg. it wouldn’t just say, “Screw my past self, my values are only about the present moment and onwards.” So I see your current set of values as being the most “accurate”, in which case regret minimization seems like it makes sense.
I think we’re getting closer to agreement as I’m starting to see what you’re getting at. My comment here would be that yes, your values can shift, and they have shifted after thinking hard about what post-Singularity life will be like and getting all excited. But the shift it has caused is a larger multiplier in front of the temporal discounted integral, not the disabling of temporal discounting altogether.
Yes, but I don’t think there is any layer of reasoning beneath that preference. Evading death is just something that is very much hard-coded into us by evolution.
I don’t think that’s true. Crucially, there is no knowledge being gained over the course of dinner, only value shift. It’s not like you didn’t know beforehand that pizza was unhealthy, or that you will regret your decision. And if post-dinner self does not take explicit steps to manipulate future value, the situation will repeat itself the next day, and the day after, and so on for hundreds of times.
I’m in agreement here! Some follow up questions: what are your thoughts on how much discounting should be done? Relatedly, what are your thoughts on how much we should value life? Is it obvious that past eg. 500 years, it’s far enough into the future that it becomes negligible? If not, why aren’t these things that are discussed? Also, do you share my impression that people (on LW) largely assume that life expectancy is something like 80 years and life is valued at something like $10M?
Regardless of whether it stems from a layer of reasoning or whether it is hard-coded, doesn’t it imply that you aren’t doing too much temporal discounting? If you did a lot of temporal discounting and didn’t value the years beyond eg. 250 years old very much, then death wouldn’t be that bad, right?