Sorry, I mixed up the justifications in the first two paragraphs. Or, just, both orthogonality and complexity of value are relevant to both points. In P2, I’m saying that since there’s very many utility functions and most of them are basically orthogonal to human value, and human value is pretty specific (you have to have this specific type of computation called “consciousness” for stuff to matter), the result of superintelligent optimization for most utility functions is likely (given orthogonality of values and intelligence, and given that stuff comes apart at the tails) to be irrelevant to human value. Like, Goodharting a near-miss utility function seems likely to neutralize value in the limit of superintelligent optimization. (I’m not that confident of this. Maybe consciousness that we care about is just much more common in computation-space, or much more of a natural category, so that the AIs values are likely to point to consciousness.)
And on a more poetic note, this is such a crappy time to be alive… Specially for the 1% of us who take s-risk seriously. When I take a walk, I look at the people… We could have been in a right path, you know? (At least as far as Liberal democracies are concerned). We could have been building something good, if it wasn’t for this huge monster over our heads that most are too dumb, or too coward, to believe in.
Maybe start telling people that we can’t play God is a good start. (Not at least until hundreds of years from now till we have the mathematical and the social proof to build God).
Evolution might have not been perfect, allowing things like torture due to an obsolete-for-humans (and highly abusable) survival mechanism called pain. But at least there are balances. It gave us compassion, or more skeptically the need to vomit when we see atrocities. It gave us death so you can actually escape. It gave us shock so you have a high probability of dying quick in extreme cases. There is kind of a balance, even if weak. I see the possibility of that balance being broken with AGI or even just nano by itself.
If only it was possible to implement David Pearce’s abolitionist project of anihilating the molecular substrates of below 0 hedonic level, with several safeguards. That used to be my only hope but I think chaos will arrive way first.
Why is it crappy to be alive now? If you want a nice life, now’s fairly okay, esp. compared to most of history. If you’re worried about the future, best to be in the time that most matters, which is now, so you can do the most good. It does suck that there’s all that wasted potential though.
And hey, by the Doomsday anthropic argument, we’re probably the people who take over the universe! Or something.
On acausal trade, what I meant was, if you believe that it is possible for it to work BETWEEN a human and an ASI (apparently you do?). I’ve heard people say it doesn’t, because you are not intelligent enough to model an ASI. Only an ASI is. Which is what I’m more inclined to believe, also adding that humans are not computers and therefore can’t run any type of realistic simulations. But I agree that committing to no blackmail is the correct option anyway.
On AGI timelines, do you feel safe that it’s extremely unlikely to arrive tomorrow? Do you often find yourself looking out the window for the nano-swarms, like I do? GPT-3 scares the hell out of me. Do you feel safe that it’s at least 5 years? I’d like to have a more technical understanding to have a better idea on that, which can be hard when you’re not into computer science.
I don’t think this by itself should scare you very much. Why does it scare you? When there’s a surprising capability, we have to update up up on “woah, AI is more impressive than I thought”, and also up on “oh, that task is not as hard as I thought”. It turns out that sounding natural in written text without close reading, isn’t that hard. GPT-3 is going to tell you very little that’s true and useful and that it didn’t basically read somewhere(s). It can make new puns and similar, but that should not be surprising given word2vec (which is a decade old), and it can remix stuff by shallow association (to little benefit AFAIK). If someone wants to show that it’s doing something that’s more impressive than that, I’m interested, but you have to show that it’s not “getting it directly from the dataset”, which isn’t a perfectly clear test but I think is the right sort of question. (Examples of things that don’t get their impressiveness, if they have any, “directly from the dataset”: successful RL agents; automated theorem provers.) In other words, I think “scaling laws” are probably somewhat silly (or at least, the (possibly very straw) version that I’m imagining some people think). Algorithms that we know so far all seem to hit outer limits. It’s not like AlphaZero is effectively infinitely good at Chess or Go; it got really good really fast, and then hit an asymptote. (Though this is weak evidence, because it could also have hit some sort of actual upper limit, though that seems implausible.) GPT will get as far as you can get by squeezing more shallow patterns out of text, and then basically stop, is my retro- / pre-diction.
GPT-3 by itself shouldn’t scare you very much, I think, but as part of a pattern I think it’s scary.
“GPT-3 by itself shouldn’t scare you very much, I think, but as part of a pattern I think it’s scary.”
Exactly. Combining it with other parts, like an agent and something else, like an AI researcher whose name I can’t recall said in YouTube interview that I watched (titled something like “GPT-3 is the fire alarm for AGI” (reasons: GPT-2 was kinda dumb and just scalling the model turned into something drastically better, plus the combination aspect that I mentioned).
“Why is it crappy to be alive now? If you want a nice life, now’s fairly okay, esp. compared to most of history. If you’re worried about the future, best to be in the time that most matters, which is now, so you can do the most good. It does suck that there’s all that wasted potential though.”
Well isn’t it easy to tell?? Life is certainly more comfortable now, and mine certainly has been, but there’s a) the immense gloom of knowing the current situation, I don’t think any other human in history thought he or his fellow humans might come to face s-risk scenarios (except maybe those fooled by promises of Hell in religions, but I never saw any theist seriously stressed over it)
b) the possibility of being caught in a treacherous turn ending in s-risk scenario, making you paranoid 24⁄7 and considering… Well, bad things. That vastly outweighs any comfort advantage. Specially when your personal timelines are as short as mine.
And about helping… Again, sorry for being extremely depressing, but it’s just how it is: I don’t see any hope, don’t see any way out, specially because of, again, my short timelines, say 5-10 years. I’m with Eliezer that only a miracle can save us at this point. I started praying to a benevolent creator that might be listening, started hoping for aliens to save us, started hoping for the existence of Illuminati to save us, etc. such is my despair.
Sorry, I mixed up the justifications in the first two paragraphs. Or, just, both orthogonality and complexity of value are relevant to both points. In P2, I’m saying that since there’s very many utility functions and most of them are basically orthogonal to human value, and human value is pretty specific (you have to have this specific type of computation called “consciousness” for stuff to matter), the result of superintelligent optimization for most utility functions is likely (given orthogonality of values and intelligence, and given that stuff comes apart at the tails) to be irrelevant to human value. Like, Goodharting a near-miss utility function seems likely to neutralize value in the limit of superintelligent optimization. (I’m not that confident of this. Maybe consciousness that we care about is just much more common in computation-space, or much more of a natural category, so that the AIs values are likely to point to consciousness.)
And on a more poetic note, this is such a crappy time to be alive… Specially for the 1% of us who take s-risk seriously. When I take a walk, I look at the people… We could have been in a right path, you know? (At least as far as Liberal democracies are concerned). We could have been building something good, if it wasn’t for this huge monster over our heads that most are too dumb, or too coward, to believe in.
Maybe start telling people that we can’t play God is a good start. (Not at least until hundreds of years from now till we have the mathematical and the social proof to build God).
Evolution might have not been perfect, allowing things like torture due to an obsolete-for-humans (and highly abusable) survival mechanism called pain. But at least there are balances. It gave us compassion, or more skeptically the need to vomit when we see atrocities. It gave us death so you can actually escape. It gave us shock so you have a high probability of dying quick in extreme cases. There is kind of a balance, even if weak. I see the possibility of that balance being broken with AGI or even just nano by itself.
If only it was possible to implement David Pearce’s abolitionist project of anihilating the molecular substrates of below 0 hedonic level, with several safeguards. That used to be my only hope but I think chaos will arrive way first.
Why is it crappy to be alive now? If you want a nice life, now’s fairly okay, esp. compared to most of history. If you’re worried about the future, best to be in the time that most matters, which is now, so you can do the most good. It does suck that there’s all that wasted potential though.
And hey, by the Doomsday anthropic argument, we’re probably the people who take over the universe! Or something.
Strongly agree on everything.
2 last clarifications:
On acausal trade, what I meant was, if you believe that it is possible for it to work BETWEEN a human and an ASI (apparently you do?). I’ve heard people say it doesn’t, because you are not intelligent enough to model an ASI. Only an ASI is. Which is what I’m more inclined to believe, also adding that humans are not computers and therefore can’t run any type of realistic simulations. But I agree that committing to no blackmail is the correct option anyway.
On AGI timelines, do you feel safe that it’s extremely unlikely to arrive tomorrow? Do you often find yourself looking out the window for the nano-swarms, like I do? GPT-3 scares the hell out of me. Do you feel safe that it’s at least 5 years? I’d like to have a more technical understanding to have a better idea on that, which can be hard when you’re not into computer science.
I don’t think this by itself should scare you very much. Why does it scare you? When there’s a surprising capability, we have to update up up on “woah, AI is more impressive than I thought”, and also up on “oh, that task is not as hard as I thought”. It turns out that sounding natural in written text without close reading, isn’t that hard. GPT-3 is going to tell you very little that’s true and useful and that it didn’t basically read somewhere(s). It can make new puns and similar, but that should not be surprising given word2vec (which is a decade old), and it can remix stuff by shallow association (to little benefit AFAIK). If someone wants to show that it’s doing something that’s more impressive than that, I’m interested, but you have to show that it’s not “getting it directly from the dataset”, which isn’t a perfectly clear test but I think is the right sort of question. (Examples of things that don’t get their impressiveness, if they have any, “directly from the dataset”: successful RL agents; automated theorem provers.) In other words, I think “scaling laws” are probably somewhat silly (or at least, the (possibly very straw) version that I’m imagining some people think). Algorithms that we know so far all seem to hit outer limits. It’s not like AlphaZero is effectively infinitely good at Chess or Go; it got really good really fast, and then hit an asymptote. (Though this is weak evidence, because it could also have hit some sort of actual upper limit, though that seems implausible.) GPT will get as far as you can get by squeezing more shallow patterns out of text, and then basically stop, is my retro- / pre-diction.
GPT-3 by itself shouldn’t scare you very much, I think, but as part of a pattern I think it’s scary.
“GPT-3 by itself shouldn’t scare you very much, I think, but as part of a pattern I think it’s scary.”
Exactly. Combining it with other parts, like an agent and something else, like an AI researcher whose name I can’t recall said in YouTube interview that I watched (titled something like “GPT-3 is the fire alarm for AGI” (reasons: GPT-2 was kinda dumb and just scalling the model turned into something drastically better, plus the combination aspect that I mentioned).
“Why is it crappy to be alive now? If you want a nice life, now’s fairly okay, esp. compared to most of history. If you’re worried about the future, best to be in the time that most matters, which is now, so you can do the most good. It does suck that there’s all that wasted potential though.”
Well isn’t it easy to tell?? Life is certainly more comfortable now, and mine certainly has been, but there’s a) the immense gloom of knowing the current situation, I don’t think any other human in history thought he or his fellow humans might come to face s-risk scenarios (except maybe those fooled by promises of Hell in religions, but I never saw any theist seriously stressed over it)
b) the possibility of being caught in a treacherous turn ending in s-risk scenario, making you paranoid 24⁄7 and considering… Well, bad things. That vastly outweighs any comfort advantage. Specially when your personal timelines are as short as mine.
And about helping… Again, sorry for being extremely depressing, but it’s just how it is: I don’t see any hope, don’t see any way out, specially because of, again, my short timelines, say 5-10 years. I’m with Eliezer that only a miracle can save us at this point. I started praying to a benevolent creator that might be listening, started hoping for aliens to save us, started hoping for the existence of Illuminati to save us, etc. such is my despair.