I don’t think “Eliezer is wrong about something” implies that the field is ridiculous or easily solved. Many people in the field (including the plurality with meaningful AI experience) disagree with Eliezer’s basic perspective and think he is wildly overconfident.
I basically agree with Eliezer that if arguments for doom look convincing then you should focus on something more like improving the log odds of survival, and preparing to do the hard work to take advantage of the actual kind of miracles that might actually occur, rather than starting to live in fantasy-land where things are not as they seem. But “let’s just assume this whole alignment thing isn’t real” is a particularly extreme instance of the behavior that Eliezer is criticizing.
So to get there, I think you’d have to both reject Eliezer’s basic perspective, and take Eliezer as representing the entire field so strongly that rejecting his perspective means rejecting the whole field. I don’t think that’s reasonable, unless your interest in AI safety was primarily driven by deference to Eliezer. If so, I do think it makes sense to stop deferring and just figure out how you want to help the world, and it’s reasonable if that’s not AI safety.
As an aside, if your goal is earning to give, I think it currently looks unwise to go into distributed systems or cryptography rather than AI (though I don’t know your situation, this is just based on similarly-skilled people making much more money right now in AI and that looking like it’s going to hold up or get even more extreme over the next 20 years).
But “let’s just assume this whole alignment thing isn’t real” is a particularly extreme instance of the behavior that Eliezer is criticizing.
I endorse this response, and want to highlight it. Specifically, there are two separate errors in “if Eliezer, an expert in that field. is correct, then it’s hopeless for us to try. So our only hope is that he’s wrong and the field is somewhat ridiculous or easily solved.”:
1. The post doesn’t say it’s hopeless to try (‘low probability of success’ is not equivalent to ‘negligible probablity of success’).
2. Per my comment in another thread, the exact mental move the OP is criticizing is ‘if you’re right then we’re screwed regardless, so we might as well adopt a bunch of more-optimistic assumptions’.
What I’d recommend is just to start building your own mental models of AI risk, especially if your initial reason for concern was mainly based on deference to others.
If you get enough evidence to decide that alignment is difficult and necessary for an awesome future (based on a mix of your more-fleshed-out models of the object level and your more-fleshed-out models of which people tend to be right about which things), then work on it insofar as this is higher-EV than whatever you’d be doing instead, even if you end up agreeing with Eliezer that the absolute odds of success are low.
In the process, be wary of flailing and of rushing to epistemic closure; and see also https://www.lesswrong.com/posts/st7DiQP23YQSxumCt/on-doing-the-improbable. That post was about people preemptively giving up even in the face of moderate difficulties, where it’s especially obvious that the psychological processes at work are maladaptive; but the same processes are also in play when there are larger difficulties, even if they’re harder to spot because the problem genuinely is harder and less obviously tractable.
I think you’d have to both reject Eliezer’s basic perspective, and take Eliezer as representing the entire field so strongly that rejecting his perspective means rejecting the whole field.
I think this is mostly false, in that it’s not sufficient for ‘I shouldn’t work on AI alignment’ (Eliezer can be wrong about a lot of core things and it still be the case that AI alignment is important and difficult), and in that it doesn’t connect up to joraine’s ‘I infer from this post that I shouldn’t work on AI alignment’.
I don’t think that’s reasonable, unless your interest in AI safety was primarily driven by deference to Eliezer.
Likewise, this seems false to me because it suggests that past Eliezer-deference is sufficient to justify ‘alignment isn’t important if Eliezer is wrong’ (which it’s not), and/or it suggests that Eliezer-deferential people should conclude from the OP ‘ah, Eliezer is wrong about AI’ (without appeal to your own personal model of AI that disagrees with Eliezer).
‘Our odds of success were pretty low and have gotten lower’ is just a possible state of the world; it isn’t a ridiculous-on-its-face view. So while there are reasonable paths to arriving at the conclusion ‘I shouldn’t work on alignment’, I don’t think any of them have much to do with this post.
I guess the most reasonable would be ‘I don’t care that much about longevity or the long-term future, so I’ll work on alignment if I think it’s easy but not if I think it’s hard; and the OP updated me toward thinking it’s hard; so now I’m going to go play video games and chill instead of worrying about the long-term future.’
If so, I do think it makes sense to stop deferring
I don’t know what specific things ‘If so’ is referring back to, and I also recommended trying to build one’s own models of the space. But it again seems to me like this can be misinterpreted as saying ‘the OP contains an obvious reasoning error such that if you came to the post thinking EY was right about p(doom), you should leave it thinking he’s wrong about p(doom)’. That seems obviously wrong to me, whether or not EY is actually correct about p(doom).
The reason you should build your own models of this space is because there’s high expected information value (both for you and for others) in figuring out what’s up here. If you’re a smart technical person who hasn’t done much of their own model-building about AI, then IMO that’s very likely to be true regardless of what your current p(doom) is.
As an aside, if your goal is earning to give, I think it currently looks unwise to go into distributed systems or cryptography rather than AI
True, though the other two are much less likely to cause an increase in existential risk. I would pretty strongly prefer we don’t randomly send people who are interested in helping the world to work on the single most dangerous technology that humanity is developing, without a plan for how to make it go better. My current guess is it’s worth it to just buy out marginal people, especially people with somewhat of an interest in AGI, from working in AI at all, so going into AI seems pretty net-negative to me, even if you make a good amount of money.
(I think one could make an argument that general non-AGI related AI work isn’t that bad for the world, but my sense is that a lot of the highest-paying AI jobs have some pretty decent impact on AI capabilities, either by directly involving research, or by substantially increasing the amount of commercialization and therefore funding and talent that goes into AI. I bet there are some AI jobs you can find that won’t make any substantial difference on AGI, so if you keep that in mind and are pretty conservative with regards to improving commercialization of AI technologies or attracting more talent to the field, you might be fine.)
Apologize for long wall of text, at the bottom I dived into your aside more as that’s highly relevant to deciding the course of my next 10 years and would appreciate your weighing-in.
Pre-Lesswrong/my entire life I’ve been really interested in longevity, and I would do anything to help people have more time with their loved ones (and as a child I thought solving this was the only worthy kind of fame I’d ever want.) I didn’t know how to get there, but it was probably somewhere in math and science so I decided I had to do anything to get into MIT. My hobbies ended up being CS-y instead of biology-y, and I realized that not only was CS profitable for earn to give, but it also might be the best shot for longevity since AI was just infinitely better at problem solving. So that’s where my AI interest comes from. Not in being afraid of it but in using it to solve mortal problems. But the AI safety thing is something that I of course just hear smart people like Eliezer mention and then I think to myself “hmm well they know more about AI than me and I can’t use it to cure aging without the AI also maybe destroying us so I should look into that.”
Your crypto comment is surprising though and I’d like to go further on that. I should be more clear, I’m pretty interested in cryptocurrency not just cryptography and so far trading it has been really profitable, and this summer I’m essentially trying to decide if I’ll stop my schooling to do a crypto startup or if I’ll do my Masters in AI (or potentially also a crypto thing).
Startups seem like the best thing to do for profit and people are falling over themselves to fund them nowadays so I assumed given how many people have offered me funding to do so, that the crypto startup thing would be far easier to profit from than an ML startup (with ML maybe overtaking it in 7 years or so)
If this isn’t the case, or we’re a year away from the flip to ML being the easier startup, I’d love to know, because I’m right on the precipice between pursuing as much ML knowledge as I can and trying to get a pHd (probably eventually do an ML spin-off), versus trying to crypto startup earn-to-give ala FTX Sam.
My claim about AI vs crypto was just a misunderstanding. I still think of “cryptography” and “distributed systems” with their historical meaning rather than “cryptocurrency startup” or “cryptocurrency trading,” but in the context of earning to give I think that should have been clear to me :)
I’d still guess an AI career is generally the better way to make money, but I don’t have a strong take / think it depends on the person and situation / am no longer confused by your position.
Crypto generally has been unusually profitable (as a career or investment), but much of this simply stems from the rising tide raising all boats. Given that crypto market caps can only grow about another 10x or so before approaching major currency status and financial system parity at which point growth will slow—it seems likely that much of the glory days of crypto are behind us.
AI/ML is pretty clearly the big thing happening on Earth.
Admittedly in the short term it may be easier to cash in on a quick crypto career, but since most startups fail, consider whether you’d rather try but fail to have an impact on AI vs trying but failing at crypto wealth.
I don’t think “Eliezer is wrong about something” implies that the field is ridiculous or easily solved. Many people in the field (including the plurality with meaningful AI experience) disagree with Eliezer’s basic perspective and think he is wildly overconfident.
I basically agree with Eliezer that if arguments for doom look convincing then you should focus on something more like improving the log odds of survival, and preparing to do the hard work to take advantage of the actual kind of miracles that might actually occur, rather than starting to live in fantasy-land where things are not as they seem. But “let’s just assume this whole alignment thing isn’t real” is a particularly extreme instance of the behavior that Eliezer is criticizing.
So to get there, I think you’d have to both reject Eliezer’s basic perspective, and take Eliezer as representing the entire field so strongly that rejecting his perspective means rejecting the whole field. I don’t think that’s reasonable, unless your interest in AI safety was primarily driven by deference to Eliezer. If so, I do think it makes sense to stop deferring and just figure out how you want to help the world, and it’s reasonable if that’s not AI safety.
As an aside, if your goal is earning to give, I think it currently looks unwise to go into distributed systems or cryptography rather than AI (though I don’t know your situation, this is just based on similarly-skilled people making much more money right now in AI and that looking like it’s going to hold up or get even more extreme over the next 20 years).
I endorse this response, and want to highlight it. Specifically, there are two separate errors in “if Eliezer, an expert in that field. is correct, then it’s hopeless for us to try. So our only hope is that he’s wrong and the field is somewhat ridiculous or easily solved.”:
1. The post doesn’t say it’s hopeless to try (‘low probability of success’ is not equivalent to ‘negligible probablity of success’).
2. Per my comment in another thread, the exact mental move the OP is criticizing is ‘if you’re right then we’re screwed regardless, so we might as well adopt a bunch of more-optimistic assumptions’.
What I’d recommend is just to start building your own mental models of AI risk, especially if your initial reason for concern was mainly based on deference to others.
If you get enough evidence to decide that alignment is difficult and necessary for an awesome future (based on a mix of your more-fleshed-out models of the object level and your more-fleshed-out models of which people tend to be right about which things), then work on it insofar as this is higher-EV than whatever you’d be doing instead, even if you end up agreeing with Eliezer that the absolute odds of success are low.
In the process, be wary of flailing and of rushing to epistemic closure; and see also https://www.lesswrong.com/posts/st7DiQP23YQSxumCt/on-doing-the-improbable. That post was about people preemptively giving up even in the face of moderate difficulties, where it’s especially obvious that the psychological processes at work are maladaptive; but the same processes are also in play when there are larger difficulties, even if they’re harder to spot because the problem genuinely is harder and less obviously tractable.
I think this is mostly false, in that it’s not sufficient for ‘I shouldn’t work on AI alignment’ (Eliezer can be wrong about a lot of core things and it still be the case that AI alignment is important and difficult), and in that it doesn’t connect up to joraine’s ‘I infer from this post that I shouldn’t work on AI alignment’.
Likewise, this seems false to me because it suggests that past Eliezer-deference is sufficient to justify ‘alignment isn’t important if Eliezer is wrong’ (which it’s not), and/or it suggests that Eliezer-deferential people should conclude from the OP ‘ah, Eliezer is wrong about AI’ (without appeal to your own personal model of AI that disagrees with Eliezer).
‘Our odds of success were pretty low and have gotten lower’ is just a possible state of the world; it isn’t a ridiculous-on-its-face view. So while there are reasonable paths to arriving at the conclusion ‘I shouldn’t work on alignment’, I don’t think any of them have much to do with this post.
I guess the most reasonable would be ‘I don’t care that much about longevity or the long-term future, so I’ll work on alignment if I think it’s easy but not if I think it’s hard; and the OP updated me toward thinking it’s hard; so now I’m going to go play video games and chill instead of worrying about the long-term future.’
I don’t know what specific things ‘If so’ is referring back to, and I also recommended trying to build one’s own models of the space. But it again seems to me like this can be misinterpreted as saying ‘the OP contains an obvious reasoning error such that if you came to the post thinking EY was right about p(doom), you should leave it thinking he’s wrong about p(doom)’. That seems obviously wrong to me, whether or not EY is actually correct about p(doom).
The reason you should build your own models of this space is because there’s high expected information value (both for you and for others) in figuring out what’s up here. If you’re a smart technical person who hasn’t done much of their own model-building about AI, then IMO that’s very likely to be true regardless of what your current p(doom) is.
True, though the other two are much less likely to cause an increase in existential risk. I would pretty strongly prefer we don’t randomly send people who are interested in helping the world to work on the single most dangerous technology that humanity is developing, without a plan for how to make it go better. My current guess is it’s worth it to just buy out marginal people, especially people with somewhat of an interest in AGI, from working in AI at all, so going into AI seems pretty net-negative to me, even if you make a good amount of money.
(I think one could make an argument that general non-AGI related AI work isn’t that bad for the world, but my sense is that a lot of the highest-paying AI jobs have some pretty decent impact on AI capabilities, either by directly involving research, or by substantially increasing the amount of commercialization and therefore funding and talent that goes into AI. I bet there are some AI jobs you can find that won’t make any substantial difference on AGI, so if you keep that in mind and are pretty conservative with regards to improving commercialization of AI technologies or attracting more talent to the field, you might be fine.)
Apologize for long wall of text, at the bottom I dived into your aside more as that’s highly relevant to deciding the course of my next 10 years and would appreciate your weighing-in.
Pre-Lesswrong/my entire life I’ve been really interested in longevity, and I would do anything to help people have more time with their loved ones (and as a child I thought solving this was the only worthy kind of fame I’d ever want.)
I didn’t know how to get there, but it was probably somewhere in math and science so I decided I had to do anything to get into MIT.
My hobbies ended up being CS-y instead of biology-y, and I realized that not only was CS profitable for earn to give, but it also might be the best shot for longevity since AI was just infinitely better at problem solving.
So that’s where my AI interest comes from. Not in being afraid of it but in using it to solve mortal problems. But the AI safety thing is something that I of course just hear smart people like Eliezer mention and then I think to myself “hmm well they know more about AI than me and I can’t use it to cure aging without the AI also maybe destroying us so I should look into that.”
Your crypto comment is surprising though and I’d like to go further on that. I should be more clear, I’m pretty interested in cryptocurrency not just cryptography and so far trading it has been really profitable, and this summer I’m essentially trying to decide if I’ll stop my schooling to do a crypto startup or if I’ll do my Masters in AI (or potentially also a crypto thing).
Startups seem like the best thing to do for profit and people are falling over themselves to fund them nowadays so I assumed given how many people have offered me funding to do so, that the crypto startup thing would be far easier to profit from than an ML startup (with ML maybe overtaking it in 7 years or so)
If this isn’t the case, or we’re a year away from the flip to ML being the easier startup, I’d love to know, because I’m right on the precipice between pursuing as much ML knowledge as I can and trying to get a pHd (probably eventually do an ML spin-off), versus trying to crypto startup earn-to-give ala FTX Sam.
My claim about AI vs crypto was just a misunderstanding. I still think of “cryptography” and “distributed systems” with their historical meaning rather than “cryptocurrency startup” or “cryptocurrency trading,” but in the context of earning to give I think that should have been clear to me :)
I’d still guess an AI career is generally the better way to make money, but I don’t have a strong take / think it depends on the person and situation / am no longer confused by your position.
Yeah I saw this post:
https://www.lesswrong.com/posts/MR6cJKy2LE6kF24B7/why-hasn-t-deep-learning-generated-significant-economic
So I’m somewhat confused on how profitable AI is, but maybe I can just start asking random experts and researching AI startups
Crypto generally has been unusually profitable (as a career or investment), but much of this simply stems from the rising tide raising all boats. Given that crypto market caps can only grow about another 10x or so before approaching major currency status and financial system parity at which point growth will slow—it seems likely that much of the glory days of crypto are behind us.
AI/ML is pretty clearly the big thing happening on Earth.
Admittedly in the short term it may be easier to cash in on a quick crypto career, but since most startups fail, consider whether you’d rather try but fail to have an impact on AI vs trying but failing at crypto wealth.