They predict that AI will only come from GOFAI AIXI-likes with utility functions that will bootstrap recursively.
Do you have a link for this prediction? (Or are you just referring to, e.g., Eliezer’s dismissive attitude toward neural networks, as expressed in the Sequences?)
They predict fast takeoff and FOOM. … Deep Learning systems don’t look like they FOOM.
It’s not clear that deep learning systems get us to AGI, either. There doesn’t seem to be any good reason to be sure, at this time, that we won’t get “fast takeoff and FOOM”, does it? (Indeed it’s my understanding that Eliezer still predicts this. Or is that false?)
Stochastic Gradient Descent doesn’t look like it will treacherous turn.
It… doesn’t? What do you mean by this? I’ve seen no reason to be optimistic on this point—quite the opposite!
So what am I supposed to extract from this pattern of behaviour?
I think that at least some of the things you take to be obvious conclusions that Eliezer/MIRI should’ve drawn, are in fact not obvious, and some are even plausibly false.
You also make some good points. But there isn’t nearly so clear a pattern as you suggest.
It… doesn’t? What do you mean by this? I’ve seen no reason to be optimistic on this point—quite the opposite!
As I understand the argument, it goes like the following:
For evolutionary methods, you can’t predict the outcome of changes before they’re made, and so you end up with ‘throw the spaghetti at the wall and see what sticks’. At some point, those changes accumulate to a mind that’s capable of figuring out what environment it’s in and then performing well at that task, so you get what looks like an aligned agent while you haven’t actually exerted any influence on its internal goals (i.e. what it’ll do once it’s out in the world).
For gradient-descent based methods, you can predict the outcome of changes before they’re made; that’s the gradient part. It’s overall less plausible that the system you’re building figures out generic reasoning and then applies that generic reasoning to a specific task, compared to figuring out the specific reasoning for the task that you’d like solved. Jumps in the loss look more like “a new cognitive capacity has emerged in the network” and less like “the system is now reasoning about its training environment”.
Of course, that “overall less plausible” is making a handwavy argument about what simplicity metric we should be using and which design is simpler according to that metric. Related, earlier research: Are minimal circuits deceptive?
IMO this should be somewhat persuasive but not conclusive. I’m much happier with a transformer shaped by a giant English text corpus than I am with whatever is spit out by a neural-architecture-search program pointed at itself! But for cognitive megaprojects, I think you probably have to have something-like-a-mind in there, even if you got to it by SGD.
Do you have a link for this prediction? (Or are you just referring to, e.g., Eliezer’s dismissive attitude toward neural networks, as expressed in the Sequences?)
It’s not clear that deep learning systems get us to AGI, either. There doesn’t seem to be any good reason to be sure, at this time, that we won’t get “fast takeoff and FOOM”, does it? (Indeed it’s my understanding that Eliezer still predicts this. Or is that false?)
It… doesn’t? What do you mean by this? I’ve seen no reason to be optimistic on this point—quite the opposite!
I think that at least some of the things you take to be obvious conclusions that Eliezer/MIRI should’ve drawn, are in fact not obvious, and some are even plausibly false.
You also make some good points. But there isn’t nearly so clear a pattern as you suggest.
As I understand the argument, it goes like the following:
For evolutionary methods, you can’t predict the outcome of changes before they’re made, and so you end up with ‘throw the spaghetti at the wall and see what sticks’. At some point, those changes accumulate to a mind that’s capable of figuring out what environment it’s in and then performing well at that task, so you get what looks like an aligned agent while you haven’t actually exerted any influence on its internal goals (i.e. what it’ll do once it’s out in the world).
For gradient-descent based methods, you can predict the outcome of changes before they’re made; that’s the gradient part. It’s overall less plausible that the system you’re building figures out generic reasoning and then applies that generic reasoning to a specific task, compared to figuring out the specific reasoning for the task that you’d like solved. Jumps in the loss look more like “a new cognitive capacity has emerged in the network” and less like “the system is now reasoning about its training environment”.
Of course, that “overall less plausible” is making a handwavy argument about what simplicity metric we should be using and which design is simpler according to that metric. Related, earlier research: Are minimal circuits deceptive?
IMO this should be somewhat persuasive but not conclusive. I’m much happier with a transformer shaped by a giant English text corpus than I am with whatever is spit out by a neural-architecture-search program pointed at itself! But for cognitive megaprojects, I think you probably have to have something-like-a-mind in there, even if you got to it by SGD.