Humans are already capable of self-improvement. This argument would suggest that the smartest human (or the one who was best at self-improvement, if you prefer) should have undergone fast takeoff and become seriously overpowered, but this doesn’t seem to have happened.
In a world where the limiting factor is researcher talent, not compute
Compute is definitely a limiting factor currently. Why would that change?
Humans are not currently capable of self improvement in the understanding your our own source code sense. The “self improvement” section in bookstores doesn’t change the hardware or the operating system, it basically adds more data.
Of course talent and compute both make a difference, in the sense that ∂o/∂i>0 and ∂o/∂r>0. I was talking about the subset of worlds where research talent was by far the most important. ∂o/∂r<<∂o/∂i.
In a world where researchers have little idea what they are doing, and are running a new AI every hour hoping to stumble across something that works, the result holds.
In a world where research involves months thinking about maths, then a day writing code, then an hour running it, this result holds.
In a world where everyone knows the right algorithm, but it takes a lot of compute, so AI research consists of building custom hardware and super-computing clusters, this result fails.
Currently, we are somewhere in the middle. I don’t know which of these options future research will look like, although if its the first one, friendly AI seems unlikely.
In most of the scenarios where the first smarter than human AI, is orders of magnitude faster than a human, I would expect a hard takeoff. As we went from having no algorithms that could say (tell a cat from a dog) straight to having algorithms superhumanly fast at doing so, there was no algorithm that worked, but took supercomputer hours, this seems like a plausible assumption.
Humans are not currently capable of self improvement in the understanding your o. I was talking about the subset of worlds where research talent ense. The “self improvement” section in bookstores doesn’t change the hardware or the operating system, it basically adds more data.
I’m not sure I understand this. Are you claiming δoδr is not positive for humans?
In most of the scenarios where the first smarter than human AI, is orders of magnitude faster than a human, I would expect a hard takeoff.
This sounds like “conditioned on a hard takeoff, I expect a hard takeoff”. It’s not exactly saying that, since speed could be different from intelligence, but you need to argue for the premise too: nearly all of the arguments in the linked post could be applied to your premise as well.
In a world where researchers have little idea what they are doing, and are running a new AI every hour hoping to stumble across something that works, the result holds.
In a world where research involves months thinking about maths, then a day writing code, then an hour running it, this result holds.
Agreed on both counts, and again I think the arguments in the linked posts suggest that the premises are not true.
As we went from having no algorithms that could say (tell a cat from a dog) straight to having algorithms superhumanly fast at doing so, there was no algorithm that worked, but took supercomputer hours, this seems like a plausible assumption.
This seems false to me. At what point would you say that we had AI systems that could tell a cat from a dog? I don’t know the history of object recognition, but I would guess that depending on how you operationalize it, I think the answer could be anywhere between the 60s and “we still can’t do it”. (Though it’s also possible that people didn’t care about object recognition until the 21st century, and only did other types of computer vision in the 60s-90s. It’s quite strange that object recognition is an “interesting” task, given how little information you get from it.)
My claim at the start had a typo in it. I am claiming that you can’t make a human seriously superhuman with a good education. Much like you can’t get a chimp up to human level with lots of education and “self improvement”. Serious genetic modification is another story, but at that point, your building an AI out of protien.
It does depend where you draw the line, but the for a wide range of performance levels, we went from no algorithm at that level, to a fast algorithm at that level. You couldn’t get much better results just by throwing more compute at it.
I am claiming that you can’t make a human seriously superhuman with a good education.
Is the claim that δo/δr for humans goes down over time so that o eventually hits an asymptote? If so, why won’t that apply to AI?
Serious genetic modification is another story, but at that point, your building an AI out of protien.
But it seems quite relevant that we haven’t successfully done that yet.
You couldn’t get much better results just by throwing more compute at it.
Okay, so my new story for this argument is:
For every task T, there are bottlenecks that limit its performance, which could be compute, data, algorithms, etc.
For the task of “AI research”, compute will not be the bottleneck.
So, once we get human-level performance on “AI research”, we can apply more compute to get exponential recursive self-improvement.
Is that your argument? If so, I think my question would be “why didn’t the bottleneck in point 2 vanish in point 3?” I think the only way this would be true would be if the bottleneck was algorithms, and there was a discontinuous jump in the capability of algorithms. I agree that in that world you would see a hard/fast/discontinuous takeoff, but I don’t see why we should expect that (again, the arguments in the linked posts argue against that premise).
Humans are already capable of self-improvement. This argument would suggest that the smartest human (or the one who was best at self-improvement, if you prefer) should have undergone fast takeoff and become seriously overpowered, but this doesn’t seem to have happened.
Compute is definitely a limiting factor currently. Why would that change?
Humans are not currently capable of self improvement in the understanding your our own source code sense. The “self improvement” section in bookstores doesn’t change the hardware or the operating system, it basically adds more data.
Of course talent and compute both make a difference, in the sense that ∂o/∂i>0 and ∂o/∂r>0. I was talking about the subset of worlds where research talent was by far the most important. ∂o/∂r<<∂o/∂i.
In a world where researchers have little idea what they are doing, and are running a new AI every hour hoping to stumble across something that works, the result holds.
In a world where research involves months thinking about maths, then a day writing code, then an hour running it, this result holds.
In a world where everyone knows the right algorithm, but it takes a lot of compute, so AI research consists of building custom hardware and super-computing clusters, this result fails.
Currently, we are somewhere in the middle. I don’t know which of these options future research will look like, although if its the first one, friendly AI seems unlikely.
In most of the scenarios where the first smarter than human AI, is orders of magnitude faster than a human, I would expect a hard takeoff. As we went from having no algorithms that could say (tell a cat from a dog) straight to having algorithms superhumanly fast at doing so, there was no algorithm that worked, but took supercomputer hours, this seems like a plausible assumption.
I’m not sure I understand this. Are you claiming δoδr is not positive for humans?
This sounds like “conditioned on a hard takeoff, I expect a hard takeoff”. It’s not exactly saying that, since speed could be different from intelligence, but you need to argue for the premise too: nearly all of the arguments in the linked post could be applied to your premise as well.
Agreed on both counts, and again I think the arguments in the linked posts suggest that the premises are not true.
This seems false to me. At what point would you say that we had AI systems that could tell a cat from a dog? I don’t know the history of object recognition, but I would guess that depending on how you operationalize it, I think the answer could be anywhere between the 60s and “we still can’t do it”. (Though it’s also possible that people didn’t care about object recognition until the 21st century, and only did other types of computer vision in the 60s-90s. It’s quite strange that object recognition is an “interesting” task, given how little information you get from it.)
My claim at the start had a typo in it. I am claiming that you can’t make a human seriously superhuman with a good education. Much like you can’t get a chimp up to human level with lots of education and “self improvement”. Serious genetic modification is another story, but at that point, your building an AI out of protien.
It does depend where you draw the line, but the for a wide range of performance levels, we went from no algorithm at that level, to a fast algorithm at that level. You couldn’t get much better results just by throwing more compute at it.
Is the claim that δo/δr for humans goes down over time so that o eventually hits an asymptote? If so, why won’t that apply to AI?
But it seems quite relevant that we haven’t successfully done that yet.
Okay, so my new story for this argument is:
For every task T, there are bottlenecks that limit its performance, which could be compute, data, algorithms, etc.
For the task of “AI research”, compute will not be the bottleneck.
So, once we get human-level performance on “AI research”, we can apply more compute to get exponential recursive self-improvement.
Is that your argument? If so, I think my question would be “why didn’t the bottleneck in point 2 vanish in point 3?” I think the only way this would be true would be if the bottleneck was algorithms, and there was a discontinuous jump in the capability of algorithms. I agree that in that world you would see a hard/fast/discontinuous takeoff, but I don’t see why we should expect that (again, the arguments in the linked posts argue against that premise).