You continue to assert things without justification, which is fine insofar as your goal is not to persuade others. And perhaps this isn’t your goal! Perhaps your goal is merely to make it clear what your beliefs are, without necessarily providing the reasoning/evidence/argumentation that would convince a neutral observer to believe the same things you do.
But in that case, you are not, in fact, licensed to act surprised, and to call others “irrational”, if they fail to update to your position after merely seeing it stated. You haven’t actually given anyone a reason they should update to your position, and so—if they weren’t already inclined to agree with you—failing to agree with you is not “irrational”, “wordcel”, or whatever other pejorative you are inclined to use, but merely correct updating procedure.
So what are we left with, then? You seem to think that this sentence says something meaningful:
If ground truth reality supports 1 and 2 I am right, if it does not I am wrong.
but it is merely a tautology: “If I am right I am right, whereas if I am wrong I am wrong.” If there is additional substance to this statement of yours, I currently fail to see it. This statement can be made for any set of claims whatsoever, and so to observe it being made for a particular set of claims does not, in fact, serve as evidence for that set’s truth or falsity.
Of course, the above applies to your position, and also to my own, as well as to EY’s and to anyone else who claims to have a position on this topic. Does this thereby imply that all of these positions are equally plausible? No, I claim—no more so than, for example, “either I win the lottery or I don’t” implies a 50⁄50 spread on the outcome space. This, I claim, is structurally isomorphic to the sentence you emitted, and equally as invalid.
In order to argue that a particular possibility ought to be singled out as likelier than the others, requires more than just stating it and thereby privileging it with all of your probability mass. You must do the actual hard work of coming up with evidence, and interpreting that evidence so as to favor your model over competing models. This is work that you have not yet done, despite being many comments deep into this thread—and is therefore substantial evidence in my view that it is work you cannot do (else you could easily win this argument—or at the very least advance it substantially—by doing just that)!
Of course, you claim you are not here to do that. Too “wordcel”, or something along those lines. Well, good for you—but in that case I think the label “irrational” applies squarely to one participant in this conversation, and the name of that participant is not “Eliezer Yudkowsky”.
You’ve done an excellent job of arguing your points. It doesn’t mean they are correct, however.
Would you agree that if you made a perfect argument against the theory of relativity (numerous contemporary physicists did) it was still a waste of time?
In this context, let’s break open the object level argument. Because only the laws of physics get a vote—you don’t and I don’t.
The object level argument is that the worst of the below determines if foom is possible:
1. Compute. Right now there is a shortage of compute, and with a bit of rough estimating the shortage is actually pretty severe. Nvidia makes approximately 60 million GPUs per year, of which 500k-1000k are A/H100s. This is based on taking their data center revenue (source: wsj) and dividing by an estimated cost per chipset of (10k, 20k). Compute production can be increased, but the limit would be all the world’s 14nm or better silicon dedicated to producing AI compute. This can be increased but it takes time. Let’s estimate how many worth of labor an AI system with access to all new compute (old compute doesn’t matter due to a lack of interconnect bandwidth). If a GPT-4 instance requires a full DGX “supercompute” node, which is 8 H100s with 80 Gb of memory each, (so approximately 1T weights in fp16), how much would it require for realtime multimodal operation? Let’s assume 4x the compute, which may be a gross underestimate. So 8 more cards are running at least 1 robot in real time, 8 more are processing images for vision, and 8 more for audio i/o and helper systems for longer duration memory context.
So then if all new cards are used for inference, 1m/32 = 31,250 “instances” worth of labor. Since they operate 24 hours a day this is equivalent to perhaps 100k humans? If all of the silicon Nvidia has the contract rights to build is going into H100s, this scales by about 30 times, or 3m humans. And most of those instances cannot be involved in world takeover efforts, they have to be collecting revenue for their owners. If Nvidia gets all the silicon in the world (this may happen as it can outbid everyone else) it gives them approximately another oom. Still not enough. There are bottlenecks on increasing chip production. This also also links to my next point:
2. Algorithm search space. Every search of a possible AGI design that is better than what you have requires a massive training run. Each training run occupies tens of thousands of GPUs for around 1 month, give or take. (source: llama paper, which was sub GPT-4 in perf. They needed 2048 A100s for 3 weeks for 65b). Presumably searching this space is a game of diminishing returns : to find an algorithm better than the best you currently have requires increasingly large numbers of searches and compute. Compute that can’t be spent on exploiting the algorithm you have right now.
3. Robotics/money : for an AGI to actually take over, it has to redirect resources to itself. And this assumes humans don’t simply use CAIS and have thousands of stateless AI systems separately handling these real world tasks. Robotics is especially problematic : you know and I know how poor the current hardware is, and there are budget cuts and layoffs in many of the cutting edge labs. The best robotics hardware company, boston dynamics, keeps getting passed around as each new owner can’t find a way to make money from it. So it takes time—time to develop new robotics hardware. Time to begin mass production. Time for the new robotics produced by the first round of production to begin assisting with the manufacture of itself. Time for the equipment in the real world to begin to fail from early failures after a few thousand hours, then the design errors to be found and fixed. This puts years on the clock, likely decades. It requires humans to both build massive amounts of robotic equipment, and then put it all under the control of either insecure narrow AI task performing systems, or to stupidly centralize control to large AGIs.
4. Data. This is explained much better by https://www.lesswrong.com/posts/qpgkttrxkvGrH9BRr/superintelligence-is-not-omniscience . The chaos means that in order for any system to develop new tools for novel tasks, the system needs sufficiently high quality information about the task domain or the tool building is not possible. This prevents blind nanoforge building (what EY is talking about when he says someone could bootstrap to diamondoid nanotechnology from wet protein biology) or blind species killing bioweapon construction.
“blind” means ‘without sufficient direct observation and a large number of domain experiments’. Meaning I am claiming it is flat impossible to accomplish these things without a large amount of automated equipment or human lab technicians acting as paid willing stooges to investigate the domain in question and empirically develop step by step a path to a solution.
Speaking in probabilities, the above is a joint probability. Meaning that so long as the AGI system is limited by any of the 4 terms, it is throttled and unable to proceed with foom. Putting some terms on it:
90% for compute (we have direct evidence), 95% for robotics (direct evidence), 99% for data (near certain evidence), 50% for searching AGI spaces (quite possible there are findable much stronger algorithms in the search space)
So the probability of foom is the series probability of all 4, as all 4 must be their not form.
so 0.1 x 0.05 x 0.01 x 0.5 = 0.000025 pfoom
Notice that you can put quite a lot of doubt on my estimates. Say you do not believe me, and believe the chance I am right on each term is 25%.
pfoom = 0.32
Not enough to be as confident we’re all dead as EY is. Though I concede a 32% chance of the end of humanity isn’t super great.
As for the last part :
CAIS means you use small, isolated AI systems that tested extremely well on the training set. You check with an autoencoder how compressible to the state space of the training set the current inference set fed to the agent is for this timestamp. What you are doing is validating that it is impossible for the AI system to know that the current input is not from the training set. This both makes deception difficult and it makes the probability of success for the AI system on the current task to be similar to how it benchmarked in the training set.
If the real world input is outside that space, you transfer control to a different agent, which in the worst case is an embedded control system using classic control techniques.
This is currently how autonomous cars more or less work (depends on the team and the project).
I have several yoe actually working on embedded ML systems, and many more years on embedded controls. The above is correct. Eliezer Yudkowsky was wrong to dismiss it.
Note the Eliezer has mentioned that ML teams are going to need to find “some way” to get from—I think he estimated about an 80% chance that a GPT-3 style agent is correct on a question—to the many 9s of real world reliability.
Stateless, well isolated systems is one of the few ways human engineers know how to accomplish that. So we may get a significant amount of AI safety by default simply to meet requirements.
You continue to assert things without justification, which is fine insofar as your goal is not to persuade others. And perhaps this isn’t your goal! Perhaps your goal is merely to make it clear what your beliefs are, without necessarily providing the reasoning/evidence/argumentation that would convince a neutral observer to believe the same things you do.
But in that case, you are not, in fact, licensed to act surprised, and to call others “irrational”, if they fail to update to your position after merely seeing it stated. You haven’t actually given anyone a reason they should update to your position, and so—if they weren’t already inclined to agree with you—failing to agree with you is not “irrational”, “wordcel”, or whatever other pejorative you are inclined to use, but merely correct updating procedure.
So what are we left with, then? You seem to think that this sentence says something meaningful:
but it is merely a tautology: “If I am right I am right, whereas if I am wrong I am wrong.” If there is additional substance to this statement of yours, I currently fail to see it. This statement can be made for any set of claims whatsoever, and so to observe it being made for a particular set of claims does not, in fact, serve as evidence for that set’s truth or falsity.
Of course, the above applies to your position, and also to my own, as well as to EY’s and to anyone else who claims to have a position on this topic. Does this thereby imply that all of these positions are equally plausible? No, I claim—no more so than, for example, “either I win the lottery or I don’t” implies a 50⁄50 spread on the outcome space. This, I claim, is structurally isomorphic to the sentence you emitted, and equally as invalid.
In order to argue that a particular possibility ought to be singled out as likelier than the others, requires more than just stating it and thereby privileging it with all of your probability mass. You must do the actual hard work of coming up with evidence, and interpreting that evidence so as to favor your model over competing models. This is work that you have not yet done, despite being many comments deep into this thread—and is therefore substantial evidence in my view that it is work you cannot do (else you could easily win this argument—or at the very least advance it substantially—by doing just that)!
Of course, you claim you are not here to do that. Too “wordcel”, or something along those lines. Well, good for you—but in that case I think the label “irrational” applies squarely to one participant in this conversation, and the name of that participant is not “Eliezer Yudkowsky”.
You’ve done an excellent job of arguing your points. It doesn’t mean they are correct, however.
Would you agree that if you made a perfect argument against the theory of relativity (numerous contemporary physicists did) it was still a waste of time?
In this context, let’s break open the object level argument. Because only the laws of physics get a vote—you don’t and I don’t.
The object level argument is that the worst of the below determines if foom is possible:
1. Compute. Right now there is a shortage of compute, and with a bit of rough estimating the shortage is actually pretty severe. Nvidia makes approximately 60 million GPUs per year, of which 500k-1000k are A/H100s. This is based on taking their data center revenue (source: wsj) and dividing by an estimated cost per chipset of (10k, 20k). Compute production can be increased, but the limit would be all the world’s 14nm or better silicon dedicated to producing AI compute. This can be increased but it takes time.
Let’s estimate how many worth of labor an AI system with access to all new compute (old compute doesn’t matter due to a lack of interconnect bandwidth). If a GPT-4 instance requires a full DGX “supercompute” node, which is 8 H100s with 80 Gb of memory each, (so approximately 1T weights in fp16), how much would it require for realtime multimodal operation? Let’s assume 4x the compute, which may be a gross underestimate. So 8 more cards are running at least 1 robot in real time, 8 more are processing images for vision, and 8 more for audio i/o and helper systems for longer duration memory context.
So then if all new cards are used for inference, 1m/32 = 31,250 “instances” worth of labor. Since they operate 24 hours a day this is equivalent to perhaps 100k humans? If all of the silicon Nvidia has the contract rights to build is going into H100s, this scales by about 30 times, or 3m humans. And most of those instances cannot be involved in world takeover efforts, they have to be collecting revenue for their owners. If Nvidia gets all the silicon in the world (this may happen as it can outbid everyone else) it gives them approximately another oom. Still not enough. There are bottlenecks on increasing chip production. This also also links to my next point:
2. Algorithm search space. Every search of a possible AGI design that is better than what you have requires a massive training run. Each training run occupies tens of thousands of GPUs for around 1 month, give or take. (source: llama paper, which was sub GPT-4 in perf. They needed 2048 A100s for 3 weeks for 65b). Presumably searching this space is a game of diminishing returns : to find an algorithm better than the best you currently have requires increasingly large numbers of searches and compute. Compute that can’t be spent on exploiting the algorithm you have right now.
3. Robotics/money : for an AGI to actually take over, it has to redirect resources to itself. And this assumes humans don’t simply use CAIS and have thousands of stateless AI systems separately handling these real world tasks. Robotics is especially problematic : you know and I know how poor the current hardware is, and there are budget cuts and layoffs in many of the cutting edge labs. The best robotics hardware company, boston dynamics, keeps getting passed around as each new owner can’t find a way to make money from it. So it takes time—time to develop new robotics hardware. Time to begin mass production. Time for the new robotics produced by the first round of production to begin assisting with the manufacture of itself. Time for the equipment in the real world to begin to fail from early failures after a few thousand hours, then the design errors to be found and fixed. This puts years on the clock, likely decades. It requires humans to both build massive amounts of robotic equipment, and then put it all under the control of either insecure narrow AI task performing systems, or to stupidly centralize control to large AGIs.
4. Data. This is explained much better by https://www.lesswrong.com/posts/qpgkttrxkvGrH9BRr/superintelligence-is-not-omniscience . The chaos means that in order for any system to develop new tools for novel tasks, the system needs sufficiently high quality information about the task domain or the tool building is not possible. This prevents blind nanoforge building (what EY is talking about when he says someone could bootstrap to diamondoid nanotechnology from wet protein biology) or blind species killing bioweapon construction.
“blind” means ‘without sufficient direct observation and a large number of domain experiments’. Meaning I am claiming it is flat impossible to accomplish these things without a large amount of automated equipment or human lab technicians acting as paid willing stooges to investigate the domain in question and empirically develop step by step a path to a solution.
Speaking in probabilities, the above is a joint probability. Meaning that so long as the AGI system is limited by any of the 4 terms, it is throttled and unable to proceed with foom. Putting some terms on it:
90% for compute (we have direct evidence), 95% for robotics (direct evidence), 99% for data (near certain evidence), 50% for searching AGI spaces (quite possible there are findable much stronger algorithms in the search space)
So the probability of foom is the series probability of all 4, as all 4 must be their not form.
so 0.1 x 0.05 x 0.01 x 0.5 = 0.000025 pfoom
Notice that you can put quite a lot of doubt on my estimates. Say you do not believe me, and believe the chance I am right on each term is 25%.
pfoom = 0.32
Not enough to be as confident we’re all dead as EY is. Though I concede a 32% chance of the end of humanity isn’t super great.
As for the last part :
CAIS means you use small, isolated AI systems that tested extremely well on the training set. You check with an autoencoder how compressible to the state space of the training set the current inference set fed to the agent is for this timestamp. What you are doing is validating that it is impossible for the AI system to know that the current input is not from the training set. This both makes deception difficult and it makes the probability of success for the AI system on the current task to be similar to how it benchmarked in the training set.
If the real world input is outside that space, you transfer control to a different agent, which in the worst case is an embedded control system using classic control techniques.
This is currently how autonomous cars more or less work (depends on the team and the project).
I have several yoe actually working on embedded ML systems, and many more years on embedded controls. The above is correct. Eliezer Yudkowsky was wrong to dismiss it.
Note the Eliezer has mentioned that ML teams are going to need to find “some way” to get from—I think he estimated about an 80% chance that a GPT-3 style agent is correct on a question—to the many 9s of real world reliability.
Stateless, well isolated systems is one of the few ways human engineers know how to accomplish that. So we may get a significant amount of AI safety by default simply to meet requirements.