(1) I don’t see how an human capable or a bit smarter than human capable AI(say 50% smarter) will be a serious threat. Broadly humans are smart because of group and social behavior. So a 1.5 Human AI might be roughly as smart as two humans? Doesn’t seem too concerning.
Humans have like 4 x the brain volume of chimps, but you can’t put 4 chimps together to get 1 human. And you can’t put 100 average humans together to get 1 Einstein. Despite Einstein having a brain with about the same size and genetics of any other human. This doesn’t suggest an obvious model, but it does suggest your additive intelligence model is wrong.
(2) I don’t see how a bit smarter than humans scales to superhuman levels of intelligence.
Either Or between: Once you have the right algorithm, it really is as simple as increasing some parameter or neuron count. The AI is smarter than a smart human. Smart humans can do AI research a bit. The AI does whatever a human doing AI research would do, but quicker and better.
(a) Most things have diminishing returns, and I don’t see how one of the most elusive faculties would be an exception. (GPT-3 is a bit better than GPT-2 but GPT-3 requires 10x the resources).
I think this is somewhat to do with the underlying algorithm being not very good in a way that doesn’t apply to all algorithms. It all depends on the scales you use anyway. It takes 10x as much energy to make sound 10db louder, because decibels are a logarithmic scale.
I don’t see how superhuman intelligence by itself is necessarily a threat. Most problems cannot be solved by pure reasoning alone.
Of course most problems can’t be solved by reason alone. In a world with the internet, there is a lot of information available. The limits on how much useful information and understanding you can extract from your data are really high. So yes you need some data. This doesn’t mean lack of data is going to be a serious bottleneck, any more than lack of silicon atoms is the bottleneck on making chips. The world is awash with data. Cleaned up, sorted out data is a little rarer. Taking a large amount of data, some of tangential relevance, some with mistakes in, some fraudulent, and figuring out what is going on, that takes intelligence.
For example, a human with 150 IQ isn’t going to be much better at predicting the weather than a person with 100 IQ.
What is happening here? Are both people just looking at a picture and guessing numbers, or can the IQ 150 person program a simulation while the IQ 100 person is looking at the Navier stokes equation trying (and failing) to figure out what it means.
A lot of science seems to be done by a handful of top scientists. The likes of Einstein wouldn’t be a thing if a higher intelligence didn’t make you much better at discovering some things.
I think weather prediction isn’t a good example of intelligence in general. Take programming.Entirely discrete. No chaos. Smart humans to lots better than dumb ones. Small differences in intelligence make the difference between a loglinear sort and a quadratic one.
If you reduce the IQ of your scientists by 20 points. That doesn’t make your nukes 20% less powerful, it means you don’t get nukes.
Not all parts of the world are chaotic in the chaos theory sense. There are deep patterns and things that can be predicted. Humans have a huge practical advantage over all other animals. Real intelligence often pulls powerful and incomprehensible tricks out of nowhere. The “how the ??? did the AI manage to do that with only those tools?” Not just getting really good answers on standardized tests.
The probability it rains tomorrow might (or might not) be a special case where we are doing the best we can with the data we have. If a coin will land heads is another question where there may be no room for an AI to improve. Design an Mrna strand to reverse ageing? Is almost surely a question that leaves the AI lots of room to show off its intelligence.
Hmmmmm there is a lot here let me see if I can narrow down on some key points.
Once you have the right algorithm, it really is as simple as increasing some parameter or neuron count.
There are some problems that do not scale well(or at all). For example, doubling the computational power applied to solving the knapsack problem will let you solve a problem size that is one element bigger. Why should we presume that intelligence scales like an O(n) problem and not an O(2^n) problem?
What is happening here? Are both people just looking at a picture and guessing numbers, or can the IQ 150 person program a simulation while the IQ 100 person is looking at the Navier stokes equation trying (and failing) to figure out what it means.
I picked weather prediction as an exemplar problem because
(1) NWP programs are the product of not just single scientists but the efforts of thousands. (the intelligences of many people combined into a product that is far greater than any could produce individually)
(II) The problem is fairly well understood and observation limited. If we could simultaneously measure the properties of 1m^3 voxels of atmsophere our NWP would be dramatically improved. But our capabilities are closer to one per day(non-simultaneous) measurements of spots rougly 40 km in diameter. Access to the internet will not improve this. The measurements don’t exist. Other chaotic systems like ensembles of humans or stocks may very well have this property.
Smart humans to lots better than dumb ones. Small differences in intelligence make the difference between a loglinear sort and a quadratic one.
But hardly any programs are the result of individual efforts. They’re the product of thousands. If a quadratic sort slips through it gets caught by a profiler and someone else fixes it. (And everyone uses compilers, interpreters, libraries, etc...)
A lot of science seems to be done by a handful of top scientists. The likes of Einstein wouldn’t be a thing if a higher intelligence didn’t make you much better at discovering some things.
This is not my view of science. I tend to understand someone like Einstein as an inspirational story we tell when the history of physics in the early 20th century is fact a tale of dozens, if not hundreds. But I do think this is getting towards the crux.
There are some problems that do not scale well(or at all). For example, doubling the computational power applied to solving the knapsack problem will let you solve a problem size that is one element bigger. Why should we presume that intelligence scales like an O(n) problem and not an O(2^n) problem?
We don’t know that, P vs NP is an unproved conjecture. Most real world problems are not giant knapsack problems. And there are algorithms that quickly produce answers that are close to optimal. Actually, most of the real use of intelligence is not a complexity theory problem at all. “Is inventing transistors a O(n) or an O(2^n) problem?” Meaningless. In practice, modest improvements in intelligence seem to produce largish differences in results.
Lots of big human achievements are built by many humans. Some things are made by one exceptionally smart human. Inventing general relativity, or solving Fermat’s last theorem are the sort of problem with a smaller number of deep abstract general parts. Solving 10^12 capchas is exactly the sort of task that can be split among many people easily. This sort of problem can’t be solved by a human genius (unless they make an AI, but that doesn’t count) There are just too many separate small problems.
I think some tasks can easily be split into many small parts, some can’t. Tasks that can be split, we get a team of many humans to do. Tasks that can’t, if our smartest geniuses can’t solve them, they go unsolved.
Suppose an AI as smart as the average human. You can of course put in 10x the compute, and get an AI as smart as 10 average humans working together. I think you can also put in 1.2x the compute, and get an AI as smart as Einstein. Or put in 12x the compute and get an AI as smart as 12 Einstein’s working together. But its generally better to put a lot of you compute into one big pile and get insights far more deep and profound than any human could ever make. Lots of distinct human mind sized blobs talking is an inefficient way to use your compute, any well designed AI can do better.
This is not my view of science. I tend to understand someone like Einstein as an inspirational story we tell when the history of physics in the early 20th century is fact a tale of dozens, if not hundreds. But I do think this is getting towards the crux.
Sure, maybe as many as 100′s. Not millions. Now the differences between human brains are fairly small. This indicates that 2 humans aren’t as good as 1 human with a brain that works slightly better.
But hardly any programs are the result of individual efforts. They’re the product of thousands. If a quadratic sort slips through it gets caught by a profiler and someone else fixes it. (And everyone uses compilers, interpreters, libraries, etc...)
Isn’t that just relying on the intelligence of everyone else on the team. “Intelligence isn’t important, the project would be going just as well if Alex was as dumb as a bag of bricks. (If Alex was that dumb, everyone else would do his work.) ” Sure, lots of people work together to make code. And if all those people were 10 IQ points smarter, the code would be a lot better. If they were 10 IQ points dumber, the code would be a lot worse. Sometimes one person being much dumber wouldn’t make a difference, if all their work is being checked. Except that if they were dumb, someone would have to write new code that passes the checks. And if the checker is dumb, all the rubbish gets through.
The problem is fairly well understood and observation limited.
You picked a problem that was particularly well understood. The problems we are fundamentally confused about are the ones where better intelligence has the big wins. The problem of predicting a quantum coin toss is well understood, and totally observation limited. In other words, its a rare special case where the problem has mathematical structure allowing humans to make near optimal use of available information. Most problems aren’t like this.
Access to the internet will not improve this. The measurements don’t exist. Other chaotic systems like ensembles of humans or stocks may very well have this property.
I think there is a lot of info about humans and stocks on the internet. And quite a lot of info about weather like clouds in the background of someones holiday pics. But current weather prediction doesn’t use that data. It just uses the weather satellite data, because it isn’t smart enough to make sense of all the social media data. I mean you could argue that most of the good data is from the weather satellite. That social media data doesn’t help much even if you are smart enough to use it. If that is true, that would be a way that the weather problem differs from many other problems.
Suppose the AI is 2x as good at using data as the human. So if both human and AI had priors of 1:1 on a hypothesis, the human updates to 10:1, and the AI to 100:1. The human is 91% sure, and the AI is 99% sure. But if Both start with priors of 1:1000,000 then the human can update to 1:100 while the AI updates to 100:1. Ie the human is 1% sure while the AI is 99% sure.
A small improvement in evidence processing ability can make the difference between narrowing 1000,000 options down to 100, and narrowing them down to 1. Which makes a large difference in chance of correctness.
We don’t know that, P vs NP is an unproved conjecture. Most real world problems are not giant knapsack problems. And there are algorithms that quickly produce answers that are close to optimal. Actually, most of the real use of intelligence is not a complexity theory problem at all. “Is inventing transistors a O(n) or an O(2^n) problem?”
P vs. NP is unproven. But I disagree that “most real world problems are not giant knapsack problems”. The Cook-Levin theorem showed that many of the most interesting problems are reducible to NP-complete problems. I’m going to quote this paper by Scott-Aaronson, but it is a great read and I hope you check out the whole thing. https://www.scottaaronson.com/papers/npcomplete.pdf
Even many computer scientists do not seem to appreciate how different the world would be if we could solve NP-complete problems efficiently. I have heard it said, with a straight face, that a proof of P = NP would be important because it would let airlines schedule their flights better, or shipping companies pack more boxes in their trucks! One person who did understand was G ̈odel. In his celebrated 1956 letter to von Neumann (see [69]), in which he first raised the P versus NP question, G ̈odel says that a linear or quadratic-time procedure for what we now call NP-complete problems would have “consequences of the greatest magnitude.” For such an procedure “would clearly indicate that, despite the unsolvability of the Entscheidungsproblem, the mental effort of the mathematician in the case of yes-or-no questions could be completely replaced by machines.”
But it would indicate even more. If such a procedure existed, then we could quickly find the smallest Boolean circuits that output (say) a table of historical stock market data, or the human genome, or the complete works of Shakespeare. It seems entirely conceivable that, by analyzing these circuits, we could make an easy fortune on Wall Street, or retrace evolution, or even generate Shakespeare’s 38th play. For broadly speaking, that which we can compress we can understand, and that which we can understand we can predict. Indeed, in a recent book [12], Eric Baum argues that much of what we call ‘insight’ or ‘intelligence’ simply means finding succinct representations for our sense data. On his view, the human mind is largely a bundle of hacks and heuristics for this succinct-representation problem, cobbled together over a billion years of evolution. So if we could solve the general case—if knowing something was tantamount to knowing the shortest efficient description of it—then we would be almost like gods. The NP Hardness Assumption is the belief that such power will be forever beyond our reach.
I take the NP-hardness assumption as foundational. That being the case, a lot of talk of AI x-risk sounds to me like saying that AI will be an NP oracle. (For example, the idea that a highly intelligent expert system designing tractors could somehow “get out of the box” and threaten humanity, would require a highly accurate predictive model that would almost certainly contain one or many NP-complete subproblems)
But current weather prediction doesn’t use that data. It just uses the weather satellite data, because it isn’t smart enough to make sense of all the social media data. I mean you could argue that most of the good data is from the weather satellite. That social media data doesn’t help much even if you are smart enough to use it. If that is true, that would be a way that the weather problem differs from many other problems.
Yes I would argue that current weather prediction doesn’t use social media data because cameras at optical wavelengths cannot sound the upper atmosphere. Physics means there is no free lunch from social media data.
I would argue that most real world problems are observationally and experimentally bound. The seminal paper on photoelectric effect was a direct consequence of a series of experimental results from the 19th century. Relativity is the same story. It isn’t like there were measurements of the speed of light or the ratio of frequency to energy of photons available in the 17th century just waiting for someone with sufficient intelligence to find them in the 17th century equivalent of social media. And no amount of data on european peasants (the 17th century equivalent of facebook) would be a sufficient substitute. The right data makes all the difference.
A common AI risk problem like manipulating a programmer into escalating AI privileges is a less legible problem than examples from physics but I have no reason to think that it won’t also be observationally bound. Making an attempt to manipulate the programmer and being so accurate in the prediction that the AI is highly confident it won’t be caught would require a model of the programmer as detailed(or more) than an atmospheric model. There is no guarantee the programmer has any psychological vulnerabiliies. There is no guarantee that they share the right information on social media. Even if they’re a prolific poster, why would we think this information is sufficient to manipulate them?
The smallest boolean circuit trick is a reasonable trick, if you can get it to work efficiently. But it won’t magically be omniscient at everything. It would be just one fairly useful tool in the ML toolbox.
“Minimal circuit” based approaches will fail badly when the data comes from a simple to specify but computationally heavy function. For example, the minimal boolean circuit trained to take in a point on the complex plane (with X and Y as IEEE floats) and output if that point is in the mandelbrot set. This algorithm will fail reasonable numbers of training points.
If the 3 sat algorithm is O(n^4) then this algorithm might not be that useful compared to other approaches.
On the other hand, a loglinear algorithm that produced a circuit that was usually almost optimal, that could be useful.
What’s the runtime complexity of rowhammering your own hardware to access the internet? O(1)? Meaningless. Asymptotic runtime complexity is a mathematical tool that assumes an infinite sequence of ever harder problems. (An infinite sequence of ever longer lists to sort, or 3-sat’s to solve) There is not an infinite sequence of ever harder problems in reality. Computational complexity is mostly concerned with the worst case, and finding the absolutely perfect solution. In reality, an AI can use algorithms that find a pretty good solution most of the time.
A common AI risk problem like manipulating a programmer into escalating AI privileges is a less legible problem than examples from physics but I have no reason to think that it won’t also be observationally bound.
Obviously there is a bound based on observation for every problem, just often that bound is crazy high on our scales. For a problem humans understand well, we may be operating close to that bound. For the many problems that are “illegible” or “hard for humans to think about” or “confusing”, we are nowhere near the bound, so the AI has room to beat the pants off us with the same data.
I would argue that most real world problems are observationally and experimentally bound.
And no amount of data on european peasants (the 17th century equivalent of facebook) would be a sufficient substitute.
Consider a world in which humans, being smart but not that smart, can figure out relativity from good clear physics data. Could a superintelligence figure out relativity based on the experiences of the typical caveman? You seem to be arguing that a superintelligence couldn’t manage this because Einstein didn’t. Apparently gold wouldn’t be yellowish if it weren’t for relativistic effects on the innermost electrons. The sun is shining. Life ⇒ Evolution over millions of years. Sun’s energy density too high for chemical fuel. E=mc^2? Why the night sky is dark. Redshift and a universe with a big bang. These clues, and probably more, have been available to most humans throughout history.
These clues weren’t enough to lead Einstein to relativity, but Einstein was only human.
In reality, an AI can use algorithms that find a pretty good solution most of the time.
If you replace “AI” with “ML” I agree with this point. And yep this is what we can do with the networks we’re scaling. But “pretty good most of the time” doesn’t get you an x-risk intelligence. It gets you some really cool tools.
If the 3 sat algorithm is O(n^4) then this algorithm might not be that useful compared to other approaches.
If 3 SAT is O(n^4) then P=NP and back to Aaronson’s point; the fundamental structure of reality is much different than we think it is. (did you mean “4^N”? Plenty of common algorithms are quartic.)
For the many problems that are “illegible” or “hard for humans to think about” or “confusing”, we are nowhere near the bound, so the AI has room to beat the pants off us with the same data.
The assertion that “illegible” means “requiring more intelligence” rather than “ill-posed” or “underspecified” doesn’t seem obvious to me. Maybe you can expand on this?
Could a superintelligence figure out relativity based on the experiences of the typical caveman?..These clues weren’t enough to lead Einstein to relativity, but Einstein was only human.
I’m not sure I can draw the inference that this means it was possible to generate the theory without the key observations it is explaining. What I’m grasping at is how we can bound what cababilities more intelligence gives an agent. It seems intuitive to me that there must be limits and we can look to physics and math to try to understand them. Which leads us here:
Meaningless. Asymptotic runtime complexity is a mathematical tool that assumes an infinite sequence of ever harder problems.
I disagree. We’ve got a highly speculative question in front of us. “What can a machine intelligence greater than ours accomplish”? We can’t really know what it would be like to be twice as smart any more than an ape can. But if we stipulate that the machine is running on Turing Complete hardware and accept NP hardness then we can at least put an upper bound on the capabilities of this machine.
Concretely, I can box the machine using a post-quantum cryptographic standard and know that it lacks the computational resources to break out before the heat death of the universe. More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
If you replace “AI” with “ML” I agree with this point. And yep this is what we can do with the networks we’re scaling. But “pretty good most of the time” doesn’t get you an x-risk intelligence. It gets you some really cool tools.
On some problems, finding the exact best solution is intractable. On these problems, its all approximations and tricks that usually work. Whether the simplest dumb ML algorithm running with didly squat compute, or some vastly superintelligent AI running on a Matrioska brain.
Take hacking a computer system that controls nukes or something. The problem of finding the fastest way to hack an arbitrary computer system is NP hard. But humans sometimes hack computers without exponentially vast brains. Suppose the AI’s hacking algorithm can hack 99% of all computer systems that are theoretically possible to hack with unlimited compute. And The AI takes at most 10% longer than the theoretically minimum time on those problems.
This AI is still clearly dangerous. Especially if it isn’t just a hacking tool. It has a big picture view of the world and what it wants to achieve.
In short, maybe P!=NP and there is no perfect algoritm, but its possible to be a lot better than current ML, and a lot better than humans, and you don’t need a perfect algorithm to create an X risk.
If 3 SAT is O(n^4) then P=NP and back to Aaronson’s point; the fundamental structure of reality is much different than we think it is. (did you mean “4^N”? Plenty of common algorithms are quartic.)
If 3-sat is O(n^1000,000) then P=NP on a technicality, but the algorithm is totally useless as it is far too slow in practice. If its O(n^4) there are still some uses, but it would seriously hamper the effectiveness of the minimum circuit style predictors. Neural nets are trained on a lot of data. With an O(n^4) algorithm, training beyond 10kb of data will be getting difficult, depending somewhat on the constant.
The assertion that “illegible” means “requiring more intelligence” rather than “ill-posed” or “underspecified” doesn’t seem obvious to me. Maybe you can expand on this?
If you consider the problem of persuading a human to let you out of the box in an AI boxing scenario, well that is perfectly well posed. (There is a big red “open box” button, and either its pressed or it isn’t. ) But we don’t have enough understanding of phycology to do it. Pick some disease we don’t know the cause of or how to cure yet. There will be lots of semi relevant information in biology textbooks. There will be case reports and social media posts and a few confusing clinical trials.
In weather, we know the equations, and there are equations. Any remaining uncertainty is uncertainty about windspeed, temperature etc. But suppose you had a lot of weather data, but hadn’t invented the equations yet. You have a few rules of thumb about how weather patterns move. When you invent the equations, suddenly you can predict so much more.
It seems intuitive to me that there must be limits and we can look to physics and math to try to understand them.
Of course their are bounds. But those bounds are really high on a human scale.
I’m not sure I can draw the inference that this means it was possible to generate the theory without the key observations it is explaining.
I am arguing that there are several facts observable to the average caveman that are explainable by general relativity. Einstein needed data from experiments as well. If you take 10 clues to solve a puzzle, but once you solve it, all the pieces fit beautifully together, that indicates that the problem may have been solvable with fewer clues. It wasn’t that Pythagoras had 10 trillion theories, each as mathematically elegant as general relativity, in mind, and needed experiments to tell which one was true. Arguably Newton had 1 theory like that, and there were still a few clues available that hinted towards relativity.
Concretely, I can box the machine using a post-quantum cryptographic standard and know that it lacks the computational resources to break out before the heat death of the universe. More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
A large fraction of hacking doesn’t involve breaking cryptographic protocols in the field of cryptographic protocols, but in places where those abstractions break down. Sure you use that post quantum cryptographic standard. But then you get an attack that runs a computation that takes 5 or 30 seconds depending on a single bit of the key, and seeing if the cooling fan turns on. Data on what the cooling fan gets up to wasn’t included in the ideal mathematical model in which the problem was NP hard. Or it could be something as stupid as an easily guessable admin password. Or a cryptographic library calls a big int library. The big int library has a debug mode that logs all the arithmetic done. The debug mode can be turned on with a buffer overflow. So turn on debug and check the logs for the private keys. Most hacks aren’t about breaking NP hard math, but about the surrounding software being buggy or imperfect, allowing you do bypass the math.
The NP-hard maths is about the worst case. I can give you a 3sat of a million variables that is trivial to solve. The maths conjectures that it is hard to solve the worst case. Is hacking a particular computer or designing a bioweapon the worst case of a 3 sat problem, or is it one of the easier cases? I don’t know. Large 3 sat problems are often solved in practice, like a million variable problems are solved in minutes. Because NP hardness is about the worst case. And the practical problem people are solving isn’t the worst case.
More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
Are you claiming that designing a lethal bioweapon is NP hard? That building nanotech is NP hard? Like using hacking and social engineering to start a nuclear war is NP hard? What are these large 3 sat problems that must be solved before the world can be destroyed?
Humans have like 4 x the brain volume of chimps, but you can’t put 4 chimps together to get 1 human. And you can’t put 100 average humans together to get 1 Einstein. Despite Einstein having a brain with about the same size and genetics of any other human. This doesn’t suggest an obvious model, but it does suggest your additive intelligence model is wrong.
Either Or between: Once you have the right algorithm, it really is as simple as increasing some parameter or neuron count. The AI is smarter than a smart human. Smart humans can do AI research a bit. The AI does whatever a human doing AI research would do, but quicker and better.
I think this is somewhat to do with the underlying algorithm being not very good in a way that doesn’t apply to all algorithms. It all depends on the scales you use anyway. It takes 10x as much energy to make sound 10db louder, because decibels are a logarithmic scale.
Of course most problems can’t be solved by reason alone. In a world with the internet, there is a lot of information available. The limits on how much useful information and understanding you can extract from your data are really high. So yes you need some data. This doesn’t mean lack of data is going to be a serious bottleneck, any more than lack of silicon atoms is the bottleneck on making chips. The world is awash with data. Cleaned up, sorted out data is a little rarer. Taking a large amount of data, some of tangential relevance, some with mistakes in, some fraudulent, and figuring out what is going on, that takes intelligence.
What is happening here? Are both people just looking at a picture and guessing numbers, or can the IQ 150 person program a simulation while the IQ 100 person is looking at the Navier stokes equation trying (and failing) to figure out what it means.
A lot of science seems to be done by a handful of top scientists. The likes of Einstein wouldn’t be a thing if a higher intelligence didn’t make you much better at discovering some things.
I think weather prediction isn’t a good example of intelligence in general. Take programming.Entirely discrete. No chaos. Smart humans to lots better than dumb ones. Small differences in intelligence make the difference between a loglinear sort and a quadratic one.
If you reduce the IQ of your scientists by 20 points. That doesn’t make your nukes 20% less powerful, it means you don’t get nukes.
Not all parts of the world are chaotic in the chaos theory sense. There are deep patterns and things that can be predicted. Humans have a huge practical advantage over all other animals. Real intelligence often pulls powerful and incomprehensible tricks out of nowhere. The “how the ??? did the AI manage to do that with only those tools?” Not just getting really good answers on standardized tests.
The probability it rains tomorrow might (or might not) be a special case where we are doing the best we can with the data we have. If a coin will land heads is another question where there may be no room for an AI to improve. Design an Mrna strand to reverse ageing? Is almost surely a question that leaves the AI lots of room to show off its intelligence.
Hmmmmm there is a lot here let me see if I can narrow down on some key points.
There are some problems that do not scale well(or at all). For example, doubling the computational power applied to solving the knapsack problem will let you solve a problem size that is one element bigger. Why should we presume that intelligence scales like an O(n) problem and not an O(2^n) problem?
I picked weather prediction as an exemplar problem because
(1) NWP programs are the product of not just single scientists but the efforts of thousands. (the intelligences of many people combined into a product that is far greater than any could produce individually)
(II) The problem is fairly well understood and observation limited. If we could simultaneously measure the properties of 1m^3 voxels of atmsophere our NWP would be dramatically improved. But our capabilities are closer to one per day(non-simultaneous) measurements of spots rougly 40 km in diameter. Access to the internet will not improve this. The measurements don’t exist. Other chaotic systems like ensembles of humans or stocks may very well have this property.
But hardly any programs are the result of individual efforts. They’re the product of thousands. If a quadratic sort slips through it gets caught by a profiler and someone else fixes it. (And everyone uses compilers, interpreters, libraries, etc...)
This is not my view of science. I tend to understand someone like Einstein as an inspirational story we tell when the history of physics in the early 20th century is fact a tale of dozens, if not hundreds. But I do think this is getting towards the crux.
We don’t know that, P vs NP is an unproved conjecture. Most real world problems are not giant knapsack problems. And there are algorithms that quickly produce answers that are close to optimal. Actually, most of the real use of intelligence is not a complexity theory problem at all. “Is inventing transistors a O(n) or an O(2^n) problem?” Meaningless. In practice, modest improvements in intelligence seem to produce largish differences in results.
Lots of big human achievements are built by many humans. Some things are made by one exceptionally smart human. Inventing general relativity, or solving Fermat’s last theorem are the sort of problem with a smaller number of deep abstract general parts. Solving 10^12 capchas is exactly the sort of task that can be split among many people easily. This sort of problem can’t be solved by a human genius (unless they make an AI, but that doesn’t count) There are just too many separate small problems.
I think some tasks can easily be split into many small parts, some can’t. Tasks that can be split, we get a team of many humans to do. Tasks that can’t, if our smartest geniuses can’t solve them, they go unsolved.
Suppose an AI as smart as the average human. You can of course put in 10x the compute, and get an AI as smart as 10 average humans working together. I think you can also put in 1.2x the compute, and get an AI as smart as Einstein. Or put in 12x the compute and get an AI as smart as 12 Einstein’s working together. But its generally better to put a lot of you compute into one big pile and get insights far more deep and profound than any human could ever make. Lots of distinct human mind sized blobs talking is an inefficient way to use your compute, any well designed AI can do better.
Sure, maybe as many as 100′s. Not millions. Now the differences between human brains are fairly small. This indicates that 2 humans aren’t as good as 1 human with a brain that works slightly better.
Isn’t that just relying on the intelligence of everyone else on the team. “Intelligence isn’t important, the project would be going just as well if Alex was as dumb as a bag of bricks. (If Alex was that dumb, everyone else would do his work.) ” Sure, lots of people work together to make code. And if all those people were 10 IQ points smarter, the code would be a lot better. If they were 10 IQ points dumber, the code would be a lot worse. Sometimes one person being much dumber wouldn’t make a difference, if all their work is being checked. Except that if they were dumb, someone would have to write new code that passes the checks. And if the checker is dumb, all the rubbish gets through.
You picked a problem that was particularly well understood. The problems we are fundamentally confused about are the ones where better intelligence has the big wins. The problem of predicting a quantum coin toss is well understood, and totally observation limited. In other words, its a rare special case where the problem has mathematical structure allowing humans to make near optimal use of available information. Most problems aren’t like this.
I think there is a lot of info about humans and stocks on the internet. And quite a lot of info about weather like clouds in the background of someones holiday pics. But current weather prediction doesn’t use that data. It just uses the weather satellite data, because it isn’t smart enough to make sense of all the social media data. I mean you could argue that most of the good data is from the weather satellite. That social media data doesn’t help much even if you are smart enough to use it. If that is true, that would be a way that the weather problem differs from many other problems.
Suppose the AI is 2x as good at using data as the human. So if both human and AI had priors of 1:1 on a hypothesis, the human updates to 10:1, and the AI to 100:1. The human is 91% sure, and the AI is 99% sure. But if Both start with priors of 1:1000,000 then the human can update to 1:100 while the AI updates to 100:1. Ie the human is 1% sure while the AI is 99% sure.
A small improvement in evidence processing ability can make the difference between narrowing 1000,000 options down to 100, and narrowing them down to 1. Which makes a large difference in chance of correctness.
P vs. NP is unproven. But I disagree that “most real world problems are not giant knapsack problems”. The Cook-Levin theorem showed that many of the most interesting problems are reducible to NP-complete problems. I’m going to quote this paper by Scott-Aaronson, but it is a great read and I hope you check out the whole thing. https://www.scottaaronson.com/papers/npcomplete.pdf
I take the NP-hardness assumption as foundational. That being the case, a lot of talk of AI x-risk sounds to me like saying that AI will be an NP oracle. (For example, the idea that a highly intelligent expert system designing tractors could somehow “get out of the box” and threaten humanity, would require a highly accurate predictive model that would almost certainly contain one or many NP-complete subproblems)
Yes I would argue that current weather prediction doesn’t use social media data because cameras at optical wavelengths cannot sound the upper atmosphere. Physics means there is no free lunch from social media data.
I would argue that most real world problems are observationally and experimentally bound. The seminal paper on photoelectric effect was a direct consequence of a series of experimental results from the 19th century. Relativity is the same story. It isn’t like there were measurements of the speed of light or the ratio of frequency to energy of photons available in the 17th century just waiting for someone with sufficient intelligence to find them in the 17th century equivalent of social media. And no amount of data on european peasants (the 17th century equivalent of facebook) would be a sufficient substitute. The right data makes all the difference.
A common AI risk problem like manipulating a programmer into escalating AI privileges is a less legible problem than examples from physics but I have no reason to think that it won’t also be observationally bound. Making an attempt to manipulate the programmer and being so accurate in the prediction that the AI is highly confident it won’t be caught would require a model of the programmer as detailed(or more) than an atmospheric model. There is no guarantee the programmer has any psychological vulnerabiliies. There is no guarantee that they share the right information on social media. Even if they’re a prolific poster, why would we think this information is sufficient to manipulate them?
The smallest boolean circuit trick is a reasonable trick, if you can get it to work efficiently. But it won’t magically be omniscient at everything. It would be just one fairly useful tool in the ML toolbox.
“Minimal circuit” based approaches will fail badly when the data comes from a simple to specify but computationally heavy function. For example, the minimal boolean circuit trained to take in a point on the complex plane (with X and Y as IEEE floats) and output if that point is in the mandelbrot set. This algorithm will fail reasonable numbers of training points.
If the 3 sat algorithm is O(n^4) then this algorithm might not be that useful compared to other approaches.
On the other hand, a loglinear algorithm that produced a circuit that was usually almost optimal, that could be useful.
What’s the runtime complexity of rowhammering your own hardware to access the internet? O(1)? Meaningless. Asymptotic runtime complexity is a mathematical tool that assumes an infinite sequence of ever harder problems. (An infinite sequence of ever longer lists to sort, or 3-sat’s to solve) There is not an infinite sequence of ever harder problems in reality. Computational complexity is mostly concerned with the worst case, and finding the absolutely perfect solution. In reality, an AI can use algorithms that find a pretty good solution most of the time.
Obviously there is a bound based on observation for every problem, just often that bound is crazy high on our scales. For a problem humans understand well, we may be operating close to that bound. For the many problems that are “illegible” or “hard for humans to think about” or “confusing”, we are nowhere near the bound, so the AI has room to beat the pants off us with the same data.
Consider a world in which humans, being smart but not that smart, can figure out relativity from good clear physics data. Could a superintelligence figure out relativity based on the experiences of the typical caveman? You seem to be arguing that a superintelligence couldn’t manage this because Einstein didn’t. Apparently gold wouldn’t be yellowish if it weren’t for relativistic effects on the innermost electrons. The sun is shining. Life ⇒ Evolution over millions of years. Sun’s energy density too high for chemical fuel. E=mc^2? Why the night sky is dark. Redshift and a universe with a big bang. These clues, and probably more, have been available to most humans throughout history.
These clues weren’t enough to lead Einstein to relativity, but Einstein was only human.
If you replace “AI” with “ML” I agree with this point. And yep this is what we can do with the networks we’re scaling. But “pretty good most of the time” doesn’t get you an x-risk intelligence. It gets you some really cool tools.
If 3 SAT is O(n^4) then P=NP and back to Aaronson’s point; the fundamental structure of reality is much different than we think it is. (did you mean “4^N”? Plenty of common algorithms are quartic.)
The assertion that “illegible” means “requiring more intelligence” rather than “ill-posed” or “underspecified” doesn’t seem obvious to me. Maybe you can expand on this?
I’m not sure I can draw the inference that this means it was possible to generate the theory without the key observations it is explaining. What I’m grasping at is how we can bound what cababilities more intelligence gives an agent. It seems intuitive to me that there must be limits and we can look to physics and math to try to understand them. Which leads us here:
I disagree. We’ve got a highly speculative question in front of us. “What can a machine intelligence greater than ours accomplish”? We can’t really know what it would be like to be twice as smart any more than an ape can. But if we stipulate that the machine is running on Turing Complete hardware and accept NP hardness then we can at least put an upper bound on the capabilities of this machine.
Concretely, I can box the machine using a post-quantum cryptographic standard and know that it lacks the computational resources to break out before the heat death of the universe. More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
On some problems, finding the exact best solution is intractable. On these problems, its all approximations and tricks that usually work. Whether the simplest dumb ML algorithm running with didly squat compute, or some vastly superintelligent AI running on a Matrioska brain.
Take hacking a computer system that controls nukes or something. The problem of finding the fastest way to hack an arbitrary computer system is NP hard. But humans sometimes hack computers without exponentially vast brains. Suppose the AI’s hacking algorithm can hack 99% of all computer systems that are theoretically possible to hack with unlimited compute. And The AI takes at most 10% longer than the theoretically minimum time on those problems.
This AI is still clearly dangerous. Especially if it isn’t just a hacking tool. It has a big picture view of the world and what it wants to achieve.
In short, maybe P!=NP and there is no perfect algoritm, but its possible to be a lot better than current ML, and a lot better than humans, and you don’t need a perfect algorithm to create an X risk.
If 3-sat is O(n^1000,000) then P=NP on a technicality, but the algorithm is totally useless as it is far too slow in practice. If its O(n^4) there are still some uses, but it would seriously hamper the effectiveness of the minimum circuit style predictors. Neural nets are trained on a lot of data. With an O(n^4) algorithm, training beyond 10kb of data will be getting difficult, depending somewhat on the constant.
If you consider the problem of persuading a human to let you out of the box in an AI boxing scenario, well that is perfectly well posed. (There is a big red “open box” button, and either its pressed or it isn’t. ) But we don’t have enough understanding of phycology to do it. Pick some disease we don’t know the cause of or how to cure yet. There will be lots of semi relevant information in biology textbooks. There will be case reports and social media posts and a few confusing clinical trials.
In weather, we know the equations, and there are equations. Any remaining uncertainty is uncertainty about windspeed, temperature etc. But suppose you had a lot of weather data, but hadn’t invented the equations yet. You have a few rules of thumb about how weather patterns move. When you invent the equations, suddenly you can predict so much more.
Of course their are bounds. But those bounds are really high on a human scale.
I am arguing that there are several facts observable to the average caveman that are explainable by general relativity. Einstein needed data from experiments as well. If you take 10 clues to solve a puzzle, but once you solve it, all the pieces fit beautifully together, that indicates that the problem may have been solvable with fewer clues. It wasn’t that Pythagoras had 10 trillion theories, each as mathematically elegant as general relativity, in mind, and needed experiments to tell which one was true. Arguably Newton had 1 theory like that, and there were still a few clues available that hinted towards relativity.
A large fraction of hacking doesn’t involve breaking cryptographic protocols in the field of cryptographic protocols, but in places where those abstractions break down. Sure you use that post quantum cryptographic standard. But then you get an attack that runs a computation that takes 5 or 30 seconds depending on a single bit of the key, and seeing if the cooling fan turns on. Data on what the cooling fan gets up to wasn’t included in the ideal mathematical model in which the problem was NP hard. Or it could be something as stupid as an easily guessable admin password. Or a cryptographic library calls a big int library. The big int library has a debug mode that logs all the arithmetic done. The debug mode can be turned on with a buffer overflow. So turn on debug and check the logs for the private keys. Most hacks aren’t about breaking NP hard math, but about the surrounding software being buggy or imperfect, allowing you do bypass the math.
The NP-hard maths is about the worst case. I can give you a 3sat of a million variables that is trivial to solve. The maths conjectures that it is hard to solve the worst case. Is hacking a particular computer or designing a bioweapon the worst case of a 3 sat problem, or is it one of the easier cases? I don’t know. Large 3 sat problems are often solved in practice, like a million variable problems are solved in minutes. Because NP hardness is about the worst case. And the practical problem people are solving isn’t the worst case.
Are you claiming that designing a lethal bioweapon is NP hard? That building nanotech is NP hard? Like using hacking and social engineering to start a nuclear war is NP hard? What are these large 3 sat problems that must be solved before the world can be destroyed?