In reality, an AI can use algorithms that find a pretty good solution most of the time.
If you replace “AI” with “ML” I agree with this point. And yep this is what we can do with the networks we’re scaling. But “pretty good most of the time” doesn’t get you an x-risk intelligence. It gets you some really cool tools.
If the 3 sat algorithm is O(n^4) then this algorithm might not be that useful compared to other approaches.
If 3 SAT is O(n^4) then P=NP and back to Aaronson’s point; the fundamental structure of reality is much different than we think it is. (did you mean “4^N”? Plenty of common algorithms are quartic.)
For the many problems that are “illegible” or “hard for humans to think about” or “confusing”, we are nowhere near the bound, so the AI has room to beat the pants off us with the same data.
The assertion that “illegible” means “requiring more intelligence” rather than “ill-posed” or “underspecified” doesn’t seem obvious to me. Maybe you can expand on this?
Could a superintelligence figure out relativity based on the experiences of the typical caveman?..These clues weren’t enough to lead Einstein to relativity, but Einstein was only human.
I’m not sure I can draw the inference that this means it was possible to generate the theory without the key observations it is explaining. What I’m grasping at is how we can bound what cababilities more intelligence gives an agent. It seems intuitive to me that there must be limits and we can look to physics and math to try to understand them. Which leads us here:
Meaningless. Asymptotic runtime complexity is a mathematical tool that assumes an infinite sequence of ever harder problems.
I disagree. We’ve got a highly speculative question in front of us. “What can a machine intelligence greater than ours accomplish”? We can’t really know what it would be like to be twice as smart any more than an ape can. But if we stipulate that the machine is running on Turing Complete hardware and accept NP hardness then we can at least put an upper bound on the capabilities of this machine.
Concretely, I can box the machine using a post-quantum cryptographic standard and know that it lacks the computational resources to break out before the heat death of the universe. More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
If you replace “AI” with “ML” I agree with this point. And yep this is what we can do with the networks we’re scaling. But “pretty good most of the time” doesn’t get you an x-risk intelligence. It gets you some really cool tools.
On some problems, finding the exact best solution is intractable. On these problems, its all approximations and tricks that usually work. Whether the simplest dumb ML algorithm running with didly squat compute, or some vastly superintelligent AI running on a Matrioska brain.
Take hacking a computer system that controls nukes or something. The problem of finding the fastest way to hack an arbitrary computer system is NP hard. But humans sometimes hack computers without exponentially vast brains. Suppose the AI’s hacking algorithm can hack 99% of all computer systems that are theoretically possible to hack with unlimited compute. And The AI takes at most 10% longer than the theoretically minimum time on those problems.
This AI is still clearly dangerous. Especially if it isn’t just a hacking tool. It has a big picture view of the world and what it wants to achieve.
In short, maybe P!=NP and there is no perfect algoritm, but its possible to be a lot better than current ML, and a lot better than humans, and you don’t need a perfect algorithm to create an X risk.
If 3 SAT is O(n^4) then P=NP and back to Aaronson’s point; the fundamental structure of reality is much different than we think it is. (did you mean “4^N”? Plenty of common algorithms are quartic.)
If 3-sat is O(n^1000,000) then P=NP on a technicality, but the algorithm is totally useless as it is far too slow in practice. If its O(n^4) there are still some uses, but it would seriously hamper the effectiveness of the minimum circuit style predictors. Neural nets are trained on a lot of data. With an O(n^4) algorithm, training beyond 10kb of data will be getting difficult, depending somewhat on the constant.
The assertion that “illegible” means “requiring more intelligence” rather than “ill-posed” or “underspecified” doesn’t seem obvious to me. Maybe you can expand on this?
If you consider the problem of persuading a human to let you out of the box in an AI boxing scenario, well that is perfectly well posed. (There is a big red “open box” button, and either its pressed or it isn’t. ) But we don’t have enough understanding of phycology to do it. Pick some disease we don’t know the cause of or how to cure yet. There will be lots of semi relevant information in biology textbooks. There will be case reports and social media posts and a few confusing clinical trials.
In weather, we know the equations, and there are equations. Any remaining uncertainty is uncertainty about windspeed, temperature etc. But suppose you had a lot of weather data, but hadn’t invented the equations yet. You have a few rules of thumb about how weather patterns move. When you invent the equations, suddenly you can predict so much more.
It seems intuitive to me that there must be limits and we can look to physics and math to try to understand them.
Of course their are bounds. But those bounds are really high on a human scale.
I’m not sure I can draw the inference that this means it was possible to generate the theory without the key observations it is explaining.
I am arguing that there are several facts observable to the average caveman that are explainable by general relativity. Einstein needed data from experiments as well. If you take 10 clues to solve a puzzle, but once you solve it, all the pieces fit beautifully together, that indicates that the problem may have been solvable with fewer clues. It wasn’t that Pythagoras had 10 trillion theories, each as mathematically elegant as general relativity, in mind, and needed experiments to tell which one was true. Arguably Newton had 1 theory like that, and there were still a few clues available that hinted towards relativity.
Concretely, I can box the machine using a post-quantum cryptographic standard and know that it lacks the computational resources to break out before the heat death of the universe. More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
A large fraction of hacking doesn’t involve breaking cryptographic protocols in the field of cryptographic protocols, but in places where those abstractions break down. Sure you use that post quantum cryptographic standard. But then you get an attack that runs a computation that takes 5 or 30 seconds depending on a single bit of the key, and seeing if the cooling fan turns on. Data on what the cooling fan gets up to wasn’t included in the ideal mathematical model in which the problem was NP hard. Or it could be something as stupid as an easily guessable admin password. Or a cryptographic library calls a big int library. The big int library has a debug mode that logs all the arithmetic done. The debug mode can be turned on with a buffer overflow. So turn on debug and check the logs for the private keys. Most hacks aren’t about breaking NP hard math, but about the surrounding software being buggy or imperfect, allowing you do bypass the math.
The NP-hard maths is about the worst case. I can give you a 3sat of a million variables that is trivial to solve. The maths conjectures that it is hard to solve the worst case. Is hacking a particular computer or designing a bioweapon the worst case of a 3 sat problem, or is it one of the easier cases? I don’t know. Large 3 sat problems are often solved in practice, like a million variable problems are solved in minutes. Because NP hardness is about the worst case. And the practical problem people are solving isn’t the worst case.
More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
Are you claiming that designing a lethal bioweapon is NP hard? That building nanotech is NP hard? Like using hacking and social engineering to start a nuclear war is NP hard? What are these large 3 sat problems that must be solved before the world can be destroyed?
If you replace “AI” with “ML” I agree with this point. And yep this is what we can do with the networks we’re scaling. But “pretty good most of the time” doesn’t get you an x-risk intelligence. It gets you some really cool tools.
If 3 SAT is O(n^4) then P=NP and back to Aaronson’s point; the fundamental structure of reality is much different than we think it is. (did you mean “4^N”? Plenty of common algorithms are quartic.)
The assertion that “illegible” means “requiring more intelligence” rather than “ill-posed” or “underspecified” doesn’t seem obvious to me. Maybe you can expand on this?
I’m not sure I can draw the inference that this means it was possible to generate the theory without the key observations it is explaining. What I’m grasping at is how we can bound what cababilities more intelligence gives an agent. It seems intuitive to me that there must be limits and we can look to physics and math to try to understand them. Which leads us here:
I disagree. We’ve got a highly speculative question in front of us. “What can a machine intelligence greater than ours accomplish”? We can’t really know what it would be like to be twice as smart any more than an ape can. But if we stipulate that the machine is running on Turing Complete hardware and accept NP hardness then we can at least put an upper bound on the capabilities of this machine.
Concretely, I can box the machine using a post-quantum cryptographic standard and know that it lacks the computational resources to break out before the heat death of the universe. More abstractly, any AI risk scenario cannot require solving NP problems of more than modest size. (because of completeness, this means many problems and many of the oft-posed risk scenarios are off the table)
On some problems, finding the exact best solution is intractable. On these problems, its all approximations and tricks that usually work. Whether the simplest dumb ML algorithm running with didly squat compute, or some vastly superintelligent AI running on a Matrioska brain.
Take hacking a computer system that controls nukes or something. The problem of finding the fastest way to hack an arbitrary computer system is NP hard. But humans sometimes hack computers without exponentially vast brains. Suppose the AI’s hacking algorithm can hack 99% of all computer systems that are theoretically possible to hack with unlimited compute. And The AI takes at most 10% longer than the theoretically minimum time on those problems.
This AI is still clearly dangerous. Especially if it isn’t just a hacking tool. It has a big picture view of the world and what it wants to achieve.
In short, maybe P!=NP and there is no perfect algoritm, but its possible to be a lot better than current ML, and a lot better than humans, and you don’t need a perfect algorithm to create an X risk.
If 3-sat is O(n^1000,000) then P=NP on a technicality, but the algorithm is totally useless as it is far too slow in practice. If its O(n^4) there are still some uses, but it would seriously hamper the effectiveness of the minimum circuit style predictors. Neural nets are trained on a lot of data. With an O(n^4) algorithm, training beyond 10kb of data will be getting difficult, depending somewhat on the constant.
If you consider the problem of persuading a human to let you out of the box in an AI boxing scenario, well that is perfectly well posed. (There is a big red “open box” button, and either its pressed or it isn’t. ) But we don’t have enough understanding of phycology to do it. Pick some disease we don’t know the cause of or how to cure yet. There will be lots of semi relevant information in biology textbooks. There will be case reports and social media posts and a few confusing clinical trials.
In weather, we know the equations, and there are equations. Any remaining uncertainty is uncertainty about windspeed, temperature etc. But suppose you had a lot of weather data, but hadn’t invented the equations yet. You have a few rules of thumb about how weather patterns move. When you invent the equations, suddenly you can predict so much more.
Of course their are bounds. But those bounds are really high on a human scale.
I am arguing that there are several facts observable to the average caveman that are explainable by general relativity. Einstein needed data from experiments as well. If you take 10 clues to solve a puzzle, but once you solve it, all the pieces fit beautifully together, that indicates that the problem may have been solvable with fewer clues. It wasn’t that Pythagoras had 10 trillion theories, each as mathematically elegant as general relativity, in mind, and needed experiments to tell which one was true. Arguably Newton had 1 theory like that, and there were still a few clues available that hinted towards relativity.
A large fraction of hacking doesn’t involve breaking cryptographic protocols in the field of cryptographic protocols, but in places where those abstractions break down. Sure you use that post quantum cryptographic standard. But then you get an attack that runs a computation that takes 5 or 30 seconds depending on a single bit of the key, and seeing if the cooling fan turns on. Data on what the cooling fan gets up to wasn’t included in the ideal mathematical model in which the problem was NP hard. Or it could be something as stupid as an easily guessable admin password. Or a cryptographic library calls a big int library. The big int library has a debug mode that logs all the arithmetic done. The debug mode can be turned on with a buffer overflow. So turn on debug and check the logs for the private keys. Most hacks aren’t about breaking NP hard math, but about the surrounding software being buggy or imperfect, allowing you do bypass the math.
The NP-hard maths is about the worst case. I can give you a 3sat of a million variables that is trivial to solve. The maths conjectures that it is hard to solve the worst case. Is hacking a particular computer or designing a bioweapon the worst case of a 3 sat problem, or is it one of the easier cases? I don’t know. Large 3 sat problems are often solved in practice, like a million variable problems are solved in minutes. Because NP hardness is about the worst case. And the practical problem people are solving isn’t the worst case.
Are you claiming that designing a lethal bioweapon is NP hard? That building nanotech is NP hard? Like using hacking and social engineering to start a nuclear war is NP hard? What are these large 3 sat problems that must be solved before the world can be destroyed?