It’s still extremely relevant since they have to grapple with watered-down versions of many of the exact same problems.
You might be concerned that a non-FAI will optimize for some scoring function and do things you don’t want while they’re actually dealing with the actual nuts and bolts of making modern AI’s where they want to make sure they don’t optimize for some scoring function and do things you don’t want (on a more mundane level). That kind of problem is in the first few pages of many AI textbooks yet the applause lights here hold that almost all AI researchers are blind to such possibilities.
There’s no need to convince me that general AI is possible in principle. We can use the same method to prove that nanobots and self replicating von neumann machines are perfectly possible but we’re still a lot way from actually building them.
it’s just frustrating: like watching someone trying to explain why proving code is important in the control software of a nuclear reactor (extremely true) who has no idea how code is proven, has never written even a hello-world program and every now and then talks as if they believe that exception handling is unknown to programmers mixed with references to magic. They’re making a reasonable point but mixing their language with references to magic and occasional absurdities.
Still, if someone is capable of grasping the argument “Any kind of software failure in the control systems of a nuclear reactor could have disastrous consequences; the total amount of software required isn’t too enormous; therefore it is worth going to great lengths, including formal correctness proofs, to ensure that the software is correct” then they’re right to make that argument even if their grasp of what kind of software is used for controlling a nuclear reactor is extremely tenuous. And if they say ”… because otherwise the reactor could explode and turn everyone in the British Isles into a hideous mutant with weird superpowers” then of course they’re hilariously wrong, but their wrongness is about the details of the catastrophic disaster rather than the (more important) fact that a catastrophic disaster could happen and needs preventing.
That’s absolutely true but it leads to two problems.
First: the obvious lack of experience/understanding of the nuts and bolts of it makes people from outside less likely to take the realistic parts of their warning seriously and may even lead to it being viewed as a subject of mockery which works against you.
Second: The failure modes that people suggest due to their lack of understanding can also be hilariously wrong like “a breach may be caused by the radiation giving part of the shielding magical superpowers which will then cause it to gain life and open a portal to R’lyeh” and they may even spend many paragraphs on talking about how serious that failure mode it while others who also don’t actually understand politely aplaude. This has some of the same unfortunate side effects: it makes people who are totally unfamiliar with the subject less likely to take the realistic parts of their warning seriously.
It’s still extremely relevant since they have to grapple with watered-down versions of many of the exact same problems.
You might be concerned that a non-FAI will optimize for some scoring function and do things you don’t want while they’re actually dealing with the actual nuts and bolts of making modern AI’s where they want to make sure they don’t optimize for some scoring function and do things you don’t want (on a more mundane level). That kind of problem is in the first few pages of many AI textbooks yet the applause lights here hold that almost all AI researchers are blind to such possibilities.
There’s no need to convince me that general AI is possible in principle. We can use the same method to prove that nanobots and self replicating von neumann machines are perfectly possible but we’re still a lot way from actually building them.
it’s just frustrating: like watching someone trying to explain why proving code is important in the control software of a nuclear reactor (extremely true) who has no idea how code is proven, has never written even a hello-world program and every now and then talks as if they believe that exception handling is unknown to programmers mixed with references to magic. They’re making a reasonable point but mixing their language with references to magic and occasional absurdities.
Yeah, I understand the frustration.
Still, if someone is capable of grasping the argument “Any kind of software failure in the control systems of a nuclear reactor could have disastrous consequences; the total amount of software required isn’t too enormous; therefore it is worth going to great lengths, including formal correctness proofs, to ensure that the software is correct” then they’re right to make that argument even if their grasp of what kind of software is used for controlling a nuclear reactor is extremely tenuous. And if they say ”… because otherwise the reactor could explode and turn everyone in the British Isles into a hideous mutant with weird superpowers” then of course they’re hilariously wrong, but their wrongness is about the details of the catastrophic disaster rather than the (more important) fact that a catastrophic disaster could happen and needs preventing.
That’s absolutely true but it leads to two problems.
First: the obvious lack of experience/understanding of the nuts and bolts of it makes people from outside less likely to take the realistic parts of their warning seriously and may even lead to it being viewed as a subject of mockery which works against you.
Second: The failure modes that people suggest due to their lack of understanding can also be hilariously wrong like “a breach may be caused by the radiation giving part of the shielding magical superpowers which will then cause it to gain life and open a portal to R’lyeh” and they may even spend many paragraphs on talking about how serious that failure mode it while others who also don’t actually understand politely aplaude. This has some of the same unfortunate side effects: it makes people who are totally unfamiliar with the subject less likely to take the realistic parts of their warning seriously.