“Who can say what science does not know? There is far too much science for any one human being to learn. Who can say that we are not ready for a scientific revolution, in advance of the surprise? And if we cannot make progress on Friendly AI because we are not prepared, this does not mean we do not need Friendly AI. Those two statements are not at all equivalent!”
“And if we cannot make progress on Friendly AI because we are not prepared, this does not mean we do not need Friendly AI.”
Friendly AI will be the only thing we need, and unfriendly AI will be the worst possible thing that could happen. If the problem seems impossible to solve, that doesn’t matter because this isn’t about diminishing returns or ROI, it’s about humanity’s final chance at figuring things out. The payoff of a single working solution will be a payoff that continues yielding dividends for billions of years.
“Who can say what science does not know? There is far too much science for any one human being to learn. Who can say that we are not ready for a scientific revolution, in advance of the surprise? And if we cannot make progress on Friendly AI because we are not prepared, this does not mean we do not need Friendly AI. Those two statements are not at all equivalent!”
EY, AI as a pos neg factor, 2006ish
“And if we cannot make progress on Friendly AI because we are not prepared, this does not mean we do not need Friendly AI.”
Friendly AI will be the only thing we need, and unfriendly AI will be the worst possible thing that could happen. If the problem seems impossible to solve, that doesn’t matter because this isn’t about diminishing returns or ROI, it’s about humanity’s final chance at figuring things out. The payoff of a single working solution will be a payoff that continues yielding dividends for billions of years.