There are multiple concepts in the potential of ad-hoc. There is Strong AI, Good AI (Strong AI that has a positive effect), and Useful AI (Strong AI that can be used as a prototype or inspiration for Good AI, but can go Paperclip maximizer if allowed to grow). These concepts can be believed to be in quite different relations to each other.
Your irony states that there is no potential for any Strong AI in ad-hoc. Given that stupid evolution managed to get there, I think that with enough brute force of technology it’s quite feasible to get to Strong AI via this road.
Many reckless people working on AGI think that Strong AI is likely to also be a Good AI.
My previous position was that ad-hoc gives a good chance (in the near future) for Strong AI that is likely a Useful AI, but unlikely a Good AI. My current position is that ad-hoc has a small but decent chance (in the near future) for Strong AI, that is unlikely to be either Useful AI or Good AI.
Good AI is a category containing Friendly AI, that doesn’t require the outcome to be precisely right. This separates more elaborated concept of Friendly AI from an informal concept (requirement) of good outcome.
I believe the concepts are much more close than it seems, that is it’s hard to construct an AI that is not precisely Friendly, but still Good.
FAI is about being reliably harmless. Whether the outcome seems good in the short term is tangential. Even a “good” AI ought to be considered unfriendly if it’s opaque to proof—what can you possibly rely upon? No amount of demonstrated good behavior can be trusted. It could be insincere, it could be sincere but fatally misguided, it could have a flaw that will distort its goals after a few recursions. We would be stupid to just run it and see.
There are multiple concepts in the potential of ad-hoc. There is Strong AI, Good AI (Strong AI that has a positive effect), and Useful AI (Strong AI that can be used as a prototype or inspiration for Good AI, but can go Paperclip maximizer if allowed to grow). These concepts can be believed to be in quite different relations to each other.
Your irony states that there is no potential for any Strong AI in ad-hoc. Given that stupid evolution managed to get there, I think that with enough brute force of technology it’s quite feasible to get to Strong AI via this road.
Many reckless people working on AGI think that Strong AI is likely to also be a Good AI.
My previous position was that ad-hoc gives a good chance (in the near future) for Strong AI that is likely a Useful AI, but unlikely a Good AI. My current position is that ad-hoc has a small but decent chance (in the near future) for Strong AI, that is unlikely to be either Useful AI or Good AI.
BTW, none of the above classifications are “friendly”.
Good AI is a category containing Friendly AI, that doesn’t require the outcome to be precisely right. This separates more elaborated concept of Friendly AI from an informal concept (requirement) of good outcome.
I believe the concepts are much more close than it seems, that is it’s hard to construct an AI that is not precisely Friendly, but still Good.
FAI is about being reliably harmless. Whether the outcome seems good in the short term is tangential. Even a “good” AI ought to be considered unfriendly if it’s opaque to proof—what can you possibly rely upon? No amount of demonstrated good behavior can be trusted. It could be insincere, it could be sincere but fatally misguided, it could have a flaw that will distort its goals after a few recursions. We would be stupid to just run it and see.
At which point you are starting to think of what it takes to make not just informally “Good” AI, but an actually Friendly AI.