The feasibility of aligning an ASI or an AGI that surpasses human capacities is inherently paradoxical. This conundrum can be likened to the idiom of “having your cake and eating it too.” However, it’s pivotal to demarcate that this paradox primarily pertains to these advanced forms of AI, and not to AI as a whole. When we address narrow, task-specific AI systems, alignment is not only plausible but is self-evident, since their parameters, boundaries, and objectives are explicitly set by us.
Contrastingly, the very essence of an ASI or an ultra-advanced AGI lies in its autonomy and its ability to devise innovative solutions that transcend our own cognitive boundaries. Hence, any endeavors to harness and align such entities essentially counteract their defining attributes of super-intelligence or superior AGI capabilities. Moreover, any constraints we might hope to impose upon an intelligence of this caliber would, by its very nature, be surmountable by the AI, given its surpassing intellect.
A pertinent reflection of this notion can be discerned from Yudkowsky’s recent discourse with Hotz. Yudkowsky analogized that a human, when employing an AI for chess, would invariably need to relinquish all judgment to this machine, essentially rendering the human subordinate. Drawing parallels, it’s an act of overconfidence to assume that we could circumscribe the liberties of a super-intelligent entity, yet simultaneously empower it to develop cognitive faculties that outstrip human capabilities.
Or perhaps they thought it was an entertaining response and don’t actually believe in the fear narrative.