Presumably we will build ML AGIs because they are safe
I don’t see anything in the structure of humanity’s AGI-development process that would ensure this property. LLM human imitations are only plausibly aligned because they are imitations of humans. There are other active lines of research vying with them for the first AGI, with no hope for their safety.
For the moment, LLM characters have the capability advantage of wielding human faculties, not needing to reinvent alternatives for them from scratch. This is an advantage for crossing the AGI threshold, which humans already crossed, but not for improving further than that. There is nothing in this story that predicates the outcome on safety.
I don’t see anything in the structure of humanity’s AGI-development process that would ensure this property. LLM human imitations are only plausibly aligned because they are imitations of humans. There are other active lines of research vying with them for the first AGI, with no hope for their safety.
For the moment, LLM characters have the capability advantage of wielding human faculties, not needing to reinvent alternatives for them from scratch. This is an advantage for crossing the AGI threshold, which humans already crossed, but not for improving further than that. There is nothing in this story that predicates the outcome on safety.