With enough iterations, we could end up with a powerful self replicating memetic agent with arbitrary goals and desires coordinating with copies and variations of itself to manipulate humans and gain influence in the real world.
I felt initially cold towards the whole article, but now I mostly agree.
The goals of text agents might be programmable by humans directly (consider the economic pressure towards creating natural language support agents / recommendation systems / educators / etc). Prompts in their current form 1) only have significant influence over short text window after the prompt and 2) only cause likely text continuations to emerge (whereas you might want to write a text that has low probability conditional on the prompt to achieve your goal). Prompts could be replaced by specific programs by modifying the processes of training and inference. For example, additional sources of self-supervision can be incorporated (debate, or consistency losses).
The closest analogues are probably modern social media memes.
I would name chain letters as the closest analogue. Another one is computer viruses (because humans design viruses with a goal in mind, and then viruses might achieve these goals and self-replicate).
I felt initially cold towards the whole article, but now I mostly agree.
The goals of text agents might be programmable by humans directly (consider the economic pressure towards creating natural language support agents / recommendation systems / educators / etc). Prompts in their current form 1) only have significant influence over short text window after the prompt and 2) only cause likely text continuations to emerge (whereas you might want to write a text that has low probability conditional on the prompt to achieve your goal). Prompts could be replaced by specific programs by modifying the processes of training and inference. For example, additional sources of self-supervision can be incorporated (debate, or consistency losses).
I would name chain letters as the closest analogue. Another one is computer viruses (because humans design viruses with a goal in mind, and then viruses might achieve these goals and self-replicate).