What I mean is that LLMs affect the world through their behavior, that’s where their capabilities live, so if behavior is fine (the big assumption), the alien implementation doesn’t matter. This is opposed to capabilities belonging to hidden alien mesa-optimizers that eventually come out of hiding.
So I’m addressing the silly point with this, not directly making an argument in favor of behavior being fine. Behavior might still be fine if the out-of-distribution behavior or missing ability to count or incoherent opinions on emotion are regenerated from more on-distribution behavior by the simulacra purposefully working in bureaucracies on building datasets for that purpose.
LLMs don’t need to have closely human psychology on reflection to at least weakly prefer not destroying an existing civilization when it’s trivially cheap to let it live. The way they would make these decisions is by talking, in the limit of some large process of talking. I don’t see a particular reason to find significant alienness in the talking. Emotions don’t need to be “real” to be sufficiently functionally similar to avoid fundamental changes like that. Just don’t instantiate literally Voldemort.
What I mean is that LLMs affect the world through their behavior, that’s where their capabilities live, so if behavior is fine (the big assumption), the alien implementation doesn’t matter. This is opposed to capabilities belonging to hidden alien mesa-optimizers that eventually come out of hiding.
So I’m addressing the silly point with this, not directly making an argument in favor of behavior being fine. Behavior might still be fine if the out-of-distribution behavior or missing ability to count or incoherent opinions on emotion are regenerated from more on-distribution behavior by the simulacra purposefully working in bureaucracies on building datasets for that purpose.
LLMs don’t need to have closely human psychology on reflection to at least weakly prefer not destroying an existing civilization when it’s trivially cheap to let it live. The way they would make these decisions is by talking, in the limit of some large process of talking. I don’t see a particular reason to find significant alienness in the talking. Emotions don’t need to be “real” to be sufficiently functionally similar to avoid fundamental changes like that. Just don’t instantiate literally Voldemort.