That is a fascinating take! I haven’t heard it put that way before. I think that perspective is a way to understand the gap between old-school agent foundations folks’ high p(doom) and new school LLMers relatively low p(doom) - something I’ve been working to understand, and hope to publish on soon.
To the extent this is true, I think that’s great, because I expect to see some real insights on intelligence as LLMs are turned into functioning minds in cognitive architectures.
Do you have any refs for that take, or is it purely a gestalt?
If it is not a false memory, I’ve seen this on twitter of either EY or Rob Bensinger, but it’s unlikely I find source now, it was in the middle of discussion.
Fair enough, thank you! Regardless, it does seem like a good reason to be concerned about alignment. If you have no idea how intelligence works, how in the world would you know what goals your created intelligence is going to have? At that point, it is like alchemy—performing an incantation and hoping not just that you got it right, but that it does the thing you want.
That is a fascinating take! I haven’t heard it put that way before. I think that perspective is a way to understand the gap between old-school agent foundations folks’ high p(doom) and new school LLMers relatively low p(doom) - something I’ve been working to understand, and hope to publish on soon.
To the extent this is true, I think that’s great, because I expect to see some real insights on intelligence as LLMs are turned into functioning minds in cognitive architectures.
Do you have any refs for that take, or is it purely a gestalt?
If it is not a false memory, I’ve seen this on twitter of either EY or Rob Bensinger, but it’s unlikely I find source now, it was in the middle of discussion.
Fair enough, thank you! Regardless, it does seem like a good reason to be concerned about alignment. If you have no idea how intelligence works, how in the world would you know what goals your created intelligence is going to have? At that point, it is like alchemy—performing an incantation and hoping not just that you got it right, but that it does the thing you want.