And this is where the fundamental AGI-doom arguments – all these coherence theorems, utility-maximization frameworks, et cetera – come in. At their core, they’re claims that any “artificial generally intelligent system capable of autonomously optimizing the world the way humans can” would necessarily be well-approximated as a game-theoretic agent.
These “coherence theorems” rest on very shaky ground. See this post. I don’t think it is appropriate to merely gesture at the existence of such theorems, as if they are widely accepted as unproblematic, without pointing to a specific one. There are other, informal, ways to make a similar point, but acting as if there are well-established, substantial proofs of the matter is not justified.
That post is clickbait. It only argues that the incompleteness money pump doesn’t work. The reasons the incompleteness money pump does work is well summarized at a high level here or here (more specifically here and here, if we get into details).
Admittedly, I can’t judge all the technical details. But I notice that neither So8res nor Wentworth have engaged with the EJT post (neither directly in the comments nor in their posts you have linked), despite being published later. And EJT’s engagement with the Wentworth post didn’t elicit much of a reaction either. So from an outside view, the viability of coherence arguments seems questionable.
My third link is down the thread from your link. I agree that from an outside view it’s difficulty to work out who is right. Unfortunately in this case one has to actually work through the details.
These “coherence theorems” rest on very shaky ground. See this post. I don’t think it is appropriate to merely gesture at the existence of such theorems, as if they are widely accepted as unproblematic, without pointing to a specific one. There are other, informal, ways to make a similar point, but acting as if there are well-established, substantial proofs of the matter is not justified.
That post is clickbait. It only argues that the incompleteness money pump doesn’t work. The reasons the incompleteness money pump does work is well summarized at a high level here or here (more specifically here and here, if we get into details).
Admittedly, I can’t judge all the technical details. But I notice that neither So8res nor Wentworth have engaged with the EJT post (neither directly in the comments nor in their posts you have linked), despite being published later. And EJT’s engagement with the Wentworth post didn’t elicit much of a reaction either. So from an outside view, the viability of coherence arguments seems questionable.
My third link is down the thread from your link. I agree that from an outside view it’s difficulty to work out who is right. Unfortunately in this case one has to actually work through the details.