None of this should be surprising. It’s always been the case that an AI can’t be perfectly rational , because perfect rationality isn’t computable. It’s never been the case that an AI has to be an agent at all, or must be an efficient optimiser of a utility function. The ubiquitous utility function is a piece of rationalist mythos.
The better your epistemology is, the less surprised you have to be.
Of course there would be AIs like this, and of course that shouldn’t be surprising. But that doesn’t mean it’s incorrect to worry about the capability accumulation of powerfully rational AIs.
It’s so dubious to use assumptions about the rationality of unknown future AIs as a way of reasoning apriori about them. There is very little you can know about unknown future AIs, and for that reason all claims stated with high probability are dubious. Also, “(ir)rational” should not be treated as a binary.
Highly rational is the direction towards capabilities. It’s perfectly reasonable to ask what highly rational AIs can do as long as you also don’t forget to ask about irrational AIs. And agreed that these are directions not binary attributes.
None of this should be surprising. It’s always been the case that an AI can’t be perfectly rational , because perfect rationality isn’t computable. It’s never been the case that an AI has to be an agent at all, or must be an efficient optimiser of a utility function. The ubiquitous utility function is a piece of rationalist mythos.
The better your epistemology is, the less surprised you have to be.
Of course there would be AIs like this, and of course that shouldn’t be surprising. But that doesn’t mean it’s incorrect to worry about the capability accumulation of powerfully rational AIs.
It’s so dubious to use assumptions about the rationality of unknown future AIs as a way of reasoning apriori about them. There is very little you can know about unknown future AIs, and for that reason all claims stated with high probability are dubious. Also, “(ir)rational” should not be treated as a binary.
Highly rational is the direction towards capabilities. It’s perfectly reasonable to ask what highly rational AIs can do as long as you also don’t forget to ask about irrational AIs. And agreed that these are directions not binary attributes.