Of course there would be AIs like this, and of course that shouldn’t be surprising. But that doesn’t mean it’s incorrect to worry about the capability accumulation of powerfully rational AIs.
It’s so dubious to use assumptions about the rationality of unknown future AIs as a way of reasoning apriori about them. There is very little you can know about unknown future AIs, and for that reason all claims stated with high probability are dubious. Also, “(ir)rational” should not be treated as a binary.
Highly rational is the direction towards capabilities. It’s perfectly reasonable to ask what highly rational AIs can do as long as you also don’t forget to ask about irrational AIs. And agreed that these are directions not binary attributes.
Of course there would be AIs like this, and of course that shouldn’t be surprising. But that doesn’t mean it’s incorrect to worry about the capability accumulation of powerfully rational AIs.
It’s so dubious to use assumptions about the rationality of unknown future AIs as a way of reasoning apriori about them. There is very little you can know about unknown future AIs, and for that reason all claims stated with high probability are dubious. Also, “(ir)rational” should not be treated as a binary.
Highly rational is the direction towards capabilities. It’s perfectly reasonable to ask what highly rational AIs can do as long as you also don’t forget to ask about irrational AIs. And agreed that these are directions not binary attributes.