it has one goal in mind, maximizing the number of paperclips in the universe
In which universe? It doesn’t know. And it may have uncertainty with regards to true number. There’s going to be hypothetical universes that produce same observations but have ridiculously huge amounts of invisible paperclips at stake, which are influenced by paperclipper’s actions (it may even be that the simplest extra addition that makes agent’s actions influence invisible paperclips would utterly dominate all theories starting from some length, as it leaves most length for a busy beaver like construction that makes the amount of insivisible paperclips ridiculously huge. One extra bit for a busy beaver is seriously a lot more paperclips). So given some sort of length prior that ignores size of hypothetical universe (the kind that won’t discriminate against MWI just because its big), those aren’t assigned low enough prior, and dominate it’s expected utility calculations.
In which universe? It doesn’t know. And it may have uncertainty with regards to true number. There’s going to be hypothetical universes that produce same observations but have ridiculously huge amounts of invisible paperclips at stake, which are influenced by paperclipper’s actions (it may even be that the simplest extra addition that makes agent’s actions influence invisible paperclips would utterly dominate all theories starting from some length, as it leaves most length for a busy beaver like construction that makes the amount of insivisible paperclips ridiculously huge. One extra bit for a busy beaver is seriously a lot more paperclips). So given some sort of length prior that ignores size of hypothetical universe (the kind that won’t discriminate against MWI just because its big), those aren’t assigned low enough prior, and dominate it’s expected utility calculations.