For #2: Let’s assume for the sake of simplicity that Bob is a perfect rational agent with a stable utility function—Bob’s utility function (the function itself, not the output) doesn’t change over time. If Bob goes FOOM, Bob is still going to support his original utility function (by definition, because I stipulated that it was stable wrt time above.) I think that you’re wondering whether this stability conflicts with Bob being a perfect (instrumentally) rational agent. It doesn’t—Bob being a perfect rational agent just means that he makes the best choices possible to maximize his utility function given the information and computing power that he has.
You can have a rational paperclip maximizer, or a rational bunny maximizer...pretty much whatever you like.
Note: The whole utility function thing gets more complicated for humans, because human’s utility functions tend not to be stable wrt time, and it gets tricky when you utility is changing while you’re trying to maximize it. Also, we have intuitions which make us happier/more content/whatever when we’re trying to maximize certain types of utility functions, and less so with others (having a nice meal will probably make you happier than would a close friend dying, to use an extreme example) -- we usually want to consider this when picking utility functions.
For #2: Let’s assume for the sake of simplicity that Bob is a perfect rational agent with a stable utility function—Bob’s utility function (the function itself, not the output) doesn’t change over time. If Bob goes FOOM, Bob is still going to support his original utility function (by definition, because I stipulated that it was stable wrt time above.) I think that you’re wondering whether this stability conflicts with Bob being a perfect (instrumentally) rational agent. It doesn’t—Bob being a perfect rational agent just means that he makes the best choices possible to maximize his utility function given the information and computing power that he has.
You can have a rational paperclip maximizer, or a rational bunny maximizer...pretty much whatever you like.
Note: The whole utility function thing gets more complicated for humans, because human’s utility functions tend not to be stable wrt time, and it gets tricky when you utility is changing while you’re trying to maximize it. Also, we have intuitions which make us happier/more content/whatever when we’re trying to maximize certain types of utility functions, and less so with others (having a nice meal will probably make you happier than would a close friend dying, to use an extreme example) -- we usually want to consider this when picking utility functions.