(I am in the midst of reading the EY-RH “FOOM” debate, so some of the following may be less informed than would be ideal.)
From a purely technical standpoint, one problem is that if you permit self-modification, and give the baby AI enough insight into its own structure to make self-modification remotely a useful thing to do (as opposed to making baby repeatedly crash, burn, and restore from backup), then you cannot guarantee that utility() won’t be modified in arbitrary ways. Even if you store the actual code implementing utility() in ROM, baby could self-modify to replace all references to that fixed function with references to a different (modifiable) one.
What you need is for utility() to be some kind of fixed point in utility-function space under whatever modification regime is permitted, or… something. This problem seems nigh-insoluble to me, at the moment. Even if you solve the theoretical problem of preserving those aspects of utility() that ensure Friendliness, a cosmic-ray hit might change a specific bit of memory and turn baby into a monster. (Though I suppose you could arrange, mathematically, for that particular possibility to be astronomically unlikely.)
Survey taken. I hope I didn’t break it—I am a committed atheist, but also an active member of a Unitarian Universalist congregation, and I indicated that in spite of the explicit request for atheists not to answer the denomination question. (Atheist UUs are very common, and people on the “agnostic or less religious” side of the spectrum probably make up around 40% of the UU congregations I’m familiar with.)