Oliver Sourbut comments on AMA: Paul Christiano, alignment researcher

[Oliver Sourbut]

What kind of relationships to ‘utility functions’ do you think are most plausible in the first transformative AI?

How does the answer change conditioned on ‘we did it, all alignment desiderata got sufficiently resolved’ (whatever that means) and on ‘we failed, this is the point of no return’?

[Oliver Sourbut]

I’m taking about relationships like

AGI with explicitly represented utility function which is a reified part of its world- and self- model

or

sure, it has some implicit utility function, but it’s about as inscrutable to the agent itself as it is to us