Humans are a proof by example that you can have a general AI which is not a very good utility function maximizer; arguably they even suggest that this is what you get by default.
I mostly buy that eventually you get a “spark of consequentialism” in some sense, but that might actually happen at a substantially superhuman level.
I’m not sure this actually extends timelines very much if it is true, but I’m updating towards expecting the foom to happen a little later into the AI improvement curve.
That’s fair, my brain might be doing great at maximizing something which isn’t especially correlated with what I actually want / what actually makes me happy.
Humans are a proof by example that you can have a general AI which is not a very good utility function maximizer; arguably they even suggest that this is what you get by default.
I mostly buy that eventually you get a “spark of consequentialism” in some sense, but that might actually happen at a substantially superhuman level.
I’m not sure this actually extends timelines very much if it is true, but I’m updating towards expecting the foom to happen a little later into the AI improvement curve.
...to the extent that evolved intelligence is similar to AI, at least.
[edited]
That’s fair, my brain might be doing great at maximizing something which isn’t especially correlated with what I actually want / what actually makes me happy.