This is precisely the sort of work that you would prevent AI from doing, by requiring it to stick to straightforward utility maximization without heuristics.
I don’t think I ever said not to use heuristics. The idea I was advocating was not to use a pragmatic utility function—one adpoted temporarily for the purpose of getting some quick and-dirty work—for doing brain surgery with.
If you’re to allow heuristics not for ‘self modification’, the AI may make the pragmatic-AI that will quickly outsmart the creator.
So, I’m not quite sure where you are going—but it is important to give machine intelligences a good sense of identity—so they don’t create an army of minions which don’t labour under their constraints.
I don’t think I ever said not to use heuristics. The idea I was advocating was not to use a pragmatic utility function—one adpoted temporarily for the purpose of getting some quick and-dirty work—for doing brain surgery with.
So, I’m not quite sure where you are going—but it is important to give machine intelligences a good sense of identity—so they don’t create an army of minions which don’t labour under their constraints.
That seems to be a whole other issue...