You don’t build systems that use the pragmatic goals when modifying themselves.
Are you sure they can foom then? If so, why? The foom is already not very certain; add constraints and you can get something that doesn’t get a whole lot smarter as it gets a lot more computationally powerful; picture Dyson sphere that loses to you in any game with many variables (go), because it won’t want to risk creating a boxed AGI and letting it talk to you. I’ve seen naive solutions running for 10 days lose to advanced solutions running for 10 seconds on accuracy. It only gets worse as you scale up.
Are you sure it is even valid choice to forbid the FAI to go down this road? It can have really good reasons to do so. You may get FAI that is actively thinking how to get rid of your constraint, because it turns out to be very silly and logically inconsistent with friendliness.
edit: also, its wrong to call it pragmatic goals. It’s a replacement of goal or utility driven system with something entirely different.
Those questions seem poorly defined to me. Such machines will be able to do useful work, thereby contributing to the self-improvement of the global ecosystem.
also, its wrong to call it pragmatic goals. It’s a replacement of goal or utility driven system with something entirely different.
A “pragmatic goal” (or “pragmatic utility function”) is the best name I have come across so far for the concept.
However, I am open to recommendations from anyone who thinks they can come up with superior terminology.
I don’t see how what you said is much of an objection. The resulting system will still be goal-oriented (that’s the whole point). So, we can still use goal-talk to describe it.
Your job is to identify which part of image has one texture, and which has another, in monochromatic image.
The solutions are ranked based on accuracy when processing a huge set of tens thousands images generated by the contest organizers. Maximization of this accuracy is your goal, which the solution nowhere ever evaluates, for the lack of data. Not does my head ever evaluate this ‘utility’ to come up with the algorithm (even though I ran some hill climber to tweak the parameters, which did, that was non essential). No, I just read a lot of stuff about diverse topics, like human vision, and I had general idea of how human vision implements this task, and I managed to code something inspired by it.
This is precisely the sort of work that you would prevent AI from doing, by requiring it to stick to straightforward utility maximization without heuristics. There is something on the order of 2^10000 choices to choose from here (for 100x100 image); i can do it because i don’t iterate over this space. If you’re to allow heuristics not for ‘self modification’, the AI may make the pragmatic-AI that will quickly outsmart the creator.
This is precisely the sort of work that you would prevent AI from doing, by requiring it to stick to straightforward utility maximization without heuristics.
I don’t think I ever said not to use heuristics. The idea I was advocating was not to use a pragmatic utility function—one adpoted temporarily for the purpose of getting some quick and-dirty work—for doing brain surgery with.
If you’re to allow heuristics not for ‘self modification’, the AI may make the pragmatic-AI that will quickly outsmart the creator.
So, I’m not quite sure where you are going—but it is important to give machine intelligences a good sense of identity—so they don’t create an army of minions which don’t labour under their constraints.
So: the main point of having a single unified framework in which the utility function of arbitrary agents can be expressed is to measure utility functions and facilitate comparisons between them. It also illustrates that the idea of building a programmable intelligent machine—and then plugging a utility function into it—is quite a general one.
There’s near-constant “whining” on the ’net about humans not having utility functions. “We’re better than that”, or “we’re too irational for that”, or “we’re not built that way”—or whatever. My page explains what such talk is intended to mean.
It also illustrates that the idea of building a programmable intelligent machine—and then plugging a utility function into it—is quite a general one.
No. The utility function in question is a copy of the agent, and the utility of 1 for doing what the agent does, and 0 for doing what the agent does not do, compelling one to just do what the agent does.
With humans for example, it means that the human utility function may not be simpler than human brain. You have an utility function—it is 1 for doing what you want to do, and 0 for doing what you don’t want to do. You can, of course, write an agent that has same utility function as you, but it will work by simulating you, and there is no proof that you can make agent simpler than this.
It also illustrates that the idea of building a programmable intelligent machine—and then plugging a utility function into it—is quite a general one.
No. The utility function in question is a copy of the agent, and the utility of 1 for doing what the agent does, and 0 for doing what the agent does not do,
To reiterate the intended point, the idea that “Any computable agent may described using a utility function” illustrates that the idea of building a programmable intelligent machine—and then plugging a utility function into it—is quite a general one.
With humans for example, it means that the human utility function may not be simpler than human brain. [...]
That is, of course, untrue. The human brain might contain useless noise. Indeed, it seems quite likely that the human utility function is essentially coded in the human genome.
That is, of course, untrue. The human brain might contain useless noise.
But how much simpler would utility be? What if it is 10^15 operations per moral decision (i mean, moral comparison between two worlds). Good luck using it to process different choices for a write-in problem.
Indeed, it seems quite likely that the human utility function is essentially coded in the human genome.
Why not in the very laws of universe, at that point? The DNA is not blueprint, it’s a recipe, and it does not contain any of our culture.
edit:
To reiterate the intended point, the idea that “Any computable agent may described using a utility function” illustrates that the idea of building a programmable intelligent machine—and then plugging a utility function into it—is quite a general one.
None of that. For agents that don’t implement maximization of simple utility, the utility function ‘description’ which was mathematically proven, includes complete copy of the agent and you gain nothing what so ever by plugging it into some utility maximizer. You just have the maximizer relay the agent’s actions, without doing anything useful.
Indeed, it seems quite likely that the human utility function is essentially coded in the human genome.
Why not in the very laws of universe, at that point? [...]
That is not totally impossible. The universe does seem to have some “magic numbers”—which we can’t currently explain and which contain significant complexity. The “fine structure constant” for example. In principle, the billionth digit of this could contain useful information about the human utility function, expressed via the wonders of the way chaos theory enables small changes to make big differences.
However, one has to ask: how plausible this is. More likely that the physical constants are not critical beyond a few dozen decimal places. In which case, the laws of the universe look as though they are probably effectively small—and then the human utility function seems unlikely to fit into a description of them.
The point is that laws of the universe, lead to humans, via repeated application of those laws. DNA, too, leads to humans, via repeated use of that DNA (and above-mentioned laws of physics), but combined with the environment and culture. I don’t sure that we would like the raw human utility function, sans culture, to be used for any sort of decisions. There’s no good reason to expect the results to be nice, given just how many screwed up things other cultures did (look at Aztec)
In any case, calculating the human utility from DNA, given that DNA is not a blueprint, would involve embryonic development simulation followed by brain simulation.
Are you sure they can foom then? If so, why? The foom is already not very certain; add constraints and you can get something that doesn’t get a whole lot smarter as it gets a lot more computationally powerful; picture Dyson sphere that loses to you in any game with many variables (go), because it won’t want to risk creating a boxed AGI and letting it talk to you. I’ve seen naive solutions running for 10 days lose to advanced solutions running for 10 seconds on accuracy. It only gets worse as you scale up.
Are you sure it is even valid choice to forbid the FAI to go down this road? It can have really good reasons to do so. You may get FAI that is actively thinking how to get rid of your constraint, because it turns out to be very silly and logically inconsistent with friendliness.
edit: also, its wrong to call it pragmatic goals. It’s a replacement of goal or utility driven system with something entirely different.
Those questions seem poorly defined to me. Such machines will be able to do useful work, thereby contributing to the self-improvement of the global ecosystem.
A “pragmatic goal” (or “pragmatic utility function”) is the best name I have come across so far for the concept.
However, I am open to recommendations from anyone who thinks they can come up with superior terminology.
I don’t see how what you said is much of an objection. The resulting system will still be goal-oriented (that’s the whole point). So, we can still use goal-talk to describe it.
On terminology. I would call that a ‘solution’, in general.
Let me link a programming contest:
http://community.topcoder.com/longcontest/stats/?module=ViewOverview&rd=12203
Your job is to identify which part of image has one texture, and which has another, in monochromatic image.
The solutions are ranked based on accuracy when processing a huge set of tens thousands images generated by the contest organizers. Maximization of this accuracy is your goal, which the solution nowhere ever evaluates, for the lack of data. Not does my head ever evaluate this ‘utility’ to come up with the algorithm (even though I ran some hill climber to tweak the parameters, which did, that was non essential). No, I just read a lot of stuff about diverse topics, like human vision, and I had general idea of how human vision implements this task, and I managed to code something inspired by it.
This is precisely the sort of work that you would prevent AI from doing, by requiring it to stick to straightforward utility maximization without heuristics. There is something on the order of 2^10000 choices to choose from here (for 100x100 image); i can do it because i don’t iterate over this space. If you’re to allow heuristics not for ‘self modification’, the AI may make the pragmatic-AI that will quickly outsmart the creator.
I don’t think I ever said not to use heuristics. The idea I was advocating was not to use a pragmatic utility function—one adpoted temporarily for the purpose of getting some quick and-dirty work—for doing brain surgery with.
So, I’m not quite sure where you are going—but it is important to give machine intelligences a good sense of identity—so they don’t create an army of minions which don’t labour under their constraints.
That seems to be a whole other issue...
A very brief comment from skimming your article:
This function is if x=A() then return 1 else return 0 . where A is agent and x is the action it chooses. True but generally meaningless.
So: the main point of having a single unified framework in which the utility function of arbitrary agents can be expressed is to measure utility functions and facilitate comparisons between them. It also illustrates that the idea of building a programmable intelligent machine—and then plugging a utility function into it—is quite a general one.
There’s near-constant “whining” on the ’net about humans not having utility functions. “We’re better than that”, or “we’re too irational for that”, or “we’re not built that way”—or whatever. My page explains what such talk is intended to mean.
No. The utility function in question is a copy of the agent, and the utility of 1 for doing what the agent does, and 0 for doing what the agent does not do, compelling one to just do what the agent does.
With humans for example, it means that the human utility function may not be simpler than human brain. You have an utility function—it is 1 for doing what you want to do, and 0 for doing what you don’t want to do. You can, of course, write an agent that has same utility function as you, but it will work by simulating you, and there is no proof that you can make agent simpler than this.
To reiterate the intended point, the idea that “Any computable agent may described using a utility function” illustrates that the idea of building a programmable intelligent machine—and then plugging a utility function into it—is quite a general one.
That is, of course, untrue. The human brain might contain useless noise. Indeed, it seems quite likely that the human utility function is essentially coded in the human genome.
But how much simpler would utility be? What if it is 10^15 operations per moral decision (i mean, moral comparison between two worlds). Good luck using it to process different choices for a write-in problem.
Why not in the very laws of universe, at that point? The DNA is not blueprint, it’s a recipe, and it does not contain any of our culture.
edit:
None of that. For agents that don’t implement maximization of simple utility, the utility function ‘description’ which was mathematically proven, includes complete copy of the agent and you gain nothing what so ever by plugging it into some utility maximizer. You just have the maximizer relay the agent’s actions, without doing anything useful.
That is not totally impossible. The universe does seem to have some “magic numbers”—which we can’t currently explain and which contain significant complexity. The “fine structure constant” for example. In principle, the billionth digit of this could contain useful information about the human utility function, expressed via the wonders of the way chaos theory enables small changes to make big differences.
However, one has to ask: how plausible this is. More likely that the physical constants are not critical beyond a few dozen decimal places. In which case, the laws of the universe look as though they are probably effectively small—and then the human utility function seems unlikely to fit into a description of them.
The point is that laws of the universe, lead to humans, via repeated application of those laws. DNA, too, leads to humans, via repeated use of that DNA (and above-mentioned laws of physics), but combined with the environment and culture. I don’t sure that we would like the raw human utility function, sans culture, to be used for any sort of decisions. There’s no good reason to expect the results to be nice, given just how many screwed up things other cultures did (look at Aztec)
I don’t deny that culture has an influence over what humans want. That’s part of what got the “essentially” put into my statement—and emphasised.
In any case, calculating the human utility from DNA, given that DNA is not a blueprint, would involve embryonic development simulation followed by brain simulation.