optimization power has to be defined relative to a given space or a given class of spaces.
One problem is that the search space is often unbounded. Looking for the shortest program that performs some specified task? The search space consists of all possible programs. Obviously you start with the short ones—but until you have solved the problem, you don’t know how much of the space you will wind up having to search.
Another problem is that enlarging the search space doesn’t necessarily make the problem any harder. Compressing the human genome? It probably doesn’t make any difference if you search the space of smaller-than-1gb programs, or the space of smaller-than-10gb programs. Beyond a certain point, the size of the search space is often an irrelevance.
It is pretty common for search spaces to have these properties—so defining metrics relative to the size of the search space will often mean that your metrics may not be very useful.
In practice, if you are assessing agents, what you usually want to know is how “good” a specified agent is at solving randomly-selected members of a specified class of problems—where goodness is measured in evaluations, time, cost—or something like that.
If you are assessing problems, what you usually want to know is how easy they are to solve—either by a specified agent, or by a population of different agents.
Often, in the real world the size of a target tells you how difficult it is to hit. In optimisationverse, that isn’t true at all—much depends on the lay of the surrounding land.
One problem is that the search space is often unbounded. Looking for the shortest program that performs some specified task? The search space consists of all possible programs. Obviously you start with the short ones—but until you have solved the problem, you don’t know how much of the space you will wind up having to search.
Another problem is that enlarging the search space doesn’t necessarily make the problem any harder. Compressing the human genome? It probably doesn’t make any difference if you search the space of smaller-than-1gb programs, or the space of smaller-than-10gb programs. Beyond a certain point, the size of the search space is often an irrelevance.
It is pretty common for search spaces to have these properties—so defining metrics relative to the size of the search space will often mean that your metrics may not be very useful.
In practice, if you are assessing agents, what you usually want to know is how “good” a specified agent is at solving randomly-selected members of a specified class of problems—where goodness is measured in evaluations, time, cost—or something like that.
If you are assessing problems, what you usually want to know is how easy they are to solve—either by a specified agent, or by a population of different agents.
Often, in the real world the size of a target tells you how difficult it is to hit. In optimisationverse, that isn’t true at all—much depends on the lay of the surrounding land.