“ask for too much and you wind up with nothing” is a fine fairy tale moral. Does it actually hold in these particular circumstances?
Imagine that there’s a landscape of possible words. There is a function (A) on this landscape, we don’t know how to define it, but it is how much we truly would prefer a world if only we knew. Somewhere this function has a peak, the most ideal “eutopia”. There is another function. This one we do define. It is intended to approximate the first function, but it does not do so perfectly. Our “acceptability criteria” is to require that this second function (B) has a value at least some threshold.
Now as we raise the acceptability criteria (threshold for function B), we might expect there to be two different regimes. In a first regime with low acceptability criteria, Function B is not that bad a proxy for function A, and raising the threshold increases the average true desirability of the worlds that meet it. In a second regime with high acceptability criteria, function B ceases to be effective as a proxy. Here we are asking for “too much”. The peak of function B is at a different place than the peak of function A, and as we raise the threshold high enough we exclude the peak of A entirely. What we end up with is a world highly optimized for B and not so well optimized for A—a “marketing world”.
So, we must conclude, like you and Stuart Armstrong, that asking for “too much” is bad and we’d better set a lower threshold. Case closed, right?
Wrong.
The problem is that the above line of reasoning provides no reason to believe that the “marketing world” at the peak of function B is any worse than a random world at any lower threshold. As we relax the threshold on B, we include more worlds that are better in terms of A but also more that are worse. There’s no particular reason to believe, simply because the peak of B is at a different place than the peak of A, that the peak of B is at a valley of A. In fact, if B represents our best available estimate of A, it would seem that, even though the peak of B is predictably a marketing world, it’s still our best bet at getting a good value of A. A random world at any lower threshold should have a lower expected value of A.
The problem is that the above line of reasoning provides no reason to believe that the “marketing world” at the peak of function B is any worse than a random world at any lower threshold.
True. Which is why I added arguments pointing that a marketing world will likely be bad. Even on your terms, a peak of B will probably involve a diversion of effort/energy that could have contributed to A, away from A. eg if A is apples and B is bananas, the world with the most bananas is likely to contain no apples at all.
“ask for too much and you wind up with nothing” is a fine fairy tale moral. Does it actually hold in these particular circumstances?
Imagine that there’s a landscape of possible words. There is a function (A) on this landscape, we don’t know how to define it, but it is how much we truly would prefer a world if only we knew. Somewhere this function has a peak, the most ideal “eutopia”. There is another function. This one we do define. It is intended to approximate the first function, but it does not do so perfectly. Our “acceptability criteria” is to require that this second function (B) has a value at least some threshold.
Now as we raise the acceptability criteria (threshold for function B), we might expect there to be two different regimes. In a first regime with low acceptability criteria, Function B is not that bad a proxy for function A, and raising the threshold increases the average true desirability of the worlds that meet it. In a second regime with high acceptability criteria, function B ceases to be effective as a proxy. Here we are asking for “too much”. The peak of function B is at a different place than the peak of function A, and as we raise the threshold high enough we exclude the peak of A entirely. What we end up with is a world highly optimized for B and not so well optimized for A—a “marketing world”.
So, we must conclude, like you and Stuart Armstrong, that asking for “too much” is bad and we’d better set a lower threshold. Case closed, right?
Wrong.
The problem is that the above line of reasoning provides no reason to believe that the “marketing world” at the peak of function B is any worse than a random world at any lower threshold. As we relax the threshold on B, we include more worlds that are better in terms of A but also more that are worse. There’s no particular reason to believe, simply because the peak of B is at a different place than the peak of A, that the peak of B is at a valley of A. In fact, if B represents our best available estimate of A, it would seem that, even though the peak of B is predictably a marketing world, it’s still our best bet at getting a good value of A. A random world at any lower threshold should have a lower expected value of A.
True. Which is why I added arguments pointing that a marketing world will likely be bad. Even on your terms, a peak of B will probably involve a diversion of effort/energy that could have contributed to A, away from A. eg if A is apples and B is bananas, the world with the most bananas is likely to contain no apples at all.