Considered. Not convinced. If that was intended as an argument, then EY was having a very bad day.
He is welcome to his opinion but he is not welcome to substitute his for mine.
The ending was particularly bizarre. It sounded like he was saying that treasury bills don’t pay enough interest to make up for the risk that the US may not be here 300 years from now. But we should, for example, consider the projected enjoyment of people we imagine visiting our nature preserves 500 years from now, as if their enjoyment were as important as our own, not discounting at all for the risk that they may not even exist.
But we should, for example, consider the projected enjoyment of people we imagine visiting our nature preserves 500 years from now, as if their enjoyment were as important as our own, not discounting at all for the risk that they may not even exist.
Eliezer doesn’t disagree: as he says more than once, he’s talking about pure preferences, intrinsic values. Other risks do need to be incorporated, but it seems better to do so directly, rather than through a discounting heuristic. Larks seems to implicitly be doing this with his P(AGI) = 10^-9.
Time travel, the past “still existing”—and utilitariainism? I don’t buy any of that either—but in the context of artificial intelligence, I do agree that building discounting functions into the agent’s ultimate values looks like bad news.
Discounting functions arise because agents don’t know about the future—and can’t predict or control it very well. However, the extent to which they can’t predict or control it is a function of the circumstances and their own capabilities. If you wire temporal discounting into the ultimate preferences of super-Deep Blue—then it can’t ever self-improve to push its prediction horizon further out as it gets more computing power! You are unnecessarily building limitations into it. Better to have no temporal discounting wired in—and let the machine itself figure out to what extent it can predict and control the future—and so figure out the relative value of the present.
For 2, perhaps consider:
http://lesswrong.com/lw/n2/against_discount_rates/
Considered. Not convinced. If that was intended as an argument, then EY was having a very bad day.
He is welcome to his opinion but he is not welcome to substitute his for mine.
The ending was particularly bizarre. It sounded like he was saying that treasury bills don’t pay enough interest to make up for the risk that the US may not be here 300 years from now. But we should, for example, consider the projected enjoyment of people we imagine visiting our nature preserves 500 years from now, as if their enjoyment were as important as our own, not discounting at all for the risk that they may not even exist.
Eliezer doesn’t disagree: as he says more than once, he’s talking about pure preferences, intrinsic values. Other risks do need to be incorporated, but it seems better to do so directly, rather than through a discounting heuristic. Larks seems to implicitly be doing this with his P(AGI) = 10^-9.
Time travel, the past “still existing”—and utilitariainism? I don’t buy any of that either—but in the context of artificial intelligence, I do agree that building discounting functions into the agent’s ultimate values looks like bad news.
Discounting functions arise because agents don’t know about the future—and can’t predict or control it very well. However, the extent to which they can’t predict or control it is a function of the circumstances and their own capabilities. If you wire temporal discounting into the ultimate preferences of super-Deep Blue—then it can’t ever self-improve to push its prediction horizon further out as it gets more computing power! You are unnecessarily building limitations into it. Better to have no temporal discounting wired in—and let the machine itself figure out to what extent it can predict and control the future—and so figure out the relative value of the present.