Upvoted for raising some very important topics. But I disagree on a few points.
One is the assumption that ‘subjective time’ is related to the discount rate—that if a super-intelligence can do as much thinking in a day as we can do in a century, then it will care as little about tomorrow as we care about the next century. I would make a different assumption—that the ‘natural’ discount rate is more closely related to the machine’s expected lifetime (when it expects indexical utility flows to cease) and to its planning horizon (when its expectations regarding the future environment become no better than guesses).
The second is the failure to distinguish communication latencies from communication bandwidths. Both are important, but they play different roles. According to some theories of consciousness, it is an essentially serial phenomenon, and hence latencies matter a lot. So, while it may be possible to construct a mind whose physical substrate is distributed between Earth and Jupiter’s moons, it probably won’t be possible to construct a consciousness divided in this way. At least not a consciousness that could pass a Turing test.
I didn’t mean to imply that subjective time is related to the discount rate, and I tend to agree that the ‘natural’ discount rate and planning horizon is probably related to expected lifetime for most agents. But it’s difficult to to show why this should always tend to be so.
The time dilation for extremely fast thinkers will slow down the subjective rate of return of Moore’s Law type investments just as much as space expansion type investments, that’s not really the core of the argument against expansion.
The second is the failure to distinguish communication latencies from communication bandwidths.
Where did I confuse these two? I discussed both. Latency subjectively increases with rate of thought and bandwidth decreases, respectively. They both contribute to divergence.
Upvoted for raising some very important topics. But I disagree on a few points.
One is the assumption that ‘subjective time’ is related to the discount rate—that if a super-intelligence can do as much thinking in a day as we can do in a century, then it will care as little about tomorrow as we care about the next century. I would make a different assumption—that the ‘natural’ discount rate is more closely related to the machine’s expected lifetime (when it expects indexical utility flows to cease) and to its planning horizon (when its expectations regarding the future environment become no better than guesses).
The second is the failure to distinguish communication latencies from communication bandwidths. Both are important, but they play different roles. According to some theories of consciousness, it is an essentially serial phenomenon, and hence latencies matter a lot. So, while it may be possible to construct a mind whose physical substrate is distributed between Earth and Jupiter’s moons, it probably won’t be possible to construct a consciousness divided in this way. At least not a consciousness that could pass a Turing test.
I completely agree with your points.
I didn’t mean to imply that subjective time is related to the discount rate, and I tend to agree that the ‘natural’ discount rate and planning horizon is probably related to expected lifetime for most agents. But it’s difficult to to show why this should always tend to be so.
The time dilation for extremely fast thinkers will slow down the subjective rate of return of Moore’s Law type investments just as much as space expansion type investments, that’s not really the core of the argument against expansion.
Where did I confuse these two? I discussed both. Latency subjectively increases with rate of thought and bandwidth decreases, respectively. They both contribute to divergence.