Could you frame the debate to avoid ambiguity? What argument do you refer to (in your own words)? In what way is it circular? (I feel that the structure of the argument is roughly that the answer to the question “what is 2+2?” is “4″, because the algebraic laws assumed in the question imply 4 as the answer, even though other algebraic laws can lead to other answers.)
For example, if some other rule turns out to be even more conducive to evolution, the anthropic question arises: why aren’t we in that world instead of this one?
We just aren’t, this question has no meaning. Why are you you, and not someone else? When you refer to yourself, you identify a particular concept (of “yourself”). That concept is distinct from other concepts, and that’s the end of it. Two given concepts are not identical, as defined.
It’s entirely possible that other rules (measures) are also conductive to evolution, but look at them as something happening “far away”, like in universes with different fundamental constants. And over there, other creatures could’ve also biologically evolved. I’m not arguing with that, so finding other rules that produce good-enough physical processes doesn’t answer any questions. Why am I a human, and not a dolphin?
An FAI that doesn’t help our world is a big fat piece of fail. Can I please have a machine that’s based on less lofty abstractions, but actually does stuff?
We can’t outright assume anything about preference. We need to actually understand it. Powerful optimization is bound to be weird, so absurdity heuristic goes out the window. And correspondingly, the necessary standard of understanding goes up a dozen of notches. We are so far away from the adequate level that if a random AGI is built 30 year from now, we still almost certainly fail to beat it. Maybe 50 or 100 years (at which point uploads start influencing progress) sounds more reasonable, judging by the rate of progress in mathematics. We need to work faster.
You are committing the general error of prematurely declaring a question “dissolved”. It’s always better to err in the other direction. That’s how I come up with all my weird models, anyway.
I just took a little walk outside and this clarification occurred to me: imagine an algorithm (Turing machine) running on a classical physical computer, sitting on a table in our quantum universe. The computer has the interesting property that it is “stable” under the Born rule: a weighted-majority of near futures ranked by the 2-norm have the computer correctly executing the next few steps of the computation, but for the 1-norm this isn’t necessarily the case—the computer will likely glitch or self-destruct. (All computers built by humans probably have this property. Also note that it can be defined in terms of the wavefunction alone, without assuming weights a priori.) Then the algorithm will have “subjective anticipation” of a weird kind: conditioned on the algorithm itself running faithfully in the future, it can conclude that future histories with higher Born-weight are more likely.
This idea has the drawback that it doesn’t look at histories of the outside world, only the computer’s internals. But maybe it can be extended to include observations somehow?
You are committing the general error of prematurely declaring the question “dissolved”. It’s always better to err in the other direction.
“Beginning to feel” that the question is dissolved is far from the level of certainty required to “declare it dissolved”, merely a hunch that it’s the right direction to look for the answer (not that it’s a question I’m especially interested in, but it might be useful to understand it better).
I agree with your description in the second paragraph, but don’t clearly see what you wanted to communicate through it. (Closest salient idea is Hanson’s “mingled worlds”.)
Could you frame the debate to avoid ambiguity? What argument do you refer to (in your own words)? In what way is it circular? (I feel that the structure of the argument is roughly that the answer to the question “what is 2+2?” is “4″, because the algebraic laws assumed in the question imply 4 as the answer, even though other algebraic laws can lead to other answers.)
We just aren’t, this question has no meaning. Why are you you, and not someone else? When you refer to yourself, you identify a particular concept (of “yourself”). That concept is distinct from other concepts, and that’s the end of it. Two given concepts are not identical, as defined.
It’s entirely possible that other rules (measures) are also conductive to evolution, but look at them as something happening “far away”, like in universes with different fundamental constants. And over there, other creatures could’ve also biologically evolved. I’m not arguing with that, so finding other rules that produce good-enough physical processes doesn’t answer any questions. Why am I a human, and not a dolphin?
We can’t outright assume anything about preference. We need to actually understand it. Powerful optimization is bound to be weird, so absurdity heuristic goes out the window. And correspondingly, the necessary standard of understanding goes up a dozen of notches. We are so far away from the adequate level that if a random AGI is built 30 year from now, we still almost certainly fail to beat it. Maybe 50 or 100 years (at which point uploads start influencing progress) sounds more reasonable, judging by the rate of progress in mathematics. We need to work faster.
You are committing the general error of prematurely declaring a question “dissolved”. It’s always better to err in the other direction. That’s how I come up with all my weird models, anyway.
I just took a little walk outside and this clarification occurred to me: imagine an algorithm (Turing machine) running on a classical physical computer, sitting on a table in our quantum universe. The computer has the interesting property that it is “stable” under the Born rule: a weighted-majority of near futures ranked by the 2-norm have the computer correctly executing the next few steps of the computation, but for the 1-norm this isn’t necessarily the case—the computer will likely glitch or self-destruct. (All computers built by humans probably have this property. Also note that it can be defined in terms of the wavefunction alone, without assuming weights a priori.) Then the algorithm will have “subjective anticipation” of a weird kind: conditioned on the algorithm itself running faithfully in the future, it can conclude that future histories with higher Born-weight are more likely.
This idea has the drawback that it doesn’t look at histories of the outside world, only the computer’s internals. But maybe it can be extended to include observations somehow?
“Beginning to feel” that the question is dissolved is far from the level of certainty required to “declare it dissolved”, merely a hunch that it’s the right direction to look for the answer (not that it’s a question I’m especially interested in, but it might be useful to understand it better).
I agree with your description in the second paragraph, but don’t clearly see what you wanted to communicate through it. (Closest salient idea is Hanson’s “mingled worlds”.)