The true point of no return has to be indeed much later than we believe it to be now.
Who is “we”, and what do “we” believe about the point of no return? Surely you’re not talking about ordinary doctors pronouncing medical death, because that’s just irrelevant (pronouncements of medical death are assertions about what current medicine can repair, not about information-theoretic death). But I don’t know what other consensus you could be referring to.
I think your answer is in The Domain of Your Utility Function. That post isn’t specifically about cryonics, but is about how you can care about possible futures in which you will be dead. If you understand both of the perspectives therein and are still confused, then I can elaborate.
Why would a self-improving agent not improve its own decision-theory to reach an optimum without human intervention, given a “comfortable” utility function in the first place?
A self-improving agent does improve its own decision theory, but it uses its current decision theory to predict which self-modifications would be improvements, and broken decision theories can be wrong about that. Not all starting points converge to the same answer.
That strategy is optimal if and only if the probably of success was reasonably high after all. Otoh, if you put an unconditional extortioner in an environment mostly populated by decision theories that refuse extortion, then the extortioner will start a war and end up on the losing side.
Jbay didn’t specify that the drug has to leave people able to answer questions about their own emotional state. And in fact there are some people who can’t do that, even though they’re otherwise functional.
There are many such operators, and different ones give different answers when presented with the same agent. Only a human utility function distinguishes the right way of interpreting a human mind as having a utility function from all of the wrong ways of interpreting a human mind as having a utility function. So you need to get a bunch of Friendliness Theory right before you can bootstrap.
fanficdownloader. I haven’t tried the webapp version of it, but I’m happy with the CLI.
I donated $40,000.00
If you can encode microstate s in n bits, that implies that you have a prior that assigns P(s)=2^-n. The set of all possible microstates is countably infinite. There is no such thing as a uniform distribution over a countably infinite set. Therefore, even the ignorance prior can’t assign equal length bitstrings to all microstates.
Can we instead do “probability distribution over equivalence classes of models of L”, where equivalence is determined by agreement on the truth-values of all first order sentences? There’s only 2^ℵ₀ of those, and the paper never depends on any distinction within such an equivalence class.
Yes, that’s the usual application, but it’s the wrong level of generality to make them synonyms. “Fully general counterargument” is one particular absurdity that you can reduce things to. Even after you’ve specified that you’re performing a reductio ad absurdum against the proposition “argument X is sound”, you still need to say what the absurd conclusion is, so you still need a term for “fully general counterargument”.
Why should you not have preferences about something just because you can’t observe it? Do you also not care whether an intergalactic colony-ship survives its journey, if the colony will be beyond the cosmological horizon?
Here’s a citation for the claim of DRAM persisting with >99% accuracy for seconds at operating temperature or hours at LN2. (The latest hardware tested there is from 2007. Did something drastically change in the last 6 years?)
What relevance does personal identity have to TDT? TDT doesn’t depend on whether the other instances of TDT are in copies of you, or in other people who merely use the same decision theory as you.
That works with caveats: You can’t just publish the seed in advance, because that would allow the player to generate the coin in advance. You can’t just publish the seed in retrospect, because the seed is an ordinary random number, and if it’s unknown then you’re just dealing with an ordinary coin, not a logical one. So publish in advance the first k bits of the pseudorandom stream, where k > seed length, thus making it information-theoretically possible but computationally intractable to derive the seed; use the k+1st bit as the coin; and then publish the seed itself in retrospect to allow verification.
Possible desiderata that are still missing: If you take multiple coins from the same pseudorandom stream, then you can’t allow verification until the end of the whole experiment. You could allow intermediate verification by committing to N different seeds and taking one coin from each, but that fails wedrifid’s desideratum of a single indexable problem (which I assume is there to prevent Omega from biasing the result via nonrandom choice of seed?).
I can get both of those desiderata at once using a different protocol: Pick a public key cryptosystem, a key, and a hash function with a 1-bit output. You need a cryptosystem where there’s only one possible signature of any given input+key, i.e. one that doesn’t randomize encryption. To generate the Nth coin: sign N, publish the signature, then hash the signature.
In fact, the question itself seems superficially similar to the halting problem, where “running off the rails” is the analogue for “halting”
If you want to draw an analogy to halting, then what that analogy actually says is: There are lots of programs that provably halt, and lots that provably don’t halt, and lots that aren’t provable either way. The impossibility of the halting problem is irrelevant, because we don’t need a fully general classifier that works for every possible program. We only need to find a single program that provably has behavior X (for some well-chosen value of X).
If you’re postulating that there are some possible friendly behaviors, and some possible programs with those behaviors, but that they’re all in the unprovable category, then you’re postulating that friendliness is dissimilar to the halting problem in that respect.
Then what you should be asking is “which problems are in BQP?” (if you just want a summary of the high level capabilities that have been proved so far), or “how do quantum circuits work?” (if you want to know what role individual qubits play). I don’t think there’s any meaningful answer to “a qubit’s specs” short of a tutorial in the aforementioned topics. Here is one such tutorial I recommend.
“do not affect anything outside of this volume of space”
Suppose you, standing outside the specified volume, observe the end result of the AI’s work: Oops, that’s an example of the AI affecting you. Therefore, the AI isn’t allowed to do anything at all. Suppose the AI does nothing: Oops, you can see that too, so that’s also forbidden. More generally, the AI is made of matter, which will have gravitational effects on everything in its future lightcone.
Suppose I say “I prefer state X to Z, and don’t express a preference between X and Y, or between Y and Z.” I am not saying that X and Y are equivalent; I am merely refusing to judge.
If the result of that partial preference is that you start with Z and then decline the sequence of trades Z->Y->X, then you got dutch booked.
Otoh, maybe you want to accept the sequence Z->Y->X if you expect both trades to be offered, but decline each in isolation? But then your decision procedure is dynamically inconsistent: Standing at Z and expecting both trade offers, you have to precommit to using a different algorithm to evaluate the Y->X trade than you will want to use once you have Y.
Use a prefix-free encoding for the hypotheses. There’s not 2^n hypotheses of length n: Some of the length-n bitstrings are incomplete and you’d need to add more bits in order to get a hypothesis; others are actually a length <n hypothesis plus some gibberish on the end.
Then the sum of the probabilities of all programs of all lengths combined is 1.0. After excluding the programs that don’t halt, the normalization constant is Chaitin’s Omega.