How do you interpret “soundness”? It’s being used to mean that a proof of X implies X, for any statement X in the formal language of the theory. And yes, Löb’s Theorem directly shows that PA cannot prove its own soundness for any set of statements save a subset of its own theorems.
hairyfigment
Karma: 2,077
Go ahead and test the prediction from the start of that thread, if you like, and verify that random people on the street will often deny the existence of the other two types. (The prediction also says not everyone will deny the same two.) You already know that NTs—asked to imagine maximal, perfect goodness—will imagine someone who gets upset about having the chance to save humanity by suffering for a few days, but who will do it anyway if Omega tells him it can’t be avoided.
It sure sounds like you think outsiders would typically have the “common sense” to avoid Ziz. What do you think such an outsider would make of this comment?
There’s this guy Michael Vassar who strikes me—from afar—as a failed cult leader, and Ziz as a disciple of his who took some followers in a different direction. Even before this new information, I thought her faith sounded like a breakaway sect of the Church of Asmodeus.
Michael Vassar was one of the inspirations for Eliezer’s Professor Quirrell, but otherwise seems to have little influence.
While it’s arguably good for you to understand the confusion which led to it, you might want to actually just look up Solomonoff Induction now.
>Occam’s razor. Is it saying anything other than
P(A) >= P(A & B)?Yes, this is the same as the argument for (the abstract importance of) Solomonoff Induction. (Though I guess you might not find it convincing.)
We have an intuitive sense that it’s simpler to say the world keeps existing when you turn your back on it. Likewise, it’s an intuitively simpler theory to say the laws of physics will continue to hold indefinitely, than to say the laws will hold up until February 12, 2023 at midnight Greenwich Mean Time. The law of probability which you cited only captures a small part of this idea, since it says that last theory has at least as much probability as the ‘simple’ version. We could add a rule saying nearly all of that probability accrues to the model where the laws keep holding, but what’s the general form of that rule?
Occam’s original formulation about not multiplying entities doesn’t help us much, as we could read this to say we shouldn’t assume the world keeps existing when unobserved. That’s the opposite of what we want. Newton’s version was arguably better, talking about “properties” which should be tentatively assumed to hold generally when we can’t find any evidence to the contrary, but then we could reasonably ask what “property” means.
SI comes from the idea that we should look for the minimum message length which predicts the data, and we could see SI in particular as an attempt to solve the grue-bleen problem. The “naturalized induction” school says this is technically still wrong, but it represents a major advance.
Except, if you Read The Manual, you might conclude that in fact those people also can’t understand you exist.
Well, current events seem to have confirmed that China couldn’t keep restrictions in place indefinitely, and the fact that they even tried—together with the cost of stopping—suggest that it would’ve been a really good idea to protect their people using the best vaccine. China could presumably have just stuck it in everyone by force of law. What am I missing here?
Didn’t you complain about Biden relaxing intellectual property laws for our more-effective vaccines? Why doesn’t that solve the problem? According to what you told us before, shouldn’t China be able to copy the best vaccine without losing face by admitting that’s what they’re doing?
The correct alternative was absolutely to not apply such constraints, but that’s because supply should have been a non-issue. Paying $500/shot, for a course of vaccination begun in the first month, would have cost much less than 0.3 taken off of the 1.9 trillion dollar COVID relief bill of early 2021. This should have been literally free.
Zvi spends a lot of time talking about the problems of choosing scarcity—and dishonestly ignoring the evidence that African-Americans were more likely to die if they got infected—but the actual question should be why this wasn’t trivially solved in the run-up to January 20, 2021, aside from that spot of bother concerning the changeover.
I mean, that makes sense—perhaps more so than it does for Hells, if we allow arbitrarily smart deceptive adversaries—but now I’m wondering if your first sentence is a strawman.
I’m glad Jacob agrees that empowerment could theoretically help arbitrary entities achieve arbitrary goals. (I recall someone who was supposedly great at board games recommending it as a fairly general strategy.) I don’t see how, if empowerment is compatible with almost any goal, it could prevent the AI from changing our goals whenever this is convenient.
Perhaps he thinks we can define “empowerment” to exclude this? Quick reaction: that seems likely to be FAI-complete, and somewhat unlikely to be a fruitful approach. My understanding of physics says that pretty much action has a physical effect on our brains. Therefore, the definition of which changes to our brains “empower” and which “disempower” us, may be doing all of the heavy lifting. How does this become easier to program than CEV?
Jacob responds: The distribution shift from humans born in 0AD to humans born in 2000AD seems fairly inconsequential for human alignment.
I now have additional questions. The above seems likely enough in the context of CEV (again), but otherwise false.
>FDT has bigger problems then that.
Does it. The post you linked does nothing to support that claim, and I don’t think you’ve presented any actual problem which definitively wouldn’t be solved by logical counterfactuals. (Would this problem also apply to real people killing terrorists, instead of giving in to their demands? Because zero percent of the people obeying FDT in that regard are doing so because they think they might not be real.) This post is actually about TDT, but it’s unclear to me why the ideas couldn’t be transferred.
I also note that 100% of responses in this thread, so far, appear to assume that your ghosts would need to have qualia in order for the argument to make sense. I think your predictions were bad. I think you should stop doing that, and concentrate on the object-level ideas.
Again, it isn’t more resilient, and thinking you doubt a concept you call “qualia” doesn’t mean you can doubt your own qualia. Perhaps the more important point here is that you are typically more uncertain of mathematical statements, which is why you haven’t removed and cannot remove the need for logical counterfactuals.
Real humans have some degree of uncertainty about most mathematical theorems. There may be exceptions, like 0+1=1, or the halting problem and its application to God, but typically we have enough uncertainty when it comes to mathematics, that we might need to consider counterfactuals. Indeed, this seems to be required by the theorem alluded to at the above link—logical omniscience seems logically impossible.
For a concrete (though unimportant) example of how regular people might use such counterfactuals in everyday life, consider P=NP. That statement is likely false. Yet, we can ask meaningful-sounding questions about what its truth would mean, and even say that the episode of ‘Elementary’ which dealt with that question made unjustified leaps. “Even if someone did prove P=NP,” I find myself reasoning, “that wouldn’t automatically entail what they’re claiming.”
Tell me if I’ve misunderstood, but it sounds like you’re claiming we can’t do something which we plainly do all the time. That is unconvincing. It doesn’t get any more convincing when you add that maybe my experience of doing so isn’t real. I am very confident that you will convince zero average people by telling them that they might not actually be conscious. I’m skeptical that even a philosopher would swallow that.
If you think you might not have qualia, then by definition you don’t have qualia. This just seems like a restatement of the idea that we should act as if we were choosing the output of a computation. On its face, this is at least as likely to be coherent as ‘What if the claim we have the most certainty of were false,’ because the whole point of counterfactuals in general is to screen off potential contradictions.
The problem arises because, for some reason, you’ve assumed the ghosts have qualia. Now, that might be a necessary assumption if you require us to be uncertain about our degree of ghostliness. Necessary or not, though, it seems both dubious and potentially fatal to the whole argument.
That is indeed somewhat similar to the “Hansonian adjustment” approach to solving the Mugging, when larger numbers come into play. Hanson originally suggested that, conditional on the claim that 3^^^^3 distinct people will come into existence, we should need a lot of evidence to convince us we’re the one with a unique opportunity to determine almost all of their fates. It seems like such claims should be penalized by a factor of 1/3^^^^3. We can perhaps extend this so it applies to causal nodes as well as people. That idea seems more promising to me than bounded utility, which implies that even a selfish agent would be unable to share many goals with its future self (and technically, even a simple expected value calculation takes time.)
Your numbers above are, at least, more credible than saying there’s a 1⁄512 chance someone will offer you a chance to pick between a billion US dollars and one hundred million.
I may reply to this more fully, but first I’d like you to acknowledge that you cannot in fact point to a false prediction by EY here, and in the exact post you seemed to be referring to, he says that his view is compatible with this sort of AI producing realistic sculptures of human faces!
Do you also believe that if you could “prove” eating babies was morally required, eating babies would be morally required? PA obviously believes Lob’s theorem itself, and indeed proves the soundness of all its actual proofs, which is what I said above. What PA doesn’t trust is hypothetical proofs.