I don’t think it being unfalsifiable is a problem. I think this is more of a definition than a derivation. Morality is a fuzzy concept that we have intuitions about, and we like to formalize these sorts of things into definitions. This can’t be disproven any more than the definition of a triangle can be disproven.
What needs to be done instead is show the definition to be incoherent or that it doesn’t match our intuition.
Can you explain why that’s a misconception? Or at least point me to a source that explains it?
I’ve started working with neural networks lately and I don’t know too much yet, but the idea that they recreate the generative process behind a system, at least implicitly, seems almost obvious. If I train a neural network on a simple linear function, the weights on the network will probably change to reflect the coefficients of that function. Does this not generalize?
It fits with the idea of the universe having an orderly underlying structure. The simulation hypothesis is just one way that can be true. Physics being true is another, simpler explanation.
Neural networks may very well turn out to be the easiest way to create a general intelligence, but whether they’re the easiest way to create a friendly general intelligence is another question altogether.
Many civilizations may fear AI, but maybe there’s a super-complicated but persuasive proof of friendliness that convinces most AI researchers, but has a well-hidden flaw. That’s probably a similar thing to what you’re saying about unpredictable physics though, and the universe might look the same to us in either case.
Not necessarily all instances. Just enough instances to allow our observations to not be incredibly unlikely. I wouldn’t be too surprised if out of a sample of 100 000 AIs none of them managed to produce successful vNP before crashing. In addition to the previous points the vNP would have to leave the solar system fast enough to avoid the AI’s “crash radius” of destruction.
Regarding your second point, if it turns out that most organic races can’t produce a stable AI, then I doubt an insane AI would be able to make a sane intelligence. Even if it had the knowledge to, its own unstable value system could cause the VNP to have a really unstable value system too.
It might be the case that the space of self-modifying unstable AI has attractor zones that cause unstable AI of different designs to converge on similar behaviors, none of which produce VNP before crashing.
Your last point is an interesting idea though.
That’s a good point. Possible solutions:
AI just don’t create them in the first place. Most utility functions don’t need non-evolving von Neumann probes, and instead the AI itself leads the expansion.
AI crash before creating von Neumann probes. There are lots of destructive technologies an AI could get to before being able to build such probes. An unstable AI that isn’t in the attractor zone of self-correcting fooms would probably become more and more unstable with each modification, meaning that the more powerful it becomes the more likely it is to destroy itself. von Neumann probes may simply be far beyond this point.
Any von Neumann probes that could successfully colonize the universe would have to have enough intelligence to risk falling into the same trap as their parent AI.
It would only take one exception, but the second and third possibilities are probably strong enough to handle it. A successful von Neumann probe would be really advanced, while an increasingly insane AI might get ahold of destructive nanotech and nukes and all kinds of things before then.
Infinity is really confusing.
My statement itself isn’t something I believe with certainty, but adding that qualifier to everything I say would be a pointless hassle, especially for things that I believe with a near-enough certainty that my mind feels it is certain. The part with the “ALL” is itself a part of the statement I believe with near certainty, not a qualifier of the statement I believe. Sorry I didn’t make that clearer.
The idea of ALL beliefs being probabilities on a continuum, not just belief vs disbelief.
Suppose I’m destructively uploaded. Let’s assume also that my consciousness is destroyed, a new consciousness is created for the upload, and there is no continuity. The upload of me will continue to think what I would’ve thought, feel what I would’ve felt, choose what I would’ve chosen, and generally optimize the world in the way I would’ve. The only thing it would lack is my “original consciousness”, which doesn’t seem to have any observable effect in the world. Saying that there’s no conscious continuity doesn’t seem meaningful. The only actual observation we could make is that the process I tend to label “me” is made of different matter, but who cares?
I think a lot of the confusion about this is treating consciousness as an actual entity separate from the process it’s identified with, which somehow fails to transfer over. I think that if consciousness is something worth talking about, then it’s a property of that process itself, and is agnostic toward what’s running the process.
I expect that most people are biased when it comes to judging how attractive they are. Asking people probably doesn’t help too much, since people are likely to be nice, and close friends probably also have a biased view of ones attractiveness. So is there a good way to calibrate your perception of how good you look?
One thing that helped me a lot was doing some soul-searching. It’s not so much about finding something to protect so much as realizing what I already care about, even if there are some layers of distance between my current feelings and that thing. I think that a lot of that listless feeling of not having something to protect is just sort of being distracted from what we actually care about. I would recommend just looking for anything you care about at all, even slightly, and just focusing on that feeling.
At least that makes sense and works for me.
There are a lot of ways to be irrational, and if enough people are being irrational in different ways, at least some of them are bound to pay off. Using your example, some of the people with blind idealism may get stuck to an idea that they can accomplish, but most of them fail. The point of trying to be rational isn’t to do everything perfectly, but to systematically increasing your chances of succeeding, even though in some cases you might get unlucky.
I think the biggest reason we have to assume that the universe is empty is that the earth hasn’t already been colonized.
Ah I see. I was thinking of motte and bailey as something like a fallacy or a singular argument tactic, not a description of a general behavior. The name makes much more sense now. Thank you. Also, you said it’s called that “everywhere except the Scottosphere”. Could you elaborate on that?
What does the tern “doctrine” mean in this context anyways? It’s not exactly a belief or anything, just a type of argument. I’ve seen that it’s called that but I don’t understand why.
Is this the same thing as the motte and bailey argument?
You cite the language’s tendency to borrow foreign terms as a positive thing. Wouldn’t that require an inconsistent orthography?
Robin, or anyone who agrees with Robin:
What evidence can you imagine would convince you that AGI would go FOOM?