In some cases something like this might work:
“The plumber says it’s fixed, so hopefully it is”
Or
“The plumber says it’s fixed, so it probably is”
Which I think conveys”there’s an assumption I’m making here, but I’m just putting a flag in the ground to return to if things don’t play out as expected”
I don’t share your intuition here. I think many people would see blue as the “band together” option and would have confidence that others will do the same. For the average responder, the question would reduce to “choose blue to signal trust in humanity, choose red to signal selfish cowardice”.
“Innate faith in human compassion, especially in a crisis” is the co-ordination mechanism, and I think there is pretty strong support for that notion if you look at how we respond to crises in real life and how we depict them in fiction. That is the narrative we tell ourselves at least, but narrative is what’s important here.
I would be surprised if blue was less than 30%, and would predict around 60%.
I’m not quite sure what you mean by that.
Unless I expect the pool of responders to be 100% rational and choose red, then I should expect some to choose blue. Since I (and presumably other responders) do expect some to choose blue, that makes >50% blue the preferred outcome. Universal red is just not a realistic outcome.
Whether or not I choose blue then depends on factors like how I value the lives of others compared to mine, the number of responders, etc—as in the equations in your post.
Emperically, as GeneSmith points out, something is wrong with WalterL’s suggestion that red is the obvious choice no matter your reasoning. Applying his logic would push the twitter poll away from the realistically ideal outcome of >50% blue and closer to the worst possible outcome (51% red).
No matter what the game theory says, a non-zero number of people will choose blue and thus die under this equilibrium. This fact—that getting to >50% blue is the only way to save absolutely everyone—is enough for me to consider choosing blue and hope that others reason the same (which, in a self-fulfilling way, strengthens the case for choosing blue).
Edit—This turned out pretty long, so in short:
What reason do we have to believe that humans aren’t already close to maxing out the gains one can achieve from intelligence, or at least having those gains in our sights?
One crux of the ASI risk argument is that ASI will be unfathomably powerful thanks to its intelligence. One oft-given analogy for this is that the difference in intelligence between humans and chimps is enough that we are able to completely dominate the earth while they become endangered as an unintended side effect. And we should expect the difference between us and ASI to be much much larger than that.
How do we know that intelligence works that way in this universe? What if it’s like opposable thumbs? It’s very difficult to manipulate the world without them. Chimps have them, but don’t seem as dexterous with them as humans are, and they are able to manipulate the world only in small ways. Humans have very dexterous thumbs and fingers and use them to dominate the earth. But if we created a robot with 1000 infinitely dexterous hands… well, it doesn’t seem like it would pose an existential risk. Humans seem to have already maxed out the effective gains from hand dexterity.
Imagine the entire universe consisted of a white room with a huge pile of children’s building blocks. A mouse in this room can’t do anything of note. A chimp can use its intelligence and dexterity to throw blocks around, and maybe stack a few. A human can use its intelligence and dexterity to build towers, walls, maybe a rudimentary house. Maybe given enough time a human could figure out how to light a fire, or sharpen the blocks.
But it’s hard to see how a super intelligent robot could achieve more than a human could in the children’s block-universe. The scope of possibilities in that world is small enough that humans were able to max it out, and superintelligence doesn’t offer any additional value.
What reason do we have to believe that humans aren’t already close to maxing out the gains one can achieve from intelligence?
(I can still see ways that ASI could be very dangerous or very useful even if this was the case, but it does seem like a big limiting factor. It would make foom unlikely, for one. And maybe an ASI can’t outstrategise humanity in real life for the same reason it couldn’t outstrategise us in Tic Tac Toe.)
---
(I tried to post the below as a separate comment to make this less of a wall of text, but I’m rate limited so I’ll edit it in here instead)
One obvious response is to point to phenomena that we know is physically possible, but humans haven’t figured out yet. Eg. fission. But that doesn’t seem like something an ASI could blindside us with—it would be one of the first things that humans would try to get out of AI and I expect us to succeed using more rudimentary AI well before we reach ASI level. If not, then at least we can keep an eye on any fission-y type projects an ASI seems to be plotting and use it as a huge warning sign.
”You don’t know exactly how Magnus Carlsen will beat you in chess, but you know that he’ll beat you” is a hard problem.
”You don’t know exactly how Magnus Carlsen will beat you in chess, but you know that he’ll develop nuclear fission to do it” is much easier to plan for at least.
It seems like the less survivable worlds hinge more on there being unknown unknowns, like some hack for our mental processes, some way of speeding up computer processing a million times, nanotechnology etc. So to be more specific:
What reason do we have to believe that humans aren’t already close to maxing out the gains one can achieve from intelligence, or at least having those gains in our sights?
I lost 25kg in 9 months using a very similar method. Some suggestions to help with the hunger/willpower:
1. Brush your teeth straight after dinner. It adds a point of friction between you and more food (“I’d have to brush my teeth again”). Then drink herbal teas in the evening.
2. You don’t have to feel hungry all the time. Choose when you consume your calories so that you’re hungry at the least inconvenient times and for the fewest waking hours. I usually eat a decent breakfast as late as I can, because I’m not as hungry in the mornings. Then I eat two half-sized meals, one at noon and one mid afternoon. Followed by a large dinner. I’m most hungry between noon and mid afternoon, but it’s not too hard to bare because I know I’ve got that smaller mid-afternoon meal coming up, and then a satisfying dinner.
3. Use exercise to “earn” treats. If you want ice-cream for dessert, calculate how many calories you’ll need to expend to make up for it and go for an appropriately long jog beforehand. Don’t do this the other way around (eat a treat, promising yourself that you’ll burn it off tomorrow).