existential-risk.org by Nick Bostrom
existential-risk.org
(Updated 2011-12-16 due to a comment by Nick Bostrom.)
‘Existential Risk FAQ’ by Nick Bostrom
(2011) Version 1.0
Short answers to common questions
‘Existential Risk Prevention as the Most Important Task for Humanity’ by Nick Bostrom
(2011) Working paper (revised)
ABSTRACT
Existential risks are those that threaten the entire future of humanity. Many theories of value imply that even relatively small reductions in net existential risk have enormous expected value. Despite their importance, issues surrounding human-extinction risks and related hazards remain poorly understood. In this paper, I clarify the concept of existential risk and develop an improved classification scheme. I discuss the relation between existential risks and basic issues in axiology, and show how existential risk reduction (via the maxipok rule) can serve as a strongly action-guiding principle for utilitarian concerns. I also show how the notion of existential risk suggests a new way of thinking about the ideal of sustainability.
The site is meant to be improved over time, so any comments that could help do that—nitpicky or not—are very welcome. I especially agree that all unnecessary crankiness should be eliminated.
Reg. .com or .org—If there is a clear sense that the .org would be more appropriate, I could look into obtaining that domain name and moving the site there. This should ideally be done soon, while the site is still young. So far, it seems nobody thinks .com is better all things considered, and some people think .org would be a slightly better; but I’d like a little more data before taking the plunge.
Reg. typos in FAQ—Will be corrected over time as they are discovered.
Reg. design—It could be better, but I’m not a web designer. If somebody here would make a nicer version, I’d gratefully replace the current design.
Reg. comments on draft paper ‘The Concept of Existential Risk’ - this is the main new original content. It is currently under review for a journal, and I might make some revisions when preparing the final version.
I’ve moved the site to http://www.existential-risk.org/, changed the visuals, and made some other improvements in response to the suggestions that were made here.
There is also a revised version of the new paper (now titled “Existential RIsk Prevention as the Most Important Task for Humanity”): http://www.existential-risk.org/concept.pdf.
Minor issue- The signaling of using a ”.com” rather than a ”.org” address makes the website appear to be slightly cranky. It is a tiny signal but it is one of the first things people will notice. I don’t quite understand what Bostrom was thinking.
Edit: Similar signaling issues occur elsewhere, such as a disturbing number of typos. For example, in the FAQ- “This suggests and important point”—presumably “and” should be “an”.
Edit: I’m also confused about who this is aimed at. In http://www.existentialrisk.com/concept.html he talks about both “singletons” and “simulators” in passing without explaining them or linking to anything. I suspect that most people who know those terms already have a fair bit of basic background about existential risk. This could use a lot of hyperlinking.
Additionally, the website doesn’t conform to current website aesthetic standards, another cranky signal.
Agree, but less of a problem than having the term ‘Singularity’ in the name of your charity.
ETA: Seriously, I feel ashamed every time I link someone to the SIAI website, just because of the impression I expect it to have on the person I want to convince of risks from AI. “Singularity” is such a loaded term if you are not a cosmologist...rapture of the nerds, anyone?
I so dislike that phrase. Unfortunately, you’re probably right about its referent’s signaling implications.
On the other hand, I’ve generally found it more productive to assume that there are coherent reasons behind any particular decision. Perhaps Eliezer et al. thought they could get more mileage out of the word’s positive associations among adherents than they’d lose from its negative associations among skeptics—or perhaps it’s just that it was a less loaded concept back in 2000. Probably both.
Nope, actually Eliezer also feels bad about SIAI name. He jokingly suggested it should have been named the Good Institute, after I. J. Good.
No public comments along those lines though, AFAICS.
Oh, yeah, it wasn’t a public comment. He was just visiting the rationality mini-camp and the topic came up.
If SIAI were being founded today, it would surely be called the Rationality Institute.
yup, I almost always preface discussion of something involving the word ‘singularity’ with “I really hate the word singularity, but...”
That risks activating the heuristic “If someone starts a discussion with “I’m not X but …” , (s)he is probably X ”. Urban dictionary : “I’m not a racist but...”
The “Rapture” and the “Singularity” Have Much in Common.
.com seems to have established itself as the generic domain that’s used for everything. I doubt most people even notice (I didn’t), even though .org would probably be slightly better.
“Homo sapience” → “Homo Sapiens”
I personally like the website design.
The summaries of academic papers look serious (not cranky) and the FAQ is well-written.
I think it’s a good resource for pointing people to; it would be a pity if LW reaction was dominated by nit-picks.
Unfortunately one is talking about a series of ideas that can already easily trigger weirdness heuristics and feelings of crankiness whether or not they exist. Every little feature that reinforces that (such as typos, references to undefined terms, etc.) will make that worse, possibly at a very rapid rate.
In what way would that be a pity? The negative impact of prevalence of nitpicking is in being misinterpreted as signaling negative overall impression or in making the signal about overall impression hard to hear. Whose mistake or difficulty to estimate the impression are we talking about here? I’d guess that most would agree that the site potentially makes a positive contribution, and would be even better if some low-hanging fruit is additionally collected.
Why Our Kind Can’t Cooperate may be relevant here. The good things about the site deserve praise as much as the bad parts deserve criticism.
(Edited after Vladimir Nesov replied)
(Would’ve upvoted for the reference, downvoted for “true” given without further detail.)
I suspect that for most people it has the opposite effect, if any.
Sorry, can you expand on that? Are you saying that most people would consider .com to be less cranky than .org? Can you explain that? My gut reaction when I see something that is potentially crank is that a .org is more likely to have multiple people behind it whereas a .com is likely to be whatever some person bought. Moreover, there’s a definite tendency for cranks to be a few years behind the times in regards to website design, so the popularity of .com addresses in the late 90s also comes into play.
I’d wager most people wonder wtf .org is all about and why it’s not a .com like all the others. But then again those people are not the ones that are gonna wind up at the site. So I find it most likely you two are just imagining two different sets of ‘most people’.
Sadly, the site seems to be down.