Well, I have a privileged position of being able to derive it from the first principles, so it is “true” given certain rather mild assumptions about the way the universe works, which stem from some observations (speed of light is constant, observations leading to the Maxwell equations, etc.) leading to the relativistic free particle Lagrangian, and confirmed by others (e.g. atmospheric cosmic ray muon decay). So this is not an isolated belief, but more like an essential part of the model of the world. Without it the whole ontology falls apart. And so does epistemology.
Shmi
Karma: 28,836
Given that there is no known physical theory that allows deliberate time travel (rather than being stuck in a loop forever to begin with), I am confused as to how you can estimate the cost of it.
I am all for squelching terrorism, but this site is probably not the right venue to discuss ways of killing people.
A more realistic and rational outcome: Alice is indeed an ass and it’s not fun to be around her. Bob walks out and blocks her everywhere. Now, dutchbook this!
It’s a good start, but I don’t think this is a reasonably exhaustive list, since I don’t find myself on it :)
My position is closest to your number 3: “ASI will not want to take over or destroy the world.” Mostly because “want” is a very anthropomorphic concept. The Orthogonality Thesis is not false, but inapplicable, since AI are so different from humans. They did not evolve to survive, they were designed to answer questions.
It will be possible to coordinate to prevent any AI from being given deliberately dangerous instructions, and also any unintended consequences will not be that much of a problem
I do not think it will be possible, and I expect some serious calamities from people intentionally or accidentally giving an AI “deliberately dangerous instructions”. I just wouldn’t expect it to result in systematic extermination of all life on earth, since the AI itself does not care in the same way humans do. Sure, it’s a dangerous tool to wield, but it is not a malevolent one. Sort of 3-b-iv, but not quite.
But mostly the issue with doomerism I see is that the Knightian uncertainty on any non-trivial time frame: there will be black swans in all directions, just like there have been lately (for example, no one expected near-human-level LARPing that LLMs do, while not being in any way close to a sentient agent).
To be clear, I expect the world to change quickly and maybe even unrecognizably in the next decade or two, with lots of catastrophic calamities, but the odds of complete “destruction of all value”, the way Zvi puts it, cannot be evaluated at this point with any confidence. The only way to get this confidence is to walk the walk. Pausing and being careful and deliberate about each step does not seem to make sense, at least not yet.
Yeah, that looks like a bizarre claim. I do not think there is any reason whatsoever to doubt yours or Ben’s integrity.
That looks and reads… very corporate.
When has this become a “gossip about the outgroup” site?
Thought I’d comment in brief. I very much enjoyed your post and I think it is mostly right on point. I agree that EA does not have a great epistemic hygiene, given what their aspirations are, and the veganism discussion is a case in point. (Other issues related to EA and CEA have been brought up lately in various posts, and are not worth rehashing here.)
As far as the quoted exchange with me, I agree that I have not stated a proper disclaimer, which was quite warranted, given the thrust of the post. My only intended point was that, while a lot of people do veganism wrong and some are not suited to it at all, an average person can be vegan without adverse health effects, as long as they eat varied and enriched plant-based diet and periodically check their vitamins/nutrients/minerals levels and make dietary adjustments as necessary. Some might find out that they are in the small minority for whom vegan diet is not feasible, and they would do well to focus on what works for them and contribute to EA in other ways. Again, I’m sorry this seems to have come across wrong.
Oh, and cat veganism is basically animal torture, those who want to wean cats off farmed animal food should focus on vat-grown meat for pet food etc.
Sure, it’s not necessary that a sufficiently advance AI has to work like the brain, but there has to be an intuition about why is not need it to at least create an utility maximizer.
Octopus’ brain(s) is nothing like that of mammals, and yet it is equally intelligent.
“Sanity” may not be a useful concept in edge cases, but yes, being able to trust your mind to autopilot is definitely within the central definition of sanity, it’s a good observation.
You may also be interested in Scott’s post series on the topic, the latest being https://www.astralcodexten.com/p/contra-kirkegaard-on-evolutionary
FTFY: “Smile at strangers iff it has non-negative EV, because smiling is cheap and sometimes it does”.
“I am going to read you mind and if you believe in a decision theory that one-boxes in Newcomb’s Paradox I will leave you alone, but if you believe in any other decision theory I will kick you in the dick”
Sure, that’s possible. Assuming there are no Newcomb’s predictors in that universe, but only DK, rational agents believe in two-boxing. I am lost as to how it is related to your original point.
Let me clarify what I said. Any decision theory or no decision theory at all that results in someone one-boxing is rewarded. Examples: Someone hates touching transparent boxes. Someone likes a mystery of an opaque box. Someone thinking that they don’t deserve a guaranteed payout and hoping for an empty box. Someone who is a gambler. Etc. What matters is the outcome, not the thought process.
How well can you predict what will be good in 100 years? For perspective, given the contemporary zeitgeist of 1923, rather than knowing all we know now, what would have been considered a great policy to enact then that would take effect in 2023?
What if you believe in DKRUDT, the “Dick Kick’em rewards you” decision theory?
Seriously though, Newcomb’s setup is not adversarial in the same way, the predictor rewards or punishes you for actions, not beliefs. Your internal reasoning does not matter, as long as you end up one-boxing you walk away with more money.
As someone who gets rate-limited due to downvoted comments occasionally, I can see a rather strong correlation between the style/tone and the response. It is very much possible to express controversial, contrarian or even outright silly views without being downvoted. The rule of thumb is to be respectful and charitable to your opponents and, well, read the room. The more your view diverges from the local mainstream, the better, tighter and clearer your argument must be.
It is standard practice when writing scientific papers to first demonstrate intimate familiarity with the views you argue against later. This is what you would want from someone else writing a refutation of your argument, right? So give them the same courtesy. You can cut corners if your post or comment is close to the mainstream, but it is not a great habit. If you want to learn how to do it right, read ACX, Scott is a master of this art.
it is a common pitfall to blame the society/subculture for the negative reaction you get. It is almost never a constructive way to proceed.
If you are in doubt as to how your post would be received, and you want to get through to more people, consider reaching out to someone familiar with this site to review your draft. Or to anyone, really. The bar for writing a post or a comment with non-negative expected karma is pretty low.
it’s not a probability question. Blood tests exist.
Your potential contribution to timeline shortening as an undergrad is probably negligible on the margins, unless you are in the top 1% of all applicants, or even then. The field is crowded by very intelligent, well paid and very determined people who do this for a living. So whatever considerations you have, they need not be influenced by the AGI killeveryoneism.
Huh, I never heard of this umbrella Effective Ventures Foundation before. Let alone about its ability to muzzle individual speech.