Anders_Sandberg

Karma: 294

Does davidad’s uploading moonshot work?

Bird Concept, lisathiergart, Anders_Sandberg, davidad and Arenamontanus

Nov 3, 2023, 2:21 AM

146 points

35 comments25 min readLW link

Anders_Sandberg Sep 22, 2008, 5:19 PM
3 points
on: How Many LHC Failures Is Too Many?
I did a calculation here:
http://tinyurl.com/3rgjrl
and concluded that I would start to believe there was something to the universe-destroying scenario after about 30 clear, uncorrelated mishaps (even when taking a certain probability of foul play into account).

Anders_Sandberg Sep 22, 2008, 4:05 PM
5 points
on: Horrible LHC Inconsistency
I like Roko’s suggestion that we should look at how many doomsayers actually predicted a danger (and how early). We should also look at how many dangers occurred with no prediction at all (the Cameroon lake eruptions come to mind).

Overall, the human error rate is pretty high: http://panko.shidler.hawaii.edu/HumanErr/ Getting the error rate under 0.5% per statement/action seems very unlikely, unless one deliberately puts it into a system that forces several iterations of checking and correction (Panko’s data suggests that error checking typically finds about 80% of the errors). For scientific papers/arguments one bad per thousand is probably conservative (My friend Mikael claimed the number of erroneous maths papers are far less than this level because of the peculiarities of the field, but I wonder how many orders of magnitude they can buy).

At least to me this seems to suggest that in the absence of any other evidence, assigning a prior probability much less than 1/1000 to any event we regard as extremely unlikely is overconfident. Of course, as soon as we have a bit of evidence (cosmic rays, knowledge of physics) we can start using smaller priors. But uninformative priors are always going to be odd and silly.

Anders_Sandberg Jun 24, 2008, 6:55 PM
0 points
on: LA-602 vs. RHIC Review
A new report (Steven B. Giddings and Michelangelo M. Mangano, Astrophysical implications of hypothetical stable TeV-scale black holes, arXiv:0806.3381 ) does a much better job at dealing with the black hole risk than the old “report” Eliezer rightly slammed. It doesn’t rely on Hawking radiation (but has a pretty nice section showing why it is very likely) but instead calculates how well black holes can be captured by planets, white dwarves and neutron stars (based on AFAIK well-understood physics, besides the multidimensional gravity one has to assume in order to get the threat in the first place). The derivation does not assume that Eddington luminosity slows accretion and does a good job at examining how fast black holes can be slowed—it turns out that white dwarves and neutron stars are good at slowing them. This is used to show how dangerously fast planetary accretion rates are incompatible with the observed lifetime of white dwarves and neutron stars.

The best argument for Hawking radiation IMHO is that particle physics is time-reversible, so if there exist particle collisions producing black holes there ought to exist black holes decaying into particles.

Anders_Sandberg Oct 19, 2007, 1:13 AM
0 points
on: Congratulations to Paris Hilton
If this is not a hoax or she does a Leary, we will have her around for a long time. Maybe one day she will even grow up. But seriously, I think Eli is right. In a way, given that I consider cryonics likely to be worthwhile, she has demonstrated that she might be more mature than I am.

To get back to the topic of this blog, cryonics and cognitive biases is a fine subject. There is a lot of biases to go around here, on all sides.

Anders_Sandberg Oct 18, 2007, 9:03 AM
3 points
on: “Can’t Say No” Spending
“If intelligence is an ability to act in the world, if it refer to some external reality, and if this reality is almost infinitely malleable, then intelligence cannot be purely innate or genetic.”

This misses the No Free Lunch theorems, which state that there is no learning system that outperforms any other in general. Yes, full human intelligence, AI superintelligence, earthworms and selecting actions at random are just as good. The trick is “in general”, since that covers an infinity of patternless possible worlds. Worlds with (to us) learnable and understandable patterns is a minuscule minority.

Clearly intelligence needs input from an external world. But it has been shaped by millions of years of evolution within a particular kind of world, and there is quite a bit of information in our genes about how to make a brain that can process this kind of world. Beings that are born with perfectly general brains will not learn how to deal with the world until it is too late, compared to beings with more specialised brains. This is actually a source of our biases, since the built in biases that reduce learning time may not be perfectly aligned with the real world or the new world we currently inhabit.

Conversely, it should not be strange that there is variation in the genes that enable our brains to form and that this produces different biases, different levels of adaptivity and different “styles” of brains. Just think of trying to set the optimal learning rate, discount rate and exploration rate of reinforcement agents.

I agree with Watson that it would be very surprising if intelligence-related genes were perfectly equally distributed. At the same time there are a lot of traits that are surprisingly equally distributed. At the same time, the interplay between genetics, environment, schooling, nutrition, rich and complex societies etc. is complex and accounts for a lot. We honestly do not understand it and its limits at present.

Anders_Sandberg Oct 16, 2007, 11:31 PM
2 points
on: How to Seem (and Be) Deep
People have apparently argued for a 300 to 30,000 years storage limit due to free radicals due to cosmic rays, but the uncertainty is pretty big. Cosmic rays and background radiation are likely not as much a problem as carbon-14 and potassium-40 atoms anyway, not to mention the freezing damage. http://www.cryonics.org/1chapter2.html has a bit of discussion of this. The quick way of estimating the damage is to assume it is time compressed, so that the accumulated yearly dose is given as an acute dose.

Anders_Sandberg Oct 16, 2007, 11:38 AM
5 points
on: The Logical Fallacy of Generalization from Fictional Evidence
I think Kaj has a good point. In a current paper I’m discussing the Fermi paradox and the possibility of self-replicating interstellar killing machines. Should I mention Saberhagen’s berserkers? In this case my choice was pretty easy, since beyond the basic concept his novels don’t contain that much of actual relevance to my paper, so I just credit him with the concept and move on.

The example of Metamorphosis of Prime Intellect seems deeper, since it would be a example of something that can be described entirely theoretically but becomes more vivid and clearly understandable in the light of a fictional example. But I suspect the problem here is the vividness: it would produce a bias towards increasing risk estimates for that particular problem as a side effect of making the problem itself clearer. Sometimes that might be worth it, especially if the analysis is strong enough to rein in wild risk estimates, but quite often it might be counterproductive.

There is also a variant of absurdity bias in referring to sf: many people tend to regard the whole argument as sf if there is an sf reference in it. I noticed that some listeners to my talk on berserkers did indeed not take the issue of whether there are civilization-killers out there very seriously, while they might be concerned about other “normal” existential risks (and of course, many existential risks are regarded as sf in the first place).

Maybe a rule of thumb is to limit fiction references to where they 1) say something directly relevant, 2) there is a valid reason for crediting them, 3) the biasing effects do not reduce the ability to think rationally about the argument too much.

Anders_Sandberg Oct 16, 2007, 9:41 AM
13 points
0
on: The Logical Fallacy of Generalization from Fictional Evidence
Another reason people overvalue science fiction is the availability bias due to the authors who got things right. Jules Verne had a fairly accurate time for going from the Earth to the Moon, Clarke predicted/invented geostationary satelites, John Brunner predicted computer worms. But of course this leaves out all space pirates using slide rules for astrogation (while their robots serve rum), rays from unknown parts of the electromagnetic spectrum and gravity-shielding cavorite. There is a vast number of quite erroneous predictions.

I have collected a list of sf stories involving cognition enhancement. They are all over the place in terms of plausibility, and I was honestly surprised by how little useful ideas of the impact of enhancement they had. Maybe it is easier to figure out the impact of spaceflight. I think the list might be useful as a list of things we might want to invent and common tropes surrounding enhancement rather than any start for analysis of what might actually happen.

Still, sf might be useful in the same sense that ordinary novels are: creating scenarios and showing more or less possible actions or ways of relate to events. There are a few studies showing that reading ordinary novels improves empathy, and perhaps sf might improve “future empathy”, our ability to consider situations far away from our here-and-now situation.

Anders_Sandberg Oct 15, 2007, 11:09 AM
1 point
on: How to Seem (and Be) Deep
In think the “death gives meaning to life” meme is a great example of “standard wisdom”. It is apparently paradoxical (right form to be “deep”), it provides a comfortable consolation for a nasty situation. But I have seldom seen any deep defense for it in the bioethical literature. Even people who strongly support it and ought to work very hard to demonstrate to fellow philosophers that it is a true statement seem to be content to just rattle it off as self-evident (or that people not feeling it in their guts are simply superficial).

Being a hopeless empiricist I would like to check whether people today feel life being less meaningful than a century ago, and whether people in countries with short life expectancy feel more meaning than in countries with none. I’m pretty certain the later is not true, and the first looks iffy (hard to check, and lots of confounders like changed social and cultural values, though). I did some statistics on the current state, http://www.aleph.se/andart/archives/2006/12/a_long_and_happy_life.html and found no link between longer life and ennui, at least on a national level.

Anders_Sandberg Oct 14, 2007, 8:18 PM
37 points
0
on: How to Seem (and Be) Deep
I have played with the idea of writing a “wisdom generator” program for a long time. A lot of “wise” statements seem to follow a small set of formulaic rules, and it would not be too hard to make a program that randomly generated wise sayings. A typical rule is to create a paradox (“Seek freedom and become captive of your desires. Seek discipline and find your liberty”) or just use a nice chiasm or reversal (“The heart of a fool is in his mouth, but the mouth of the wise man is in his heart”). This seems to fit in with your theory: the structure given by the form is enough to trigger recognition that a wise saying will now arrive. If the conclusion is weird or unfamiliar, so much the better.

Currently reading Raymond Smullyan’s The Tao is Silent, and I’m struck by how much less wise taoism seems when it is clearly explained.

Anders_Sandberg Oct 14, 2007, 10:15 AM
2 points
0
on: Original Seeing
There is much to be said for looking at the super-specific. All the interesting complexity is found in the specific cases, while the whole often has less complexity (i.e. the algorithmic complexity of a list of the integers is much smaller than the algorithmic complexity of most large integers). While we might be trying to find good compressed descriptions of the whole, if we do not see how specific cases can be compressed and how they relate to each other we do not have much of a starting point, given that the whole usually overwhelms our limited working memories.

Staring at walls is underrated. But I tend to get distracted from my main project by all the interesting details in the walls.

Anders_Sandberg Oct 10, 2007, 1:59 PM
2 points
on: Priming and Contamination
It appears that priming can be reduced by placing words into a context: priming for words previously seen in a text (or even a nonsense jumble) is weaker than when seen individually.

Anders_Sandberg Oct 2, 2007, 12:42 AM
3 points
on: Recommended Rationalist Reading
I constantly buy textbooks and use them as bedtime reading. A wonderful way to pick up the fundamentals (or at least a superficial familiarity) with many subjects. However, just reading any textbook is unlikely to actually give a great insight into any field. Doing exercises, and in particular having a teacher or mentor point out what is important, is necessary for actually getting anywhere.

To add at least some thread-relevant material, I’d like to recommend Eliezer’s web page “An Intuitive Explanation of Bayesian Reasoning” at http://yudkowsky.net/bayes/bayes.html

I’m reading Piattelli Palmarini’s “Inevitable Illusions” right now, but I’m not that impressed so far. Most of the contents seem to be familiar from this list.

Anders_Sandberg Sep 29, 2007, 8:54 PM
4 points
on: 9/26 is Petrov Day
In my opinion a full scale thermonuclear war would likely neither have wiped out humanity (I’m reading the original nuclear winter papers as well as their criticisms right now) nor wiped out civilization. It would have been terribly bad for both though. I did a small fictional writeup of such a scenario for a roleplaying game, http://www.nada.kth.se/~asa/Game/Fukuyama/bigd.html based in turn on the information in “The Effects of Nuclear War” (OTA 1979). That scenario may have been too optimistic, but it is hard to tell. It seems that much would depend on exact timing and level of forewarning. But even in the most optimistic scenario the repercussions on human progress would have been severe since human capital is disproportionally concentrated in cities that are likely to be devastated. This can in turn make other threats to human flourishing more serious. For example, in my scenario AIDS is likely to become a far more devastating epidemic than in our world since the rate of research into it has been much reduced and the seriousness of the epidemic is overshadowed by war-related conditions.

Anders_Sandberg Sep 25, 2007, 10:44 AM
11 points
on: Einstein’s Arrogance
I agree with Tom that there isn’t that much room to change the field equations once you have decided on the Riemannian tensor framework: gravity cannot be expressed as first-order differential equations and still fit with observation, while number of objects to build a set of second-order equations is very limited. The equations are the simplest possibility (with the cosmological constant as a slight uglification, but it is just a constant of integration).

But selecting the tensor framework, that is of course where all the bits had to go. It is not an obvious choice at all.

It is interesting to note that Einstein’s last paper, “On the relativistic theory of the non-symmetric field” includes a discussion of the “strength” of different theories in terms of how many undetermined degrees of freedom they have. http://books.google.com/books?id=tB9Roi3YnAgC&pg=PA131&lpg=PA131&dq=%22relativistic+theory+of+the+non+symmetric+field%22&source=web&ots=EkMv5tudsI&sig=lkTQE94Ay1h2-qS0mcbGT3xa22M If I recall right, he finds his own theory to be rather flabby.

Anders_Sandberg 25 Sep 2007 10:23 UTC
2 points
on: How Much Evidence Does It Take?
Yes, publication bias matters. But it also applies to the p<0.001 experiment—if we have just a single publication, should we believe that the effect is true and just one group has done the experiment, or that the effect is false and publication bias has prevented the publication of the negative results? If we had a few experiments (even with different results) it would be easier to estimate this than in the one published experiment case.

Anders_Sandberg 24 Sep 2007 11:40 UTC
9 points
0
on: How Much Evidence Does It Take?
This also shows why independently replicated scientific experiments (more independent boxes) are more important than experiments with high p-values (boxes with better likeliehood ratios).

Anders_Sandberg 14 Sep 2007 20:48 UTC
0 points
on: Human Evil and Muddled Thinking
While Eliezer and I may be approaching the topic differently, I think we have very much the same aim. My approach will however never produce anything worthy to go into anybody’s quote file.

Anders_Sandberg 14 Sep 2007 19:41 UTC
9 points
on: Human Evil and Muddled Thinking
David Brin has a nice analysis in his book The Transparent Society of what makes open societies work so well (no doubt distilled from others). Essentially it is the freedom to criticize and hold accountable that keeps powerful institutions honest and effective. While most people do not care or dare enough there are enough “antibodies” in a healthy open society to maintain it, even when the “antibodies” themselves may not always be entirely sane (there is a kind of social “peer review” going on here among the criticisms).

Muddled thinking affects this process in several ways. It weakens the ability to perform and react to criticism, and may contribute to reducing the signal-to-noise ratio among whistleblowers by reducing the social “peer review”. This is how muddled thinking can promote the loss of openness, democracy and accountability, in the long run leading to non-accountable leaders that have little valid feedback or can just ignore it.

But are biases the main source of muddled thinking? I think muddle is the sum of many different factors: biases, lack of knowledge, communications problems etc. In any situation one or a few factors are the most serious causes of muddle, but they may differ between issues—the biases we have discussed relating to new technology are different from the biases in conspiracy theory or everyday political behavior. To reduce muddle in a situation we ought to reduce the main muddling component(s), but that may be very different in different situations. Sometimes biases are the main problem, sometimes it might just be lack of communication ability. It might be more cost-effective giving people in a developing country camera cellphones than teaching them about availability biases—while in another country the reverse may be true. But clearly overcoming biases is a relevant component in attacking many forms of societally dangerous muddle.

Anders_Sandberg

Does davi­dad’s up­load­ing moon­shot work?

Does davidad’s uploading moonshot work?