What makes you think so? The main reason I can see why the death of less than 100% of the population would stop us from getting back is if it’s followed by a natural event that finishes off the rest. However 25% of current humanity seems much more than enough to survive all natural disasters that are likely to happen in the following 10,000 years. The black death killed about half the population of Europe and it wasn’t enough even to destroy the pre-existing social institutions.
Squark
Karma: 1,117
Hi Peter! I am Vadim, we met in a LW meetup in CFAR’s office last May.
You might be right that SPARC is important but I really want to hear from the horse’s mouth what is their strategy in this regard. I’m inclined to disagree with you regarding younger people, what makes you think so? Regardless of age I would guess establishing a continuous education programme would have much more impact than a two-week summer workshop. It’s not obvious what is the optimal distribution of resources (many two week workshops for many people or one long program for fewer people) but I haven’t seen such an analysis by CFAR.
The body of this worthy man died in August 2014, but his brain is preserved by Alcor. May a day come when he lives again and death is banished forever.
It feels like there is an implicit assumption in CFAR’s agenda that most of the important things are going to happen in one or two decades from now. Otherwise it would make sense to place more emphasis on creating educational programs for children where the long term impact can be larger (I think). Do you agree with this assessment? If so, how do you justify the short term assumption?
On the other hand, articles and books can reach a much larger number of people (case in point: the Sequences). I would really want to see a more detailed explanation by CFAR of the rationale behind their strategy.
Thank you for writing this. Several questions.
How do you see CFAR in the long term? Are workshops going to remain in the center? Are you planning some entirely new approaches to promoting rationality?
How much do you plan to upscale? Are the workshops intended to produce a rationality elite or eventually become more of a mass phenomenon?
It seem possible that revolutionizing the school system would have much higher impact on rationality than providing workshops for adults. SPARC might be one step in this direction. What are you thoughts / plans regarding this approach?
Facebook event: https://www.facebook.com/events/796399390482188/
!!! It is October 27, not 28 !!!
Also, it’s at 19:00
Sorry but it’s impossible to edit the post.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
This is not quite what happens. When you do UDT properly, the result is that the Tegmark level IV multiverse has finite capacity for human lives (when human lives are counted with 2^-{Kolomogorov complexity} weights, as they should). Therefore the “bare” utility function has some kind of diminishing returns but the “effective” utility function is roughly linear in human lives once you take their “measure of existence” into account.
I consider it highly likely that bounded utility is the correct solution.
If you have trouble finding the location, feel free to call me (Vadim) at 0542600919.
In order for the local interpretation of Sleeping Beauty to work, it’s true that the utility function has to assign utilities to impossible counterfactuals. I don’t think this is a problem...
It is a problem in the sense that there is no canonical way to assign these utilities in general.
In the utility functions I used as examples above (winning bets to maximize money, trying to watch a sports game on a specific day), the utility for these impossible counterfactuals is naturally specified because the utility function was specified as a sum of the utilities of local properties of the universe. This is what both allows local “consequences” in Savage’s theorem, and specifies those causally-inaccessible utilities.
True. As a side note, the Savage theorem is not quite the right thing here since it produces both probabilities and utilities while in our situations the utilities are already given.
This raises the question of whether, if you were given only the total utilities of the causally accessible histories of the universe, it would be “okay” to choose the inaccessible utilities arbitrarily such that the utility could be expressed in terms of local properties.
The problem is that different extensions produce complete different probabilities. For example, suppose U(AA) = 0, U(BB) = 1. We can decide U(AB)=U(BA)=0.5 in which case the probability of both copies is 50%. Or, we can decide U(AB)=0.7 and U(BA)=0.3 in which case the probability of the first copy is 30% and the probability of the second copy is 70%.
The ambiguity is avoided if each copy has an independent source of random because this way all of the counterfactuals are “legal.” However, as the example above shows, these probabilities depend on the utility function. So, even if we consider sleeping beauties with independent sources of random, the classical formulation of the problem is ambiguous since it doesn’t specify a utility function. Moreover, if all of the counterfactuals are legal then it might be the utility function doesn’t decompose into a linear combination over copies, in which case there is no probability assignment at all. This is why Everett branches have well defined probabilities but e.g. brain emulation clones don’t.
It’s also a valid interpretation to have the “outcome” be whether Sleeping Beauty wins, loses, or doesn’t take an individual bet about what day it is (there is a preference ordering over these things), the “action” being accepting or rejecting the bet, and the “event” being which day it is (the outcome is a function of the chosen action and the event).
In Savage’s theorem acts are arbitrary functions from the set of states to the set of consequences. Therefore to apply Savage’s theorem in this context you have to consider blatantly inconsistent counterfactuals in which the sleeping beauty makes difference choices in computationally equivalent situations. If you have an extension of the utility function to these counterfactuals and it happens to satisfy the conditions of Savage’s theorem then you can assign probabilities. This extension is not unique. Moreover, in some anthropic scenarios in doesn’t exist (as you noted yourself).
...argument in favor have to either resort to Cox’s theorem (which I find more confusing), or engage in contortions about games that counterfactually could be constructed.
Cox’s theorem only says that any reasonable measure of uncertainty can be transformed into a probability assignment. Here there is no such measure of uncertainty. Different counterfactual games lead to different probability assignments.
I’m not asking researchers to predict what they will discover. There are different mindsets of research. One mindset is looking for heuristics that maximize short term progress on problems of direct practical relevance. Another mindset is looking for a rigorously defined overarching theory. MIRI is using the latter mindset while most other AI researchers are much closer to the former mindset.
I disagree with the part “her actions lead to different outcomes depending on what day it is.” The way I see it, the “outcome” is the state of the entire multiverse. It doesn’t depend on “what day it is” since “it” is undefined. The sleeping beauty’s action simultaneously affects the multiverse through several “points of interaction” which are located in different days.
Hi Charlie! Actually I complete agree with Vladimir on this: subjective probabilities are meaningless, meaningful questions are decision theoretic. When the sleeping beauty is asked “what day is it?” the question is meaningless because she is simultaneously in several different days (since identical copies of her are in different days).
A “coincidence” is an a priori improbable event in your model that has to happen in order to create a situation containing a “copy” of the observer (which roughly means any agent with a similar utility function and similar decision algorithm).
Imagine two universe clusters in the multiverse: one cluster consists of universe running on fragile physics, another cluster consists of universes running on normal physics. The fragile cluster will contain much less agent-copies than the normal cluster (weighted by probability). Imagine you have to make a decision which produces different utilities depending on whether you are in the fragile cluster or the normal cluster. According to UDT, you have to think as even you are deciding for all copies. In other words, if you make decisions under the assumption you are in the fragile cluster, all copies make decisions under this assumption, if you make decisions under the assumption you are in the normal cluster, all copies make decisions under this assumption. Since the normal cluster is much more “copy-dense”, it pays off much more to make decisions as if you are in the normal cluster (since utility is aggregated over the entire multiverse).
The weighting comes from the Solomonoff prior. For example, see the paper by Legg.
I did a considerable amount of software engineer recruiting during my career. I only called the references at an advanced stage, after an interview. It seems to me that calling references before an interview would take too much of their time (since if everyone did this they would be called very often) and too much of my time (since I think their input would rarely disqualify a candidate at this point). The interview played the most important role in my final decision, but when a reference mentioned something negative which resonated with something that concerned me after the interview, this was often a reason to reject.
I’m digging into this a little bit, but I’m not following your reasoning. UDT from what I see doesn’t mandate the procedure you outline. (perhaps you can show an article where it does) I also don’t see how which decision theory is best should play a strong role here.
Unfortunately a lot of the knowledge on UDT is scattered in discussions and it’s difficult to locate good references. The UDT point of view is that subjective probabilities are meaningless (the third horn of the anthropic trilemma) thus the only questions it make sense to ask are decision-theoretic questions. Therefore decision theory does play a strong role in any question involving anthropics. See also this.
But anyways I think the heart of your objection seems to be “Fragile universes will be strongly discounted in the expected utility because of the amount of coincidences required to create them”. So I’ll free admit to not understanding how this discounting process works...
The weight of a hypothesis in the Solomonoff prior equals N 2^{-(K + C)} where K is its Kolomogorov complexity, C is the number of coin flips needed to produce the given observation and N is the number of different coin flip outcomes compatible with the given observation. Your fragile universes have high C and low N.
...but I will note that current theoretical structures (standard model inflation cosmology/string theory) have a large amount of constants that are considered coincidences and also produce a large amount of universes like ours in terms of physical law but different in terms of outcome.
Right. But these are weak points of the theory, not strong points. That is, if we find an equally simple theory which doesn’t require these coincidences it will receive substantially higher weight. Anyway your fragile universes have a lot more coincidences than any conventional physical theory.
I would also note that fragile universe “coincidences” don’t seem to me to be more coincidental in character than the fact we happen to live on a planet suitable for life.
In principle hypotheses with more planets suitable for life also get higher weight, but the effect levels off when reaching O(1) civilizations per current cosmological horizon because it is offset by the high utility of having the entire future light cone to yourself. This is essentially the anthropic argument for a late filter in the Fermi paradox, and the reason this argument doesn’t work in UDT.
Lastly I would also note that at this point we don’t have a good H1 or H2.
All of the physical theories we have so far are not fragile, therefore they are vastly superior to any fragile physics you might invent.
“Neural networks” vs. “Not neural networks” is a completely wrong way to look at the problem.
For one thing, there are very different algorithms lumped under the title “neural networks”. For example Boltzmann machines and feedforward networks are both called “neural networks” but IMO it’s more because it’s a fashionable name than because of actual similarity in how they work.
More importantly, the really significant distinction is making progress by trail and error vs. making progress by theoretical understanding. The goal of AI safety research should be shifting the balance towards the second option, since the second option is much more likely to yield results that are predictable and satisfy provable guarantees. In this context I believe MIRI correctly identified multiple important problems (logical uncertainty, decision theory, naturalized induction, Vingean reflection). I am mildly skeptical about the attempts to attack these problems using formal logic, but the approaches based on complexity theory and statistical learning theory that I’m pursuing seem completely compatible with various machine learning techniques including ANNs.