jacobt

Karma: 119

jacobt 27 Mar 2013 19:19 UTC
0 points
in reply to: ThrustVectoring’s comment on: A Difficulty in the Concept of CEV
Arrow’s Theorem doesn’t say anything about strategic voting. The only reasonable non-strategic voting system I know of is random ballot (pick a random voter; they decide who wins). I’m currently trying to figure out a voting system that is based on finding the Nash equilibrium (which may be mixed) of approval voting, and this system might also be strategy-free.

When I said linear combination of utility functions, I meant that you fix the scaling factors initially and don’t change them. You could make all of them 1, for example. Your voting system (described in the last paragraph) is a combination of range voting and IRV. If everyone range votes so that their favorite gets 1 and everyone else gets −1, then it’s identical to IRV, and shares the same problems such as non-monotonicity. I suspect that you will also get non-monotonicity when votes aren’t “favorite gets 1 and everyone else gets −1”.

EDIT: I should clarify: it’s not 1 for your favorite and −1 for everyone else. It’s 1 for your favorite and close to −1 for everyone else, such that when your favorite is eliminated, it’s 1 for your next favorite and close to −1 for everyone else after rescaling.

jacobt 27 Mar 2013 19:11 UTC
4 points
on: Upgrading moral theories to include complex values

It is bad to create a small population of creatures with humane values (that has positive welfare) and a large population of animals that are in pain. For instance, it is bad to create a population of animals with −75 total welfare, even if doing so allows you to create a population of humans with 50 total welfare.

Why do you believe this? I don’t. Due to wild animal suffering, this proposition implies that it would have been better if no life had appeared on Earth, assuming average human/animal welfare and the human/animal ratio don’t dramatically change in the future.

jacobt 27 Mar 2013 1:40 UTC
3 points
on: A Difficulty in the Concept of CEV
I couldn’t access the “Aggregation Procedure for Cardinal Preferences” article. In any case, why isn’t using an aggregate utility function that is a linear combination of everyone’s utility functions (choosing some arbitrary number for each person’s weight) a way to satisfy Arrow’s criteria?

It should also be noted that Arrow’s impossibility theorem doesn’t hold for non-deterministic decision procedures. I would also caution against calling this an “existential risk”, because while decision procedures that violate Arrow’s criteria might be considered imperfect in some sense, they don’t necessarily cause an existential catastrophe. Worldwide range voting would not be the best way of deciding everything, but it most likely wouldn’t be an existential risk.

jacobt 11 Mar 2013 21:05 UTC
2 points
in reply to: Eliezer Yudkowsky’s comment on: You only need faith in two things
Ok, I agree with this interpretation of “being exposed to ordered sensory data will rapidly promote the hypothesis that induction works”.

jacobt 11 Mar 2013 18:30 UTC
3 points
in reply to: Eliezer Yudkowsky’s comment on: You only need faith in two things
You could choose to single out a single alternative hypothesis that says the sun won’t rise some day in the future. The ratio between P(sun rises until day X) and P(sun rises every day) will not change with any evidence before day X. If initially you believed a 99% chance of “the sun rises every day until day X” and a 1% chance of Solomonoff induction’s prior, you would end up assigning more than a 99% probability to “the sun rises every day until day X”.

Solomonoff induction itself will give some significant probability mass to “induction works until day X” statements. The Kolmogorov complexity of “the sun rises until day X” is about the Kolmogorov complexity of “the sun rises every day” plus the Kolmogorov complexity of X (approximately log2(x)+2log2(log2(x))). Therefore, even according to Solomonoff induction, the “sun rises until day X” hypothesis will have a probability approximately proportional to P(sun rises every day) / (X log2(X)^2). This decreases subexponentially with X, and even slower if you sum this probability for all Y >= X.

In order to get exponential change in the odds, you would need to have repeatable independent observations that distinguish between Solomonoff induction and some other hypothesis. You can’t get that in the case of “sun rises every day until day X” hypotheses.

jacobt 11 Mar 2013 17:34 UTC
0 points
in reply to: DaFranker’s comment on: You only need faith in two things
You’re making the argument that Solomonoff induction would select “the sun rises every day” over “the sun rises every day until day X”. I agree, assuming a reasonable prior over programs for Solomonoff induction. However, if your prior is 99% “the sun rises every day until day X”, and 1% “Solomonoff induction’s prior” (which itself might assign, say, 10% probability to the sun rising every day), then you will end up believing that the sun rises every day until day X. Eliezer asserted that in a situation where you assign only a small probability to Solomonoff induction, it will quickly dominate the posterior. This is false.

most of the evidence given by sunrise accrues to “the sun rises every day”, and the rest gets evenly divided over all non-falsified “Day X”

Not sure exactly what this means, but the ratio between the probabilities “the sun rises every day” and “the sun rises every day until day X” will not be affected by any evidence that happens before day X.

jacobt 11 Mar 2013 10:07 UTC
3 points
on: You only need faith in two things

Because being exposed to ordered sensory data will rapidly promote the hypothesis that induction works

Not if the alternative hypothesis assigns about the same probability to the data up to the present. For example, an alternative hypothesis to the standard “the sun rises every day” is “the sun rises every day, until March 22, 2015″, and the alternative hypothesis assigns the same probability to the data observed until the present as the standard one does.

You also have to trust your memory and your ability to compute Solomonoff induction, both of which are demonstrably imperfect.

jacobt 16 Feb 2013 20:57 UTC
3 points
in reply to: loup-vaillant’s comment on: A Series of Increasingly Perverse and Destructive Games
For every n, a program exists that will solve the halting problem for programs up to length n, but the size of this program must grow with n. I don’t really see any practical way for a human to write this program other than generating an extremely large number and then testing all programs up to length n for halting within this bound, in which case you’ve already pretty much solved the original problem. If you use some proof system to try to prove that programs halt and then take the maximum running time of only those, then you might as well use a formalism like the calculus of constructions.

jacobt 15 Feb 2013 21:35 UTC
4 points
on: A Series of Increasingly Perverse and Destructive Games
Game1 has been done in real life (without the murder): http://djm.cc/bignum-results.txt

Also:

Write a program that generates all programs shorter than length n, and finds the one with the largest output.

Can’t do that, unless you already know the programs will halt. The winner of the actual contest used a similar strategy, using programs in the calculus of constructions so they are guaranteed to halt.

For Game2, if your opponent’s program (say there are only 2 players) says to return your program’s output + 1, then you can’t win. If your program ever halts, they win. If it doesn’t halt, then you both lose.

jacobt 16 Jan 2013 2:24 UTC
0 points
in reply to: Nisan’s comment on: A fungibility theorem

But if the choices only have the same expectation of v2, then you won’t be optimizing for v1.

Ok, this correct. I hadn’t understood the preconditions well enough. It seems that now the important question is whether things people intuitively think of as different values (my happiness, total happiness, average happiness) satisfy this condition.

jacobt 15 Jan 2013 7:22 UTC
0 points
in reply to: Nisan’s comment on: A fungibility theorem
You would if you could survive for v1*v2 days.

jacobt 14 Jan 2013 22:40 UTC
1 point
in reply to: DaFranker’s comment on: A fungibility theorem
I do think that everything should reduce to a single utility function. That said, this utility function is not necessarily a convex combination of separate values, such as “my happiness”, “everyone else’s happiness”, etc. It could contain more complex values such as your v1 and v2, which depend on both x and y.

In your example, let’s add a choice D: 50% of the time it’s A, 50% of the time it’s B. In terms of individual happiness, this is Pareto superior to C. It is Pareto inferior for v1 and v2, though.

EDIT: For an example of what I’m criticizing: Nisan claims that this theorem presents a difficulty for avoiding the repugnant conclusion if your desiderata are total and average happiness. If v1 = total happiness and v2 = average happiness, and Pareto optimality is desirable, then it follows that utility is a*v1 + b*v2. From this utility function, some degenerate behavior (blissful solipsist or repugnant conclusion) follows. However, there is nothing that says that Pareto optimality in v1 and v2 is desirable. You might pick a non-linear utility function of total and average happiness, for example atan(average happiness) + atan(total happiness). Such a utility function will sometimes pick policies that are Pareto inferior with respect to v1 and v2.

jacobt 14 Jan 2013 22:35 UTC
3 points
in reply to: AlexMennen’s comment on: A fungibility theorem
I didn’t say anything about risk aversion. This is about utility functions that depend on multiple different “values” in some non-convex way. You can observe that, in my original example, if you have no water, then utility (days survived) is linear with respect to food.

jacobt 14 Jan 2013 18:55 UTC
0 points
in reply to: DaFranker’s comment on: A fungibility theorem
I think we agree. I am just pointing out that Pareto optimality is undesirable for some selections of “values”. For example, you might want you and everyone else to both be happy, and happiness of one without the other would be much less valuable.

I’m not sure how you would go about deciding if Pareto optimality is desirable, now that the theorem proves that it is desirable iff you maximize some convex combination of the values.

jacobt 14 Jan 2013 6:35 UTC
2 points
on: A fungibility theorem
I think that, depending on what the v’s are, choosing a Pareto optimum is actually quite undesirable.

For example, let v1 be min(1000, how much food you have), and let v2 be min(1000, how much water you have). Suppose you can survive for days equal to a soft minimum of v1 and v2 (for example, 0.001 v1 + 0.001 v2 + min(v1, v2)). All else being equal, more v1 is good and more v2 is good. But maximizing a convex combination of v1 and v2 can lead to avoidable dehydration or starvation. Suppose you assign weights to v1 and v2, and are offered either 1000 of the more valued resource, or 100 of each. Then you will pick the 1000 of the one resource, causing starvation or dehydration after 1 day when you could have lasted over 100. If which resource is chosen is selected randomly, then any convex optimizer will die early at least half the time.

A non-convex aggregate utility function, for example the number of days survived (0.001 v1 + 0.001 v2 + min(v1, v2)), is much more sensible. However, it will not select Pareto optima. It will always select the 100 of each option; always selecting 1000 of one leads to greater expected v1 and expected v2 (500 for each).

jacobt 23 Sep 2012 19:12 UTC
1 point
in reply to: Vladimir_Nesov’s comment on: No Anthropic Evidence
Actually you’re right, I misread the problem at first. I thought that you had observed yourself not dying 1000 times (rather than observing “heads” 1000 times), in which case you should keep playing.

Applying my style of analyzing anthropic problems to this one: Suppose we have 1,000,000 * 2^1000 players. Half flip heads initially, half flip tails. About 1,000,000 will get heads 1,000 times. Of them, 500,000 will have flipped heads initially. So, your conclusion is correct.

jacobt 23 Sep 2012 18:37 UTC
1 point
on: No Anthropic Evidence
I think you’re wrong. Suppose 1,000,000 people play this game. Each of them flips the coin 1000 times. We would expect about 500,000 to survive, and all of them would have flipped heads initially. Therefore, P(I flipped heads initially | I haven’t died yet after flipping 1000 coins) ~= 1.

This is actually quite similar to the Sleeping Beauty problem. You have a higher chance of surviving (analogous to waking up more times) if the original coin was heads. So, just as the fact that you woke up is evidence that you were scheduled to wake up more times in the Sleeping Beauty problem, the fact that you survive is evidence that you were “scheduled to survive” more in this problem.

On the other hand, each “heads” you observe doesn’t distinguish the hypothetical where the original coin was “heads” from one where it was “tails”.

This is the same incorrect logic that leads people to say that you “don’t learn anything” between falling asleep and waking up in the Sleeping Beauty problem.

I believe the only coherent definition of Bayesian probability in anthropic problems is that P(H | O) = the proportion of observers who have observed O, in a very large universe (where the experiment will be repeated many times), for whom H is true. This definition naturally leads to both ²⁄₃ probability in the Sleeping Beauty problem and “anthropic evidence” in this problem. It is also implied by the many-worlds interpretation in the case of quantum coins, since then all those observers really do exist.

jacobt 20 Jul 2012 7:28 UTC
16 points
on: Imperfect Voting Systems
I vote for range voting. It has the lowest Bayesian regret (best expected social utility). It’s also extremely simple. Though it’s not exactly the most unbiased source, rangevoting.org has lots of information about range voting in comparison to other methods.

jacobt 6 Jun 2012 21:22 UTC
3 points
on: Open Problems Related to Solomonoff Induction
For aliens with a halting oracle:

Suppose the aliens have this machine that may or may not be a halting oracle. We give them a few Turing machine programs and they decide which ones halt and which ones don’t. Then we run the programs. Sure enough, none of the ones they say run forever halt, and some of them they say don’t run forever will halt at some point. Suppose we repeat this process a few times with different programs.

Now what method should we use to predict the point at which new programs halt? The best strategy seems to be to ask the aliens which ones halt, give the non-halting ones a 0 probability of halting at every step, and give the other ones some small nonzero probability of halting on every step. Of course this strategy only works if the aliens actually have a halting oracle so it its predictions should be linearly combined with a fallback strategy.

I think that Solomonoff induction will find this strategy, because the hypothesis that the aliens have a true halting oracle is formalizable. Here’s how: we learn a function from aliens’ answers → distribution over when the programs halt. We can use the strategy of predicting that the ones that the aliens say don’t halt don’t halt, and using some fallback mechanism for predicting the ones that do halt. This strategy is computable so Solomonoff induction will find it.

For Berry’s paradox:

This is a problem with every formalizable probability distribution. You can always define a sequence that says: see what bit the predictor predicts with a higher probability, then output the opposite. Luckily Solomonoff induction does the best it can by having the estimates converge to 50%. I don’t see what a useful solution to this problem would even look like; it seems best to just treat the uncomputable sequence as subjectively random.

jacobt 2 Jun 2012 0:57 UTC
0 points
on: The Truth Points to Itself, Part I
I found this post interesting but somewhat confusing. You start by talking about UDT in order to talk about importance. But really the only connection from UDT to importance is the utility function, so you might as well start with that. And then you ignore utility functions in the rest of your post when you talk about Schmidhuber’s theory.

It just has a utility function which specifies what actions it should take in all of the possible worlds it finds itself in.

Not quite. The utility function doesn’t specify what action to take, it specifies what worlds are desirable. UDT also requires a prior over worlds and a specification of how the agent interacts with the world (like the Python programs here). The combination of this prior and the expected value computations that UDT does would constitute “beliefs”.

Informally, your decision policy tells you what options or actions to pay most attention to, or what possibilities are most important.

I don’t see how this is. Your decision policy tells you what to do once you already know what you can do. If you’re using “important” to mean “valuable” just say that instead.

I do like the idea of modelling the mind as an approximate compression engine. This is great for reducing some thought processes to algorithms. For example I think property dualism can be thought of as a way to compress the fact that I am me rather than some other person, or at least make explicit the fact that this must be compressed.

Schmidhuber’s theory is interesting but incomplete. You can create whatever compression problem you want through a template, e.g. a pseudorandom sequence you can only compress by guessing the seed. Yet repetitions of the same problem template are not necessarily interesting. It seems that some bits are more important than other bits; physicists are very interested in compressing the number of spacial dimensions in the universe even though this quantity can be specified in a few bits. I don’t know any formal approaches to quantifying the importance of compressing different things.

I wrote a paper on this subject (compression as it relates to theory of the mind). I also wrote this LessWrong post about using compression to learn values.