Alternate dialogue for the last two panels:
Kimiko [thinking quickly]: I think, if you meta-analyze for a—ah—Poisson process intensity marginal likelih -- [disappears]
Inquisitor-Superintendent [off-panel]: Post-hoc.
(This joke would work better if there were clearer hints that past appellant hypotheses had also been rejected, though.)
Previous arguments by Egan:
http://metamagician3000.blogspot.com/2009/09/interview-with-greg-egan.html
Sept. 2009, from an interview in Aurealis.
http://metamagician3000.blogspot.com/2008/04/transhumanism-still-at-crossroads.html
From April 2008. Only in the last few comments does Egan actually express an argument for the key intuition that has been driving the entire rest of his reasoning.
(To my eyes, this intution of Egan’s refers to a completely irrelevant hypothetical, in which humans somehow magically and reliably are always able to acquire possession of and make appropriate use of any insentient software tools that will be required, at any given moment, in order for humans to maintain hypothetical strategic parity with any contemporary AI’s.)
Gambit said the only equilibrium was mixed, with 1⁄5 each of (blue sword, blue armor), (blue sword, green armor), (yellow sword, yellow armor), (green sword, yellow armor), and (green sword, green armor).
With a stylin’ bonus of ε points per duel (if a win is 1 point and a loss is −1 points), Gambit says for ε≤1/4 the equilibrium is:
(blue sword, blue armor): 1/5−(4/5)ε
(blue sword, green armor): 1/5−(3/5)ε
(yellow sword, yellow armor): 1/5+(4/5)ε
(green sword, yellow armor): 1/5+(3/5)ε
(green sword, green armor): 1⁄5
I wish there is a “public drafts” feature on LessWrong
Seconded. See also JenniferRM on editorial-level versus object-level comments.
If someone has made a claim that they have life experience and their opinion, that is an observation, so its significance as at least outside-view evidence can still be interpreted in Bayesian terms. That means there is still analyzable evidence on the table, so the conversation doesn’t have to halt: you can quote their claim as an observation, and argue about its interpretation as evidence. “So then, what are the odds that people with your claimed amount of life experience would have your opinion if that opinion were true, versus if that opinion were false? Are there other things that might cause people with your claimed amount of life experience to have your opinion without its still being true?”
I once heard of people at UCSD who had plans to experiment with inverted spectrum goggles. Jonathan Cohen would know more.
I think peoples’ decision about whether to accept or resist the AGW proposition is being complicated by an implicit negotiation over political power that’s inevitably attached to that decision.
Because the scientific projections are still vague, people feel as if their decision about whether to believe in AGW is underdetermined by the evidence, in such a way that political actors in the future will feel entitled to retrospectively interpret their decision for purposes of political precedent. (“Were they forced by the evidence, or did they feel weak enough that they made a concession they didn’t have to make?”) And the precedent won’t be induced in terms of the mental states that a perfect decision theorist, thinking about the AGW mitigation decision problem, would have had. The precedent will be in terms of the mental states that a normal non-scientifically-trained (but politically active) human would have had. One of those mental states would be uncertainty about whether scientists (unconsciously intuited as potentially colluding with, and/or hoping to become, power-grubbing environmental regulators) are just making AGW up. In that context, agreeing that AGW is probably real feels like ceding one’s right of objection to whatever seizures of power someone’s found some vague scientific way of justifying.
It becomes a signaling game, in which each choice of belief will be understood as exactly how you would communicate a particular choice of political move, and the costs of making the wrong political move feel very high. So the belief decisions and the political actions become tangled up.
Roughly, people have no way of saying:
I believe that in terms of pure decision theory, the predicted AGW damage and costs of further investigation and costs of delay are high enough that mitigation attempts should start now. But I don’t want to give up my {economic privileges / substantive national sovereignty / chance to get the standard of living of past carbon-emitting nations} without a fight, because I don’t want groups in the future like {scientists / profit-hating hippie tree-huggers / freedom-hating U.N. environmental bureaucrats / greedy unfair first-world hypocrites} to think I’ll just roll over when they try to impose concessions on me, in the name of premises that will feel psychologically as though they might just as well have been made up. In that future situation, it will be important for me to be able to credibly threaten outrage at being forced into such concessions. But as long as nobody else is going to take me for their fool, the sacrifices needed to prevent AGW are fine with me; we could start today.
So instead, they say:
I believe that the case for AGW isn’t strong enough. I demand clearer proof.
If it were possible to negotiate separately about AGW action and about precedents of policy concessions to e.g. scientists’ claims, then you might see less decision-theoretic insanity around the AGW action question itself.
(Note—most of this analysis is not on the basis of such data as opinion polls or controlled studies. It’s just from introspecting on my experience of attempting to empathize with the state of mind of AGW disputants, as recalled mostly from Internet forums.)
Some comments:
We will then compare the likelihoods achieved by the two methods; higher likelihood wins. If there is ambiguity concerning the validity of the result, then we will compress the data set using compression algorithms based on the models and compare compression rates. Constructing a compressor from a statistical model is essentially a technical exercise; I can provide a Java implementation of arithmetic encoding. The compression rates must take into account the size of the compressor itself.
“Likelihood” is ambiguous: a Bayes net can be defined with a single hidden node for “object ID”, and all the observed nodes dependent only on the object ID, with conditional probability tables lifted from large look-up tables. This Bayes net would have a low probability from a prior belief distribution over Bayes nets but a high likelihood to the data.
To define compression unambiguously, you should agree on a programming language or executable format, and on runtime and memory bounds on a reference computer, as in the Hutter Prize.
An alternative to a test of compression would be a test of sequential prediction of e.g. individual real-valued or integer measurements, conditional on values of previous measurements according to some ordering. For each measurement, the predictor program would generate a predictive probability distribution, and a judge program would normalize the distribution or test that it summed to 1. The predictor’s score would be the log of the product of the sequential likelihoods. Compared to arithmetic compressor/decompressor pairs, sequential predictors might be less technically demanding to program (no problem of synchronizing the compressor and decompressor), but might be more technically demanding to judge (a predictive distribution from Monte Carlo sampling may depend on allowed computation time). The size of the predictor might still need to be included.
The Bayes Net contestant should be permitted to use prior belief distributions over network structures (as here). (This can be encoded as a causal belief about a hidden cause of patterns of causation.)
The Bayes Net contestant may wish to use network search algorithms or priors that hypothesize causal models with unobserved latent nodes.
The Bayes Net contestant should remember that the Skeptic contestant can use compression schemes based on reconstructability analysis, which is relatively powerful, and Markov graphical models, which include all simple Bayes nets for purposes of this problem.
[. . .] while the belief network paradigm is mathematically elegant and intuitively appealing, it is NOT very useful for describing real data.
The challenge is to prove the above claim wrong. [. . .]
The challenge hinges on the meaning of the word “real data”. [. . .] Other than that, there are no limitations—it can be image data, text, speech, machine learning sets, NetFlix, social science databases, etc.
Some data are about objects whose existence or relations are causally dependent on objects’ properties. Simple Bayesian networks cannot usually represent knowledge of such causal dependencies, but there are more general families of Bayesian models which can. Would it be in the spirit of the challenge for the Bayes Net contestant to use these more general models? Would it be out of the spirit of the challenge for the data to be about such a collection of objects?
PyMC is a DSL in python for (non-recursive) Bayesian models and Bayesian probability computations. I have been thinking of trying to make an ad-hoc collaborative scenario projection tool with PyMC and ikiwiki. Users would edit Literate Python (e.g. PyLit or Ly) wiki pages that defined PyMC model modules, and ikiwiki triggers would maintain Monte Carlo sampling computations and update results pages. But it won’t be enough for real argument mapping without decision theory (and possibly some other things).
Also: if mathematics in contact only with mathematics becomes “less mathematical” than mathematics in contact with praxis, then how can praxis in contact with mathematics become more practical than praxis out of contact with mathematics?
While I agree, where could the earth be getting its strength from?
In my experience, people who don’t compartmentalize tend to be cranks.
[. . .]
Most people who try to make all their beliefs fit with all their other beliefs, end up forcing some of the puzzle pieces into wrong-shaped holes. Their favorite part of their mental map of the world is locally consistent, but the farther-out parts are now WAY off, thus the crank-ism.
And that’s just the physical world. When we get to human values, some of them REALLY ARE in conflict with others[. . .]
The post “Reason as memetic immune disorder” was related. I’ll quote teasers so that you’ll read it:
People who grow up with a religion learn how to cope with its more inconvenient parts by partitioning them off, rationalizing them away, or forgetting about them. Religious communities actually protect their members from religion in one sense—they develop an unspoken consensus on which parts of their religion members can legitimately ignore. New converts sometimes try to actually do what their religion tells them to do.
[. . .]
The reason I bring this up is that intelligent people sometimes do things more stupid than stupid people are capable of. There are a variety of reasons for this; but one has to do with the fact that all cultures have dangerous memes circulating in them, and cultural antibodies to those memes. The trouble is that these antibodies are not logical[. . . .] They are the blind spots that let us live with a dangerous meme without being impelled to action by it.
And my comment there:
If the culture is constrained to hold constant the religion or cultural norms, then the resulting selection will cause the culture to develop blind spots, and also develop an unspoken (because unspeakable) but viciously enforced meta-norm of not seeing the blind spots. But if the culture is constrained to hold opposite meta-norms constant, such as a norm of seeing the blind spots or a norm of actually doing what one’s religion or cultural norms tell one do do, then the resulting selection will act against the dangerous memes instead.
you put a 3D landscape (mountains, etc) with “Territory” written under it.
[...]
Voilà!
(“Ceci n’est pas un territoire”)
a Map stuck to their clothes
Then a better symbol would be a surveyor’s theodolite, maybe with a map (or paper), a ruler, and a drafting compass.
I’ve also suggested a balance scale (decision theory) with an urn full of colored balls (Bayesian inference). Another symbol could be a locket (something to protect).
The question I think is interesting is: When you throw away probability information and keep only possibility information, like when you go from “a random ace was a spade” in Scenario 2 to “there was an ace which was a spade” in Scenario 1, when does that cause bias? How do you have to think about the information you have left? When can you just condition on your information being true, and when do you have to think about the probability that you would have been left with this information and not some other information?
Sasha Chislenko, when he was still alive, took this idea in the opposite direction:
Traffic signs are a great invention embodying the ancient dream of humanity: they tell you where you should stop, when you can go, what is around the corner, and some of them—traffic lights—even predict what will happen in near future. That’s about all I would like to know on my life path. People dreamed of knowing these things before traffic signs existed, and they looked at nature trying to interpret natural events as “omens”—signs put on their path by some intelligent agencies, with the purpose to tell them where they should go and what they can expect to happen there. Alas, nature wasn’t very helpful here, even though eventually people learned to recognize some phenomena as precursors to others. So with time, people decided to take things into their own hands, and to augment natural locales with clear indicators of what should be happening here, what is around, where are other similar places, and provide any other information that may be of value to their visitor.
[. . .] A similar process once happened to primitive biological organisms, which used to suffer from lack of guidance and tended to “fall into the same pits” over and over again. Road signs would have been quite useful then as well, but putting them up was too hard. So the poor critters developed little attached signs that would see the situation and tell the bodies what is going on and what they should do. We call these “smart signs” sensors and brains[. . .]
(Sasha was deeply interested in technologies for collective epistemology. I wonder if we would be in as bad a position as we’re in now if he hadn’t died… Which also means I wonder if there’s anyone else we—(“we”, who?—) should be taking unusual effort to keep alive.)
One way to think about these ideas about identity and roles is in the context of a more general theory I want to suggest: that there is a recurring set of conditions under which of games of costly signaling of the ability to resemble prototypes of a category tend to evolve convergently.
Such a theory might be consistent with the results from experiments on attractiveness of facial symmetry and facial averageness.
It might also be consistent with some observations I make by introspecting on intuitions which predict social penalties for unusual but morally harmless behavior. (E.g. the penalties one would receive if one were to wear, without explanation, a formal business suit with details somehow precisely matching the accidents of fashion of a randomly and fairly drawn alternate history from 200 years ago, instead of a formal business suit with details precisely matching the accidents of fashion of our own local history.)
Such a theory would predict that there would be literature on a cognitive bias to prefer prototypical and central members of a category to non-prototypical and peripheral members of the category. But as far as I know, there is not very much specific literature on this question. The only specific literature I know of is work) by Jamin Halberstadt of the University of Otago (NZ) and co-authors about visual attractiveness. A summary from 2006 is “The generality and ultimate origins of the attractiveness of prototypes”. “Prototypes are attractive because they are easy on the mind” is an ungated paper from 2006 that reports an experiment controlling for the effect of preference for fluently processed stimuli, because prototypical stimuli are processed more fluently. (One possible interpretation of the result is that there may be a reason to prefer stimuli which aren’t confusing, because confusing stimuli may hide defects better.) “The face of fluency: Semantic coherence automatically elicits a specific pattern of facial muscle reactions” generalizes part of this effect to non-visual stimuli, and “Uniting the tribes of fluency to form a metacognitive nation” argues that fluency experiences affect many dimensions of social judgement.
If you want to get more meta, you could have the characters in the story use game theory to analyze the causal basis for oppobrium towards authors of self-insert fanfic (perhaps in terms of signals about underlying psychological states, and/or signals about abhorrence of some of those psychological states, and/or signals about abhorrence of inadequate abhorrence).
(And then, having analyzed the oppobrium and reduced it to game theory and social psychology, they could argue about whether or not the oppobrium “should” apply to their own author. And about whether or not having the analysis even gives them any useful information to help decide that question.)
It was me. Zack Davis was jokingly suggesting a benefit performance, you were unhappy that it was a bad play, and I was looking for things that would make it a better play, such as playing it as a farce.
And again, the whole point was that, technically, it’s not self-insert fanfic if someone else already inserted you...
What information can be derived about utility functions from behavior?
(Here, “information about utility functions” may be understood in your policy-relevant sense, of “factors influencing the course of action that rational expected-utility maximization might surprisingly choose to force upon us after it was too late to decommit.”)
Suppose you observe that some agents, when they are investing, take into account projected market rates of return when trading off gains and losses at different points in time. Here are two hypotheses about the utility functions of those agents.
Hypothesis 1: These agents happened to already have a utility function whose temporal discounting was to match what the market rate of return would be. This is to say: The utility function already assigned particular intrinsic values to hypothetical events in which assets were gained or lost at different times. The ratios between these intrinsic values were already equal to what the appropriate exponential of the integrated market rate of return would later turn out to be.
Hypothesis 2: These agents have a utility function in which assets gained or lost in the near term are valued because of an intrinsic good which could be purchased with those assets at a point in the distant future. These agents evaluate near-term investments and payoffs happening at different times in terms of market rates of return, for understandable and purely instrumental reasons relating to opportunity cost.
Neither hypothesis is quite plausible psychologically or historically, but the second hypothesis is closer to being plausible, and each hypothesis makes the same predictive distribution about the agents’ near-term investment behaviors. This is to say that the “preference likelihood” ratio between the two hypotheses is flat.
(In your apparent policy terms, this would correspond roughly to the idea that, while rational expected-utility maximization may be trying to “choose” which of these two utility functions to define as normative, so that it can then “force” the courses of action dictated by the chosen utility function “upon” the agents, in this case the balance of factors affecting rational expected-utility maximization’s “choice” evens out. Therefore, rational expected-utility maximization’s “decision” will depend on its prior disposition to “prefer” one or the other utility function, for reasons unrelated to observation.)
Now, suppose that the agents from the second hypothesis forecast market rates of return for some period, and then create new agents. These new agents have recognizable internal data structures representing utility functions in a form as per the first hypothesis, and these data structures will be queried to determine the new agents’ decisions about near-term trades. However, the new agents’ only source of information about their utility functions comes from observing their own behavior: they do not have direct introspective access to their internal data structure, and they do not know about the asset conversion event in the future. (However, they will convert their holdings at that time, as a hard-coded instinct; in terms of revealed preference, this can be interpreted as having a utility function that assigns the purchased good infinite relative value). Now, which hypothesis should we say is “really” true of these new agents’ utility functions?
(And how do we delineate what the parts of this situation even are, that supposedly “have” the utility functions we want to inquire about?)
This is a general problem with our present framework for reasoning about utility. The predictions and recommendations from a hypothesized utility function are invariant under various transformations of the hypothesis; in particular, transformations that preserve relative intervals of expected utility between available actions at each juncture. For example, for a perfect expected-utility maximizer, the reward function constructed by a perfectly trained temporal-difference reinforcement learning system motivates exactly the same behavior as the reward function whose integrals the TD learner was trained to predict. (This is quite apart from the problem of invariance under transformations that stretch or squeeze probability and reward simultaneously, such as the transformations that relate different methods of anthropic reasoning.)
As if to add to the confusion, when humans are informed about utility theory, and asked to interpret their introspective information about their preferences in terms of utility, they will report different preferences as being “intrinsic” vs. “instrumental” at different points in time [citation: folk belief]. There may be a psychological process related to temporal-difference reinforcement learning which converts preferences which introspectively appear “instrumental” into preferences which introspectively appear “intrinsic”.
Why were you so certain, in your original draft, that exponential temporal discounting behavior was a matter of intrinsic value rather than instrumental value, so that a normative framework of utilitarian reasoning would force it upon us, and the alternative possibility was not worth mentioning?