I usually call this lampshading, and I’ll link this comment to explain what I mean. Thanks!
a gently pricked vein
Karma: 235
Thank you for this comment. I went through almost exactly the same thing, and might have possibly tabled it at the “I am really confused by this post” stage had I not seen someone well-known in the community struggle with and get through it.
My brain especially refused to read past the line that said “pushing it to 50% is like throwing away information”: Why would throwing away information correspond to the magic number 50%?! Throwing away information brings you closer to maxent, so if true, what is it about the setup that makes 50% the unique solution, independent of the baseline and your estimate? That is, what is the question?
I think it’s this: in a world where people can report the probability for a claim or the negation of it, what is the distribution of probability-reports you’d see?
By banning one side of it as Rafael does, you get it to tend informative. Anyway, this kind of thinking makes it seem like it’s a fact about this flipping trick and not fundamental to probability theory. I wonder if there are more such tricks/actual psychology to adjust for to get a different answer.
While you’re technically correct, I’d say it’s still a little unfair (in the sense of connoting “haha you call yourself a rationalist how come you’re failing at akrasia”).
Two assumptions that can, I think you’ll agree, take away from the force of “akrasia is epistemic failure”:
if modeling and solving akrasia is, like diet, a hard problem that even “experts” barely have an edge on, and importantly, things that do work seem to be very individual-specific making it quite hard to stand on the shoulders of giants
if a large percentage of people who’ve found and read through the sequences etc have done so only because they had very important deadlines to procrastinate
...then on average you’d see akrasia over-represented in rationalists. Add to this the fact that akrasia itself makes manually aiming your rationality skills at what you want harder. That can leave it stable even under very persistent efforts.
I’m interested in this. The problem is that if people consider the value provided by the different currencies at all fungible, side markets will pop up that allow their exchange.
An idea I haven’t thought about enough (mainly because I lack expertise) is to mark a token as Contaminated if its history indicates that it has passed through “illegal” channels, ie has benefited someone in an exchange not considered a true exchange of value, and so purists can refuse to accept those. Purist communities, if large, would allow stability of such non-contaminated tokens.
Maybe a better question to ask is “do we have utility functions that are partial orders and thus would benefit from many isolated markets?”, because if so, you wouldn’t have to worry about enforcing anything, many different currencies will automatically come into existence and be stable.
Of course, more generally, you wouldn’t quite have complete isolation, but different valuations of goods in different currencies, without “true” fungibility. I think it is quite possibe that our preference orderings are in fact partial and the current one-currency valuation of everything might be improved.
The expectations you do not know you have control your happiness more than you know. High expectations that you currently have don’t look like high expectations from the inside, they just look like how the world is/would be.
But “lower your expectations” can often be almost useless advice, kind of like “do the right thing”.
Trying to incorporate “lower expectations” often amounts to “be sad”. How low should you go? It’s not clear at all if you’re using territory-free un-asymmetric simple rules like “lower”. Like any other attempt at truth-finding, it is not magic. It requires thermodynamic work.
The thing is, the payoff is rather amazing. You can just get down to work. As soon as you’re free of a constant stream of abuse from beliefs previously housed in your head, you can Choose without Suffering.
The problem is, I’m not sure how to strategically go about doing this, other than using my full brain with Constant Vigilance.
Coda: A large portion of the LW project (or at least, more than a few offshoots) is about noticing you have beliefs that respond to incentives other than pure epistemic ones, and trying not to reload when shooting your foot off with those. So unsurprisingly, there’s a failure mode here: when you publicly declare really low expectations (eg “everyone’s an asshole”), it works to challenge people, urges them to prove you wrong. It’s a cool trick to win games of Chicken but as usual, it works by handicapping you. So make sure you at least understand the costs and the contexts it works in.
I think a counterexample to “you should not devote cognition to achieving things that have already happened” is being angry at someone who has revealed they’ve betrayed you, which might acause them to not have betrayed you.
Is metarationality about (really tearing open) the twelfth virtue?
It seems like it says “the map you have of map-making is not the territory of map-making”, and gets into how to respond to it fluidly, with a necessarily nebulous strategy of applying the virtue of the Void.
(this is also why it always felt like metarationality seems to only provide comments where Eliezer would’ve just given you the code)
The parts that don’t quite seem to follow is where meaning-making and epistemology collide. I can try to see it as a “all models are false, some models are useful” but I’m not sure if that’s the right perspective.
strangepoop’s Shortform
I want to ask this because I think I missed it the first few times I read Living in Many Worlds: Are you similarly unsatisfied with our response to suffering that’s already happened, like how Eliezer asks, about the twelfth century? It’s boldface “just as real” too. Do you feel the same “deflation” and “incongruity”?
I expect that you might think (as I once did) that the notion of “generalized past” is a contrived but well-intentioned analogy to manage your feelings.
But that’s not so at all: once you’ve redone your ontology, where the naive idea of time isn’t necessarily a fundamental thing and thinking in terms of causal links comes a lot closer to how reality is arranged, it’s not a stretch at all. If anything, it follows that you must try and think and feel correctly about the generalized past after being given this information.
Of course, you might modus tollens here.
Soares also did a good job of impressing this in Dive In:
In my experience, the way you end up doing good in the world has very little to do with how good your initial plan was. Most of your outcome will depend on luck, timing, and your ability to actually get out of your own way and start somewhere. The way to end up with a good plan is not to start with a good plan, it’s to start with some plan, and then slam that plan against reality until reality hands you a better plan.
The idea doesn’t have to be good, and it doesn’t have to be feasible, it just needs to be the best incredibly concrete plan that you can come up with at the moment. Don’t worry, it will change rapidly when you start slamming it into reality. The important thing is to come up with a concrete plan, and then start executing it as hard as you can — while retaining a reflective state of mind updating in the face of evidence.
I don’t think the “idea of scientific thinking and evidence” has so much to do with throwing away information as adding reflection, post which you might excise the cruft.
Being able to describe what you’re doing, ie usefully compress existing strategies-in-use, is probably going to be helpful regardless of level of intelligence because it allows you to cheaply tweak your strategies when either the situation or the goal is perturbed.
To further elaborate 4: your example of the string “1” being a conscious agent because you can “unpack” it into an agent really feels like it shouldn’t count: you’re just throwing away the “1” and replaying a separate recording of something that was conscious. This sounds about as much of a non-sequitur as “I am next to this pen, so this pen is conscious”.
We could, however, make it more interesting by making the computation depend “crucially” on the input. But what counts?
Suppose I have a program that turns noise into a conscious agent (much like generative models can turn a noise vector into a face, say). If we now seed this with a waterfall, is the waterfall now a part of the computation, enough to be granted some sentience/moral patienthood? I think the usual answer is “all the non-trivial work is being done by the program, not the random seed”, as Scott Aaronson seems to say here. (He also makes the interesting claim of “has to participate fully in the arrow of time to be conscious”, which would disqualify caching and replaying.)
But this can be made a little more confusing, because it’s hard to tell which bit is non-trivial from the outside: suppose I save and encrypt the conscious-generating-program. This looks like random noise from the outside, and will pass all randomness tests. Now I have another program with the stored key decrypt it and run it. From the outside, you might disregard the random-seed-looking-thingy and instead try to analyze the decryption program, thinking that’s where the magic is.
I’d love to hear about ideas to pin down the difference between Seeding and Decrypting in general, for arbitrary interpretations. It seems within reach, and like a good first step, since the two lie on roughly opposite ends of a spectrum of “cruciality” when the system breaks down into two or more modules.
Responses to your four final notes:
1. This is, as has been remarked in another comment, pretty much Dust theory. See also Moravec’s concise take on the topic, referenced in the Dust theory FAQ. Doing a search for it on LW might also prove helpful for previous discussions.
2. “that was already there”? What do you mean by this? Would you prefer to use the term ‘magical reality fluid’ instead of “exists”/”extant”/”real”/”there” etc, to mark your confusion about this? If you instead feel like you aren’t confused about these terms, please provide (a link to) a solution. You can find the problem statement in The Anthropic Trilemma.
3. Eliezer deals with this using average utilitarianism, depending on whether or not you agree with rescuability (see below).
4. GAZP vs GLUT talks about the difference between a cellphone transmitting information of consciousness vs the actual conscious brain on the other end, and generalizes it to arbitrary “interpretations”. That is, there are parts of the computation that are merely “interpreting”, informing you about consciousness and others that are “actually” instantiating. It may not be clear what exactly the crucial difference is yet, but I think it might be possible to rescue the difference, even if you can construct continuums to mess with the notion. This is of course deeply tied to 2.
----
It may seem that my takeaway from your post is mostly negative, this is not the case. I appreciate this post, it was very well organized despite tackling some very hairy issues, which made it easier to respond to. I do feel like LW could solve this somewhat satisfactorily, perhaps some people already have and don’t bother pointing the rest of us/are lost in the noise?
it is not as though rationality consisted of some singular epistemesis score that can be raised or lowered
I feel like this is fighting the hypothesis. As Garrabrant says:
Attention is a conserved resource, and attention that I give to Bob is being taken away from attention that could be directed toward GOOD ideas.
It doesn’t matter whether or not you think it is possible to track rationality through some singular epistemesis score. The question is: you have limited attentional resources and the problem OP outlined; “rationality” is probably complicated; what do you do anyway?
How you divvy them is the score. Or, to replace the symbol with the substance: if you’re in charge of divvying those resources, then your particular algorithm will decide what your underlings consider status/currency, and can backpropagate into their minds.
I think your “attentional resources” are just being Counterfactually Mugged here, so if you’re okay with that, you ought to be okay with some attention being diverted away from “real” ideas, if you’re reasonably confident in your construction of the counterfactual “Bob’s idea might HAVE BEEN good”.
This way of looking at it also says that tenure is a bad metaphor: your confidence in the counterfactual being true can change over time.
(If you then insist that this confidence in your counterfactual is also something that affects Bob, which it kinda does, then I’m afraid we’re encountering an instance of unfair problem class in the wild and I don’t know what to do)
As an aside, this makes me think: What happens when all consumers in the market are willing to get counterfactually mugged? Where I’m not able to return my defected phone because prediction markets said it would have worked? I suppose this is not very different from the concept of force majeure, only systematized.
It’s worth noting that David Friedman’s Price Theory clearly states this in the very first chapter, just three paragraphs down:
The second half of the assumption, that people tend to find the correct way to achieve their objectives, is called rationality. This term is somewhat deceptive, since it suggests that the way in which people find the correct way to achieve their objectives is by rational analysis—analyzing evidence, using formal logic to deduce conclusions from assumptions, and so forth. No such assumption about how people find the correct means to achieve their ends is necessary.
One can imagine a variety of other explanations for rational behavior. To take a trivial example, most of our objectives require that we eat occasionally, so as not to die of hunger (exception—if my objective is to be fertilizer). Whether or not people have deduced this fact by logical analysis, those who do not choose to eat are not around to have their behavior analyzed by economists. More generally, evolution may produce people (and other animals) who behave rationally without knowing why. The same result may be produced by a process of trial and error; if you walk to work every day, you may by experiment find the shortest route even if you do not know enough geometry to calculate it. Rationality in this sense does not necessarily require thought. In the final section of this chapter, I give two examples of things that have no minds and yet exhibit rationality.
I don’t think it counts as a standard textbook, but it is meant to be a textbook.
On the whole, I think it’s perfectly okay for economists to mostly ignore how the equilibrium is achieved, since like you pointed out, there are so many juicy results popping out from just the fact that they are achieved on average.
Also, I enjoyed the examples in your post!
effortless pattern-recognition is what machine learning can do today, while effortful attention, and explicit reasoning (which seems to be a subset of effortful attention) is generally beyond ML’s current capabilities.
Just to be clear, are you or aren’t you (or neither) saying that this is only a matter of scale?
It seems to me like you’re saying it could indeed only be a matter of scale, we’re just in the stage of figuring out what the right dimension to amp up is (“be coherent for longer”).
See, the problem is now that I’ve also internalized this (seemingly true) lesson, the +15% might double-boost my ass numbers.
But maybe if we accumulate enough lessons we can get increasingly close to the truth by adding these “higher order terms”?
I don’t think so—the error bars do not necessarily diminish. For example:
Ass number for drawing ability percentile: ~70%
Dunning Kruger correction: ~50%
Double-dip correction: ~65%
Did I do it right? I have no idea. Every step might have already been taken into account in the first asstimate. Every system-2 patch that we discover might have immediately patched system-1.
One (admittedly lazy) way out is to chuck all context-sensitive formal rules like ‘add/subtract X%’ and leave it entirely to system-1: play calibration games for skill-percentiles.
I hope we haven’t forgotten Stuck in the Middle With Bruce and Soares’ Have No Excuses, which starts with a quote from Bonds That Make Us Free.
I think one reason people end up using a minimax strategy is that it’s just easier to compute than EV-maximization.
But more importantly, it just feels like there’s no downsides—it’s free insurance!
If you want to have a convincing excuse however, you might actively impede your chances. (It might be possible to distance yourself from the excuse so the insurance is actually free, but I think this is unlikely/hard.)
If you’ve already hedged so hard that you’ve bet a lot against yourself, you might have sufficiently changed the payoffs to make losing rational, especially if you also add a penalty to Being Mediocre. This is why post like this are needed.
There’s a whole section on voting in the LDT For Economists page on Arbital. Also see the one for analytic philosophers, which has a few other angles on voting.
From what I can tell from your other comments on this page, you might already have internalized all the relevant intuitions, but it might be useful anyway. Superrationality is also discussed.
Sidenote: I’m a little surprised no one else mentioned it already. Somehow arbital posts by Eliezer aren’t considered as canon as the sequences, maybe it’s the structure (rather than just the content)?