“Complaining about your trade partners” at the level of making trade decisions is clearly absurd (a type error). “Complaining about your trade partners” at the level of calling them out, suggesting in an annoyed voice they behave differently, looking miffed, and otherwise attempting to impose costs on them (as object level actions inside of an ongoing trade/interaction which you both are agreeing to) is not. These are sometimes the mechanism via which things of value are traded or negotiations are made, and may be preferred by both parties to ceasing the interaction.
query
Karma: 294
A potential explanation I think is implicit in Ziz’s writing: the software for doing coordination within ourselves and externally is reused. External pressures can shape your software to be of a certain form; for instance, culture can write itself into people so that they find some ideas/patterns basically unthinkable.
So, one possibility is that fusion is indeed superior for self-coordination, but requires a software change that is difficult to make and can have significant costs to your ability to engage in treaties externally. Increased Mana allows you to offset some of the costs, but not all; some interactions are just pretty direct attempts to check that you’ve installed the appropriate mental malware.
Assuming the money transfer actually takes place, this sounds like a description of gains from trade; the “no pareto improvement” phrasing is that when actually making the trade, you lose the option of making the trade—which is of greater than or equal value than the trade itself if the offer never expires. One avenue to get actual Pareto improvements is then to create or extend opportunities for trade.
If the money transfer doesn’t actually take place: I agree that Kaldor-Hicks improvements and Pareto improvements shouldn’t be conflated. It takes social technology to turn one into the other.
I was definitely very confused when writing the part you quoted. I think the underlying thought was that the processes of writing humans and of writing AlphaZero are very non-random; i.e., even if there’s a random number generated in some sense somewhere as part of the process, there’s other things going on that are highly constraining the search space—and those processes are making use of “instrumental convergence” (stored resources, intelligence, putting the hard drives in safe locations.) Then I can understand your claim as “instrumental convergence may occur in guiding the search for/construction of an agent, but there’s no reason to believe that agent will then do instrumentally convergent things.” I think that’s not true in general, but it would take more words to defend.
I think you have a good point, in that the VNM utility theorem is often overused/abused: I don’t think it’s clear how to frame a potentially self modifying agent in reality as a preference ordering on lotteries, and even if you could in theory do so it might require such a granular set of outcomes as to make the resulting utility function not super interesting. (I’d very much appreciate arguments for taking VNM more seriously in this context; I’ve been pretty frustrated about this.)
That said, I think instrumental convergence is indeed a problem for real world searches; the things we’re classifying as “instrumentally convergent goals” are just “things that are helpful for a large class of problems.” It turns out there are ways to do better than random search in general, and that some these ways (the most general ones) are making use of the things we’re calling “instrumentally convergent goals”: AlphaGo Zero was not a (uniformish) random search on Go programs, and humans were not a (uniformish) random search on creatures. So I don’t think this particular line of thought should make you think potential AI is less of a problem.
On equal and opposite advice: many more people want you to surrender to them than it is good for you to surrender to, and the world is full of people who will demand your apology (and make it seem socially mandatory) for things you do not or should not regret. Tread carefully with practicing surrender around people who will take advantage of it. Sometimes the apparent social need to apologize is due to a value/culture mismatch with your social group, and practicing minimal or non-internalized apologies is actually a good survival mechanic.
If you are high power, high status, low agreeableness, or inescapable to a certain set of people, and you often find yourself issuing non-apologies, then this may indicate blindspots (you’re doing things you don’t want to do without realizing) or that your exercise of power/disregard of social reality is hurting others.
If you are already highly agreeable, you currently have low mana (https://www.lesserwrong.com/posts/39aKPedxHYEfusDWo/mana/), or you feel like you can’t escape a certain set of people who you need to apologize to, then internalizing apologies may be pushing in the wrong direction and overwriting your values with theirs.
Yeah; it’s not open/shut. I guess I’d say in the current phrasing, the “but Aumann’s Agreement Theorem shows that if two people disagree, at least one is doing something wrong.” is suggesting implications but not actually saying anything interesting—at least one of them is doing something wrong by this standard whether or not they agree. I think adding some more context to make people less suspicious they’re getting Eulered (http://slatestarcodex.com/2014/08/10/getting-eulered/) would be good.
I think this flaw is basically in the original article as well, though, so it’s also a struggle between accurately representing the source and adding editorial correction.
Nitpicks aside, want to say again that this is really great; thank you!
This is completely awesome, thanks for doing this. This is something I can imagine actually sending to semi-interested friends.
Direct messaging seems to be wonky at the moment, so I’ll put a suggested correction here: for 2.4, Aumann’s Agreement Theorem does not show that if two people disagree, at least one of them is doing something wrong. From wikipedia: ” if two people are genuine Bayesian rationalists with common priors, and if they each have common knowledge of their individual posterior probabilities, then their posteriors must be equal. ” This could fail at multiple steps, off the top of my head:
The humans might not be (mathematically pure) Bayesian rationalists (and in fact they’re not.)
The humans might not have common priors (even if they satisfied 1.)
The humans might not have common knowledge of their posterior probabilities; a human saying words is a signal, not direct knowledge, so them telling you their posterior probabilities may not do the trick (and they might not know them.)
You could say failing to satisfy 1-3 means that at least one of them is “doing something wrong”, but I think it’s a misleading stretch—failing to be normatively matched up to an arbitrary unobtainable mathematical structure is not what we usually call wrong. It stuck out to me as something that would put off readers with a bullshit detector, so I think it’d be worth fixing.
I found this extremely helpful; it motivated me to go read your entire blog history. I hope you write more; I think the “dark side” is a concept I had only the rough edges of, but one that I unknowingly desired to understand better (and had seen the hints of in other’s writing around the community.) I feel like the similarly named “dark arts” may have been an occluding red herring.
The more you shine the light of legibility, required defensibility and justification, public scrutiny of beliefs, social reality that people’s judgement might be flawed and they need to distrust themselves and have the virtue of changing their minds, the more those with low mana get their souls written into by social reality.
This is something I liked seeing written. This is a trade-off I don’t often see recognized by people around me; without recognizing this, communal pursuits can destroy and subsume the weaker people within them. As one who has had their soul partly killed in the past by the legibility requirements of people who were alien to me, I value this lesson and anything that helps people develop protective immunity.
Beautifully written; thank you for sharing this.
EDIT: On reflection, I want to tap out of this conversation. Thanks for the responses.
Does this cause any updating in decreasing the likelihood of nightmare scenarios like the one you described?
Effectively no. I understand that you’re aware of these risks and are able to list mitigating arguments, but the weight of those arguments does not resolve my worries. The things you’ve just said aren’t different in gestalt from what I’ve read from you.
To be potentially more helpful, here’s a few ways the arguments you just made fall flat for me:
I only incidentally mention rationality, such as when I speak of Rationality Dojo as a noun. I also generally do not talk of cognitive biases, and use other euphemistic language, such as referring to thinking errors, as in this article for Salon. So this gets at the point of watering down rationality.
Connectivity to the rationalist movement or “rationality” keyword isn’t necessary to immunize people against the ideas. You’re right that if you literally never use the word “bias” then it’s unlikely my nightmare imaginary conversational partner will have a strong triggered response against the word “bias”, but if they respond the same way to the phrase “thinking errors” or realize at some point that’s the concept I’m talking about, it’s the same pitfall. And in terms of catalyzing opposition, there is enough connectivity for motivated antagonists to make such connections and use every deviation from perfection as ammunition against even fully correct forms of good ideas.
For example, in this article, I specifically discuss research studies as a key way of validating truth claims. Recall that we are all suffering from a position of curse of knowledge on this point. How can we expect to teach people who do not know what science-based means without teaching it to them in the first place? Do you remember when you were at a stage when you did not know the value of scientific studies, and then came to learn about them as a useful way of validating evidence? This is what I’m doing in that article above. Hope this helps address some of the concerns about arguing from authority.
I can’t find any discussion in the linked article about why research is a key way of validating truth claims; did you link the correct article? I also don’t know if I understand what you’re trying to say; to reflect back, are you saying something like “People first need to be convinced that scientific studies are of value, before we can teach them why scientific studies are of value.” ? I … don’t know about that, but I won’t critique that position here since I may not be understanding.
(...) Hope this helps address the concerns about the writing style and the immunization of people to good ideas, since the readers of this content are specifically looking for this kind of writing style.
You seem to be saying that since the writing is of the form needed to get on Lifehack, and since in fact people are reading it on Lifehack, that they will then not suffer from any memetic immunization via the ideas. First, not all immunization is via negative reactions; many people think science is great, but have no idea how to do science. Such people can be in a sense immunized from learning to understand the process; their curiosity is already sated, and their decisions made. Second, as someone mentioned somewhere else on this comment stream, it’s not obvious that the Lifehack readers who end up looking at your article will end up liking or agreeing with your article.
You’re clearly getting some engagement, which is suggestive of positive responses, but what if the distribution of response is bimodal, with some readers liking it a little bit and some readers absolutely loathing it to the point of sharing their disgust with friends? Google searches reveal negative reactions to your materials as well. The net impact is not obviously positive.
I have not a clue whether this sort of marketing is a good idea. Let me be clear what I mean: I think there’s maybe a 30-40% chance that Gleb is having a net positive impact through these outreach efforts. I also think there’s maybe a 10-20% chance that he’s having a horrific long-term negative impact through these outreach efforts. Thus the whole thing makes me uncomfortable.
So here’s some of the concerns I see; I’ve gone to some effort to be fair to Gleb, and not assume anything about his thoughts or motivations:
By presenting these ideas in weakened forms (either by giving short or invalid argumentation, or putting it in venues or contexts with negative associations), he may be memetically immunizing people against the stronger forms of the ideas.
By teaching people using arguments from authority, he may be worsening the primary “sanity waterline” issues rather than improving them. The articles, materials, and comments I’ve seen make heavy use of language like “science-based”, “research-based” and “expert”. The people reading these articles in general have little or no skill at evaluating such claims, so that they effectively become arguments from authority. By rhetorically convincing them to adopt the techniques or thoughts, he’s spreading quite possibly helpful ideas, but reinforcing bad habits around accepting ideas.
Gleb’s writing style strikes me as very unauthentic feeling. Let me be clear I don’t mean to accuse him of anything negative; but I intuitively feel a very negative reaction to his writing. It triggers emotional signals in me of attempted deception and rhetorical tricks (whether or not this is his intent!) His writing risks associating “rationality” with such signals (should other people share my reactions) and again causing immunization, or even catalyzing opposition.
An illustration of the nightmare scenario from such an outreach effort would be that, 3 years from now when I attempt to talk to someone about biases, they respond by saying “Oh god don’t give me that ‘6 weird tips’ bullshit about ‘rational thinking’, and spare me your godawful rhetoric, gtfo.”
Like I said at the start, I don’t know which way it swings, but those are my thoughts and concerns. I imagine they’re not new concerns to Gleb. I still have these concerns after reading all of the mitigating argumentation he has offered so far, and I’m not sure of a good way to collect evidence about this besides running absurdly large long-term “consumer” studies.
I do imagine he plans to continue his efforts, and thus we’ll find out eventually how this turns out.
I disagree with your conclusion. Specifically, I disagree that
This is, literally, infinitely more parsimonious than the many worlds theory
You’re reasoning isn’t tight enough to have confidence answering questions like these. Specifically,
What do you mean by “simpler”?
Specifically how does physics “take into account the entire state of the universe”?
In order to actually say anything like the second that’s consistent with observations, I expect your physical laws become much less simple (re: Bell’s theorem implying non-locality, maybe, see Scott Aaronson’s blog.)
A basic error you’re making is equating simplicity of physical laws with small ontology. For instance, Google just told me there’s ~10^80 atoms in the observable universe (+- a few orders of magnitude), but this is no blow against the atomic theory of matter. You can formalize this interplay via “minimum message length” for a finite, fully described system; check Wikipedia for details.
Even though MWI implies a large ontology, it’s just a certain naive interpretation of our current local description of quantum mechanics. It’s hard to see how there could be a global description that is simpler, though I’d be interested to see one. (Here local/global mean “dependent on things nearby” vs “dependent on things far away”, which of course is assuming that ontology.)
With all kindness, the strength of your conclusion is far out of scope with the argument you’ve made. The linked paper looks like nonsense to me. I would recommend studying some basic textbook math and physics if you’re truly interested in this subject, although be prepared for a long and humbling journey.
Your question of “after finishing the supertask, what is the probability that 0 stays in place” doesn’t yet parse as a question in ZFC, because you haven’t specified what is meant by “after finishing the supertask”. You need to formalize this notion before we can say anything about it.
If you’re saying that there is no formalization you know of that makes sense in ZFC, then that’s fine, but that’s not necessarily a strike against ZFC unless you have a competitive alternative you’re offering. The problem could just be that it’s an ill-defined concept to begin with, or you just haven’t found a good formalization. Just because your brain says “that sounds like it make sense”, doesn’t mean it actually makes sense.
To show that ZFC is inconsistent, you would need to display a formal contradiction deduced from the ZFC axioms. “I can’t write down a formalization of this natural sounding concept” isn’t a formal contradiction; the failure is at the modeling step, not inside the logical calculus.
The model is that persistent reflexes interact with the environment to give black swans; singular events with extremely high legal consequence. To effectively avoid all of them preemptively requires training the stable reflexes, but it could be that “editing out” only a few 10 minute periods retroactively would still be enough (those few periods when reflexes and environment interact extremely negatively.) So I think the “very regular basis” claim isn’t substantiated.
That said, we cant actually retroactively edit anyways.
Being to vague to be wrong is bad. Especially when you want to speak in favor of science.
I agree, it’s good to pump against entropy with things that could be “Go Science!” cheers. I think the author’s topic is not too vague to discuss, but his argument isn’t strong or specific enough that you should leap to action based solely on it. I think it’s a fine thing to post to Discussion though; maybe this indicate we have ideal different standards for Discussion posts?
There no reason to say “well maybe the author meant to say X” when he didn’t say X.
Sure there is! Principle of charity, interpreting what they said in different language to motivate further discussion, rephrasing for your own understanding (and opening yourself to being corrected). Sometimes someone waves their hands in a direction, and you say “Aha, you mean...”
Above the author says “I think query worded it better”, which is the sort of thing I was aiming to accomplish.
Actually, this illustrates scientific thinking; the doctor forms a hypothesis based on observation and then experimentally tests that hypothesis.
Most interactions in the world are of the form “I have an idea of what will happen, so I do X, and later I get some evidence about how correct I was”. So, taking that as a binary categorization of scientific thinking is not so interesting, though I endorse promoting reflection on the fact that this is what is happening.
I think the author intends to point out some of the degrees of scientiificism by which things vary: how formal is the hypothesis, how formal is the evidence gathering, are analytical techniques being applied, etc. Normal interactions with doctors are low on scientificism in this sense, though they are heavily utilizing the output of previous scientificism to generate a judgement.
I think it would be good to separate the analysis into FGCA’s which are always fallacious, versus those that are only warning signs/rude. For instance, the fallacy of grey is indeed a fallacy, so using it as a counter-argument is a wrong move regardless of its generality.
However, it may in fact be that your opponent is a very clever arguer or that the evidence they present you has been highly filtered. Conversationally, using these as a counter-argument is considered rude (and rightly so), and the temptation to use them is often a good internal warning sign; however you don’t want to drop consideration of them from your mental calculus. For instance, perhaps you should be motivated after the conversation to investigate alternative evidence if you’re suspicious that the evidence presented to you was highly filtered.
Upfront note: I’ve enjoyed the circling I’ve done.
One reason to be cautious of circling: dropping group punishment norms for certain types of manipulation is extremely harmful. From my experience of circling (which is limited to a CFAR workshop), it provides plausible cover for very powerful status grabs under the aegis of “(just) expressing feelings and experiences”; I think the strongest usual defense against this is actually group disapproval. If someone is able to express such a status grab without receiving overt disapproval, they have essentially succeeded unless everyone in the group truly is superhuman at later correcting for this. If mounting the obvious self-defense against the status grab is taken off the table, then you may just lose painfully unless you can out-do them.
Normalizing circling (or NVC) too much could lead to externalities, where this happens outside of an actual circling context. This could lead to people losing face who normally wouldn’t, along with arms races that turn an X community into a circling-skill community.
If people are allowed to fish sell you (https://www.lesserwrong.com/posts/aFyWFwGWBsP5DZbHF/circling#E9dqjhm8Ca3HkFRMZ), and walking away loses you social status, and other people look on expectantly for your answer as you are fish sold instead of saying “Stop, they don’t want to buy your fish”, then depending on the type of fish and what escape routes to other social circles you have available, you may be in a hellishly difficult situation.
Note that I think this is bad regardless of your personal skill at resisting social pressure. The social incentive landscape changing leads to worse outcomes for everyone, even if you can individually get better outcomes for yourself by better learning to resist social pressure. That better outcome may be moving to a different community instead of being continually downgraded in status, which is a worse outcome than the community never having that bad incentive landscape to begin with.