I read the first post, which is excellent. Thanks for sharing.
siIver
Karma: 161
Thanks! So UDT is integrated. That’s good to hear.
Can someone briefly explain to me the difference between functional and updateless decision theory / where FDT performs better? That would be much appreciated. I have not yet read FDT because it does not mention UDT (I checked) and I want to understand why UDT needs “fixing” before I invest the time.
Ugh. When I heard about this first I naively thought it was great news. Now I see it’s a much harder question.
In my situation, it is the same: you can “determine” whether your dial is set to the first or second position by making a decision about whether to smoke.
No.
You can not. You can’t.
I’m struggling with this reply. I almost decided to stop trying to convince you. I will try one more time, but I need you to consider the possibility that you are wrong before you continue to the next paragraph. Consider the outside view: if you were right, Yudkowksy would be wrong, Anna would be wrong, everyone who read your post here and didn’t upvote this revolutionary, shocking insight would be wrong. Are you sufficiently more intelligent than any of them to be confident in your conclusion? I’m saying this only so you to consider the possibility, nothing more.
You do not have an impact. The reason why you believe otherwise is probably that in Newcomb’s problem, you do have an impact in an unintuitive way, and you generalized this without fully understanding why you have an impact in Newcomb’s problem. It is not because you can magically choose to live in a certain world despite no causal connection.
In Newcomb’s problem, the kind of person you are causally determines the contents of the opaque box, and it causally determines your decision to open them. You have the option to change the kind of person you are, i.e. decide you’ll one-box in Newcomb’s problem at any given moment before you are confronted with it (such as right now), therefore you causally determine how much money you will receive once you play it in the future. The intuitive argument “it is already decided, therefore it doesn’t matter what I do” is actually 100% correct. Your choice to one-box or two-box has no influence on the contents of the opaque box. But the fact that you are the kind of person who one-boxes does, and it happens to be that you (supposedly) can’t two-box without being the kind of person who two-boxes.
In the Smoking Lesion, in your alien scenario, this impact is not there. An independent source determines both the state of your box and your decision to smoke or not to smoke. A snapshot at all humans at any given time, with no forecasting ability, reveals exactly who will die of cancer and who won’t. If superomega comes from the sky and convinces everyone to stop smoking, the exact same people will die as before. If everyone stopped smoking immediately, the exact same people will die as before. In the future, the exact same people who would otherwise have died still die. People with the box on the wrong state who decide to stop smoking still die.
it will mean that everyone in the future had their dial set to the second position.
No it won’t. Nothing you wrote into the story indicates that you can change the box (in case of no forecaster). If you could, that would change everything (and it wouldn’t be the smoking lesion anymore).
I know it was the intention, but it doesn’t actually work the way you think.
The thing that causes the confusion is that you introduced an infallible decision maker into the brain that takes all autonomy away from the human (in case of there being no forecaster). This is basically a logical impossibility, which is why I just said “this is newcomb’s problem”. There has to be a forecaster. But okay, suppose not. I’ll show you why this does make a difference.
In Newcomb’s problem, you do in fact influence the contents of the opaque box. Your decision doesn’t, but the fact that you are the kind of person who makes this decision does. Your algorithm does. In the Alien Implant scenario case no forecaster, you don’t affect the state of your box at all.
If there was a forecaster, you could prevent people from dying of cancer by telling them about Timeless Decision Theory. Their choice not to smoke wouldn’t affect the state of their box, but the fact that you convince them would: the forecaster predicts that you prevent them from smoking, therefore they do not smoke, therefore it predicts they don’t smoke, therefore the box is on state 2.
If there was no forecaster, whether or not you smoke has no effect on your box, causally or otherwise. The state of their box is already determined; if you convinced them not to smoke, they would still get cancer and die and the box would be on state 1. Now this never happens in your scenario, which like I said is pretty close to being impossible, hence the confusion.
But it doesn’t matter! Not smoking means you live, smoking means you die!
No, it doesn’t. Suppose the decision maker was infallible. Everyone who smokes dies. Sooner or later people would all stop smoking. And this is where the scenario doesn’t work anymore. Because the number of people dying can’t go down. So either it must be impossible to convince people – in that case, why try? – or the decision maker becomes fallible, in which case your whole argument breaks apart. You don’t smoke and still die.
Think about this fact again: no forecaster means there is a fixed percentage of the population who has their box on state 1. If you are still not convinced, consider that “winning” by not smoking would then have to mean that someone else gets cancer instead, since you cannot change the number of people. Obviously, this is not what happens.
If there was a forecaster and everyone stopped smoking, no-one would die. If everyone one-boxes in Newcomb’s problem, everyone gets rich.
I’m afraid you misunderstand the difference between the Smoking Lesion and Newcomb’s problem. In the Smoking Lesion, if you are the kind of person who is affected by the thing which causes lung cancer and the desire to smoke, and you resist this desire, you still die of cancer. Your example is just Newcomb’s problem with an infallible forecaster, where if you don’t smoke you don’t die of cancer. This is an inherent difference. They are not the same.
I’m pretty happy with this article… though one of my concerns is that the section on how exactly AI could wipe out humanity was a bit short. It wants to cure cancer, it kills all humans, okay, but a reader might just think “well this is easy, tell it not to harm humans.” I’d have liked if the article had at least hinted at why the problem is more difficult.
Still, all in all, this could have been much worse.
I feel like I am repeating myself. Here is the chain of arguments
1) A normal person seeing this article and its upvote count will walk away having a very negative view of LessWrong (reasons in my original reply)
2) Making the valid points of this article is in no way dependent on the negative consequences of 1). You could do the same (in fact, a better job at the same) without offending anyone.
3) LessWrong can be a gateway for people to care about existential risk and AI safety.
4) AI safety is arguably the biggest problem in the world right now and extremely low efforts go into solving it, globally speaking.
5) Due to 4) getting people to care about AI safety is extrmely important. Due to that and 3), harming the reputation of LessWrong is really bad
6) Therefore, this article is awful, harmful, and should be resented by everyone.
No, I fully acknowledge that the post tries to do those things, see the second half of my reply. I argue that it fails at doing so but is harmful for our reputation etc.
It’s about a set of mannerisms which many people on LW have that are really bad. I don’t know what you mean by woke.
L. : While obviously being rational is good, LW as a community seems to be promoting elitism and entitlement.
s: Rationality can be scary that way. But it is about seeking truth, and the community does happen to consist of smart people. Denying that is false humility. Similarly, a lot of ideas many people support just happen to be false. It’s not our fault that our society got it wrong on so many issues. We’re just after truth.
L. : How does it serve truth to label people which aren’t smart as mentally ill?
s: That’s terrible, of course. But that’s not a flaw of rationality, nothing about rationality dictates “you have to be cruel to other people”. In fact if you think about this really hard you’ll see that rationality usually dictates being nice.
L: Then how come this post on LessWrong is the most upvoted thing of the last 20 submissions?
s: …
s: I can’t defend that.
L. : Oh, okay. So I’m right and Yudkowsky’s site does promote entitlement and sexism.
s: wait, sexism?
L. : Yeah. The last thing I saw from LW was two men talking about what a woman needs to do to fit the role they want her to have in society.
s: Okay, but that’s not Yudkowsky’s fault! He is not responsible for everyone on LW! The sequences don’t promote sexism-
L. : I heard HPMoR is sexist, too.
s: That’s not remotely true. It actually promotes feminism. Hermione is-
L. : I’m sorry, but I think I value the thoughts of other people who are more knowledgeable about sexism over yours. At least you condemn this article, but you still hang out on this site.
Scott Alexander has said that it’s society’s biggest mistake to turn away from intelligence (can’t find the article). Even minor increases of intelligence correlate meaningfully with all sorts of things (a negative correlation with crime being one of them afaik). Intelligence is the most powerful force in the universe. A few intelligence points on the people working on Friendly AI right now could determine the fate of our entire species. I want to make it extra clear that I think intelligence is ultra-important and almost universally good.
None of this excuses this article. None of it suggests that it’s somehow okay to label stupid people as mentally ill. Rationality is about winning, and this article is losing in every sense of the word. It won’t be good for the reputation of LW, it won’t be good for our agenda, and it won’t be good for the pursuit of truth. The only expected positive effect is making people who read it feel good. It essentially says “being intelligent is good. Being stupid is bad. Other people are stupid. They are the problem. We are better than them.” Which is largely true, but as helpful as making an IQ test, and emailing a friend saying “look here I am verifiable smarter than you and being smart is the most important thing in our society!”
Okay, but that’s not a content critique. I just said I think this is bad and went from there. If the article was actually making a strong case, well then it could still be bad for having an unnecessarily insulting and harmful framing that is bad for our cause, but it might be defend-able on other grounds. Maybe. We want to do both; to win and to pursue truth, and those aren’t the same thing. But I strongly feel the article doesn’t succeed on that front, either. Let’s take a look.
It’s great to make people more aware of bad mental habits and encourage better ones, as many people have done on LessWrong.
sure.
The way we deal with weak thinking is, however, like how people dealt with depression before the development of effective anti-depressants:
seems to be true.
“Stupidity,” like “depression,” is a sloppy “common-sense” word that we apply to different conditions, which may be caused by genetics (for instance, mutations in the M1 or M3 pathways, or two copies of Thr92Ala), deep subconscious conditioning (e.g., religion), general health issues (like not getting enough sleep), environment (ignorance, lack of reward for intelligent behavior), or bad habits of thought.
There is an implicit assumption here that being stupid requires some kind of explanation, but nothing at all in the article provides a reason of why this would be the case. Stupidity is not depression. The reason why it makes sense to label depression as a mental illness is (I assume) that it corresponds to an anomaly in the territory. Suppose we had a function, depressedness(human, time) which displayed how depressed each person on earth has been for, say, the past 10 years. I would expect to see weird behavior of that function, strange peaks over intervals of time on various people, many of whom don’t have unusually high values most of the time. This would suggest that it is something to be treated.
If you did the same for intelligence, I’d expect relatively low change on the time axis (aside from an increase at young age and a decrease in the case of actual mental illnesses) and some kind of mathematically typical distribution among the person axis ranging from 60 to dunno 170 or something. I feel really strange about having to make this argument, but this is really the crux of the problem here. The article doesn’t argue “here and here are stats suggesting that there are anomalies with this function, therefore there is a thing which we could sensibly describe as a mental illness” it just says “some people are dumb, here are some dumb things they do, let’s label that mental illness.” To sum the fallacy committed here up in one sentence, it talks about a thing without explaining why that thing should exist.
It is implied that people being ashamed of admitting to depression is a problem, and I infer that the intention is to make being stupid feel less bad by labeling their condition a “mental illness.” But it clearly fails in this regard, and is almost certainly more likely to do the opposite.. It’s sort of a Lose-Lose dynamic: it implies that there is some specific thing influencing a natural distribution of intelligence, some special condition that covers “stupid “people which explains why they are stupid – which likely isn’t the case, in that way having low IQ is probably worse than the article was meant to imply, since there is no special condition, you just got the lower end of the stick – while also being framed in such a way that it will make unintelligent people feel worse than before, not better.
And where is the reverse causation of believing in religion causing stupidity coming from? Postulating an idea like this ought to require evidence.
The article goes on to say that we should do something to make people smarter. I totally, completely, whole-heartedly agree. But saying high IQ is better than low IQ is something that can and has been done without all of the other stuff attached to it. And research in that direction is being done already. If you wanted to make a case for why we should have more of that, then you could do that so much more effectively without all the negativity attached to it.
Here are the accusations I am making. I accuse this article of not making a good case for anything that is both true and non-obvious, on top of being offensive and harmful for our reputation, and consequently our agenda. (Even if it is correct and there is an irregularity in the intelligence function, it doesn’t make a good case.) I believe that if arguments of the same quality were brought forth on any other topic, the article would be treated the same way most articles with weak content are treated: with indifference, few upvotes, and perhaps one or two comments pointing out some flaws in it (if Omega appeared before me, I would bet a lot of money on that theory with a pretty poor ratio). I’ll go as far as to accuse upvoting this as a failure of rationality. I agree with Pimgd on everything they said, but I feel like it is important to point out how awful this article is, rather than treating it as a simple point of disagreement. The fact that this has 12 upvotes is really, really really bad, and a symptom of a much larger problem.
This is not how you are being nice. This is not how you promote rationality. This is not how you win.
Changes in AI Safety Funding
I agree that it’s clear that you should one box – I’m more talking about justifying why one-boxing is in fact correct when it can’t logically influence whether there is money in the box. Initially I found this to be unnerving initially, but maybe I was the only one.
Reposting this from last week’s open thread because it seemed to get buried
Is Newcomb’s Paradox solved? I don’t mean from a decision standpoint, but the logical knot of “it is clearly, obviously better two one-box, and it is clearly, logically proven better to two-box”. I think I have a satisfying solution, but it might be old news.
But what makes you think that more complex story types allow many more possibilities?
Isn’t that an inherent property of complexity? A larger set of elements → a larger powerset of elements → more possibilities. In fact the size of the powerset grows at 2^x. I think a second game of thrones would be less groundbreaking, but doesn’t have to be worse… and the same goes for the 1000th GoT.
There seems to be a slowdown in more arty / complex stories this decade (than compared to the 90′s for example).
With film and television creation being more democratized than ever, I don’t see a reason why the creation of these type of films would slow down apart from the remaining stories requiring more complexity and skill to write than ever.
I don’t know as much as you about the industry. These sound worrisome.
I still think it is more likely that there is another reason (not that bold of an assumption) than that we really run out of complex things to write, because that just doesn’t seem to be true looking at how complexity works and how much seems to be doable just by tweaking those more complex pieces we have. Adaption is another great example.
But, I might be suffering from bias here, because I much prefer the world where I’m right to the one where I’m wrong.
Well, there is a provably finite possibility space for stories. You only have so many ways to arrange letters in a script. The question is whether it’s meaningful.
To use some completely made-up numbers, I think the current possibility space for movies produced by the bottom 80% of people with the current style may be 90% covered. The space for the top 2%, on the other hand, is probably covered for less than 0.1% (and I resisted putting in more zeros there).
To get more concrete, I’ll name some pieces (which I avoided doing in my initial post). Take Game of Thrones. It’s a huge deal – why? Well, because there isn’t really anything like it, But when you get rid of all the typical story tropes, like main characters with invulnerability, predictable plot progressions, a heroic minority lucking out against an evil majority, typical villains, etc etc, not only does the result get better, the possibility space actually widens. (I’m not saying scripts of this quality don’t exist, but it seems to be the only show where a great script and a huge budget and a competent team came together. There could be thousands of shows like this, and there is just one).
Or take the movie Being John Malkovich. Basically, there is one supernatural element placed in an otherwise scientifically operating world, and you have a bunch of character who act like normal humans, meaning largely selfish and emotionally driven, acting over that element. Just thinking about how much you could do following that formula opens up a large area in that seems to be largely untouched.
I think we’re shooting at Pluto over and over again while (for the most part) ignoring the rest of the universe. And it still works, because production quality and effects are still improving.
(edited)
I’d say no to both. I don’t think any genre has come meaningfully close to completion, though I don’t know classic of jazz very well.
Let’s talk film. If I take a random movie that I didn’t like, I find it very similar to others. If, however, I take one that I really like, I find that frustratingly few movies exist that are even similar.
I consider the possibility space to be a function of creativity/intelligence/competence (let’s call it skill) of writing, and one that grows faster-than-linearly. The space of medium-skill writing may be nearing completion (though I think this is arguable, too), but the space for more intelligent writing is vast and largely unexplored.
Just think of how many similarities most movies have, starting with the Hero’s Journey archetype. This need not be. My two favorite non-animated pieces in film both don’t have a main character.
I often ask myself the question of “is this really a thing” when it comes to high level concepts like this. I’m very unsure on Akrasia, and you make a decent enough argument. It could very well not actually be a thing (beyond referring to sub-things).
More importantly, though, even if it were a thing, I agree that the strategy you suggest of focusing on the smaller issues is likely the better one.