I know; I know; I know. This is exactly what makes this topic so frustratingly difficult to explain, and so convenient to ignore.
The thing I am trying to say is that if a real monster would come to this community, sufficiently intelligent and saying the right keywords, we would spend all our energy inventing alternative explanations. That although in far mode we admit that the prior probability of a monster is nonzero (I think the base rate is somewhere around 1-4%), in near mode we would always treat it like zero, and any evidence would be explained away. We would congratulate ourselves for being nice, but in reality we are just scared to risk being wrong when we don’t have convincingly sounding verbal arguments on our side. (See Geek Social Fallacy #1, but instead of “unpleasant” imagine “hurting people, but only as much as is safe in given situation”.) The only way to notice the existence of the monster is probably if the monster decides to bite you personally in the foot. Then you will realize with horror that now all other people are going to invent alternative explanations why that probably didn’t happen, because they don’t want to risk being wrong in a way that would feel morally wrong to them.
I don’t have a good solution here. I am not saying that vigilantism is a good solution, because the only thing the monster needs to draw attention away is to accuse someone else of being a monster, and it is quite likely that the monster will sound more convincing. (Reversed stupidity is not intelligence.) Actually, I believe this happens rather frequently. Whenever there is some kind of a “league against monsters”, it is probably a safe bet that there is a monster somewhere at the top. (I am sure there is a TV Tropes page or two about this.)
So, we have a real danger here, but we have no good solution for it. Humans typically cope with such situations by pretending that the danger doesn’t exist. I wish we had a better solution.
I can believe that 1% − 4% of people have little or no empathy and possibly some malice in addition. However, I expect that the vast majority of them don’t have the intelligence/social skills/energy to become the sort of highly destructive person you describe below.
That’s right. The kind of person I described seems like combination of sociopathy + high intelligence + maybe something else. So it is much less than 1% of population.
(However, their potential ratio in rationalist community is probably greater than in general population, because our community already selects for high intelligence. So, if high intelligence would be the only additional factor—which I don’t know whether it’s true or not—it could again be 1-4% among the wannabe rationalists.)
The kind of person you described has extraordinary social skills as well as being highly (?) intelligent, so I think we’re relatively safe. :-)
I can hope that a people in a rationalist community would be better than average at eventually noticing they’re in a mind-warping confusion and charisma field, but I’m really hoping we don’t get tested on that one.
Returning to the original question (“Where are you right, while most others are wrong? Including people on LW!”), this is exactly the point where my opinion differs from the LW consensus.
I can hope that a people in a rationalist community would be better than average at eventually noticing they’re in a mind-warping confusion and charisma field
For a sufficiently high value of “eventually”, I agree. I am worried about what would happen until then.
I’m really hoping we don’t get tested on that one.
I’m hoping that this is not the best answer we have. :-(
To what extent is that sort of sociopath dependent on in-person contact?
Thinking about the problem for probably less than five minutes, it seems to me that the challenge is having enough people in the group who are resistant to charisma. Does CFAR or anyone else teach resistance to charisma?
Would noticing when one is confused and writing the details down help?
In addition to what I wrote in the other comment, a critical skill is to imagine the possibility that someone close to you may be manipulating you.
I am not saying that you must suspect all people all the time. But when strange things happen and you notice that you are confused, you should assign a nonzero value to this hypothesis. You should alieve that this is possible.
If I may use the fictional evidence here, the important thing for Rational!Harry is to realize that someone close to him may be Voldemort. Then it becomes a question of paying attention, good bookkeeping, gathering information, and perhaps making a clever experiment.
As long as Harry alieves that Voldemort is far away, he is likely to see all people around him as either NPCs or his party members. He doesn’t expect strategic activity from the NPCs, and he believes that his party members share the same values even if they have a few wrong beliefs which make cooperation difficult. (For example, he is frustrated that Minerva doesn’t trust him more, or that Dumbledore is okay with the idea of death, but he wouldn’t expect either of them trying to hurt him. And the list of nice people includes also Quirrell, which is the most awesome of them all.) He alieves that he lives in a relatively safe bubble, that Voldemort is somewhere outside of the bubble, and that if Voldemort tried to enter the bubble, it would be an obviously extraordinary event that he would notice. (Note: This is no longer true in the recent chapters.)
Harry also just doesn’t want to believe that Quirrell might be very bad news. (Does he consider the possibility that Quirrell is inimical, but not Voldemort?) Harry is very attached to the only person who can understand him reliably.
Does he consider the possibility that Quirrell is inimical, but not Voldemort?
This was unclear—I meant that Quirrell could be inimical without being Voldemort.
The idea of Voldemort not being a bad guy (without being dead)-- he’s reformed or maybe he’s developed other hobbies—would be an interesting shift. Voldemort as a gigantic force for good operating in secret would be the kind of shift I’d expect from HPMOR, but I don’t know of any evidence for it in the text.
Perhaps we should taboo “resistance to charisma” first. What specifically are we trying to resist?
Looking at an awesome person and thinking “this is an awesome person” is not harmful per se. Not even if the person uses some tricks to appear even more awesome than they are. Yeah, it would be nice to measure someone’s awesomeness properly, but that’s not the point. A sociopath may have some truly awesome traits, for example genuinely high intelligence.
So maybe the thing we are trying to resist is the halo effect. An awesome person tells me X, and I accept it as true because it would be emotionally painful to imagine that an awesome person would lie to me. The correct response is not to deny the awesomeness, but to realize that I still don’t have any evidence for X other than one person saying it is so. And that awesomeness alone is not expertise.
But I think there is more to a sociopath than mere charisma. Specifically, the ability to lie and harm people without providing any nonverbal cues that would probably betray a neurotypical person trying to do the same thing. (I suspect this is what makes the typical heuristics fail.)
Would noticing when one is confused and writing the details down help?
Yes, I believe so. If you already have a suspicion that something is wrong, you should start writing a diary. And a very important part would be, for every information you have, write down who said that to you. Don’t report your conclusions; report the raw data you have received. This will make it easier to see your notes later from a different angle, e.g. when you start suspecting someone you find perfectly credible today. Don’t write “X”, write “Joe said: X”, even if you perfectly believe him at the moment. If Joe says “A” and Jane says “B”, write “Joe said A. Jane said B” regardless of which one of them makes sense and which one doesn’t. If Joe says that Jane said X, write “Joe said that Jane said X”, not “Jane said X”.
Also, don’t edit the past. If you wrote “X” yesterday, but today Joe corrected you that he actually said “Y” yesterday but you have misunderstood it, don’t erase the “X”, but simply write today “Joe said he actually said Y yesterday”. Even if you are certain that you really made a mistake yesterday. When Joe gives you a promise, write it down. When there is a perfectly acceptable explanation later why the promise couldn’t be fulfilled, accept the explanation, but still record that for perfectly acceptable reasons the promise was not fulfilled. Too much misinformation is a red flag, even if there is always a perfect explanation for each case. (Either you are living in a very unlikely Everett branch, or your model is wrong.) Even if you accept an excuse, make a note of the fact that something had to be excused.
Generally, don’t let the words blind you from facts. Words are also a kind of facts (facts about human speech), but don’t mistake “X” for X.
I think gossip is generally a good thing, but only if you can follow these rules. When you learn about X, don’t write “X”, but write “my gossiping friend told me X”. It would be even better to gossip with friends who follow similar rules; who can make a distinction between “I have personally seen X” and “a completely trustworthy person said X and I was totally convinced”. But even when your friends don’t use this rule, you can still use it when speaking with them.
The problem is that this kind of journaling has a cost. It takes time; you have to protect the journal (the information it contains could harm not only you but also other people mentioned there); and you have to keep things in memory until you get to the journal. Maybe you could have some small device with you all day long where you would enter new data; and at home you would transfer the daily data to your computer and erase the device.
But maybe I’m overcomplicating things and the real skill is the ability to think about anyone you know and ask yourself a question “what if everything this person ever said to me (and to others) was a lie; what if the only thing they care about is more power or success, and they are merely using me as a tool for this purpose?” and check whether the alternative model explains the observed data better. Especially with the people you love, admire, of depend on. This is probably useful not only against literally sociopaths, but other kinds of manipulators, too.
But I think there is more to a sociopath than mere charisma. Specifically, the ability to lie and harm people without providing any nonverbal cues that would probably betray a neurotypical person trying to do the same thing. (I suspect this is what makes the typical heuristics fail.)
I don’t think “no nonverbal cues” is accurate. A psychopath shows no signs of emotional distress when he lies. On the other hand if they say something that should go along with a emotion if a normal person says it, you can detect that something doesn’t fit.
In the LW community however, there are a bunch of people with autism that show strange nonverbals and don’t show emotions when you would expect a neurotypical person to show emotions.
But maybe I’m overcomplicating things and the real skill is the ability to think about anyone you know and ask yourself a question “what if everything this person ever said to me (and to others) was a lie; what if the only thing they care about is more power or success, and they are merely using me as a tool for this purpose?”
I think that’s a strawman. Not having long-term goals is a feature of psychopaths. The don’t have a single purpose according to which they organize things. The are impulsive.
Not having long-term goals is a feature of psychopaths. The don’t have a single purpose according to which they organize things. The are impulsive.
That seems correct according to what I know (but I am not an expert). They are not like “I have to maximize the number of paperclips in the universe in the long term” but rather “I must produce some paperclips, soon”. Given sufficiently long time interval, they would probably fail at Marshmallow test.
Then I suspect the difference between a successful and an unsuccessful one is whether their impulses executed with their skills are compatible with what the society allows. If the impulse is “must get drunk and fight with people”, such person will sooner or later end in prison. If the impulse is “must lie to people and steal from them”, with some luck and skill, such person could become rich, if they can recognize situations where it is safe to lie and steal. But I’m speculating here.
Rather than thinking “I must steal” the impulse is more likely to be “I want to have X” and a lack of inhibition for stealing.
Psychopath usually don’t optimize for being evil.
Are you suggesting journaling about all your interactions where someone gives you information? That does sound exhausting and unnecessary. It might make sense to do for short periods for memory training.
Another possibility would be to record all your interactions—this isn’t legal in all jurisdictions unless you get permission from the other people being recorded, but I don’t think you’re likely to be caught if you’re just using the information for yourself.
Journaling when you have reason to suspicious of someone is another matter, and becoming miserable and confusing for no obvious reason is grounds for suspicion. (The children of such manipulators are up against a much more serious problem.)
It does seem to me that this isn’t exactly an individual problem if what you need is group resistance to extremely skilled manipulators.
Ironically, now I will be the one complaining that this definition of a “sociopath” seems to include too many people to be technically correct. (Not every top manager is a sociopath. And many sociopaths don’t make it into corporate positions of power.)
I agree that making detailed journals is probably not practical in real life. Maybe some mental habits would make it easier. For example, you could practice the habit of remembering the source of information, at least until you get home to write your diary. You could start with shorter time intervals; have a training session where people will tell you some information, and at the end you have an exam where you have to write an answer to the question and the name of the person who told you that.
If keeping the diary itself turns out to be good for a rationalist, this additional skill of remembering sources could be relatively easier, and then you will have the records you can examine later.
the challenge is having enough people in the group who are resistant to charisma.
Since we are talking about LW, let me point out that charisma in meatspace is much MUCH more effective than charisma on the ’net, especially in almost-purely-text forums.
Ex-cult members seem to have fairly general antibodies vs “charisma.” Perhaps studying cults without being directly involved might help a little as well, it would be a shame if there was no substitute for a “school of hard knocks” that actual cult membership would be.
Incidentally, cults are a bit of a hobby of mine :).
Whenever there is some kind of a “league against monsters”, it is probably a safe bet that there is a monster somewhere at the top. (I am sure there is a TV Tropes page or two about this.)
I know; I know; I know. This is exactly what makes this topic so frustratingly difficult to explain, and so convenient to ignore.
The thing I am trying to say is that if a real monster would come to this community, sufficiently intelligent and saying the right keywords, we would spend all our energy inventing alternative explanations. That although in far mode we admit that the prior probability of a monster is nonzero (I think the base rate is somewhere around 1-4%), in near mode we would always treat it like zero, and any evidence would be explained away. We would congratulate ourselves for being nice, but in reality we are just scared to risk being wrong when we don’t have convincingly sounding verbal arguments on our side. (See Geek Social Fallacy #1, but instead of “unpleasant” imagine “hurting people, but only as much as is safe in given situation”.) The only way to notice the existence of the monster is probably if the monster decides to bite you personally in the foot. Then you will realize with horror that now all other people are going to invent alternative explanations why that probably didn’t happen, because they don’t want to risk being wrong in a way that would feel morally wrong to them.
I don’t have a good solution here. I am not saying that vigilantism is a good solution, because the only thing the monster needs to draw attention away is to accuse someone else of being a monster, and it is quite likely that the monster will sound more convincing. (Reversed stupidity is not intelligence.) Actually, I believe this happens rather frequently. Whenever there is some kind of a “league against monsters”, it is probably a safe bet that there is a monster somewhere at the top. (I am sure there is a TV Tropes page or two about this.)
So, we have a real danger here, but we have no good solution for it. Humans typically cope with such situations by pretending that the danger doesn’t exist. I wish we had a better solution.
I can believe that 1% − 4% of people have little or no empathy and possibly some malice in addition. However, I expect that the vast majority of them don’t have the intelligence/social skills/energy to become the sort of highly destructive person you describe below.
That’s right. The kind of person I described seems like combination of sociopathy + high intelligence + maybe something else. So it is much less than 1% of population.
(However, their potential ratio in rationalist community is probably greater than in general population, because our community already selects for high intelligence. So, if high intelligence would be the only additional factor—which I don’t know whether it’s true or not—it could again be 1-4% among the wannabe rationalists.)
I would describe that person as a charismatic manipulator. I don’t think it requires being a sociopath, though being one helps.
The kind of person you described has extraordinary social skills as well as being highly (?) intelligent, so I think we’re relatively safe. :-)
I can hope that a people in a rationalist community would be better than average at eventually noticing they’re in a mind-warping confusion and charisma field, but I’m really hoping we don’t get tested on that one.
Returning to the original question (“Where are you right, while most others are wrong? Including people on LW!”), this is exactly the point where my opinion differs from the LW consensus.
For a sufficiently high value of “eventually”, I agree. I am worried about what would happen until then.
I’m hoping that this is not the best answer we have. :-(
To what extent is that sort of sociopath dependent on in-person contact?
Thinking about the problem for probably less than five minutes, it seems to me that the challenge is having enough people in the group who are resistant to charisma. Does CFAR or anyone else teach resistance to charisma?
Would noticing when one is confused and writing the details down help?
In addition to what I wrote in the other comment, a critical skill is to imagine the possibility that someone close to you may be manipulating you.
I am not saying that you must suspect all people all the time. But when strange things happen and you notice that you are confused, you should assign a nonzero value to this hypothesis. You should alieve that this is possible.
If I may use the fictional evidence here, the important thing for Rational!Harry is to realize that someone close to him may be Voldemort. Then it becomes a question of paying attention, good bookkeeping, gathering information, and perhaps making a clever experiment.
As long as Harry alieves that Voldemort is far away, he is likely to see all people around him as either NPCs or his party members. He doesn’t expect strategic activity from the NPCs, and he believes that his party members share the same values even if they have a few wrong beliefs which make cooperation difficult. (For example, he is frustrated that Minerva doesn’t trust him more, or that Dumbledore is okay with the idea of death, but he wouldn’t expect either of them trying to hurt him. And the list of nice people includes also Quirrell, which is the most awesome of them all.) He alieves that he lives in a relatively safe bubble, that Voldemort is somewhere outside of the bubble, and that if Voldemort tried to enter the bubble, it would be an obviously extraordinary event that he would notice. (Note: This is no longer true in the recent chapters.)
Harry also just doesn’t want to believe that Quirrell might be very bad news. (Does he consider the possibility that Quirrell is inimical, but not Voldemort?) Harry is very attached to the only person who can understand him reliably.
This was unclear—I meant that Quirrell could be inimical without being Voldemort.
The idea of Voldemort not being a bad guy (without being dead)-- he’s reformed or maybe he’s developed other hobbies—would be an interesting shift. Voldemort as a gigantic force for good operating in secret would be the kind of shift I’d expect from HPMOR, but I don’t know of any evidence for it in the text.
Perhaps we should taboo “resistance to charisma” first. What specifically are we trying to resist?
Looking at an awesome person and thinking “this is an awesome person” is not harmful per se. Not even if the person uses some tricks to appear even more awesome than they are. Yeah, it would be nice to measure someone’s awesomeness properly, but that’s not the point. A sociopath may have some truly awesome traits, for example genuinely high intelligence.
So maybe the thing we are trying to resist is the halo effect. An awesome person tells me X, and I accept it as true because it would be emotionally painful to imagine that an awesome person would lie to me. The correct response is not to deny the awesomeness, but to realize that I still don’t have any evidence for X other than one person saying it is so. And that awesomeness alone is not expertise.
But I think there is more to a sociopath than mere charisma. Specifically, the ability to lie and harm people without providing any nonverbal cues that would probably betray a neurotypical person trying to do the same thing. (I suspect this is what makes the typical heuristics fail.)
Yes, I believe so. If you already have a suspicion that something is wrong, you should start writing a diary. And a very important part would be, for every information you have, write down who said that to you. Don’t report your conclusions; report the raw data you have received. This will make it easier to see your notes later from a different angle, e.g. when you start suspecting someone you find perfectly credible today. Don’t write “X”, write “Joe said: X”, even if you perfectly believe him at the moment. If Joe says “A” and Jane says “B”, write “Joe said A. Jane said B” regardless of which one of them makes sense and which one doesn’t. If Joe says that Jane said X, write “Joe said that Jane said X”, not “Jane said X”.
Also, don’t edit the past. If you wrote “X” yesterday, but today Joe corrected you that he actually said “Y” yesterday but you have misunderstood it, don’t erase the “X”, but simply write today “Joe said he actually said Y yesterday”. Even if you are certain that you really made a mistake yesterday. When Joe gives you a promise, write it down. When there is a perfectly acceptable explanation later why the promise couldn’t be fulfilled, accept the explanation, but still record that for perfectly acceptable reasons the promise was not fulfilled. Too much misinformation is a red flag, even if there is always a perfect explanation for each case. (Either you are living in a very unlikely Everett branch, or your model is wrong.) Even if you accept an excuse, make a note of the fact that something had to be excused.
Generally, don’t let the words blind you from facts. Words are also a kind of facts (facts about human speech), but don’t mistake “X” for X.
I think gossip is generally a good thing, but only if you can follow these rules. When you learn about X, don’t write “X”, but write “my gossiping friend told me X”. It would be even better to gossip with friends who follow similar rules; who can make a distinction between “I have personally seen X” and “a completely trustworthy person said X and I was totally convinced”. But even when your friends don’t use this rule, you can still use it when speaking with them.
The problem is that this kind of journaling has a cost. It takes time; you have to protect the journal (the information it contains could harm not only you but also other people mentioned there); and you have to keep things in memory until you get to the journal. Maybe you could have some small device with you all day long where you would enter new data; and at home you would transfer the daily data to your computer and erase the device.
But maybe I’m overcomplicating things and the real skill is the ability to think about anyone you know and ask yourself a question “what if everything this person ever said to me (and to others) was a lie; what if the only thing they care about is more power or success, and they are merely using me as a tool for this purpose?” and check whether the alternative model explains the observed data better. Especially with the people you love, admire, of depend on. This is probably useful not only against literally sociopaths, but other kinds of manipulators, too.
I don’t think “no nonverbal cues” is accurate. A psychopath shows no signs of emotional distress when he lies. On the other hand if they say something that should go along with a emotion if a normal person says it, you can detect that something doesn’t fit.
In the LW community however, there are a bunch of people with autism that show strange nonverbals and don’t show emotions when you would expect a neurotypical person to show emotions.
I think that’s a strawman. Not having long-term goals is a feature of psychopaths. The don’t have a single purpose according to which they organize things. The are impulsive.
That seems correct according to what I know (but I am not an expert). They are not like “I have to maximize the number of paperclips in the universe in the long term” but rather “I must produce some paperclips, soon”. Given sufficiently long time interval, they would probably fail at Marshmallow test.
Then I suspect the difference between a successful and an unsuccessful one is whether their impulses executed with their skills are compatible with what the society allows. If the impulse is “must get drunk and fight with people”, such person will sooner or later end in prison. If the impulse is “must lie to people and steal from them”, with some luck and skill, such person could become rich, if they can recognize situations where it is safe to lie and steal. But I’m speculating here.
Human behavior is more complex than that.
Rather than thinking “I must steal” the impulse is more likely to be “I want to have X” and a lack of inhibition for stealing. Psychopath usually don’t optimize for being evil.
Are you suggesting journaling about all your interactions where someone gives you information? That does sound exhausting and unnecessary. It might make sense to do for short periods for memory training.
Another possibility would be to record all your interactions—this isn’t legal in all jurisdictions unless you get permission from the other people being recorded, but I don’t think you’re likely to be caught if you’re just using the information for yourself.
Journaling when you have reason to suspicious of someone is another matter, and becoming miserable and confusing for no obvious reason is grounds for suspicion. (The children of such manipulators are up against a much more serious problem.)
It does seem to me that this isn’t exactly an individual problem if what you need is group resistance to extremely skilled manipulators.
http://www.ribbonfarm.com/the-gervais-principle/-- some detailed analysis of sociopathy in offices.
Ironically, now I will be the one complaining that this definition of a “sociopath” seems to include too many people to be technically correct. (Not every top manager is a sociopath. And many sociopaths don’t make it into corporate positions of power.)
I agree that making detailed journals is probably not practical in real life. Maybe some mental habits would make it easier. For example, you could practice the habit of remembering the source of information, at least until you get home to write your diary. You could start with shorter time intervals; have a training session where people will tell you some information, and at the end you have an exam where you have to write an answer to the question and the name of the person who told you that.
If keeping the diary itself turns out to be good for a rationalist, this additional skill of remembering sources could be relatively easier, and then you will have the records you can examine later.
Since we are talking about LW, let me point out that charisma in meatspace is much MUCH more effective than charisma on the ’net, especially in almost-purely-text forums.
Well, consider who started CFAR (and LW for that matter) and how he managed to accomplish most of what he has.
Ex-cult members seem to have fairly general antibodies vs “charisma.” Perhaps studying cults without being directly involved might help a little as well, it would be a shame if there was no substitute for a “school of hard knocks” that actual cult membership would be.
Incidentally, cults are a bit of a hobby of mine :).
https://allthetropes.orain.org/wiki/Hired_to_Hunt_Yourself