Oh yes, that reminds me—I’ve always wondered if MoR was a waste of time or not in terms of community-building. So let’s divide the dataset into people who were referred to LW by MoR and people who weren’t...
Summary: they are younger, lower karma, lower karma per month participating (karma log-transformed or not), more likely to be students; but they have the same IQ (self-report & test) as the rest.
So, Eliezer is successfully corrupting the youth, but it’s not clear they are contributing very much yet.
R> lw <- read.csv("lw-survey/2012.csv")
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> hpmor <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
R> t.test(hpmor$IQ, hpmor$IQ)
Welch Two Sample t-test
data: hpmor$IQ and hpmor$IQ
t = 0.5444, df = 99.28, p-value = 0.5874
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.614 4.591
sample estimates:
mean of x mean of y
139.1 138.1
R> t.test(as.integer(as.character(hpmor$IQTest)), as.integer(as.character(hpmor$IQTest)))
Welch Two Sample t-test
data: as.integer(as.character(hpmor$IQTest)) and as.integer(as.character(hpmor$IQTest))
t = −0.0925, df = 264.8, p-value = 0.9264
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
−2.802 2.551
sample estimates:
mean of x mean of y
125.6 125.8
R> t.test(as.numeric(as.character(hpmor$Income)), as.numeric(as.character(hpmor$Income)))
Welch Two Sample t-test
data: as.numeric(as.character(hpmor$Income)) and as.numeric(as.character(hpmor$Income))
t = −4.341, df = 314.3, p-value = 1.917e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
−29762 −11197
sample estimates:
mean of x mean of y
33948 54427
R> t.test(hpmor$Age, hpmor$Age)
Welch Two Sample t-test
data: hpmor$Age and hpmor$Age
t = −7.033, df = 484.4, p-value = 6.93e-12
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
−5.318 −2.995
sample estimates:
mean of x mean of y
24.51 28.67
R> t.test(as.character(hpmor$WorkStatus) == “Student”, as.character(hpmor$WorkStatus) == “Student”)
Welch Two Sample t-test
data: as.character(hpmor$WorkStatus) == “Student” and as.character(hpmor$WorkStatus) == “Student”
t = 4.154, df = 389.8, p-value = 4.018e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.0791 0.2213
sample estimates:
mean of x mean of y
0.5224 0.3723
R> hpmortime ← hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime ← hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime ← hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime ← hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)
Welch Two Sample t-test
data: hpmortime and hpmortime
t = 1.05, df = 642.7, p-value = 0.2942
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
−4.257 14.036
sample estimates:
mean of x mean of y
17.69 12.80
R> hpmortime ← log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime ← hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime ← log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime ← hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)
Welch Two Sample t-test
data: hpmortime and hpmortime
t = 2.263, df = 396.9, p-value = 0.02416
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.03366 0.47878
sample estimates:
mean of x mean of y
1.1978 0.9415
Mean karma doesn’t seem like the relevant metric; that reflects something like the contributions of the typical MoR user, which seems less important to me than the contributions of the top MoR users. The top users in a community generally contribute disproportionately, so a more relevant metric might be the proportion of top users who were referred here from MoR.
The average user that sticks around might matter a lot, but people with low karma are probably less likely to stick around so they’ll have less of an impact (positive or negative) on the community. So maybe look at the distribution of karma, but among veteran users resp. veteran MoR users?
I imagine that when you divide karma by months in the community (while still restricting yourself to the top ten percent of absolute karma) the MoR contributors will look better. I’ll do it tonight if you don’t.
Looking at the four CFAR questions (described here), accuracy rates were:
74% OB folks (“Been here since it was started in the Overcoming Bias days”, n=253) 64% MoR folks (“Referred by Harry Potter and the Methods of Rationality”, n=253) 66% everyone else
So the original OB folks did better, but Methods influx is as good as the other sources of new readers. Breaking it down by question:
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> hpmor <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
Is this a typo? Or some text that was lost in the copy-paste?
Oh yes, that reminds me—I’ve always wondered if MoR was a waste of time or not in terms of community-building. So let’s divide the dataset into people who were referred to LW by MoR and people who weren’t...
Summary: they are younger, lower karma, lower karma per month participating (karma log-transformed or not), more likely to be students; but they have the same IQ (self-report & test) as the rest.
So, Eliezer is successfully corrupting the youth, but it’s not clear they are contributing very much yet.
Mean karma doesn’t seem like the relevant metric; that reflects something like the contributions of the typical MoR user, which seems less important to me than the contributions of the top MoR users. The top users in a community generally contribute disproportionately, so a more relevant metric might be the proportion of top users who were referred here from MoR.
The average user matters a lot, I think… But since you insist, here’s the top 10% of each category:
The top MoR referral user is somewhere around 10th place in the other group (which is 3.3x larger).
The average user that sticks around might matter a lot, but people with low karma are probably less likely to stick around so they’ll have less of an impact (positive or negative) on the community. So maybe look at the distribution of karma, but among veteran users resp. veteran MoR users?
What’s ‘veteran’? (And how many ways do you want to slice the data anyway...)
I imagine that when you divide karma by months in the community (while still restricting yourself to the top ten percent of absolute karma) the MoR contributors will look better. I’ll do it tonight if you don’t.
They do a bit better at the top; the sample size at “top 10%” is getting small enough that tests are losing power, though:
The interesting question might be whether people whose primary interest is HPMOR are understanding and using ideas about rationality from it.
Not sure how one would test that, aside from the CFAR questions which I don’t know how to use.
Looking at the four CFAR questions (described here), accuracy rates were:
74% OB folks (“Been here since it was started in the Overcoming Bias days”, n=253)
64% MoR folks (“Referred by Harry Potter and the Methods of Rationality”, n=253)
66% everyone else
So the original OB folks did better, but Methods influx is as good as the other sources of new readers. Breaking it down by question:
Question 1: disjunctive reasoning
OB: 52%
MoR: 42%
Other: 44%
Question 2: temporal discounting
OB: 94%
MoR: 89%
Other: 91%
Question 3: law of large numbers
OB: 92%
MoR: 85%
Other: 81%
Question 4: decoy effect
OB: 57%
MoR: 41%
Other: 49%
One possibility would be for Eliezer to ask people about it in his author’s notes when he updates HPMOR.
On the second reading, I realize that I’m asking about HPMOR and spreading rationality rather than HPMOR and community building.
Is this a typo? Or some text that was lost in the copy-paste?
Typo. I was operating on two variables,
hpmorandothers, but I guess a search-replace went awry...