There’s lots of statistical data already in the post about evidence that you will be glad you went. That wasn’t what Silas Barta asked, and frankly I’m not sure this thread is going to be productive given the way the opening question was framed.
How long would it take Anna to email the attendees and ask them to reply back about their current life status as compare to a year ago, so as to avoid proceeding with misleading evidence?
Edit in reply to the unmarked update to EY’s comment:
Like thomblake noted, the evidence being cited is not of the kind I asked for, which was why I framed my question with a link to Yvain’s lucid explanation of the problems of sorting out good retreats from bad. The exact evidence produced can be found just the same from (less useful) Christian brainwashing retreats, which is why I wasn’t impressed the last time around.
I do appreciate your efforts to apply the methods of rationality to your own endeavors.
Apparently the one-year followup is currently underway—a Minicamp attendee volunteered to do it.
This is pretty strong evidence of itself—people almost never volunteer for things and do them.
EDIT: OOPS! Anna said that an RBC attendee volunteered to do the RBC followup. RBC as you know was less successful than Minicamp, and we do not have someone doing the Minicamp followup yet.
I will remark that it’s more time-intensive than you seem to think—this is something that gets done after successfully hiring an executive assistant; we have a candidate but the hiring hasn’t yet occurred.
7b) Is there any evidence I’ll be glad I went that a Christian retreat could not produce just as easily?
Eliezer_Yudkowsky:
Apparently the one-year followup is currently underway—a Minicamp attendee volunteered to do it. This is pretty strong evidence of itself—people almost never volunteer for things and do them.
Yes, people usually don’t do that. On the other hand, it isn’t implausible that someone who just returned from a “Christian retreat” and who is “on fire for God” to “volunteer for things and do them”. SilasBarta isn’t merely asking for evidence that the camp provides benefits; he is asking for a reason to think it has benefits that exceed those that can be obtained at other kinds of events (specifically, a “Christian retreat”).
he is asking for a reason to think it has benefits that exceed those that can be obtained at other kinds of events
Or, rather, that exceeds that that can be so obtained. That is, SB’s7b relates to the relative quality of the reason for belief, not the relative quality of the benefits.
But you’re right that (for example) Christian retreats routinely get people to volunteer to do things and do them, so the simple fact of a Minicamp attendee doing so is not by itself strong evidence of a difference between the two events.
OTOH, there may well be sufficient differences between the two communities that the similarity of results is such evidence. That is, if event X1 gets result Y1 from a member of community Z1, while X2 gets Y2 from a member of Z2, the similarity of Y1 and Y2 given significant relevant differences between Z1 and Z2 suggests equally significant differences between X1 and X2. If Z2 is consistently more inclined to cooperate than Z1, and Y1/Y2 demonstrate willing cooperation, I conclude that X1 is more effective at inducing cooperation than X2.
(OTOOH, a lot depends on why Z2 cooperates more reliably. If it turns out that cooperation is primarily caused by the quality of Z2′s events, then that’s evidence against there being a significant difference between X1 and X2.)
Yes, after I said “After Minicamp you will be able to explain the math behind what you do”, thus answering the original question, whereupon I was directed to answer other questions instead.
Assuming this is true, do you have a good model for why this was the case?
It is certainly the case that those at RBC were exposed to more of the “rationality teaching” material than those at Minicamp, so if this is true then it should probably be worrying.
I think it was a selection effect: at Mini-Camp, the people who went were chosen from a broad pool, of anyone who could take a week off at the beginning of summer. But the only people who went to Mega-Camp were the types of people who could afford to take a whole summer off. So the Mega-Camp attendees were younger, more likely to be a student, less likely to have other things going on in their lives.
(I was a Mega-Camp attendee.)
Other potential reasons: it started to suck living & eating together in a tight, crowded space. It’s tolerable (and even fun!) for a week, but after a few weeks, privacy & space become an issue.
These are all good reasons why RBC would seem less awesome than Mini-Camp, but they aren’t actually good reasons why it should have been less effective at teaching people rationality. If anything, surely one would expect people who were younger and had less going on in their lives to benefit more from rationality training.
Basically, I agree with you that these are the reasons that Eliezer describes RBC as “less of a success”, but this just means that Silas is right, and the measure of “success” being used is “how awesome did everyone think it was”, not “how much rationality did we manage to teach”.
Yes, we have a good model for why this is the case, and it involves specific managers performing worse than others so the social cost of explaining our model is not zero.
We tried two experiments. The first one worked better, so we’re repeating that one instead of the second one.
I will remark that it’s more time-intensive than you seem to think—this is something that gets done after successfully hiring an executive assistant; we have a candidate but the hiring hasn’t yet occurred.
I’m open to the planning fallacy as anyone, but really, how long does it take to email everyone for their current life status?
I am not sure this thread is helpful, particularly given the way the opening question was framed.
I agree. But Silas is just doing his due diligence to ask that sort of question every time one of these things is mentioned, and surely that’s valuable to have around.
There’s lots of statistical data already in the post about evidence that you will be glad you went. That wasn’t what Silas Barta asked
I left out the clause “that a Christian brainwashing retreat could not produce just as easily” in my retelling, since I was just noting an additional constraint. I don’t think the sort of evidence in the post above actually satisfies that criterion.
There’s lots of statistical data already in the post about evidence that you will be glad you went. That wasn’t what Silas Barta asked, and frankly I’m not sure this thread is going to be productive given the way the opening question was framed.
What would be a better framing?
How long would it take Anna to email the attendees and ask them to reply back about their current life status as compare to a year ago, so as to avoid proceeding with misleading evidence?
Edit in reply to the unmarked update to EY’s comment:
Like thomblake noted, the evidence being cited is not of the kind I asked for, which was why I framed my question with a link to Yvain’s lucid explanation of the problems of sorting out good retreats from bad. The exact evidence produced can be found just the same from (less useful) Christian brainwashing retreats, which is why I wasn’t impressed the last time around.
I do appreciate your efforts to apply the methods of rationality to your own endeavors.
Apparently the one-year followup is currently underway—a Minicamp attendee volunteered to do it.
This is pretty strong evidence of itself—people almost never volunteer for things and do them.
EDIT: OOPS! Anna said that an RBC attendee volunteered to do the RBC followup. RBC as you know was less successful than Minicamp, and we do not have someone doing the Minicamp followup yet.
I will remark that it’s more time-intensive than you seem to think—this is something that gets done after successfully hiring an executive assistant; we have a candidate but the hiring hasn’t yet occurred.
It would be strong evidence if the volunteer had completed the “do them” part, certainly.
Fair enough. One cannot update on evidence one has not yet received.
SilasBarta:
Eliezer_Yudkowsky:
Yes, people usually don’t do that. On the other hand, it isn’t implausible that someone who just returned from a “Christian retreat” and who is “on fire for God” to “volunteer for things and do them”. SilasBarta isn’t merely asking for evidence that the camp provides benefits; he is asking for a reason to think it has benefits that exceed those that can be obtained at other kinds of events (specifically, a “Christian retreat”).
Or, rather, that exceeds that that can be so obtained. That is, SB’s7b relates to the relative quality of the reason for belief, not the relative quality of the benefits.
But you’re right that (for example) Christian retreats routinely get people to volunteer to do things and do them, so the simple fact of a Minicamp attendee doing so is not by itself strong evidence of a difference between the two events.
OTOH, there may well be sufficient differences between the two communities that the similarity of results is such evidence. That is, if event X1 gets result Y1 from a member of community Z1, while X2 gets Y2 from a member of Z2, the similarity of Y1 and Y2 given significant relevant differences between Z1 and Z2 suggests equally significant differences between X1 and X2. If Z2 is consistently more inclined to cooperate than Z1, and Y1/Y2 demonstrate willing cooperation, I conclude that X1 is more effective at inducing cooperation than X2.
(OTOOH, a lot depends on why Z2 cooperates more reliably. If it turns out that cooperation is primarily caused by the quality of Z2′s events, then that’s evidence against there being a significant difference between X1 and X2.)
Yes, after I said “After Minicamp you will be able to explain the math behind what you do”, thus answering the original question, whereupon I was directed to answer other questions instead.
Assuming this is true, do you have a good model for why this was the case?
It is certainly the case that those at RBC were exposed to more of the “rationality teaching” material than those at Minicamp, so if this is true then it should probably be worrying.
I think it was a selection effect: at Mini-Camp, the people who went were chosen from a broad pool, of anyone who could take a week off at the beginning of summer. But the only people who went to Mega-Camp were the types of people who could afford to take a whole summer off. So the Mega-Camp attendees were younger, more likely to be a student, less likely to have other things going on in their lives.
(I was a Mega-Camp attendee.)
Other potential reasons: it started to suck living & eating together in a tight, crowded space. It’s tolerable (and even fun!) for a week, but after a few weeks, privacy & space become an issue.
These are all good reasons why RBC would seem less awesome than Mini-Camp, but they aren’t actually good reasons why it should have been less effective at teaching people rationality. If anything, surely one would expect people who were younger and had less going on in their lives to benefit more from rationality training.
Basically, I agree with you that these are the reasons that Eliezer describes RBC as “less of a success”, but this just means that Silas is right, and the measure of “success” being used is “how awesome did everyone think it was”, not “how much rationality did we manage to teach”.
Agreed.
Yes, we have a good model for why this is the case, and it involves specific managers performing worse than others so the social cost of explaining our model is not zero.
We tried two experiments. The first one worked better, so we’re repeating that one instead of the second one.
I’m open to the planning fallacy as anyone, but really, how long does it take to email everyone for their current life status?
Try it and see :)
I agree. But Silas is just doing his due diligence to ask that sort of question every time one of these things is mentioned, and surely that’s valuable to have around.
I left out the clause “that a Christian brainwashing retreat could not produce just as easily” in my retelling, since I was just noting an additional constraint. I don’t think the sort of evidence in the post above actually satisfies that criterion.