True. For that to be an effective communication channel, there would need to be a control group. As for how to create that control group or run any sort of blind (let alone double-blind) testing… yeah, I have no idea. Definitely a problem.
ETA: By “I have no idea”, I mean “Let me find my five-minute clock and I’ll get back to you on this if anything comes up”.
So I thought for five minutes, then looked at what’s been done in other websites before.
The best I have is monthly surveys with randomized questions from a pool of stuff that matters for LessWrong (according to the current or then-current staff, I would presume) with a few community suggestions, and then possibly later implementation of a weighing algorithm for diminishing returns when multiple users with similar thread participation (e.g. two people that always post in the same thread) give similar feedback.
The second part is full of holes and horribly prone to “Death by Poking With Stick”, but an ideal implementation of this seems like it would get a lot more quality feedback than what little gets through low-bandwidth in-person conversations.
There are other, less practical (but possibly more accurate) alternatives, of course. Like picking random LW users every so often, appearing at their front door, giving them a brain-scan headset (e.g. an Emotiv Epoc), and having them wear the headset while being on LW so you can collect tons of data.
I’d stick with live feedback and simple surveys to begin with.
True. For that to be an effective communication channel, there would need to be a control group. As for how to create that control group or run any sort of blind (let alone double-blind) testing… yeah, I have no idea. Definitely a problem.
ETA: By “I have no idea”, I mean “Let me find my five-minute clock and I’ll get back to you on this if anything comes up”.
So I thought for five minutes, then looked at what’s been done in other websites before.
The best I have is monthly surveys with randomized questions from a pool of stuff that matters for LessWrong (according to the current or then-current staff, I would presume) with a few community suggestions, and then possibly later implementation of a weighing algorithm for diminishing returns when multiple users with similar thread participation (e.g. two people that always post in the same thread) give similar feedback.
The second part is full of holes and horribly prone to “Death by Poking With Stick”, but an ideal implementation of this seems like it would get a lot more quality feedback than what little gets through low-bandwidth in-person conversations.
There are other, less practical (but possibly more accurate) alternatives, of course. Like picking random LW users every so often, appearing at their front door, giving them a brain-scan headset (e.g. an Emotiv Epoc), and having them wear the headset while being on LW so you can collect tons of data.
I’d stick with live feedback and simple surveys to begin with.