We did indeed user test with 3-4 people within our target group and changed the survey quite a bit in response to these. I definitely believe the “basic quality” to be met but I appreciate your reciprocation either way.
Response to your feedback, in order: The career options are multiple-choice, i.e. you may associate yourself with computer science and academia and not industry which is valuable information in this case. It means we need less combinations of occupational features. From our perspective, the number is easier to estimate while time estimates (both in the future and past) generally are less accurate. In a similar vein, it’s fine that LessWrong blog posts each count as one due to their inherently more in-depth nature. I’ll add the “pick the nearest one”. The ranking was expected to feel like that which is in accordance with our priors on the question. There are reasons for most questions’ features as well.
And thank you, I believe the same. Compared to how data-driven we believe we are, there is very little data from within the community, as far as I see it. And we should work harder to positively encourage more action in a similar vein.
After the fact, I realize that I might have replied too defensively to your feedback as a result of the tone so sorry for that!
I sincerely do thank you for all the points and we have updated the survey with a basis in this feedback, i.e. 1) career = short text answer, 2) knowledge level = which of these learning tasks have you completed and 4) they are now independent rankings.
Thank you for the valuable feedback!
We did indeed user test with 3-4 people within our target group and changed the survey quite a bit in response to these. I definitely believe the “basic quality” to be met but I appreciate your reciprocation either way.
Response to your feedback, in order: The career options are multiple-choice, i.e. you may associate yourself with computer science and academia and not industry which is valuable information in this case. It means we need less combinations of occupational features. From our perspective, the number is easier to estimate while time estimates (both in the future and past) generally are less accurate. In a similar vein, it’s fine that LessWrong blog posts each count as one due to their inherently more in-depth nature. I’ll add the “pick the nearest one”. The ranking was expected to feel like that which is in accordance with our priors on the question. There are reasons for most questions’ features as well.
And thank you, I believe the same. Compared to how data-driven we believe we are, there is very little data from within the community, as far as I see it. And we should work harder to positively encourage more action in a similar vein.
Cool that you did user testing! I’ll leave this this thread here.
After the fact, I realize that I might have replied too defensively to your feedback as a result of the tone so sorry for that!
I sincerely do thank you for all the points and we have updated the survey with a basis in this feedback, i.e. 1) career = short text answer, 2) knowledge level = which of these learning tasks have you completed and 4) they are now independent rankings.