Q10: misspelling: “monogomous” should be “monogamous”
Q27: not clear what “as a community initiative” means. My actual impression is that LW2.0 is a project a few people are working on, but that most of “the community” has little visibility of it or input into it.
Q31: really needs an “insufficient data for meaningful answer” option.
Q32: fails to distinguish between “this is an incredibly important goal” and “this is the only goal that matters”. If I became 100% convinced that no version of LW was ever going to raise the general sanity waterline I would be disappointed (note: I am already somewhat disappointed) but I wouldn’t leave because I like being part of the community for other reasons too.
Q33: totally unclear what it’s asking, and it seems like there’s far too little information to give an answer that means anything. What is “this forum”? Specifically cafechesscourt (about which I know nothing), or is the picture just there to add colour? The question seems to presuppose that I can tell whether a forum would be worth participating in given only the identity of the person running it, which is bonkers.
Q33ff: there seems to be an underlying assumption here that what LW primarily needs is a technically better forum. I think the problems LW has faced have mostly been social rather than technical, and the main impact technical problems have had is that they have made it harder to address some social issues. (For instance, it was difficult to react appropriately to sustained malfeasance by a political fanatic because the LW server was (a) under the control of people who had little stake in its success and (b) running code that was painful to work with. But the actual problem was the political fanatic’s behaviour.)
Q36: shouldn’t forbid comments on unchosen options. Don’t you want to know why someone doesn’t care about something that sounds like it might matter?
Mental health questions: Might be worth distinguishing between “confident I would get a formal diagnosis if examined” and “think I probably have something similar”. (The depression question 51, if you pick either version of “yes”, opens up a question 52 asking whether you “still qualify for a depression diagnosis”. If that distinction doesn’t matter, maybe it should say something more like “still suffer from depression”. Incidentally, I don’t know why that question does this but the others don’t. I guess depression more often goes away completely than the others.)
Q67: for some of these, some people with sufficient natural aptitude might have attained a high level of skill without the extreme effort implied by the rightmost column. (E.g., I’m a professional mathematician; I never “practised calculus every day” so far as I can recall.) I am guessing that in such cases you still want that rightmost column ticked (and maybe I should consider effort on more difficult branches of mathematics related to calculus to qualify) but it’s not entirely clear. It’s also unclear how broadly to take some of the categories; e.g., if someone has put a lot of effort into musical performance, does that count as “painting/drawing/etc.”? (Depends whether “etc.” is meant to cover other artistic endeavours or other forms of visual art or what.)
Q71,72: I wonder whether you want one more question, asking to what extent people think research into friendly AI is important. (So then you have a three-level funnel: worth working on friendly AI? worth doing it mathematically? is MIRI doing that well? rather than a two-level funnel.)
Q92: is kinda meaningless without knowing what country the respondent is from. (At least in so far as their opinion on this is an actual opinion rather than a mere party label.)
Q100ff: I think the biggest barriers to feasibility of replacing humans in many occupations, even discounting cases where being nonhuman is as such an obstacle, will likely not be down to intelligence unless that’s broadened a great deal. Consider, e.g., opera singers or prostitutes. Machines trying to do these jobs will need to look and sound human not only because of people saying “ewwww, fake person” but also because the jobs intrinsically require specifically human-like performance.
Q108: crying out for a probability estimate rather than yes/no. I suspect most people signed up for cryonics think it probably won’t work.
Q115ff: unclear (and I think it matters a lot for many people’s opinions and feelings) what sort of modification is envisaged and how reliably it’s supposed to be known to work. E.g., I have a tweenaged child and if someone said “here is a thing we can do that will increase her intelligence” then, quite apart from the fact that it would need to be her decision as well as mine, I would be incredibly skeptical and greatly disinclined to try the treatment for fear of unforeseen adverse consequences. And even if it were somehow known to be safe, not to mess up people’s personalities, etc., it would feel like a big scary intervention in an existing person’s life, whereas an in utero treatment might feel quite different. I would guess that this sort of thing will only (at least in the foreseeable future) be feasible in utero, or perhaps even in vitro before implantation, but the question doesn’t make it clear what hypothetical we’re supposed to be entertaining. (And the reference to “your child” makes it hard for those who have children not to think in terms of modifications to the children they already have, however old.)
Q122: really needs a “don’t know / no strong opinion / it varies” option.
Q124: really needs probabilities or something. I can totally envisage scenarios where “the end of work” is a big win and others where it’s a huge disaster.
Q127: really wants to be accompanied by some sort of estimate of how probable the event is. If someone thinks there’s a 10^-6 chance of any of these things, it doesn’t matter much which they think most likely.
Thanks for your feedback. While I agree with you at many (even most) points, there are several considerations to keep in mind:
It is not possible for me to change the questions at this point in time, There are nearly 200 responses at this point and it would be completely unfair of me to force everyone to retake the survey so I can fix most of the offered feedback. As a consequence I can only take these as potential improvements for the next survey.
The questions in the AI Progress section are ripped directly from an associated study, I have no control over their content or methodology besides replicating them as accurately and faithfully as I can.
Some simplification of scenarios is necessary to make them fit into a <150 question survey. There are also real limits to how much effort I can expect from people in terms of engaging with a scenario and that is why certain things such as the probability that a genetic treatment will be safe aren’t included. Many things could use a probability but if I asked for one every time people would probably get frustrated with the survey and give up.
In the future we’re looking at changing platforms so that the survey can be offered in a ‘module’ format which allows it to be taken in chunks over a much longer period of time with more detail in individual sections.
Quite often it is useful for a survey to explicitly not include a neutral option. Picking between two things is mentally difficult and a neutral option offers a path of least resistance which people are more likely to click than put in the effort even if that doesn’t represent their true opinion.
At analysis time I have access to a respondents previous answers.
Our survey software prevents me from doing certain things which would be desirable.
Responding to certain specific questions:
Q33: Yes, CafeChesscourt. The question is “given nothing but the appearance of this forum” (CafeChesscourt put approximately zero effort into software, making it a useful control about how important software is to a forums success), would you use it if one of these “celebrity users” listed below were someone that ran it and used it regularly? The purpose of this question is to gauge how useful it is to get people who are popular to endorse a discussion forum. And while you might think it’s bonkers to choose a forum that way, empirically many people will show up to places that are often reviled like Tumblr if the right person asks them to.
Q33ff: I agree that is an assumption many people are making in regards to this issue, that assumption is not there in the question however as its purpose was me teasing at the idea that perhaps focusing on technical excellence is the wrong metric.
Q36: I believe this is a limitation of the software, though I’ll go ahead and double check since letting people write stuff in wouldn’t materially effect the survey results.
Q52: Depression does go away more often than the others. But more importantly on the 2016 survey we had somewhat horrifying rates of depression. It was difficult to distinguish if this was because everybody gets depression at least once and then ‘gets over it’ or because that many LessWrongers are actually depressed. This question lets us tell the difference.
Q67: Yes. As you can tell this is a hard question to ask people but I think it’s important enough to be worth asking even if imperfectly. In a future survey Calculus could probably be changed to “Calculus or ‘higher maths’”. The painting/etc option is meant to apply to visual arts. In the future it would probably be better to more accurately specify. In the grand scheme of things it will probably not spoil this years survey results to have the occasional musician pick it under the impression it counts. (I should also add music, thanks for the tip.)
Update (Fri Sep 15 13:46:59 PDT 2017): The issue in question 36 turned out to be fixable, thanks for the help!
For the avoidance of doubt, I appreciate your “several considerations” and in particular was not suggesting that you should hack the questions about mid-survey. And, er, I realise that I just posted a bunch of criticism without adding: thank you very much for doing the survey; I think it will be interesting and useful; the fact that I have some quibbles doesn’t make that any less true. So please consider that added :-).
As a single data point, on Q33 I attempted to answer as if the question meant something like “If all you knew about a new forum was that X was running the show, would you be likely to check it out for that reason?” on the grounds that that was the most non-bonkers interpretation I could give the question. If it was meant to be more like “If X was running the show, and the forum had no other merits, would X’s leadership be enough to make you use it and stick around despite its lack of other merits?” then my answer, at least, will not be informative. I suspect I’m not alone :-).
A few comments:
Q10: misspelling: “monogomous” should be “monogamous”
Q27: not clear what “as a community initiative” means. My actual impression is that LW2.0 is a project a few people are working on, but that most of “the community” has little visibility of it or input into it.
Q31: really needs an “insufficient data for meaningful answer” option.
Q32: fails to distinguish between “this is an incredibly important goal” and “this is the only goal that matters”. If I became 100% convinced that no version of LW was ever going to raise the general sanity waterline I would be disappointed (note: I am already somewhat disappointed) but I wouldn’t leave because I like being part of the community for other reasons too.
Q33: totally unclear what it’s asking, and it seems like there’s far too little information to give an answer that means anything. What is “this forum”? Specifically cafechesscourt (about which I know nothing), or is the picture just there to add colour? The question seems to presuppose that I can tell whether a forum would be worth participating in given only the identity of the person running it, which is bonkers.
Q33ff: there seems to be an underlying assumption here that what LW primarily needs is a technically better forum. I think the problems LW has faced have mostly been social rather than technical, and the main impact technical problems have had is that they have made it harder to address some social issues. (For instance, it was difficult to react appropriately to sustained malfeasance by a political fanatic because the LW server was (a) under the control of people who had little stake in its success and (b) running code that was painful to work with. But the actual problem was the political fanatic’s behaviour.)
Q36: shouldn’t forbid comments on unchosen options. Don’t you want to know why someone doesn’t care about something that sounds like it might matter?
Mental health questions: Might be worth distinguishing between “confident I would get a formal diagnosis if examined” and “think I probably have something similar”. (The depression question 51, if you pick either version of “yes”, opens up a question 52 asking whether you “still qualify for a depression diagnosis”. If that distinction doesn’t matter, maybe it should say something more like “still suffer from depression”. Incidentally, I don’t know why that question does this but the others don’t. I guess depression more often goes away completely than the others.)
Q67: for some of these, some people with sufficient natural aptitude might have attained a high level of skill without the extreme effort implied by the rightmost column. (E.g., I’m a professional mathematician; I never “practised calculus every day” so far as I can recall.) I am guessing that in such cases you still want that rightmost column ticked (and maybe I should consider effort on more difficult branches of mathematics related to calculus to qualify) but it’s not entirely clear. It’s also unclear how broadly to take some of the categories; e.g., if someone has put a lot of effort into musical performance, does that count as “painting/drawing/etc.”? (Depends whether “etc.” is meant to cover other artistic endeavours or other forms of visual art or what.)
Q71,72: I wonder whether you want one more question, asking to what extent people think research into friendly AI is important. (So then you have a three-level funnel: worth working on friendly AI? worth doing it mathematically? is MIRI doing that well? rather than a two-level funnel.)
Q92: is kinda meaningless without knowing what country the respondent is from. (At least in so far as their opinion on this is an actual opinion rather than a mere party label.)
Q100ff: I think the biggest barriers to feasibility of replacing humans in many occupations, even discounting cases where being nonhuman is as such an obstacle, will likely not be down to intelligence unless that’s broadened a great deal. Consider, e.g., opera singers or prostitutes. Machines trying to do these jobs will need to look and sound human not only because of people saying “ewwww, fake person” but also because the jobs intrinsically require specifically human-like performance.
Q108: crying out for a probability estimate rather than yes/no. I suspect most people signed up for cryonics think it probably won’t work.
Q115ff: unclear (and I think it matters a lot for many people’s opinions and feelings) what sort of modification is envisaged and how reliably it’s supposed to be known to work. E.g., I have a tweenaged child and if someone said “here is a thing we can do that will increase her intelligence” then, quite apart from the fact that it would need to be her decision as well as mine, I would be incredibly skeptical and greatly disinclined to try the treatment for fear of unforeseen adverse consequences. And even if it were somehow known to be safe, not to mess up people’s personalities, etc., it would feel like a big scary intervention in an existing person’s life, whereas an in utero treatment might feel quite different. I would guess that this sort of thing will only (at least in the foreseeable future) be feasible in utero, or perhaps even in vitro before implantation, but the question doesn’t make it clear what hypothetical we’re supposed to be entertaining. (And the reference to “your child” makes it hard for those who have children not to think in terms of modifications to the children they already have, however old.)
Q122: really needs a “don’t know / no strong opinion / it varies” option.
Q124: really needs probabilities or something. I can totally envisage scenarios where “the end of work” is a big win and others where it’s a huge disaster.
Q127: really wants to be accompanied by some sort of estimate of how probable the event is. If someone thinks there’s a 10^-6 chance of any of these things, it doesn’t matter much which they think most likely.
Thanks for your feedback. While I agree with you at many (even most) points, there are several considerations to keep in mind:
It is not possible for me to change the questions at this point in time, There are nearly 200 responses at this point and it would be completely unfair of me to force everyone to retake the survey so I can fix most of the offered feedback. As a consequence I can only take these as potential improvements for the next survey.
The questions in the AI Progress section are ripped directly from an associated study, I have no control over their content or methodology besides replicating them as accurately and faithfully as I can.
Some simplification of scenarios is necessary to make them fit into a <150 question survey. There are also real limits to how much effort I can expect from people in terms of engaging with a scenario and that is why certain things such as the probability that a genetic treatment will be safe aren’t included. Many things could use a probability but if I asked for one every time people would probably get frustrated with the survey and give up.
In the future we’re looking at changing platforms so that the survey can be offered in a ‘module’ format which allows it to be taken in chunks over a much longer period of time with more detail in individual sections.
Quite often it is useful for a survey to explicitly not include a neutral option. Picking between two things is mentally difficult and a neutral option offers a path of least resistance which people are more likely to click than put in the effort even if that doesn’t represent their true opinion.
At analysis time I have access to a respondents previous answers.
Our survey software prevents me from doing certain things which would be desirable.
Responding to certain specific questions:
Q33: Yes, CafeChesscourt. The question is “given nothing but the appearance of this forum” (CafeChesscourt put approximately zero effort into software, making it a useful control about how important software is to a forums success), would you use it if one of these “celebrity users” listed below were someone that ran it and used it regularly? The purpose of this question is to gauge how useful it is to get people who are popular to endorse a discussion forum. And while you might think it’s bonkers to choose a forum that way, empirically many people will show up to places that are often reviled like Tumblr if the right person asks them to.
Q33ff: I agree that is an assumption many people are making in regards to this issue, that assumption is not there in the question however as its purpose was me teasing at the idea that perhaps focusing on technical excellence is the wrong metric.
Q36: I believe this is a limitation of the software, though I’ll go ahead and double check since letting people write stuff in wouldn’t materially effect the survey results.
Q52: Depression does go away more often than the others. But more importantly on the 2016 survey we had somewhat horrifying rates of depression. It was difficult to distinguish if this was because everybody gets depression at least once and then ‘gets over it’ or because that many LessWrongers are actually depressed. This question lets us tell the difference.
Q67: Yes. As you can tell this is a hard question to ask people but I think it’s important enough to be worth asking even if imperfectly. In a future survey Calculus could probably be changed to “Calculus or ‘higher maths’”. The painting/etc option is meant to apply to visual arts. In the future it would probably be better to more accurately specify. In the grand scheme of things it will probably not spoil this years survey results to have the occasional musician pick it under the impression it counts. (I should also add music, thanks for the tip.)
Update (Fri Sep 15 13:46:59 PDT 2017): The issue in question 36 turned out to be fixable, thanks for the help!
For the avoidance of doubt, I appreciate your “several considerations” and in particular was not suggesting that you should hack the questions about mid-survey. And, er, I realise that I just posted a bunch of criticism without adding: thank you very much for doing the survey; I think it will be interesting and useful; the fact that I have some quibbles doesn’t make that any less true. So please consider that added :-).
As a single data point, on Q33 I attempted to answer as if the question meant something like “If all you knew about a new forum was that X was running the show, would you be likely to check it out for that reason?” on the grounds that that was the most non-bonkers interpretation I could give the question. If it was meant to be more like “If X was running the show, and the forum had no other merits, would X’s leadership be enough to make you use it and stick around despite its lack of other merits?” then my answer, at least, will not be informative. I suspect I’m not alone :-).