Surveys are not experiments and Acty is explicitly talking about science with control groups, etc. E.g.
compare a nice prison, nasty prison, and average-kinda-prison control group, compare reoffending rates for ex-inmates of those prisons, maybe try an intervention where kids are deterred from committing crime by visiting nasty prison and seeing what it’s like versus kids who visit the nicer prison versus a control group who don’t visit a prison and then 10 years later see what percentage of each group ended up going to prison
A survey can be a reasonably designed experiment that simply gives us a weaker result than lots of other kinds of experiments.
There are many questions about humans that I would expect to be correlated with the noises humans make when given a few choices and asked to answer honestly. In many cases, that correlation is complicated or not very strong. Nonetheless, it’s not nothing, and might be worth doing, especially in the absence of a more-correlated test we can do given our technology, resources, and ethics.
What I had in mind was the difference between passive observation and actively influencing the lives of subjects. I would consider “surveys” to be observation and “experiments” to be or contain active interventions. Since the context of the discussion is kinda-sorta ethical, this difference is meaningful.
I am not sure where is this question coming from. I am not suggesting any particular studies or ways of conducting them.
Maybe it’s worth going back to the post from which this subthread originated. Acty wrote:
If we set a benchmark that would satisfy our values … then which policy is likely to better satisfy that benchmark...? But, of course, this is a factual question. We could resolve this by doing an experiment, maybe a survey of some kind.
First, Acty is mistaken in thinking that a survey will settle the question of which policy will actually satisfy the value benchmark. We’re talking about real consequences of a policy and you don’t find out what they are by conducting a public poll.
And second, if you do want to find the real consequences of a policy, you do need to run an intervention (aka an experiment) -- implement the policy in some limited fashion and see what happens.
Oh, I guess I misunderstood. I read it as “We should survey to determine whether terminal values differ (e.g. ‘The tradeoff is not worth it’) or whether factual beliefs differ (e.g. ‘There is no tradeoff’)”
But if we’re talking about seeing whether policies actually work as intended, then yes, probably that would involve some kind of intervention. Then again, that kind of thing is done all the time, and properly run, can be low-impact and extremely informative.
Surveys are not experiments and Acty is explicitly talking about science with control groups, etc. E.g.
According to every IRB I’ve been in contact with, they are. Here’s Cornell’s, for example.
I’m talking common sense, not IRB legalese.
According to the US Federal code, a home-made pipe bomb is a weapon of mass destruction.
A survey can be a reasonably designed experiment that simply gives us a weaker result than lots of other kinds of experiments.
There are many questions about humans that I would expect to be correlated with the noises humans make when given a few choices and asked to answer honestly. In many cases, that correlation is complicated or not very strong. Nonetheless, it’s not nothing, and might be worth doing, especially in the absence of a more-correlated test we can do given our technology, resources, and ethics.
What I had in mind was the difference between passive observation and actively influencing the lives of subjects. I would consider “surveys” to be observation and “experiments” to be or contain active interventions. Since the context of the discussion is kinda-sorta ethical, this difference is meaningful.
What intervention would you suggest to study the incidence of factual versus terminal-value disagreements in opposing sides of a policy decision?
I am not sure where is this question coming from. I am not suggesting any particular studies or ways of conducting them.
Maybe it’s worth going back to the post from which this subthread originated. Acty wrote:
First, Acty is mistaken in thinking that a survey will settle the question of which policy will actually satisfy the value benchmark. We’re talking about real consequences of a policy and you don’t find out what they are by conducting a public poll.
And second, if you do want to find the real consequences of a policy, you do need to run an intervention (aka an experiment) -- implement the policy in some limited fashion and see what happens.
Oh, I guess I misunderstood. I read it as “We should survey to determine whether terminal values differ (e.g. ‘The tradeoff is not worth it’) or whether factual beliefs differ (e.g. ‘There is no tradeoff’)”
But if we’re talking about seeing whether policies actually work as intended, then yes, probably that would involve some kind of intervention. Then again, that kind of thing is done all the time, and properly run, can be low-impact and extremely informative.
--
Yep :-) That’s why GlaDOS made an appearance in this thread :-D