it should be hard or impossible for participants to look up the answers during the study
I am unclear how you are going to enforce this in practice given that the study will be online and that you’re expecting people to spend at least 30 minutes on conversation, which implies a large enough reward that it is worth hunting down answers that can’t be found on the front page of Google. The only thing that comes to my mind is asking them to make predictions about the future. Related what will your policy be on participants looking up and/or sharing references relevant to steps in their reasoning? E.g. if one of the questions is about Trump being reelected will participants be allowed to visit 538 during step 1 and/or link their partner to it in step 4?
I didn’t have enough time to properly evaluate the statistics portion, but at first glance it looks ok. Nothing seems wrong with the significance tests beyond them being significance tests. IIRC BEST addresses my main issues with them, particularly being able to indicated the absence of an effect in a way that isn’t the case for mere non-significance, but I haven’t used it in forever and don’t have time to brush up on it at the moment.
I am unclear how you are going to enforce this in practice given that the study will be online and that you’re expecting people to spend at least 30 minutes on conversation, which implies a large enough reward that it is worth hunting down answers that can’t be found on the front page of Google. The only thing that comes to my mind is asking them to make predictions about the future. Related what will your policy be on participants looking up and/or sharing references relevant to steps in their reasoning? E.g. if one of the questions is about Trump being reelected will participants be allowed to visit 538 during step 1 and/or link their partner to it in step 4?
I didn’t have enough time to properly evaluate the statistics portion, but at first glance it looks ok. Nothing seems wrong with the significance tests beyond them being significance tests. IIRC BEST addresses my main issues with them, particularly being able to indicated the absence of an effect in a way that isn’t the case for mere non-significance, but I haven’t used it in forever and don’t have time to brush up on it at the moment.