Name that study design?

Design

You want to know if men out perform women in a class on physics. So you take the class at the start of semester and give it a test on physics, give a second test half way through and then record exam results at the end of the semester.

Challenge

Find a protocol for such a study against which you can check your design!

Ruling out the obvious

There are several protocols that you can check against which it simply will not fit. It is not a clinical trial, as there is no randomisation, subjects on entering are either male or female and I am afraid giving gender altering drugs would be unethical! It is not a case-control study because the outcome variable is not male or female and there is no matching. It is not a cross-sectional  study as it does have time as a related variable.

In other words all the accepted study protocols do not apply here. This feels like madness because the study as presented here is very close to the classic case for using the T-Test in statistics 101. What is wrong is that rather than using random allocation, the study design assumes random recruitment. That is from the pool of all likely male students to take the course, everyone is equally likely to choose or not choose the course and the same for all likely female students. This is of course somewhat dubious, that is why clinical trials like to have random allocation but when you can not alter a persons status then you cannot use random allocation.

Moving Forward

So how do you tackle it. Well I can tell you the way we came up with. Firstly we stated it was a comparative study. Secondly we found that actually such studies are quite common in bio-equivalence although unlike them our null was that the two groups were the same. So we could point to other studies with the same design.

Will the Journal understand?

I am not sure. We are being open with them. Yes there is a double factorial design that we could have done if we had also run a trial without intervention (equivalent to testing a group of students who did not sit the physics class) and if we did that there would have been some random allocation but only within groups. The problem being that actually finding one group of subjects we were looking for a fairly rare event and we might well have had to sacrifice power quite substantially if we did that. However the study does not tick any of the tidy tick boxes the journal has and journals are getting more and more tick box orientated.