Click this link for Part 1 of this series on New SAT Math Problem Solving and Data Analysis.
Subjects and Treatments
This is not an official title but the name I’m giving to questions that deal with studies trying to determine cause and effect. I’m guessing that sounds pretty vague, so here is a practice question.
A high school track coach has a new training regimen in which runners are supposed to exercise twice a week riding a stationary bike for one hour instead of doing a one-hour run twice a week. Her theory is that, by biking, students will not overly exert their running muscles but will still exercise their cardiovascular system. To test this theory out she had her varsity athletes (the faster runners) incorporate the biking regimen and the junior varsity athletes (the slower runners) incorporate usual training. After three weeks, the times of her varsity athletes on a 3-mile course decreased by an average of 1-minute, whereas her junior varsity athletes decreased their time on the same 3-mile course by approximately 30 seconds.
Which of the following is an appropriate conclusion?
A) The exercise bike regimen led to the reduction of the varsity runners’ time.
B) The exercise bike regimen would have helped the junior varsity team become faster.
C) No conclusion about cause and effect can be drawn because there might be fundamental differences between the way that varsity athletes respond to training in general and the way that junior varsity athletes respond.
D) No conclusion about cause and effect can be drawn because junior varsity athletes might have decreased their speed on the 3-mile course by more than 30 seconds had they completed the biking regimen.
Before we answer this question, let’s talk about randomization. The idea of randomization is the essence, the beating heart, of determining cause and effect. It helps us more reliably answer the question of whether a certain form of treatment cause a predictable outcome in subjects.
Randomization can happen at two levels. First off, when researchers select from the population in general, they have to make sure that they are not unknowingly selecting a certain type of person. Say, for instance, that I want to know what percent of Americans use Instagram. If I walk on a college campus and ask students there, I’m not taking a randomized sample of Americans (think how different my response rate would be if I decide to poll the audience at a Rolling Stones concert).
On the other hand, if I went to a city phone directory and dropped a quarter on the page, choosing the name that the center of the quarter was closest to, I would be doing a much better job of randomizing (though, one would rightly argue, I’d still be skewing to an older age-group, assuming that most young people have only cell phones, which aren’t listed in city directories). For the sake of argument, let’s say our phone directory method is able to randomly choose for all ages.
After throwing the quarter a total of a hundred times on randomly selected pages (we wouldn’t want only people whose names begin with ‘C’, because they might share some common trait), we will have 100 subjects. If we were to ask them about their Instagram use, our findings would far more likely skew with the general population. Therefore, this method would allow us to make generalizations about the population at large.
New SAT Math Treatment
However, when dealing with cause and effect in a study, or what the SAT calls a treatment, researchers need to ensure that they randomly select amongst the participants. Imagine that we wanted to test the effects on the immune system of a new caffeinated beverage. If researchers were to break our 100 subjects into under-40 and over-40, the results would not be reliable. First off, young people are known to generally have stronger immune systems. Therefore, once we have randomly selected a group for a study, we need to further ensure that, once in the study, researchers randomly break the subjects into two groups. In this case, those who drink the newfangled beverage and those who must make do with a placebo, or beverage that is not caffeinated.
At this point, we are likely to have a group that is both representative of the overall population and will allow us to draw reliable conclusions about cause and effect.
Another scenario, and this will help us segue to the practice question above, are treatments/trials in which the subjects are not randomly chosen. For instance, in the question about the runners, clearly they are not representative of the population as a whole (I’m sure many people would never dare peel themselves of their couches to something as daft as run three miles).
Nonetheless, we can still determining cause and effect from a non-representative population (in this case runners) as long as those runners are randomly broken into two groups, exercise bike vs. usual one hour run. The problem with the study is the runner coach did not randomly assign runners, but gave the slower runners one treatment. Therefore, the observed results cannot be attributed to the bike regimen; they could likely result from the fact that the two groups are fundamentally different. Think about it: a varsity runner is already the faster runner, one who is likely to improve faster at running a three-mile course than his or her junior varsity teammate. Therefore, the answer is C).
While D) might be true, and junior varsity subjects might have become faster had they been in the bike group, it doesn’t help us identify what was flawed about the treatment in the first place: the subjects were not randomly assigned.
Here is a summary of the key points:
1) Results from a study can only be generalized to the population at large if the group of subjects was randomly selected.
2) Once subjects have been selected, whether or not they were randomly selected, cause and effect can only be determined if the subjects were randomly assigned to the groups within the experiment/study/treatment.