In science, the ideal form of research design is the experiment. Experiments allow the investigator to control the values of all of the variables and manipulate them independently of one another, and to conduct repeated trials using different combinations of variable values. Because only one independent variable value is changing in any one trial, the investigator can be absolutely certain that any changes in the dependent variable (outcome) are solely due to the changed value of the independent variable.
Ideally, social scientists would love to have this level of certainty about their own findings. Unfortunately, though, we can’t rerun history using different values: Would George W. Bush have invaded Iraq in 2003 if he had been 20 points less popular after 9/11? What if his popularity was 15 points lower? Or 10 points? We just can’t do that – it’s not possible, and even if it were, it wouldn’t be ethical. We are stuck with observational data, and we have to make the best of it.
Our solution to this problem is to control other variables as best as we can. Fortunately, multivariate statistical techniques do this, and most of the principles of qualitative case selection have this as their core idea. Both approaches have limitations, however, imposed largely by our inability to generate other cases with different combinations of variable vales. We are restricted to the set of cases that we have actually observed. We can rarely find two cases whose values on all variables (except the variable of interest) are perfectly matched. Even two observations of the same country or case at different points of time don’t match perfectly; at a minimum, they differ in time, and they differ in history since one of the cases has memory or knowledge of the outcome of the previous case. Cross-sectional (many cases observed at a single point in time) designs have even bigger problems, especially in quantitative research, because we rarely have two cases with the same exact values on a given variable. (Can you imagine two totally separate wars with identical death tolls? Or two countries whose birth rates are exactly the same?) It’s a highly improbable situation, especially for continuous variables.
Our goal is to get as close as possible to the experimental ideal so that we can have the same level of confidence in our conclusions. This just isn’t possible for most research questions. For a few questions, however, we are lucky in that natural experiments exist. The contexts are exactly the same except that some external event artificially separates the population into two groups that then receive different treatments. Examples of such external (exogenous) events include arbitrarily drawn colonial borders in Africa, which often separate members of the same cultural group or tribe into two or more states; the 2004 tsunami in the Indian Ocean, which destroyed many similar Indonesian islands but where responses and recovery efforts took very different tracks; a housing development whose residents vote in two different wards; or similar types of events that are not at all generated by the actors whose behavior we are trying to explain.
Social scientists occasionally are able to employ true experiments in their research. We can, for example, manipulate the content of campaign advertising and evaluate audience responses. We can also conduct survey experiments, in which respondents are randomly assigned to receive variations on the same question; the variations correspond to values of the independent variables that we are interested in. In both of these examples, random assignment of participants to treatments has the same effect as controlling variable values. Because the treatment (value of the investigatory variable of interest) is determined exogenously – by random assignment, rather than by some characteristic of the respondent that might be related to the outcome – we can give the results the same level of credence that we would give a ‘true’ experiment