Hard tests are ones where your theory is least likely to predict outcomes successfully. We can usually identify ‘hard’ cases because key independent variables from other theories take values that those alternative theories argue would predict against observing the outcome of interest. In the table below, that’s represented by cells in the off diagonal.
| My theory predicts —> Alt theory predicts | Yes | No |
| Yes | *** | |
| No | *** |
Successfully explaining a case that the other theory gets ‘wrong’ is a strong mark in your favor, especially if your theory doesn’t make any other mis-predictions. That’s parsimony in action: we prefer theories that get more right with the same or fewer inputs.
Of course, not all theories or outcomes have nice, neat dichotomous outcomes. In more complex forms of hard tests, competing theories or hypotheses require you to juggle between easy tests and hard tests. To simplify this discussion, let’s assume that you have two theories, yours and a best alternative theory (BAT), and you have a couple of hypotheses for each. For at least one of the tests, you want to find a case that is easy for the BAT but that is hard for your hypothesis. If possible, you should also try to have at least one test that is hard for both the BAT and your theory. If your theory can make it there, to paraphrase Frank Sinatra, “it can make it anywhere.”
Cases that are seen as ‘deviant’ outcomes, anomalous cases, or “exceptions” for some other theory can be hard tests if your theory not only predicts their outcomes but successfully predicts the outcomes of cases that the BAT – here, the one that made these cases ‘deviant’ in the first place – also correctly explains. Again, the key here is parsimony: Your theory needs to be able to explain more than the competing theories without making exceptionally large or undue demands on data inputs. The increase in inputs must be commensurate with the improvement in outcome prediction.