Mapping the Drones

Peter Bergen and Katherine Tiedemann of the New America Foundation have put together the most comprehensive data on the timing, location, targets, and casualties produced by US drone strikes in Pakistan. They have just updated the data through February, and integrated it with Google Maps. They use this to draw some conclusions about the consequences, both positive and negative, for the drone campaign. Very interesting stuff, and it's nice to see some data applied to such questions.

Big Data

The Economist has an interesting special report on the implications of the explosion of information available in digital format. Exploiting and managing this data seems to be an area where businesses and "hard sciences" are far ahead of social scientists. One of the main conclusions of the articles is that people with both expertise in a particular domain as well as strong quantitative skills are well-positioned to make big advances in knowledge.

A Qualititative Analysis Manifesto?

Andrew Exum issues an "Hippocratic Oath" for the quantitative analysis of political violence. Stephen Walt likes it. Drew Conway, Henry Farrell, and Daniel Drezner generally don't.

How about a similar qualitative manifesto? Many of Exum's complaints about quantitative analysis ring true. But qualitative work has problems too. As someone who has used both quantitative and qualitative methods in recent works, here are some of issues I've struggled with when not using numbers:

1. Remember controls. Too often qualitative works do not control for all or even most of the variables that are known to influence the outcome of interest. There is a good reason for this--more controls increases the number of cases needed to make valid inferences, and adding more cases is costly in terms of time and resources. So be super-careful about picking your cases, and be modest in your claims about cause and effect.

2. Don't select on the dependent variable. Read KKV for more.

3. Don't cite KKV ritualistically. Quants get criticized for running canned statistical analyses without fully understanding the statistical logic behind them. Fair enough. But quals do the same thing. For some, "qualitative methodology" is just a cover for doing whatever they want. Indeed, the idea that there is *a* qualitative methodology is silly. There are many approaches; the only thing they share in common is that they use qualitative data.

4. Be sure you are really process tracing. "Process tracing" can be a convenient form of hand waving--"I'm going to look at the process in detail, and see if it supports my theory". But too often this is done poorly. Doing it properly requires specifying alternative explanations and seeing if the historical record supports these.

5. Think about how you are measuring your variables. Rarely done, and rarely done well. Qualitative data makes this difficult but not impossible. Indeed, once you try to measure your variables in a systematic fashion, you start to see the value of quantitative analysis, and also better understand some of the limits of the qual approach (i.e. small number of cases) and the corresponding strengths of the quants.

6. Be modest in your claims. I would disagree with Exum on this point. He suggests that quants have great scope to make bold and inaccurate claims. Of course they do--but so do quals. One great contribution of statistical analysis is the development of some agreed-upon standards for establishing your claims. In a quant analysis, we have well-established benchmarks for determining, say, if the relationship between two variables is positive or negative or statistically significant or not. Of course there are disagreements on much of this, but the disagreements are narrower than those among quals, and quants have more of a common language for expressing their disagreements.

The last point is the most important one. Both approaches have strengths and weaknesses. As someone who reads work produced by both approaches every day, though, I would say that the big advantage of quants is transparency. It's much easier for someone to understand what a quant has done (helped by the easy dissemination of replication datasets) and if they have done it well. It's far harder for quals to do this, since it's tricky to share data, many variables are measured in unclear ways, and what factors were incorporated into a process tracing exercise are obscure.

But it is possible. A great example of the finest qual work is Daniel Lindley's Promoting Peace with Information. Lindley develops clear theory, contrasts this with rival explanations, uses these to select his cases, describes how he measures variables, and in the conclusion is very clear about where his approach adds value and where other approaches are superior.