The Manski Critique

Chuck Manski’s recent NBER working paper, “Policy Analysis with Incredible Certitude” (non-gated version) ought to be a must-read for anyone doing or interested in policy analysis.

The study is written in an accessible way, such that it can be in principle followed without explicit academic training in Economics/Econometrics (there are plenty of further references for the technical details), and essentially sums up some of Manski’s conclusions from his well known research agenda on empirical methods in social sciences such as partial identification, and using decision theory with credible assumptions, for policy inference– see for instance his books on these topics (which any applied econometrician should have on his/her shelf; though I confess, my copies are currently still in Aarhus, awaiting my shipping/bringing them to Chicago), Identification Problems in the Social Sciences (1995), Partial Identification of Probability Distributions (2003), Social Choice with Partial Knowledge of Treatment Response (2005), and Identification for Prediction and Decision (2007).

Manski catalogues the ‘incredible analytical practices’ and provides examples for each category. His set consists of “conventional certitudes”, “dueling certitudes”, “conflating science and advocacy” and “wishful extrapolation”. I find particularly compelling the sections on the conventional and respectively the dueling certitudes, which are preceded by a concise introduction on the incentives for certitude (wherein, as usually, USA presidents’ alleged statements always come in handy). Similarly, Manski pins down very well the “wishful extrapolation” practice, when often very strong, unwarranted, invariance assumptions are made (see his example on the selective incapacitation studies performed by RAND researchers in the early 80s, which gave rise to some hot political debates).

The one section that I find less thorough than the others in Manski’s paper is the one on “conflating science and advocacy”. Acknowledging that impartiality in social science (in fact, with some contextual caveats, this point is relevant for science in general) is the ideal, and that often research falls far from this ideal, I think Manski’s pointing out that conflating science and advocacy is one of the main reasons for incredible policy analysis is absolutely correct. His illustration with excerpts from Milton Friedman’s arguments for educational vouchers is also fine; indeed some of the crucial empirical evidence necessary for Friedman’s stated policy implications in that context was (and still is, as Manski also states) missing, such as whether there are significant neighborhood (or peer) externalities involved (however, for making that point stronger, plenty of better, and/or more recent examples could have been used…). What I did not particularly fancy is the between-the-lines allusion that Friedman used to do this (i.e., conflating science and advocacy) on a frequent basis or, perhaps, all/most of the time. For instance, Manski states on page 20, “Milton Friedman […] had a seductive ability to conflate science and advocacy” […]. See Krugman (2007) for a broader portrait of Friedman as scientist and advocate“. That particular NYRB article of Paul Krugman that Manski cites (presumably for other depictions of Friedman seductively conflating science with advocacy, and related– line which is otherwise not followed up or further substantiated in Manski’s paper) has however a considerable number of inaccuracies and misunderstandings, which others very well pointed out in subsequent articles, see for instance Nelson and Schwartz’s reply to Krugman’s; in fact, many would easily think that Krugman is himself guilty of conflating science and advocacy here (and elsewhere; some do actually substitute ‘conflating science and advocacy’ with ‘ignoring science for advocacy’ as practice in many of Krugman’s NYT pieces- 1st bullet point…). And yes, ok: I wrote myself a post about that Krugman portrayal of Friedman, shortly after his article appeared. All in all, however, this minor point does not in any way diminish the essence of Manski’s thesis; it’s only that if it is about impartiality and professionalism in scientific practice (which Chuck Manski is one of the champions of, no doubt), we should make sure that is also the (only) between-the-lines message.