Replication in Economics

Here's my most interesting read in the last weekend, a recent article by Dan Hamermesh, published in the Canadian Journal of Economics, on "replication in economics". It is also downloadable as PDF from Hamermesh's site (Dan Hamermesh's website contains much more information, potentially useful for any economist, albeit junior or senior-- that as a remark for those of you who did not know about this excellent online resource...). But going back to Hamermesh's viewpoint in this context: the short article linked above is extremely interesting and informative. And it contains also a proposal ("modest proposal" in the text), which I also support as very welcome: the only way to see more replication studies done is to have them commissioned (to senior researchers, with tenure...) by journal editors.

While I'd advise any empirical economist (or better: any economist that is currently/ plans in the future to also do some empirical work) to read the paper, I'll put forward below some possible omissions in Prof. Hamermesh's article, which would have been particularly interesting for me:

a. What to do about registered (say, matched employer-employee) datasets that cannot be made publicly available due to confidentiality agreements (though clear steps for accessing them can be always provided as information; in practice, some of these might take too long etc. for the replication to be worth undertaking); the use of such datasets is increasing day by day, particularly within applied microeconometrics, hence I'd say they would merit some separate discussion. Caveat lector: although their case is somehow similar to that of any proprietary database, these administrative datasets do have one common characteristic, namely that usually they are provided by the official statistics bureaus of individual countries. This might suggest possible (ad-hoc) agreements between journals and such bureaus to allow replication of studies under the same very strict confidentiality rules as those the initial author had been subjected to.

b. There is no discussion in Hamermesh's paper about structural approaches: in those cases replication, as commonly understood (particularly "scientific replication" in Hamermesh's terminology), is typically not an issue (subject to correct coding and analysis, to start with). In fact, one way to avoid frequent requests for replication using other datasets and other time periods etc, would be to have a structural framework to start with...