In a recent post we discussed the merits and limits of press freedom rankings. One of the rather sad facts is that newsrooms rarely ask how such rankings are created. Researchers seem to be more interested in this essential methodological question – but even their curiosity is sporadic. Monroe E. Price, Susan Abbott and Libby Morgan (University of Pennsylvania) recently raised this question and have tried to “evaluate the evaluators”. Four of the overall 16 contributions to their book deal explicitly with press freedom rankings. The evaluations of the researchers are surprisingly positive. Lee Becker and Tudor Vlad (both from the University of Georgia) correlate the rankings of Reporters without Borders, Freedom House and the IREX-inex and attest that they have a high reliability. However, the researchers draw awareness to the fact that the indices are based on a very Western, if not American understanding of press freedom: “The Freedom House and Reporters without Borders measures focus mostly on media freedom and independence from government control and are less reflective of the control exercised by commercial and corporate interests.”

Somewhat more critical is the analysis of the European contributors to the book. Patrick McCurdy (Erasmus-University, Rotterdam), Gerry Power (InterMediaUK) and Anna Godfrey (BBC World Service Trust) point towards inconsistencies in the questionnaires. Christina Holtz-Bacha (University of Erlangen-Nürnberg) argues that the rankings remain dubious as press freedom is not clearly defined. Pointing to the example of Finland which has occupied one of the first positions in the rankings for many years, she questions whether the government subsidies paid to newspapers in this country should not raise doubts concerning the independence of the media. It is to be hoped that the researchers themselves have not become professionally blinkered. The most sophisticated questionnaire will not allow the empirical measurement, without any doubts, of whether more press freedom is being practiced in Finland than in Austria or Switzerland. And with an even lower prospect of finding the truth may such differences be clarified in the last third of the indices. However, it is of relevance whether a country is in the first or in the last third of the ranking or somewhere in between. Therefore, it would be more honest to assign each country to a cluster without insinuating an exact ranking. However, would journalists still be interested in the topic if they could no longer provide the “news” that Switzerland or Austria have moved down a few positions? Unfortunately there would be no news value in reporting that both countries continue to be in the top ranked group – together with the Scandinavian countries, New Zealand and Germany.

Monroe E. Price/Susan Abbott/Libby Morgan (eds.) (2011): Measures of Press Freedom and Media Contributions to Development. Evaluating the Evaluators, New York u. a.: Peter Lang

Article translated from the original German Wie verlässlich sind Pressefreiheits-Rankings? by the author.  A longer, slightly different version of this posting has been published in Schweizer Journalist Nr. 2 + 3, 2013