Economists Incentivize Conformity Over Quality: The Perils of Citation-Based Academic Evaluation

This podcast episode critically examines the academic evaluation systems within economics, particularly the shift from traditional peer review to quantitative citation counts for recruitment, promotion, and research funding. Carlo D'Ippoliti argues that while peer review had issues like corruption and discrimination, the reliance on citation metrics presents even greater problems. He highlights that citation distributions are highly skewed, with a few researchers and papers accumulating disproportionately high numbers, leading to a simplistic and often misleading assumption that these aggregates accurately measure scientific quality. This system, rather than fostering excellence, risks creating more discrimination and stifling genuine intellectual inquiry.\n\nD'Ippoliti meticulously details the inherent biases within citation counts, demonstrating that they are not objective measures. Factors such as the number of authors, their age and sex, title length, and even the research topic's geographical focus (e.g., peripheral vs. US economies) significantly influence citation rates. Crucially, the system is biased against researchers who deviate from the mainstream of the profession and against women. His analysis of Italian economists reveals that institutional affiliation and ideological proximity, rather than scientific merit, strongly predict citation likelihood. This creates perverse incentives, particularly for younger and women economists, who may feel compelled to conform to certain topics or research communities to secure citations, thereby actively suppressing research freedom and intellectual diversity.\n\nTo counter these detrimental trends, the speaker advocates for a fundamental re-evaluation of how academic quality is assessed. He asserts that the only true way to gauge the scientific content of research is to read it. For broader, more massive evaluations, if absolutely necessary, he proposes developing a plurality of standardized indicators, tailored by topic and research method, rather than relying on a single, simplistic measure. A paramount objective for any research evaluation system, he argues, must be the explicit preservation of pluralism within the discipline. He also suggests that external pressure from progressive movements, trade unions, and political parties could drive the necessary changes in recruitment and funding rules.\n\nThe episode connects these academic systemic flaws to broader criticisms of economists and their past policy missteps, such as "expansionary austerity" or rigid public debt thresholds. D'Ippoliti contends that the current evaluation system discourages critical thinking and innovation, pushing economics towards a narrow, often liberal, direction due to specific funding influences. He encourages skeptical young students to pursue economics and challenge the status quo, emphasizing the need for counter-weighting funding sources, like foundations such as INET, to support innovative, radical, and unorthodox ideas. This, he concludes, is essential for increasing pluralism and ensuring that economics remains a vital discipline capable of addressing societal well-being effectively.

The picture is quite frightening, it is true.

Citation counts are supposed to be objective measures of scientific quality. But this is highly problematic.

this solution could be even worse than the problem itself, because it risks creating more discrimination than the peer review system.

citations are very much biased also against certain topics, certain fields of research.

researchers that do not fully or necessarily agree with the mainstream of the profession are less highly-cited, and this is a huge risk for the development of economics in the future.

being affiliated to the same institution, which in principle, has nothing to do with scientific quality, makes it twice as likely that two economists will cite each other, and also some measures of political and ideological proximity make it at least 5% more likely that an economist will cite someone else.

It is actively suppressing the freedom of research.

in order to really assess the scientific quality of a piece of research, it should be read.

preserving pluralism within the discipline should be a primary and explicit aim of research evaluation.

we really need the sort of students that will have a more critical stance.

we really need some counter-weighting sources of financing, that could increase pluralism within the discipline.