A measure of total research impact independent of time and discipline

This article was published as A measure of total research impact independent of time and discipline. Alberto Pepe, Michael J. Kurtz. PLoS One. 7(11): e46428. doi:10.1371/journal.pone.00464282012. (Open Access Article)

Abstract. Authorship and citation practices evolve with time and differ by academic discipline. As such, indicators of research productivity based on citation records are naturally subject to historical and disciplinary effects. We observe these effects on a corpus of astronomer career data constructed from a database of refereed publications. We employ a simple mechanism to measure research output using author and reference counts available in bibliographic databases to develop a citation-based indicator of research productivity. The total research impact (tori) quantifies, for an individual, the total amount of scholarly work that others have devoted to his/her work, measured in the volume of research papers. A derived measure, the research impact quotient (riq), is an age independent measure of an individual’s research ability. We demonstrate that these measures are substantially less vulnerable to temporal debasement and cross-disciplinary bias than the most popular current measures. The proposed measures of research impact, tori and riq, have been implemented in the Smithsonian/NASA Astrophysics Data System.

Introduction

Measuring the research performance of scholars plays a critical role in the allocation of scholarly resources at all levels “quantitative” means of measurement has long been through the use of citations (Garfield 1955, de Solla Price 1965). Citations are routinely used to evaluate the research productivity of individuals citations to measure research performance involves several confounding factors which tend to become more important as the degree of aggregation decreases. For the evaluation of individuals, important challenges are:

Discipline

Citation practices vary widely among various fields. Citation rates can vary between disciplines by an order of magnitude (Leydesdorff 2011); among sub-disciplines in the same discipline they can vary by a factor of two.

Co-Authorship

A paper can have an arbitrary number of authors, from one to several thousand. Should an author of a single authored paper receive the same credit for a citation as someone who has co-authors?

Age

The number of citations accrued by an individual scales with the square of his/her career length (Hirsch 2005, Kurtz 2005); thus, a person with a career length of 10 years will have half the citations of an equal person with a career length of 14.14 years. This age effect problem is exacerbated by the fact that the two aforementioned challenges are time dependent. For example, in the field of astrophysics, both the mean number of references and the mean number of authors have approximately doubled in the last 20 years. (Henneken 2011, Schulman 1997)

Some of the lesser challenges associated with using citations to measure research productivity of individuals are:

Self-Citation

If an author cites papers by him/herself should they count as much as citations from papers by others?

Curation

In addition to having a database of articles and citations, one must clean and curate its data. For example, an analysis of an individual’s productivity requires that one be able to exactly identify the articles written by that individual. Name changes (e.g., due to marriage) and homonyms (name clashes, where different people have the same name) can make this a serious problem.

Shot Noise

Sometimes an individual can, almost entirely by chance, become an author of one or more very highly cited papers, perhaps as a student. The citation distribution is a Zipf like power law, whereby some articles are cited thousands of times more than the median; clearly, there can be circumstances where a direct count of citations is not a fair representation of impact.

In a highly influential paper, Hirsch (Hirsch 2005) proposed a pair of citation-based measures (h, m) which: solve the shot-noise problem, substantially improve the age problem, and help with the curation difficulty, discussed above. The Hirsch index, h, is the position in a citation ranked list where the rank equals the number of citations; absent shot noise h is obviously proportional to the square root of the total number of citations, which grows linearly with career length (Hirsch 2005, Kurtz 2005). The m quotient is h divided by career length, and is a constant throughout the career of an individual with constant productivity in a constant environment.

The h-index is by far the most widel