loading page

Building Trust: Defining Subject Matter Expertise through U.S. Federal Peer Review Policy
  • Meredith Goins
Meredith Goins
Oak Ridge Associated Universities / University of Tennessee, Knoxville

Corresponding Author:[email protected]

Author Profile

Abstract

There is no quantifiable definition of what comprises a peer reviewer, leading to a lack of trust in the outcomes of peer review by the public and some government officials. Using contextual content analysis, this study uncovered concepts from federal science policy and agency peer review guidance documents to begin the creation of such a definition. The first stage determined which term is most often used for a peer reviewer as seen in the U.S. Office of Management and Budget’s (OMB) Final Information Quality Bulletin for Peer Review 2005. The second stage analyzed the U.S. Department of Energy, Energy Efficiency & Renewable Energy, the U.S. Department of Interior, Bureau of Safety and Environmental Enforcement and the U.S. Department of Justice, Office on Violence Against Women peer review guidelines to determine the number of mentions of each concept (knowledge, skills, experience), if a definition of a merit peer reviewer is provided, and descriptors associated with qualities, or sub-concepts pertaining to peer reviews including discipline, teaching, education, postdoctoral positions, certifications, grant funding, publications, presentations, awards and consulting work. Stage one results found that the term reviewer is the most used in describing a peer reviewer for all three federal agencies and the OMB bulletin. Results of stage two show that both knowledge (M=9) and experience (M=14.33) categories were the largest reported with few mentions of skills (M=3.33). Each agency provided a definition of reviewer and qualifications they must meet. This analysis of three federal peer review guidance documents shows that the federal guidance on expertise, if measured by both experience as well as knowledge, is being followed. It would be a worthy effort to do an analysis of the publications, experience and knowledge of those selected for peer review and compare it to each agencies criteria to build a profile of reviewer characteristics. In addition to contributing to the scant literature on peer review, these findings will help narrow the qualities merit peer reviewers need and will assist with future research exploring all U.S. Federal Agency peer review manuals in which qualities in both knowledge and experience could be quantified, helping to defining an accurate measurement of goodness of fit for reviewers.