Academic Peer Review

From E-Learning Faculty Modules


Contents

Module Summary

Academic publishing is a critical aspect of professional work in academia. While professional editors have important roles in “gatekeeping” academic publishing, double-blind academic peer review is important as well, to ensure that standards are maintained. This module highlights some aspects of academic peer review work, its standards, and some of the technologies that support this work.

Takeaways

Learners will...

  • consider what academic publishing is and how it relates to academic research and teaching
  • review the respective roles and responsibilities of academic peer reviewers
  • think about common professional standards for academic peer review
  • explore some of the features of web-facing technology systems for academic peer review
  • enumerate some of the benefits that accrue to academic peer reviewers


Module Pretest

1. What is academic publishing? How is it related to academic research and teaching? What is academic peer review, and how is this relevant to work in academia?

2. How do academic peer reviewers ensure publishing standards? How much influence do academic peer reviewers have? Who makes the ultimate decision about the “publishability” of an academic work?

3. What are common professional standards for academic peer review? Ethical standards? Practical standards?

4. How is academic peer review conducted via web-facing technology systems? How is a reviewer profiled? How are reviews assigned by reviewer profiles? What are some common expectations for turnaround time for reviews?

5. What are some ways that academic peer reviewers get credit and recognition for their review work? What benefits accrue to academic peer reviewers, and why?


Main Contents

1. What is academic publishing? How is it related to academic research and teaching? What is academic peer review, and how is this relevant to work in academia?

Academic publishing, as a “vertical,” involves the collection, editing, and distribution of a variety of types of writing. The more formalized works include formal articles (of various forms), book chapters, books, and collections. They also publish (online) theses and dissertations that are the product of undergraduate and graduate students in universities. The publications may be in paper or electronic forms—as books, periodicals, and stand-alone bound works. Of late, there are a number of multimedia publications as well, with embedded multimedia (like video, audio, digital objects, simulations, games, and others).

In general, virtually all academic publishing is edited. There is oversight by graduate committees. There is oversight by faculty. Editors and peer reviews serve as gatekeepers to publication. (The more elite the publication, the lower the acceptance rates. Acceptance rates are a metric maintained for a majority of academic publications.)

Academic publishing is a crucial part of higher education for several reasons:

  • It enables practitioners in academia to showcase their research and share with their peers.
  • It enables those within academia and outside to better understand the research work of those in academia (particularly in public research universities), especially since much of this work is publicly funded.
  • It enables researchers and students to learn new insights and research methods and technologies from their colleagues around the world.
  • It enables practitioners to archive their works for historical purposes and posterity.

[Some works by students, faculty, and staff are embargoed against public release and public consumption because the topic is too sensitive—such as for issues of national security. In other cases, commercially-funded research may be embargoed because of commercial advantage—and such limits are spelled out in the terms of the funding.]

Academic peer review involves the work of those in academia assessing the write-ups of others’ research or “peer research.” This process is usually done in a double-blind way, with neither the reviewer aware of the author (or authoring team), or the author / authoring team being aware of the reviewers. Only the editor and publisher is aware of both parties’ identities.

The work of peer review is “invited” by the publishers based on the publishing history, reputations, formal education, and other aspects of the peer reviewers. These are unpaid and voluntary positions. This work is rewarding for many who enjoy reading others’ research, having input on others’ work, and supporting both seasoned and up-and-coming authors / researchers.


2. How do academic peer reviewers ensure publishing standards? How much influence do academic peer reviewers have? Who makes the ultimate decision about the “publishability” of an academic work?

Academic peer reviewers ensure publishing standards in a number of ways. One of the most critical is that they are active in their professional lives and maintain active reading of the works in their main fields (and peripheral ones). They are actual local experts on particular topics. They strive to provide honest and unbiased feedback when they serve in the role of reviewer. If they are aware of the author identity of a work, they should recuse themselves. They should not have any undue influence from commercial or other entities, and if they do, they should recuse themselves from the review at the time of discovery.

Academic peer reviewers are not the sole decider of whether a work should be accepted for publication or not. Oftentimes, they are one of a few reviewers (whose identities are unknown)…or they are one of a larger team of distributed reviewers (with up to a dozen reviewers reviewing some works).

Editors (supported by publishers) make the ultimate decisions about the publishability of an academic work. There are instant “dealbreakers,” such as any sign of fraud, copyright infringement, dishonesty, and so on. Reputational signals are critical in academia as they are in other fields. There are dealbreakers for peer reviewers, too, and some of them are described in the next question-based section.


3. What are common professional standards for academic peer review? Ethical standards? Practical standards?

There are common professional standards for academic peer review. Some of them are described below: An academic peer reviewer…

  • must consider a manuscript with fairness and balance
  • must support peer researchers and academicians in an inclusive way
  • should focus on providing detailed and constructive peer critique to improve both the work and the author / researcher
  • should respect others’ voices and expertise (don’t suggest “be me” or “write like me” but encourage the development of the “best you” with your own voice and expertise…don’t be self-indulgent in the role)
  • should meet all deadlines (once he or she has agreed to take part in the peer review process)
  • should never try to game the system to help an ally or hurt a non-ally
  • should never break professional ethical standards in the field
  • should never break the double-blind peer review setup by revealing identities
  • should never misuse a shared manuscript such as by sharing it with others outside of the trust context or outside the publisher venue
  • never come at peer review when fatigued or frustrated
  • never engage in sarcasm, ad hominem attacks, gossip, or other unprofessional communications
  • must recuse himself / herself if there is any potential compromise of their objectivity, and so on

There are not only general academic standards that need to be followed but also those of the respective disciplines in academia.

Saying yes, maybe, or no. An important responsibility of the peer reviewer is to provide a recommendation for the manuscript. This summary suggestion is done with a lot of consideration and supporting details.

The outright “yes” is almost never extended. Most academic works go through a number of revisions before they are appropriate for publishing.

The “maybe” is the most common response, and in such cases, it is “accept with these conditionals” or “revise on these issues…and resubmit cold”. A cold resubmit means that the author / authoring team has to go through the process of peer review from scratch because so much of the original work had to be redone. So what are some common suggestions?

  • Offer clarity about the research methods.
  • Offer clarity about the data analytics techniques (particularly statistical methods).
  • Explain the ethical considerations for the research work and the processes to ensure these.
  • Work on the logic between the findings and the insights in the Discussion area. (If there are unsupported claims, that can be highly difficult to accept a work because professional candor is critical for all published research work. Without that basic trust, there is no standing for collegial work.)
  • Update the review of the literature to include more recent research.
  • Rework the order of the paper for clarity.
  • Work on a more consistent tone (particularly for works with multiple authors).
  • Follow clear source citation standards.

Getting to “rejection” (for cause) is a challenging call since that means that a work is irredeemably poor and unsalvageable. There are a few cases in which a rejection is the only option (in this author’s opinion).

The submitted works for Tier 1 and Tier 2 publications come from all over the world. This means that researchers and research teams hail from different social environments and different understandings of academic work. Sometimes, this means that a work may not meet the standards of human subjects review (required in the U.S.). In the same way that U.S. researchers cannot engage in research that does not meet legal standards no matter where they are in the world, they also cannot publish works that do not meet legal and ethical standards of the field.

Another deal breaker for a peer reviewer is if the data they are reading is fraudulent. How can one tell if data is fraudulent? This comes from the research methods applied, the statistical methods applied, the likelihood of outcomes, and other factors. To make an assertion this big though requires some very clear logic and a high level of provability. (If datasets are available, it would help to run the statistical analysis on that set before making any assertions.)

In some cases, an author’s work will be so muddled and the original research so poor that there is no way to salvage a work even if the author had a lot of support. There is no way to recoup lost time or lost access to research subjects or lost data, etc. If there is no way to have a researcher go back to extant information to actually process it correctly, and so on, then the “maybe” should not be extended.

If the author’s work is partially or totally plagiarized, that is a non-starter. If trust is broken, there is no way to recoup that from a distance.

In most cases, a peer reviewer is not accessible to the author / authoring team after the initial review is submitted. All interactions of the authoring team are with the editor. Actually, it’s rare for the editor himself / herself to reach out to the reviewers either for many publications… It is rare to have follow-on discussions about a reviewed work, but this depends on the prior relationships between the editor and the reviewers.


4. How is academic peer review conducted via web-facing technology systems? How is a reviewer profiled? How are reviews assigned by reviewer profiles? What are some common expectations for turnaround time for reviews?

Academic peer review for a number of mainstream publishers is done online, in bespoke systems designed for book or periodical development. These systems enable authors to submit their works, editors to receive the manuscripts, reviewers to review the works, authors to sign contracts (if their works are accepted), and so on. These technology systems maintain records of the various steps to the process.

From the academic peer reviewer side, they are usually invited by email to join. They are asked to complete a profile of their respective areas of expertise. The incoming manuscripts are assigned in part based on reviewer profiles—not in an automatic sense…but after human editor vetting (usually). In such cases, a peer reviewer is invited to review a work and is given the title and maybe an abstract of the work—to see if they are interested.

In terms of turnaround time, reviews are due usually within a few weeks to a month of agreement to review.

Each publisher has its own built-in forms which are used for the review. These include both a way to render a decision (generally: accept, accept with changes, reject). The rejection is usually for cause. For many of the systems, reviewers are also asked to rate their own expertise on the topic-at-hand as a measure of “confidence” in the review (how much weight should the editor or publisher put on the reviewer’s assessments and comments?).


5. What are some ways that academic peer reviewers get credit and recognition for their review work? What benefits accrue to academic peer reviewers, and why?

For many years, academic peer reviewers benefitted from the private satisfaction of providing peer reviews. In some cases, they were acknowledged publicly as one of a pool of peer reviewers for particular publications. In recent years, some digital badging systems have come online to allow reviewers to accrue verified “counts” of their formal reviews and to have that “credit” recorded online. The reviewed works’ titles are protected and not revealed. The outcomes of the reviews are not publicly shared. Besides such electronic credits, there are not other rewards for this work, generally speaking.

Examples

How To

The approaches to peer review differ based on the particular discipline and the reviewer. If there were one piece of advice for reviewers, it is to read broadly and in an open-minded way. It helps to approach the work with a sense of humbleness.

By its existence, an original work of research is already a rarity. While many have ambitions to write and publish, most do not follow through. Each work--which meets minimum requirements--likely has the following inputs: a review of the literature, research design, data collection, data cleaning, data analysis, original data visualizations, original diagrams, and write-ups. There were likely hundreds of hours of human effort (and collaboration). One project can involve thousands of files.

It helps to think about "survival bias" in publishing. For any one manuscript that one views, there are likely thousands of stubs and partially started works that never make it this far.

And yet, the works that are submitted are imperfect...and eminently improve-able.

Accepting a work for publication is a form of support for its bid for readers...and for posterity. And yet, even those that make it through the publication process may end up in the so-called Great Unread and never truly find an audience.

Possible Pitfalls

One of the possible pitfalls is that the technology systems supporting academic peer review are hacked and all identities are outed. This is not that far-fetched. The point is to be able to stand by one’s reviews even if they started out anonymized.

Module Post-Test

1. What is academic publishing? How is it related to academic research and teaching? What is academic peer review, and how is this relevant to work in academia?

2. How do academic peer reviewers ensure publishing standards? How much influence do academic peer reviewers have? Who makes the ultimate decision about the “publishability” of an academic work?

3. What are common professional standards for academic peer review? Ethical standards? Practical standards?

4. How is academic peer review conducted via web-facing technology systems? How is a reviewer profiled? How are reviews assigned by reviewer profiles? What are some common expectations for turnaround time for reviews?

5. What are some ways that academic peer reviewers get credit and recognition for their review work? What benefits accrue to academic peer reviewers, and why?

References

Extra Resources