Alan Warde, University of Manchester, UK
Peer review is the foundation for the legitimacy of academic knowledge, a process without which modern science could not function. Yet, in my experience, no one is ever taught explicitly how to do it. It is one of the many aspects of professional practice which we pick up on the job, a product of tacit understanding and contingent exposure rather than focused learning. Of course, we are trained to read critically, and hopefully friends, colleagues and supervisors comment on our manuscripts. But it is probably only when we submit our first journal article that we see examples of how referees report; and it can come as a shock!
Peer review comes in three main forms – evaluating research proposals for a funder, a book manuscript for a publisher and, my focus here, journal articles. The primary manifest purpose of a review is to give guidance to an editor, to help her or him make a decision which articles will best secure the cumulative aggregation of the body of knowledge, theory and methods which comprise a discipline or field. The aim, always in the context of competition for space in a journal, is to ensure high scholarly standards – especially to ensure that error and ethical misdemeanour are avoided. A second, latent, function is to provide guidance to authors when they (almost inevitably) revise their manuscripts, by making critical observations and positive suggestions for improvement. In both these regards some reviews are better than others!
Having recently become an editor (of the journal Sociology) I have read many referees’ reports in rapid succession since January (2018). These have given me a clearer sense of what comprises a good review. Not that all reviews are the same in structure, form or purpose. From them I seek to glean first of all whether the paper is sound science (i.e. rational analysis of social phenomena) including: is the interpretation of literature sources correct or defensible?; have appropriate methodological procedures of data collection and analysis been followed?; does the writing meet the objectives announced by the author – for example in answering the research questions?; are the conclusions drawn properly from the data and is the interpretation of the results framed in an adequate theoretical or disciplinary context?; and is the argument clearly and logically expressed? These are essentials. Thereafter I like to be told the strengths and weaknesses of the paper, and also what it contributes to consolidating, contesting or advancing knowledge on its topic, relevant adjacent topics and in relation to general issues of importance in Sociology more widely. The report forms that referees are asked to complete usually direct attention to these types of issue. Sociology, not untypically, asks referees to evaluate contribution to the literature, innovation, quality of argument and clarity of expression.
You may be invited by a journal editor to serve as a generalist or a specialist reviewer. Both are vital. The generalist will look at the quality of the argument, the clarity of concepts, the theoretical embeddedness of the material, methodological rigour and the potential for extrapolation to other fields or topics – in fact all the things which make for a good article. These are also tasks for the specialists, but they have extra value with regard to the literature review (and its omissions) and the unique contribution of the paper to debates in the field. As an editor I often cannot make a confident decision without such expert advice. By addressing these questions you will also provide the author with an evaluation that should prompt more or less radical revisions to enhance the paper’s quality.
The greater the detail in the report the more it helps both editor and author. I have yet to receive a referee report which I consider too long; but some have been frustratingly brief. Most journals invite some comments for the editor’s eyes only, and further more extensive comments for the author. Use the ‘For the Editor only’ box to indicate your competence (as specialist or generalist), to give a steer about the quality of the contribution, to indicate matters where you are uncertain about aspects of the paper or your own judgment. By all means tell the editor whether you think the paper should be accepted or rejected, but not the author; as you cannot necessarily anticipate that your view will be the same as all the others don’t pre-empt the editor’s decision.
When writing for the author be respectfully collegial in the tone of any criticism. Avoid writing a review which you would find personally upsetting if addressed to you. Be aware that weaker articles will probably have been eliminated in a prior process of desk rejection by editors, especially in journals where the ratio of submissions to space for publication is high. At the same time be candid in your evaluation. Be generous and supportive while making concrete and positive proposals for modification and improvement.
We editors at Sociology feel enormously grateful to reviewers who sacrifice their precious time for the collective benefit of the discipline. Authors have even been known to express their gratitude too. More common compensation for referees comes from the opportunity to reflect upon excellent new work in their own fields and from playing a part in moulding and developing the joint venture that is sociology. Only through the generous support of peer reviewers can we be assured that the discipline will continue to develop as an important, credible and insightful form of knowledge.