We will address the question of whether peer review at the Research Council of Norway within the Research Programme on Better Health and Quality of Life inhibits the diversity of perspectives and methods in the scientific production of knowledge on public health and public health policy.
It is said that when people think similarly, they do not think enough. Within the texts and speeches of academia, we keep seeing versions of this most timely reminder in everything from strategy documents to descriptions of learning outcomes, programme announcements and major speeches. Innovation, critique, transparency, interdisciplinarity and diversity of perspectives are fine words that are often heard when academic output is promoted. However, also here one may suspect that there is a gap between ideal and reality.
Based on our own experiences as applicants to the Research Council of Norway’s Research Programme on Better Health and Quality of Life (BEDREHELSE), our question in this feature article is whether the conditions for production of scientific knowledge are counterproductive in practice. More concretely, we will shed light on peer review as a compulsory point of transit, and ask: who are the referees?
We are not arguing in favour of removing the peer review process. We argue that this institutionalised convention for quality assessment in academia requires respect, appreciation and knowledge about the diversity of theoretical and scientific methods. Our question encompasses both the matter of how referees are recruited and each referee’s ability and willingness to acknowledge their own academic shortcomings.
The absence of such humility entails a risk of methodological tunnel vision, exclusion of scientific disciplines and preclusion of critical research questions – in other words, peer review may lead to people thinking too similarly and too little.
Peer review or academic border control?
The Research Council’s public health programme invites scientific production of knowledge to “[…] promote new knowledge about the prevalence and causes of ill health and health, and about the development, implementation and effect of health-promoting measures” (1). The programme announcement thus clearly falls within the framework of epidemiology, defined as “the study of occurrence, cause and control of health disorders and illness” (2).
The language of the programme announcement thus does not attract potential applicants whose research interests lie outside such an epidemiological frame, like critical analysts of public health policy as an all-encompassing state governance project. The same is true of researchers whose approaches are based on qualitative scientific methods (3).
A broader programme announcement would clearly allow more perspectives and illuminate the field of public health more widely. It would not displace epidemiologically-produced knowledge, but epidemiology clearly has epistemic limitations, for example when attempting to understand “health disobedience”, in other words why people live their everyday lives ignoring knowledge about risks, causal relationships and the effects of remedies.
It is clear that by inviting the production of knowledge within an epidemiological frame, the programme reduces the diversity of research questions and methodological approaches that reach the expert panels. The recruitment of the expert panels fosters more indirect exclusion. Referees, whose responsibilities include assessing the relevance and scientific quality of projects, are namely also recruited within an epidemiological scientific tradition. This is not particularly unusual, considering the focus of the programme announcement. It does mean, however, that projects with research questions and methods at the periphery of the programme description are peer reviewed by scientists who are referees of other disciplines. Such a gap between projects and referees entails a risk of both unqualified and hostile reading. At the same time, peer review is at risk of becoming a kind of academic border patrol that excludes the perspective of diversity from public health and public health policy.
One research project, two assessments
Together with other Nordic researchers, we submitted an application to the BEDREHELSE programme in the spring of 2016. The project was divided into three work packages. First, we wanted to investigate a selection of epistemological ‘mapping machinery’ and how it generates images of people’s health. One example of such machinery is Ungdata, a national monitoring tool that produces snapshots of young girls’ mental health (4). We then wanted to study the “public health snapshots” themselves, the materialised products of science that describe, for example, public health problems, correlations, causal relationships and the effects of remedies. In the project’s third work package, we wanted to study how public health interventions are welcomed, i.e. on both the municipal level and among “ordinary people”. What happens when epidemiologically-produced knowledge meets other forms of knowledge, people’s beliefs and doubts, and the different ways the target groups live and organise their everyday lives? In other words, we wanted to make knowledge production and public health policy our empirical field and study it within a humanist and social science framework.
The members of the project group come from disciplines such as history of ideas, linguistics, anthropology, science and technology studies, political and power analysis, and sociology. The project positioned itself clearly in the periphery of the invitation of the programme description, and funding would require appreciation of the application’s arguments regarding the relevance of the diversity of perspective and methods in public health research.
Three weeks after we submitted the application to the programme, we submitted an identical application to the Research Council’s FRIHUMSAM programme. According to the Research Council, this is a thematically ‘neutral’ programme intended to promote, among other things, “boldness in scientific thinking and innovation” in the humanities and social sciences. In other words, topics related to health are neither prioritised nor excluded in FRIHUMSAM. In November 2016, we were informed of the outcome of both of our applications. They were both rejected, but that is where the similarities between the assessments of the two expert panels end.
Both the BEDREHELSE and FRIHUMSAM expert panels assessed the relevance of the project in relation to the programme announcements. It came as no surprise that the referees for BEDREHELSE assessed its relevance as “weak”, granting it a grade of 3 on a scale of 1 to 7, where 7 is the highest grade. Such an assessment of the project’s relevance made it clear that further consideration was superfluous.
However, the Research Council’s expert panels also assess the “scientific quality” of projects they deem to be of little relevance. In our view, it is the difference between the two panels’ assessments of the quality of the projects that leads to the more general question: who are the referees?
The assessment of the BEDREHELSE expert panel was that “the project has not been presented adequately and/or has major qualitative deficiencies. It is not likely that any new knowledge will be generated”. The FRIHUMSAM panel’s assessment was that “the project’s objectives, research questions and hypotheses are very clearly presented and are based on an excellently formulated and highly original project concept”.
Here it may be particularly interesting to note the discrepancy in the assessment of the language of the application: “not been presented adequately” versus “very clearly presented”. One possible explanation is that different scientific disciplines develop their own “jargon”, which can appear to be unclear and confusing for the readers of project descriptions who do not belong to the scientific traditions of which the research projects form part.
The differences in the assessment of “scientific quality” were symptomatic of the assessment of the other criteria. “Project manager and project group”, “Implementation plan and resource parameters” and “International collaboration” were all assessed as weak/grade 3 by the referees in the BEDREHELSE programme and as very good/grade 6 in the FRIHUMSAM programme. The same was true of the grading in the “Overall assessment”. The disharmony reached its apex in the assessment of “Boldness in scientific thinking and innovation”. Here the conclusion of the FRIHUMSAM referees was a very good/grade A, and they wrote: “The project has a very high potential for scientific innovation. It is highly likely to result in substantial theoretical advancement, and/or […] a radical expansion of knowledge. The project is exceptionally creative”. Under the “Impact of the project” criterion, the BEDREHELSE referees wrote: “The project offers no significant benefit”. Having been assessed as grade 3, our project was filtered out and never made it to the programme board, according to Pål Kraft, chair of the BEDREHELSE programme board (5).
In table 1, we compare the referees’ assessments of a number of the criteria for the two programmes.
Table 1
Comparison of the peer reviews
|
Criteria
|
FRIHUMSAM
|
BEDREHELSE
|
|
Scientific quality
|
The project’s objectives, research questions and hypotheses are very clearly presented and are based on an excellently formulated and highly original project concept. The project is in the forefront of its field and will contribute to scientific innovation as well as generate important new knowledge.
|
The project has not been presented adequately and/or has major qualitative deficiencies. It is not likely that any new knowledge will be generated.
|
|
International collaboration
|
There is a satisfactory level of international collaboration, and it is of adequate quality.
|
The international collaboration activities in the project are weak, and in reality non-existent.
|
|
Boldness in scientific thinking and innovation Impact of the project
|
The project has a very high potential for scientific innovation. It is highly likely to result in substantial theoretical advancement, and/or the development of significantly new methodology and/or a radical expansion of knowledge. The project is exceptionally creative.
|
The project offers no significant benefit.
|
|
Overall assessment
|
A project at the highest international level and of the utmost interest nationally and internationally. Publications in leading journals are expected. The researchers are leaders in their field.
|
A project in need of comprehensive qualitative improvements.
|
Referees or referees of other disciplines?
How does one explain this fundamental difference between two assessments of the same research project? We are fully aware that causal analyses always offer several options, and choose not to look at what has been assessed, but at those who perform the assessments; in other words the referees, both how they are recruited and how they handle their role.
In general, the different programmes at the Research Council have several expert panels. This information is publicly available on the Research Council’s website, stating each person’s name, nationality and institution. This provides fairly easy access to the referees’ academic background, position, research interests and scientific publications.
A search of the members of the BEDREHELSE programme’s expert panel who assessed our project gives the overwhelming impression of experienced and highly-lauded researchers from a number of European universities. The same is true of the members of the FRIHUMSAM programme’s expert panel. The main difference is that the former, like the programme announcement, clearly belong to an epidemiological framework, while the latter belong to the social sciences and the humanities.
Most people who have worked on scientific commissions, expert panels and other assessment of academic texts written by themselves or others would not call peer review an exact science. This is most clear to us when the manuscripts we have submitted to scientific journals receive such contradictory responses. Even though this happens within formal frames, it is clear that just like many of the other assessments that are made in life, there is a considerable subjective component also in scientific peer review.
Self-authorisation
The inability to have a text read objectively, free of context, is nevertheless a poor argument for rejecting the process or considering all peer review to be equally valid. On the contrary, this is an argument in favour of focusing on the frames of peer review, and how they impact on the scientific production of knowledge.
Our example raises the question of whether the referees of the BEDREHELSE programme, through their epidemiological positioning, help encapsulate the phenomenon public health primarily as a matter for their own discipline. It is our opinion that an expansion of the diversity of perspectives and methods requires referees who are familiar with the perspectives and methods that are presented. There is a ‘solution’ to this qualification challenge in the “Assessment of grant application submitted to the Research Council of Norway” form, in that the referees of the BEDREHELSE programme authorise themselves when they tick “Yes” in the box for the question: “- I am/We are qualified to conduct this assessment”.
Abels tårn [Abel’s tower] is one of national broadcaster NRK’s excellent programmes on research journalism and dissemination. The element gold was the subject of one of the programmes, and we watched an excellent exemplification of the value of diversity of perspectives in science (6). With gold as the empirical pivot point, we were able to shift between the lenses of physics, geology, history, anthropology, economics and other scientific traditions. Each one illuminates gold in a different way, and together they provide a broader and deeper understanding of the phenomenon. If we swap gold with public health as the empirical point of intersection, we also see the potential offered by a diversity of perspectives and interdisciplinarity.
Today both public health and health in general are empirical fields in many scientific traditions. Nonetheless, it seems as if there is little discourse, reading or research across academic barriers. We believe that bringing different perspectives together and into a dialogue with each other is productive. In order to achieve this, it will be necessary for programme announcements to put greater priority on the diversity of perspectives in science, and for researchers to incorporate different perspectives in their projects. The assessments of projects’ relevance and quality also requires peer review to be managed in such a way as to reduce the risk of unqualified and protectionist reading. The alternative is to develop discipline-based, methodological ownership of empirical fields, with the unfortunate result that people think too similarly and too little.
