How can the examination failure rate be stabilised?

Material and method

Results

The current 65 % method resulted in variations of up to 12 % in the failure rate for the same examination in the course of study during the period investigated (Figure 1, Table 2). For example, in the examination in the 5th–6th semester in 2010–11, there were no failures, whereas in 2015, altogether 11 students (12 %) failed (Figure 1a). The failure rate declined during the course of study (Figure 1b) and remained the same for all standard-setting methods.

Figure 1 a) Proportion of medical students who have failed the annual examinations at the Norwegian University of Science and Technology (NTNU) 2010–15 with the current standard-setting method of 65 % correct answers needed for a pass grade. b) Failure rate (%) of four classes monitored over time; class 09 started in 2009, class 10 in 2010 etc. The x axis shows the examinations from which the data have been retrieved, while the y axis shows the failure rate in per cent.

The limit for a pass grade with the ‘original’ and ‘modified Cohen’ methods with K values of 0.65 and 0.70 was lower than with the ‘absolute 65 %’ limit. Use of a ‘modified Cohen’ method with a K value of 0.75 caused the limit for a pass grade to waver around the limit in use today (Figure 2). Using the ‘original Cohen’ the limit for a pass grade amounted to 57–65 % and with the ‘modified Cohen’ to 53–68 %, depending on the K value applied (Table 3).

Figure 2 Comparison of the limits for a pass grade in all examinations of medical studies at the Norwegian University of Science and Technology 2010–15 with use of different standard-setting methods. Each column represents an examination. The Y axis shows the limit for a pass grade, the X axis shows the various methods.

With the exception of the examination in the first year of study, the failure rate was lower with both the original and modified Cohen methods when compared to the current method (Figure 3, Table 3). ‘Original Cohen’ produced the same range in the failure rate as the ‘absolute 65 %’ (0–13.7 %), but reduced the average proportion of failures from 5.2 % to 3.9 %. ‘Modified Cohen’ with K values of 0.65 and 0.70 reduced the standard deviation (SD) and the total failure rate. ‘Modified Cohen’ with a K value of 0.70 has an average failure rate of 3.0 % (SD 3.1) and a range of 0–10.4 %, while the ‘absolute 65 %’ has an average failure rate of 5.2 % (SD 4.2) and a range of 0–13.7 %. ‘Modified Cohen’ with a K value of 0.75 produced a wider range in the failure rate than the ‘absolute 65 %’ method, and thus also a higher standard deviation (Table 3). The standard deviation of the failure rate decreased with a lower limit for a pass grade.

Figure 3 Average failure rate in per cent including the standard deviation for each examination with use of the different standard-setting methods at the Norwegian University of Science and Technology (NTNU) 2010–2015. The X axis shows the examinations in the course of study, while the Y axis shows the average failure rate

Discussion

We found that the proportion of medical students who fail their examinations varies from one year to the next in the same examination in the course of the medical studies. We have shown that the standard deviation of the failure rate can be reduced by using Cohen’s methods, but that this comes at the cost of a lower limit for a pass grade.

The study shows that the failure rate declines as the medical studies progress. We have not investigated the causes of this phenomenon, but they are likely to be multiple. The medical studies programme in Trondheim practises spiral learning, meaning that the same topic is studied approximately every other year. For example, cardiac physiology with clinical examples is taught during the first year and cardiology in the third year, with summary of cardiology in the final year. Through this spiral learning the students will deepen their understanding of the subject. Furthermore, from upper secondary school the students have become accustomed to a clearly defined syllabus and frequent testing. The medical studies programme at the Norwegian University of Science and Technology (NTNU), on the other hand, has learning objectives with a number of recommended textbooks and only 1–2 annual examinations which may test anything. The change in study technique will thus be a major challenge to many students, although most of them appear to learn how to cope with this during their studies. In addition, the attrition rate is highest during the first two years of study, when 2–6 students need to retake a year, while only a maximum of one student needs to retake any later years. Of those who quit or have their admission revoked because of repeated examination failures, altogether 73 % (101 of 137 based on figures from 1999–2016) had not completed the second year of study (personal communication, Mona Dalland Stormo and Marte Laugen, Student and Academic Section at the Faculty of Medicine and Health Sciences, Norwegian University of Science and Technology (NTNU). Other factors that can be assumed to contribute include experience of examinations, increasing age and the subjects being perceived as more relevant at later stages of the studies, which may help boost motivation.

The study by Cohen-Schotanus compared two cohorts at two different medical faculties in the Netherlands (5). One faculty used a reference-based method, and the average score minus one standard deviation was used to define the limit for a pass grade. This limit hence varied between 15 % and 46 %, while the failure rate remained relatively stable at 17 %. The other faculty used a pre-defined 60 % limit for a pass grade, and the failure rate amounted to 17–97 % (53 % on average). It is conceivable that the students who were subject to a higher limit for a pass grade were more knowledgeable. However, the students in these cohorts performed equally well in the national progress test which is implemented in six of the eight medical faculties in the Netherlands (5). However, students at the faculty with a 60 % pre-defined limit for a pass grade and a high failure rate spent on average one more year to complete their studies. Considering that these cohorts were equally knowledgeable in the national test, this indicates that pre-defined, absolute limits for a pass grade are a waste of public resources, and not least the students’ time and resources (5).

Both the original and modified Cohen methods reduced the standard deviation of the failure rate. With the use of these methods, fewer students would have failed. We were surprised to see that the opposite never occurred, i.e. that more students failed in examinations that nobody with an absolute limit for a pass grade failed. We believe that the Cohen method that ought to be chosen is the one that produces the largest reduction in the standard deviation of the failure rate, but the smallest change in the limit for a pass grade. The objective of this is to avoid lowering the difficulty level of the examinations while seeking to reduce the range of the failure rate. In our material, this would have been achieved with a ‘modified Cohen’ method and a K value of 0.70. This sets the limit for a pass grade at 70 % of the point score of the students in the 90th percentile.

It is difficult to assess the quality of a standard-setting method, since it is hard to tell where the ‘true’ limit to a pass grade should be for each individual examination. Cohen’s methods have the advantage of being predictable for the students, since they will know that they never need a higher percentage of correct answers than the stated K value (provided that those in the 90th or 95th percentile score perfectly). They will also know that the method corrects for the degree of difficulty of the test and that the test is not subject to the discretionary judgement of an examination committee. Let us assume that the faculty decides to use the Cohen method with a K = 0.70. Those who achieve 70 % of the scores of the students in the 90th percentile will pass, those who score lower will fail.

Another advantage of Cohen’s methods when compared to other relative methods is that they do not produce a fixed failure rate. We feel that it would be problematic to introduce a standard-setting method that lowers the existing 65 % absolute limit for a pass grade. Although the failure rate varies, the number of medical students who fail annually is nevertheless small compared to other study programmes (5, 6).

Absolute methods that involve expert panels are probably the solution most likely to produce the ‘true’ limit for a pass grade on a medical examination 2, 3). This solution is used in many places, including the United States Medical Licensing Examination (USMLE) (8). In practice, however, it would be hard to implement for all examinations in each medical faculty. Based on our findings, we believe that a 65 % absolute limit for a pass grade can be defended for as long as the failure rate remains as low as today. A standard-setting method needs to have credibility. If the variation in the failure range from one year to the next becomes excessive when testing a homogenous group of students assessed according to the admission criteria for medical studies, the examination loses its credibility (4). Cohen’s methods should be used in medical schools with an extremely high failure rate, or where there are major variations in the failure rate for the same examination in the course of study. We believe that the methods could be suitable at the Norwegian University of Science and Technology (NTNU) if the failure rate for examinations deviates considerably from what is common now.