Education authorities, students and employers are placing increasing demands on quality in higher education. A large emphasis is also being placed on students taking an active role in the development of the teaching (1 ). Student surveys suggest that medical studies in Norway are not fulfilling this goal (2 ). The medical degree programme at the University of Oslo scores relatively low on both the quality of teaching and on the students’ opportunities to influence the development of the study programme (2 ).
In the autumn of 2016, a group of students on the medical degree programme at the University of Oslo took the initiative to evaluate parts of the teaching. They were particularly interested in the lectures and prepared a criteria-based evaluation form for providing feedback to the individual lecturers. The model was based on the students’ experience with so-called objective structured clinical examination (OSCE). This is a type of examination in which the students themselves are evaluated based on objective criteria for good clinical practice. The group devised a form with corresponding criteria for a good lecture. The form was completed jointly by 1–3 students immediately after each lecture and sent by email to the relevant lecturer.
As part of the effort to further develop a system for quality in teaching, the faculty management decided to carry out a pilot project to evaluate lectures based on the students’ method. In this article, we describe this project, the method of evaluation, the results of the evaluations and the students’ and lecturers’ perceptions of the scheme.
Material and method
Twice a year, approximately 110 students are admitted to the medical degree programme at the Faculty of Medicine, University of Oslo. In the period 2013–17, the faculty carried out a comprehensive reform of the study programmes with the goal of ‘improving quality and providing more relevant teaching in line with new competence needs’. This effort entailed defining learning outcomes for all subjects (3 ).
In this project we have evaluated lectures in the first and second academic year (modules 1 and 2), which have a special emphasis on the basic medical sciences. The lectures in module 1 and in the first part of module 2 were organised jointly with the degree programme in dentistry or clinical nutrition.
Evaluation method
The primary purpose of the student evaluations was to provide the individual lecturers with structured feedback on the strengths and weaknesses of their lectures, with a special emphasis on pedagogic factors. Formative evaluations of this nature focus on dialogue and development, and differ from more summative methods of evaluation, which are aimed at checking and documenting the quality of the teaching (4 ). However, by making the evaluation systematic and the results generally accessible, as through this article, the project also involves a degree of control and demonstration of quality.
The students’ independently developed evaluation form was based on well-known and largely objective principles of pedagogy and lecture techniques, such as agenda-setting, adjusting the volume of information, student activation and punctuality (5 –8 ).
Following a revision of the questions together with members of the faculty and the cohort’s elected representatives, a simple online form was developed for evaluating both whiteboard/blackboard-based and PowerPoint-based lectures. The form also included open-ended questions about the positive aspects of the lecture and what could be improved (see Appendix 1 ). An additional question was added, asking whether the lecture was recorded on video.
Evaluation procedure
The evaluation procedure was developed in collaboration with student representatives and the study coordinators for modules 1 and 2. At the start of the semester, the associate dean of education and the study coordinators invited the cohort’s elected representatives to a dialogue meeting, primarily to introduce new representatives to the evaluation system. The study coordinator established and administered the evaluation form in the university’s online survey portal and posted a link to the evaluation form on the student cohort’s web pages. The cohort’s elected representatives organised a ‘rotation’ system, where groups consisting of three students were tasked with evaluating a lecture. They jointly completed the online form immediately after the lecture and shortly thereafter sent the completed form to the lecturer with a standardised salutation.
The evaluations for each module were archived and could be retrieved by the study coordinator for further analysis and reporting.
Students’ evaluations of lectures
The evaluations of the lectures in modules 1 and 2 in spring 2017, autumn 2017 and spring 2018 were collected and summarised. The evaluations in module 1 were somewhat incomplete since the scheme required training and incorporation into the new student cohort’s timetable. The students’ free-text responses were mapped with a simple qualitative thematic analysis by two of the authors (Stokke and Breivik), who compared the results and established a consensus.
Students’ and lecturers’ assessment of the scheme
We conducted a semi-structured group interview with the four student representatives for modules 1 and 2. We asked about their perceptions of the students’ attitudes to the project, about their experiences with organising and carrying out the evaluation, whether problems had arisen, and whether they had any ideas about how to further develop the scheme. We also asked if they had observed an improvement in the lectures.
In order to obtain the lecturers’ views and experiences, we conducted an anonymised online survey (see Appendix 2 ). The quantitative data were summarised using descriptive statistics, while the free-text responses were subjected to a simple qualitative thematic analysis (as described above).
All data were collected pursuant to the Act relating to Universities and University Colleges and processed in accordance with the Personal Data Act.
Results
During the project period, the students evaluated 445 of 1663 lectures (27 %); 167 (20 %) in module 1 and 278 (33 %) in module 2. The results of the criteria-based assessment are presented in Figure 1. The majority of the lectures met most of the criteria in the evaluation form, but a significant proportion were not adapted to the volume of information (21 %), did not point out the association with the learning outcomes in the subject (32 %), failed to activate the students (31 %) and did not provide a final summary (40 %).
Figure 1 The students’ evaluation of 445 lectures in modules 1 and 2 in the medical degree programme at the University of Oslo 2017–18. The question ‘Was the lecture linked to the learning outcomes?’ refers to whether the lecture pointed out the association with the learning outcomes in the subject. The question ‘Did the lecturer use active learning tools?’ refers to whether the lecture included elements to activate the students.
The students’ comments in the free-text field showed that they generally had a favourable impression of the lectures they had evaluated. Lecturers who used a blackboard/whiteboard and active learning tools received particularly positive feedback, and the students called for more of this. The criticism was largely related to a lack of structure, too much information and a lack of association with the learning outcomes in the subject. The students also reported that some lectures overlapped, and that there was a need for better coordination between the lecturers.
The students’ comments generally appeared to be constructive and respectful. Some feedback was relatively harsh and direct, but none was considered to be unpleasant or aggressive.
Students’ assessment of the scheme
The cohort’s elected representatives concluded in a group interview that the students generally had a positive attitude to the scheme, but also observed that not all of the students saw its value. Many of the lectures were not evaluated, and they said this was because it was difficult to maintain the commitment to evaluate, particularly late in the semester. The cohort’s elected representatives believed that the evaluation scheme had a positive social effect due to the fact that evaluation groups were made up of students from different groups of friends.
In terms of suggestions for improvements, it was proposed that the results of the evaluations should be made accessible to all students, not just the student representatives, so that more people can see the utility value of the scheme. The student representatives also called for a better technical solution, for example a mobile app for evaluations.
Lecturers’ assessment of the scheme
Of a total of 88 lecturers, 52 (59 %) responded to the survey about the student evaluations. Of these, 48 had a generally positive attitude to student evaluation. Thirty-seven stated that they found the feedback useful, and 29 had made changes to their lectures based on the evaluations. Four stated that they had received evaluations they perceived to be unpleasant or unreasonable.
The lecturers were divided with regard to the method of evaluation. While 28 liked the criteria-based solution, 24 would have preferred a more subjective evaluation of the content and quality of the lecture based on feedback from a representative sample of students. Many highlighted the rapid and direct feedback from students as a particularly positive aspect, and several reported that the free text responses were the most valuable part of the evaluation.
When asked how to best meet the challenge of lecturers receiving negative feedback, some responded that the lecturers themselves should be responsible for following up the evaluations, while others suggested guidance from colleagues and training courses. Some argued that the follow-up of the lecturers was a management responsibility and suggested that the associate dean of education should take up the matter with the individual lecturer.
Discussion
This study shows how the medical degree programme at the Faculty of Medicine in Oslo has developed and assessed a system for students’ evaluation of lectures. Both the students and the lecturers had a positive attitude to the scheme, and we have placed a large emphasis on sharing and discussing the results with the different academic communities. Most of the lecturers said the evaluations were useful, and more than half had made changes to their lectures based on the feedback. Given that most lecturers met most of the criteria in the form and generally received good feedback, we consider this to be a very positive result.
The students had chosen to focus on the lectures, which is a classic but also criticised form of teaching. Lectures are effective in the sense that one teacher can teach a large number of students, but they are also highly dependent on this person’s communication skills. Moreover, lectures are largely based on one-way communication and have limited potential for activating critical thinking and problem solving among the students. A number of studies show that lectures generally lead to poorer learning outcomes than more active learning methods (9 ). Accordingly, there is an international trend in which lectures are being replaced with student-activating teaching methods, such as team-based learning or flipped classrooms (10 ). Nevertheless, lectures still play an important role in many study programmes, including the medical degree programme at the University of Oslo. In this project, therefore, we have given the lecturers structured feedback on how they can improve their lectures, where one of the quality criteria was the use of student-activating elements.
Student evaluation as a measure of quality in education is associated with considerable uncertainty and should be interpreted with caution (4 ). Student evaluations can entail elements of a popularity contest that does not necessarily reflect the true quality and learning outcome of the teaching. This criticism is supported by studies that have examined the validity of summative student evaluations. A common weakness concerns a lack of validation of questionnaires. Students may be biased in areas such as gender and ethnicity, and evaluations conducted after the completion of a course may lack nuance and be influenced by the result of the exam (11 ).
The purpose of this project was therefore not to introduce student evaluation as an objective measure of quality in education, but to provide the individual lecturer with feedback on how the teaching can be improved. Summative evaluation of the individual teacher can be perceived as a form of control and criticism, whereas our intention was to evaluate the lectures in a way that was constructive and supportive. We assume that an evaluation form that subjectively scores the lecture on a scale from 1 to 5 would have been met with more resistance. It was also essential that the evaluation was given immediately after the lecture, directly from the students to the lecturers. The students’ initiative and involvement thereby helped to legitimise the scheme in the academic communities.
Another success criterion was that the scheme was not labour-intensive. From the lecturers’ perspective, it entailed an email from the students with specific suggestions on how the lecture could be improved. For the individual student, all that was required was 5–10 minutes to evaluate a lecture 2–3 times each semester. The task of the study coordinators was to create an online form and convene meetings with the cohort’s representatives once per semester. The main task was that of the cohort’s representatives, who organised the rotation of students and sent regular reminders to ensure that the evaluations were carried out. However, despite all their efforts, less than a third of the lectures were evaluated, and it is therefore important to find solutions that both simplify and stimulate the role of the representative as an advocate and a motivator for the continuation of the scheme.
The technical solution we chose for the evaluations was the University of Oslo’s data collection program Nettskjema . This ensured internal and secure processing of the information. During the project period, we faced challenges related to how the students could send evaluation forms anonymously to the individual lecturers, and the students chose to forward these via an external email address. This problem was solved, and we are now working on technical solutions that further simplify the evaluation procedure. In the future, we envisage the evaluation of several types of teaching, in addition to lectures, by means of a mobile app.
Strengths and weaknesses
The greatest strength of this quality development project is that it builds on the students’ commitment to improve the teaching. They have defined problems in the lectures and have initiated a constructive approach to solving them. We can also point to a relatively high response rate (59 %) in the survey among the lecturers, a large majority of which took a positive view of the evaluation of their teaching.
The project has largely been developed during the process, in a relatively complicated organisation and based on a combination of internal and external influences. It first and foremost describes a process, and not a carefully planned intervention. We present quantitative and qualitative data, collected over a limited period of time, which suggest that our intervention has helped to improve the teaching. However, we have no indication of whether the students’ learning outcomes have improved as a result of the scheme. Such an analysis would require a longitudinal study with a more rigorous research design.
Conclusion
This study has shown how student evaluation can help improve the quality of lectures in the medical degree programme. Such student evaluation is not necessarily a good measure of quality in teaching, but it can be a useful tool for dialogue and collaboration in order to develop such quality. Thereby, we have also taken an important step in the development of a ‘culture for quality’, as called for in the government report on quality in education (1 ).
