In the medical studies at the University of Oslo, the teaching of knowledge management is integrated into the course «Knowledge management, leadership and quality improvement», abbreviated KLoK (1 ). In their tenth semester, the students go on a twelve-week clinical placement period in hospitals or in general practice, where they are required to complete individual assignments for the KLoK course. The teaching of knowledge management has traditionally been based on critical appraisal of scientific articles, but experience indicated that the students failed to acquire sufficient skills in asking precise questions, searching for knowledge in suitable databases and integrating research evidence in their own clinical practice. This recognition led to a request for changing the teaching of knowledge management. Systematic reviews of various types of educational programmes indicate that clinically integrated teaching, with a focus on patient-related issues and written documentation of the process, provides knowledge and skills in knowledge management (2 – 4 ).
We developed a new learning programme, named «Evidence-based practice for medical students with the aid of e-learning resources» (KUPMESTER). During their period of clinical placement in their tenth semester, the students must complete two so-called work files to document the process leading from a clinical question to a final answer. A work file is an electronic form (MS-Word document), where the student fills in the clinical question, literature search, the search results, the selected article, a critical appraisal of the article and a conclusion with a solution to the problem, as well as an assessment of whether there is a need to change practice in the department. The students are offered supervision during the process, and are encouraged to make use of the online course www.kunnskapsbasertpraksis.no
In this article we report the results from the use of the work files for a cohort of medical students to assess their skills in formulating questions, searching for evidence and critically appraising guidelines and studies. We also report to what extent the students found answers to their clinical questions and the number of cases for which they concluded that a change of practice was warranted.
Material and method
KUPMESTER was developed in 2009 as a project headed by two medical students (the first author and Øystein Andersen) under the supervision of the second author and Arild Bjørndal, who is responsible for teaching of knowledge management. The assignment has since been permanently included in teaching during the tenth semester. The data material is based on work files submitted in the spring semester of 2011.
Teaching prior to the clinical placement period
During the seventh semester, the students attend a four-day course in evidence-based clinical practice. The course ends in a knowledge test that must be passed before the ordinary semester exam. The tenth semester includes two hours’ teaching of evidence-based practices in an introduction seminar for the KLoK programme during the week prior to the start of practical training. The teaching provides a brief repetition of the steps involved in evidence-based practice and an explanation of how the work files should be completed. Commonly, less than half of the students attend these training sessions in the seventh and tenth semester.
The students were recommended to undertake a literature search with search terms taken from a PICO question. We provide a more detailed introduction to the formulation of questions, literature searches and critical appraisal in a «Perspectives» article in this issue of this journal (5 ). The first step in the cycle from question to answer is to formulate a precise «PICO question» with a precise description of the patient/population (P), treatment alternatives (Intervention: I and Comparator: C) and outcomes/clinical end points (O). The next step is to undertake a literature search. The students are recommended to perform a «pyramid search» in the search engine McMaster PLUS with 1 – 2 simple search terms drawn from the PICO question. This is a search engine that scans relevant databases and ranks the answers at various levels of the knowledge pyramid (5 – 7 ). McMaster PLUS is thus an aid to finding pre-appraised systematic reviews and individual studies deemed to be valid and interesting to health workers. The students are encouraged to search for answers from the top to the bottom of the knowledge pyramid and select studies that have undergone quality assessment over those which have not.
A critical appraisal of validity, results and applicability should be undertaken for the selected guideline or study. The students were encouraged to use the checklists available at www.kunnskapsbasertpraksis.no
The clinical placement period
In their tenth semester, the students complete a six-week clinical placement period in ten collaborating local hospitals. No other teaching takes place during this period. The students follow normal working hours and participate in on-call duty in the hospital to which they are assigned. Of the total of 107 students in this semester, altogether 91 participated in the project. We received no work files from the remaining 16 students.
Each student was asked to complete two work files on the basis of a clinical question from their own practice, preferably a question related to treatment effect. The assignment was assessed and approved by the teacher (clinical placement teacher) at the local hospital before it was submitted to the professor at the Faculty of Medicine. The teachers had received no systematic instruction in knowledge management.
The students were encouraged to make use of the online training course at www.kunnskapsbasertpraksis.no 8 ). The online training course provides an introduction to key concepts in the six steps involved in evidence-based practice, through text modules, video lectures, assignments and literature references. We also included a question about the need to change existing practice on the basis of the conclusion in the work file. This question leads on to the KLoK assignment in the 11th semester, in which the students must develop a quality improvement project on the basis of their experience from the work placement period (1 ).
Work files
Work files for clinically integrated learning of knowledge management have been developed by the last author since 2006 as a learning tool for trainee doctors and later also for physiotherapists in Norway (Vandvik PO, Bjørndal A, lecture at the 3rd International Conference of Evidence-Based Health Care Teachers and Developers, Taormina, 2007; Olsen NR, Vandvik PO, Larun L. et al, lecture at the 5th International Conference of Evidence-Based Health Care Teachers and Developers, Taormina, 2009). The work files are intended to provide health personnel with practical skills in knowledge management based on sufficient theoretical knowledge. Concurrently, the work file is a tool for interaction between the students, the teachers and the professors, and it also enables evaluation of practical skills in knowledge management. We refined and tested these work files on several cohorts of medical students in their tenth semester during the period 2009 – 10.
Figure 1 shows the work file (which has been developed in Microsoft Word) as completed by a medical student (ER). The work file contains all the steps in the knowledge circle (5 ).
Figure 1: Example of a work file completed by a medical student (ER), with feedback.
To evaluate the students’ skills by way of the completed work files, we categorised each field in the file in accordance with the steps involved in evidence-based practice (5 ). We developed various response categories, as illustrated in our reporting of results. For example, the field «What type of core question?» contained four response categories (effect question, diagnosis question, prognosis question and «other»). For statistical processing, a numerical value was assigned to all response categories in each field. Work files with identical values in the same field could thus be summarised and used to report response proportions.
The conclusion in the work file was deemed to be satisfactory if the student had undertaken an adequate interpretation of the results from a relevant article, based on a subjective interpretation. Categorisation of the response alternatives was undertaken by the medical student (ICK) and subsequently reviewed with the responsible professor (POV). Work file no. 2 was selected for assessment and categorisation.
Results
Altogether 90 of the 91 work files described understandable and relevant clinical problems. A total of 76 work files described effects of treatment, while the remaining focused on diagnostics and prognosis.
On the basis of the clinical problems, the students developed focused questions in PICO format: 70 work files contained all four components; in 14 of them one element was missing and in seven of them two elements were missing. Population, intervention and control group were satisfactorily defined in 84, 85 and 72 work files respectively. As regards definition of patient-related outcomes, 13 work files failed to define an outcome, whereas 60 files defined one outcome, 17 files defined two and one file defined 4 – 5 outcomes. Figure 1 illustrates that the student identified only one relevant outcome.
Before the students undertook the literature search, they were asked to identify the sources that they would normally use to find answers in their usual practice. Altogether 31 would have asked experienced colleagues, 24 would have looked in textbooks, 19 would have searched the Internet and 15 would have searched for local guidelines.
In the 91 work files, literature searches were most frequently undertaken as a pyramid search in McMaster PLUS (n = 42), followed by direct searches in Up To Date or Best Practice (n = 17), PubMed or other databases (n = 15) and searches in unnamed databases (n = 17). Answers were most frequently found in evidence-based textbooks or guidelines (n = 43), followed by primary studies (n = 30) and systematic review articles (n = 15). Only three students found answers in unreported sources.
Altogether 78 of the 91 students performed a critical appraisal of the results. In the assessment of relevant issues related to validity, 29 files contained no issues, 12 contained one issue, eight contained two issues and 29 contained three or more issues. At least one effect estimate was reported in 46 work files, of which ten files included absolute as well as relative effect estimates. As regards the assessment of whether the intervention was applicable in their own practice, altogether 58 work files reported at least one relevant issue.
A total of 89 of the 91 students claimed to have found an answer to their question. We concluded that 84 of these provided an answer with an adequate and satisfactory conclusion. Figure 1 shows how this student misinterpreted the documentation and how the professor communicated this back to her.
50 of the 91 students reported what was common treatment practice in the department where they spent their work placement period. In 17 work files, the student concluded that existing practice should be changed. This included some students who failed to describe the common treatment practice in their department
None of the students reported to have contacted a professor or a librarian during their work with the file. Only five of them reported how many hours they had spent on working with the file. Figure 1 illustrates how this student received feedback on elements in the work file that needed improvement.
Discussion
Our evaluation of 91 work files submitted by a cohort of medical students in their tenth semester provides a good basis for claiming that the students acquired basic skills in knowledge management, in line with the goals of the teaching. As such, the work files are a promising tool for clinically integrated learning of knowledge management as well as for evaluation of skills.
During their work placement period, the majority of the students formulated specific questions in PICO format for clinically relevant problems. In our teaching, we emphasise the importance of assessing multiple outcomes when the advantages and disadvantages of treatment alternatives are to be considered (9 ). However, only a minority of the students reported more than one outcome of importance to the patient. This could be because the students are unfamiliar with all outcomes that can be expected from the treatment, or else they forget that treatment may also lead to undesirable consequences/adverse effects. However, some of the students described more outcomes in their reporting of the results than what they had defined in the PICO question. This is an indication of an active learning process through the steps in the knowledge circle.
Experienced colleagues and textbooks provided the most important sources of answers to clinical questions. This is in line with previously published studies (10 ). Literature searches were most frequently undertaken for effect questions with a pyramid search in McMaster PLUS. This tool is presented during the teaching, and this finding indicates that the students proceed in accordance with what they have been taught and recommended in advance. Answers were found in evidence-based textbooks or pre-appraised studies for two-thirds of the clinical questions. This finding indicates that the students have acquired good knowledge in identifying the best current research evidence. Questions related to diagnostics and prognosis presented the students with greater challenges in terms of formulating specific questions, selecting a database and critically appraising the articles. This latter finding indicates that the teaching that precedes the work placement period should place greater emphasis on diagnostics and prognosis.
In spite of the critical appraisal undertaken in 78 of the 91 work files, our evaluation identified weaknesses in the description of potentially systematic sources of error and effect estimates. This applies to the reporting and interpretation of results in particular. Sufficient understanding of both absolute and relative effect estimates as well as of statistical uncertainty are required skills in knowledge management (11 ). The findings indicate that the students need further training in critical appraisal. A possible explanation could be that only a minority of the students attend teaching sessions during the 7th semester where critical appraisal is a key topic, but also that the students made little use of the online training course and the opportunities for supervision during this work. Students should receive better information on how this training is crucial to undertake critical appraisal successfully, and that the teaching is relevant for the assignment with the work files during the 10th semester and the KLoK course in the 11th semester. Mandatory attendance of teaching sessions is another potential solution.
A need to change prevailing clinical practice was reported in nearly one-fifth of the work files. This illustrates the potential for quality improvement through searching for answers to clinical questions, as well as the need to stay updated with regard to new research evidence. This finding is somewhat uncertain, since only one-half of the students reported current clinical practice in their department. Several work files have been used for quality improvement projects for the KLoK course in the 11th semester, and some of these projects are in the process of being implemented in practice.
We believe that the study provides us with reasonably reliable knowledge about the students’ acquisition of knowledge management skills through integrated learning with work files. We deem the results based on work files submitted by 91 of 107 students to be representative, since the absent submissions were caused by a misunderstanding among some teachers. Weaknesses of the study include, for example, the choice of study design, which is not suitable for measuring the effect of this learning programme. A randomised, controlled study, in which only one-half of the students underwent the learning programme, was not considered to be feasible in our educational programme.. Measurement of the students’ knowledge and skills in evidence-based practice before and after the learning programme has been considered, but was rejected in the absence of available valid tools for such measurement..
Another weakness to consider is the students’ failure to report needs for supervision and use of the online training course during their completion of the work files. We have the impression that students who receive supervision make adequate improvements to their work files and acquire better skills in knowledge management. It is also necessary to ensure that the teachers in the local hospitals possess sufficient competence in knowledge management to permit supervision during the process as well as correct assessments during their approval of the work files. Regular Having supervisory sessions with the students during their work placement period may help provide quality assurance of the learning process.
Conclusion
This study shows that at the end of their medical studies, the majority of the students are able to formulate understandable and adequate clinical questions. The students demonstrated good knowledge about literature searches, and six out of ten students found the answer to their clinical question in pre-appraised research evidence. The critical appraisal of the articles was occasionally inadequate, and such skills should therefore be given increased emphasis in teaching. More frequent use of the online training course and supervision during the completion of the work files may further improve the student’s skills in knowledge management.
