Article review: Evaluating your written evaluation of a learner

As a new faculty, one of the first challenges that I encountered was completing evaluation forms for medical students and residents. In our department, a Daily Evaluation Card (DEC) is to be completed at the end of every shift for each learner. These DEC’s are then collated by the program directors to yield a summative final rotation evaluation.

What I wondered was: how can I best use these DEC's to help learners progress as medical professionals and at the same time provide critical information for the PD’s?

Fortunately, I stumbled upon a 2008 Medical Education paper called “Assessing the quality of supervisors' completed clinical evaluation reports” by Dr. Nancy Dudek (University of Ottawa).  This article was what I was looking for. Although this article was intended to evaluate the quality of the summative evaluation, the principles remain applicable to the DEC's.

The article is summarized below:

• End-of-rotation evaluations usually consist of a checklist/rating scale and written comments. These forms have questionable reliability and validity.
• End-of-rotation evaluations remain a valuable resource when trying to assess what a trainees "actually do" versus what they "can do" (eg. on an exam).
• The study attempted to determine the features of a high-quality evaluation and to develop an instrument to assess its quality.

Methodology:

Using brainstorming and a modified Delphi consensus technique, a focus group developed a Completed Clinical Evaluation Report Rating (CCERR) form. This form was then tested nationally and revised to yield a tool which evaluated 9-items each on a 5-point scale. This CCERR tool was found to be a reliable and valid means to differentiate superior from average from poor end-of-rotation evaluations.    

The 9-item CCERR checklist:

How would your own Daily Evaluation Card evaluations fare? 

Use a 5-point scale (1 = not at all, 3 = acceptable, 5 = exemplary).

1. Checklist/ numeric ratings show sufficient variability to allow identification of relative strengths and weaknesses of the trainee.
2. Comments are balanced providing both strengths and areas for improvement.
3. The trainee’s response to feedback and/or remediation during the rotation is described in the comments.
4. Comments justify the ratings provided.
5. Clearly explained examples of strengths using specific descriptions (not generalizations) are provided in the comments.
6. Clearly explained examples of weaknesses using specific descriptions (not generalizations) are provided in the comments.
7. Concrete recommendations for the trainee to attain a higher level of performance are provided.
8. Comments are provided in a supportive manner.
9. Overall, this end-of-rotation evaluation provides enough detail for an independent reviewer to clearly understand the trainee’s performance on the rotation.

Reference

Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisors' completed clinical evaluation reports. Med Educ. 2008 Aug;42(8):816-22. 

.

.