MEdIC Series | The Case of the Awkward Assessors – Expert Review and Curated Commentary

The Case of the Awkward Assessors outlined a scenario where faculty members are put in a difficult position as they try to provide negative feedback to a medical student working in the ED. What did the ALiEM community think of this case? This month the MEdIC team, led by Brent Thoma (@Brent_Thoma) and Teresa Chan (@TChanMD) hosted a MEdIC series discussion around this issue with insights from the ALiEM community. We are proud to present to you the Curated Community Commentary and our 2 expert opinions. Thank-you to all our participants for contributing to the very rich discussions last week.

This follow-up post includes

• Posts by our solicited expert respondents: Dr. Karen Hauer, Associate Dean for Assessment and Professor of Medicine in the School of Medicine at the University of California, San Francisco and Dr. Inna Leybell, Assistant EM Residency Program Director at NYU/Bellevue Medical Centers
• A summary of insights from the ALiEM community derived from the Twitter and blog discussions
• Freely downloadable PDF versions of the case and expert responses for use in continuing medical education activities

[su_spoiler title=”Expert response 1: Effective feedback delivery: Before, during, and after” style=”fancy” icon=”chevron-circle”]

Dr. Karen Hauer
Associate Dean for Assessment and Professor of Medicine in the School of Medicine at the University of California, San Francisco

Tom and Patrick’s scenario is realistic and occurs commonly in medical education. The fast pace of the emergency department and the often-limited contact between students and their supervisors make feedback and evaluation challenging.

Feedback is specific information about the comparison between a trainee’s observed performance and a standard, given with the intent to improve the trainee’s performance [1].  Students commonly receive verbal feedback that they are ‘doing great’ and should ‘keep reading’. This type of general feedback leaves them feeling frustrated that they haven’t received actionable feedback. They may also be surprised to find that their evaluations aren’t aligned with the verbal feedback they received from their supervisors. Students like and prefer praise, but learn more from constructive feedback [2].

Examination of the feedback given by Tom to Patrick shows several ways in which this feedback falls short.

• Specific: Toms’ feedback is very general: ‘Good’ and ‘great’ don’t specify why or how Patrick’s performance was good (or not).
• Behavior-based: Pam notes that Patrick is ‘nice enough’, but the feedback should be about specific behaviors, not the learner’s personality.
• Comparison to a standard: We do not know if Tom explained expectations to Patrick, or if Patrick knew what was expected of him.
• Intent to improve: Patrick has not been given any instructions about how to improve on his next shift or his future rotations.
• Learner reflection: Patrick has not been asked how he thinks he is doing. Tom did not even ask Patrick where he had been through the day or how he had spent his time.
• Constructive feedback: Supervisors giving feedback are often hesitant to say anything negative, and as a consequence they may choose to avoid addressing their concerns at all.
• Setting: Feedback should be delivered in an environment that is comfortable for the learner and supervisor, free of distractions and away from others who may overhear. ‘On the fly’ feedback delivered during work should be brief and focused on the task at hand, not an overall discussion of how the student is doing.

Recommendations for providing feedback

Set the expectations: Before giving feedback, it is important to set expectations. The supervisor and student should have shared expectations for the student’s role, including the activities he should be doing, and the responsibilities he should be taking on. Does Patrick know that he is expected to focus on patients in the emergency department instead of learning from consulting surgeons?

Dr. Pam and Tom should discuss expectations with their future students and together determine how the student’s learning goals can best be met. For example, Dr. Pam could ask at the beginning of a student’s clerkship: “What are you hoping to learn in this rotation?” For a student not planning to go into emergency medicine, Dr. Pam would then respond with the kinds of learning activities the student could focus on that would satisfy the clerkship requirements and also address the student’s goals. She should also include learning activities expected of all students in the clerkship, regardless of intended specialty.

Give the feedback: Feedback can be given after a period of time working together – this may be given after part of a shift, or after several shifts. Students often don’t realize that their supervisors are offering feedback; it is useful to label the feedback by starting with, ‘’Now I’m going to give you feedback.’

A useful framework for feedback is ASK TELL AGREE [3]

• Ask the student how she thinks she is doing. This question will give you information about the student’s insight into her performance. She may raise areas for improvement before you need to.
• Tell the student your feedback, based on your observations. Feedback should be about the behaviors you have observed (i.e. patient counseling, physical exam, charting) not personality traits.
• Agree on a plan for improvement for the learner.

Close the loop: The supervisor should return to the feedback in the final summary evaluation and final feedback session, if possible.  This follow-up will close the loop by helping a student determine whether he has made progress on the goals for improvement that were identified during the feedback.

Group evaluations

Tom and Dr. Pam had begun to discuss Patrick’s performance when he walked in on them. This unfortunate setting and timing for their discussion should not dissuade teachers in the emergency department or other specialties from doing group evaluations of learners. Multiple studies have shown that evaluations of learners by a group of evaluators are better than evaluations by a single individual alone [4,5]. Groups consider more perspectives. They can explore and potentially resolve discrepancies, and can calibrate outlier opinions through discussion.  Groups are more likely to identify performance concerns and areas for improvement than individual feedback providers.

In the future in this emergency department, supervisors should designate a time and place for sharing their evaluation opinions and questions. Scheduling can be analogous to ‘signout’ at the end of a shift, or can entail a larger group meeting scheduled in a room where students will not overhear.

References

1. van de Ridder JM, Stokking KM, McGaghie WC, ten Cate OT. What is feedback in clinical education? Med Educ. 2008; 42(2): 189-97. PMID: 18230092
2. Boehler ML, Rogers DA, Schwind CJ, et al. An investigation of medical student reactions to feedback: a randomised controlled trial. Med Educ. 2006; 40(8): 746-9. PMID: 16869919
3. Gigante J, Dell M, Sharkey A. Getting beyond “Good job”: how to give effective feedback. Pediatrics. 2011; 127(2): 205-7. PMID: 21242222
4. Hemmer PA, Hawkins R, Jackson JL, Pangaro LN. Assessing how well three evaluation methods detect deficiencies in medical students’ professionalism in two settings of an internal medicine clerkship. Acad Med. 2000; 75(2): 167-73. PMID: 10693850
5. Thomas MR, Beckman TJ, Mauck KF, Cha SS, Thomas KG. Group assessments of resident physicians improve reliability and decrease halo error. J Gen Intern Med. 2011; 26(7): 759-64. PMID: 21369769

[/su_spoiler]

[su_spoiler title=”Expert response 2: The Art of Feedback” style=”fancy” icon=”chevron-circle”]

Dr. Inna Leybell
Assistant EM Residency Program Director at NYU / Bellevue Medical Centers

Instead of receiving direct feedback, Patrick presumably overheard enough to understand that his performance was being discussed in unfavorable terms. The situation is awkward but salvageable.  Remaining cool and collected, I would move the conversation toward a more appropriate feedback structure. Perhaps I would start with “Tom was telling me that you did a good job with the I&D. Tom, can you give us your input on it before I give you additional feedback?”

After he had given his feedback on the procedure, I would excuse Tom and provide the negative feedback portion in private. I would ask Patrick what else he saw and did that day, confirming that he was with the surgical services and absent from the Emergency Department.  Then I would address the main issue: “I noticed you really like surgeries and procedures, and you did a good job with the I&D. I cannot evaluate your overall performance since you didn’t see other patients in the ED today. That is not acceptable behavior for our rotation. On your next shift, I’d like you to give your best in the ED and get some meaningful comments from your team.”  I would also remind Patrick that specialists benefits from knowing how to approach basic medical complaints. I would also discuss Patrick’s performance with the department’s director of medical student education instead of submitting a blank evaluation. In the future, it would be helpful if the expectations for performance and the relevance of the rotation to students not interested in emergency medicine were made clear during their initial orientation.

Conversation privacy

There are many opportunities for on-the-spot teaching in the emergency department, but it is often difficult to find space and time for in-depth feedback. It is especially difficult when we need to give negative feedback. Literature on effective feedback delivery emphasizes the importance of being in a private safe space [1,2]. For a post-case or end-of-shift feedback session in the ED, I suggest using an empty patient room (with a door), a quiet hallway, an empty resuscitation room/trauma bay, or an empty staff lounge. It can be challenging to find a safe space on a busy shift but the effort to find one will pay off in effectiveness of the session.

Feedback fails

Tom’s response – “You’re doing a good job. You did great with that I&D” -had several critical feedback fails.  To begin with, it was disingenuous and did not have quality feedback’s positive intent of contributing to Patrick’s professional growth. Tom took the easy way out of an uncomfortable conversation.  Tom also did not provide any formative assessment of Patrick’s overall performance or his procedure-related performance on the I&D.  Effective feedback needs to focus on knowledge or directly-witnessed actions that the receiver can change.  General personal praise  “good job” does not provide actionable information. Tom’s feedback would have been more effective if he had included a constructive evaluation of the procedure and an action plan for improvement.

Just like in tango, it takes two to make feedback work.  According to Sadler [3], three key factors need to be in place on the learner’s part: understanding the learning goal, being able to compare performance as objectively as possible with a higher standard, and acting to bring performance closer to the goal.  In this case, Patrick’s critical action fails as a feedback recipient include not understanding performance expectations for the EM rotation, not appreciating his performance was falling far from the goal of the rotation, and not working to close the gap.

EM and non-EM bound students

Standards are higher, scrutiny is greater, and evaluations are tougher for students interested in EM.  Basic clinical evaluation parameters, including obtaining and interpreting H&P data, creating differentials and plans for diagnosis and treatment, and procedural competency, are similar for both EM- and non-EM bound students. I pay special attention to the non-clinical characteristics of the EM-bound crowd – enthusiasm for the field, working hard, working well in a team, responsiveness to feedback – all qualities that will serve one well in the emergency department. Last but not least, professionalism is always important, regardless of the area of interest.

References

1. Bienstock JL, Katz NT, Cox SM, et al. To the point: medical education reviews–providing feedback. Am J Obstet Gynecol. 2007; 196(6): 508-13. PMID: 17547874
2. Schartel SA Giving feedback – an integral part of education. Best Pract Res Clin Anaesthesiol. 2012; 26(1): 77-87. PMID: 22559958
3. Sadler DR. Formative assessment and the design of instructional systems. Instr Sci 1989; 18:119-144. Link

[/su_spoiler]

[su_spoiler title=”The Case of the Awkward Assessors: Curated from the community” style=”fancy” icon=”chevron-circle”]

Curated by Dr. Sarah Luckett-Gatopoulos (@SLuckettG)

This month’s case focused on Patrick, a medical student rotating through emergency medicine. Patrick is not interested in emergency medicine, and spent much of his shift with consultant services. At the end of his shift, Patrick overheard an unflattering exchange about his performance between Tom, a resident, and Dr. Pam, an attending physician.

A few major themes arose from the discussion, in particular the importance of setting expectations, providing honest feedback, and identifying the appropriate setting for sensitive discussions.

In terms of setting expectations, Dr. Matthew Siedsma pointed out that trainees in all specialties are required to participate in off-service rotations. Though these rotations may seem ‘tangential to [the] ultimate career path’, they aim to improve a learner’s knowledge base and provide useful and necessary experiences. Dr. Elisha Targonsky agreed, pointing out that students must achieve core competencies, regardless of outside interests.

An appropriate time for setting expectations is prior to the emergency medicine rotation, during an initial orientation session. Students, Dr. Siedsma suggested, should know that ‘it is unreasonable to see only one patient together and then expect an evaluation’. Several commenters pointed out that expectation-setting should also occur at the beginning of a shift. Dr. Jeff Riddell believed that ‘some of this could have been avoided if Tom had given clear expectations at the beginning of the shift’. Dr. Nadim Lalani ‘diagnoses the learner’ at the beginning of each shift to ‘get a sense of their learning needs’ and how they might fit in amongst the core competences required of a rotating medical student more generally. These expectations may differ depending on whether the individual medical student is pursuing emergency medicine.

There was a general consensus that time spent achieving core competencies in the emergency department not only helps develop good clinical practice, but also facilitates accurate and useful evaluation.

Dr. Pik Mukherji was pessimistic about providing formative feedback, believing that ‘the time to set expectations, improve Patrick’s behaviour, and expose him to EM is long gone’, and that Patrick’s preceptors had ‘kind of failed Patrick here’. In terms of addressing Patrick’s absence from the ED, Dr. Mukherji acknowledged that ‘if there are professionalism…then he should get an appropriate eval[uation] and possibly make up the rotation.

Not all were so pessimistic about Patrick’s ability to hear and integrate feedback. Medical student Alvin Chin pointed out that Patrick might not have insight into the fact that ‘his assessors felt he was quite inappropriately absent during his shift’. Dr. Siedsma stated he would ‘sit Patrick down and give him the opportunity to be honest about he’s spent time in the ED during his scheduled shifts’. If Patrick is able to reflect with honesty, ‘he should be thanked for his honesty and then ask[ed] about how he can make it right’. This may require additional shifts, ‘where he demonstrates the appropriate professionalism’. Several commenters pointed out that it is necessary to ensure that ‘…Dr. Pam has correctly identified that Patrick had spent the majority of his with other specialties, outside the emergency department where he is currently placed for his rotation’. Alvin insightfully remarked that ‘the biggest problem in this situation seems to be with communication.’ Daren agreed, saying that ‘if Tom calls Pat [out] on the problem, Pat may be able to say he had no idea – he was just helping out with the tough cases and trying to make his shift applicable to him’.

Honesty was emphasized not just for Patrick, but also for his evaluators. Dr. Kaif Pardhan pointed out that ‘failing to provide an accurate assessment of a trainee, no matter what level, is doing them a disservice and may be doing patients a disservice down the line, particularly since we may be able to link them with helpful resources or remediation’. Daren pointed out that ‘most people in Tom’s position would back down and at worst write “meets expectations” – and move on’. Time pressures, cultural issues, and additional clinical and administrative duties often prevent preceptors from providing the type of honest formative feedback that allows learners to grow into competent clinicians. The challenge of justifying low ratings to administrators may contribute to evaluations that are superficial and not constructive. Dr. Pardhan emphasized the need to ‘create a space where preceptors have the opportunity to discuss their experiences with the trainees they supervise’.

Most commenters agreed that conversations surrounding evaluations should take place in a quiet area without fellow learners, patients, or allied healthcare professionals. Dr. Riddell usually ‘label[s] the conversation and take[s] it out of the work area’. Specifically, he asks the learner if they are ready to receive feedback, and finds that explicitly labelling the session in this way helps reduce some of the awkwardness involved in leaving the core work area. Dr. Lalani pointed out that the ‘attending should be careful about potentially placing a resident in the awkward position of listening to them vent about another learner’.

Alvin said he ‘would hope that in situations where a learned is confused/mistaken about these expectations, that a supervisor/preceptor would help communicate and point out the gap’. That formative feedback might have facilitated Patrick’s development as a competent clinician.

References

1. Eva KW, Regehr G. “I’ll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof. 2008; 28(1): 14-9. PMID: 18366120
2. Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med. 2005; 80(10 Suppl): S46-54. PMID: 16199457

[/su_spoiler]

Case and Responses for Download

Click Here (or on the picture below) to download the case and responses as a PDF.

Author information

The post MEdIC Series | The Case of the Awkward Assessors – Expert Review and Curated Commentary appeared first on ALiEM.