MEdIC Series: Case of the Failure to Fail – Expert Review and Curated Community Commentary

The Case of the Failure to Fail outlined a scenario of an attending emergency physician who was frustrated with the ingrained “failure to fail” culture amongst his colleagues when faced with a resident who had significantly underperformed throughout his emergency medicine rotation.

This month, the MEdIC team (Tamara McColl, Teresa Chan, Sarah Luckett-Gatopoulos, Eve Purdy, John Eicken, Alkarim Velji, and Brent Thoma), hosted a discussion around this case with insights from the ALiEM community. We are proud to present to you the curated community commentary and our expert opinions. Thank-you to all participants for contributing to the very rich discussions surrounding this case!

This follow-up post includes

• Responses from our solicited experts:
  • Dr. Nancy Dudek is an Associate Professor in the Faculty of Medicine at the University of Ottawa and Fellow of the Royal College of Physicians and Surgeons of Canada in the specialty of Physical Medicine and Rehabilitation. She focuses her clinical practice on Amputee Rehabilitation, Prosthetics and Orthotics. Academically, her work on the assessment of medical students and residents in the workplace has been recognized with numerous grants, publications and awards. Dr. Dudek currently works as a Clinician Educator for the Royal College of Physicians and Surgeons of Canada focusing on improving workplace based assessment.
  • Dr. Jessica Trier (@jesstrier) is an Assistant Professor in the Department of Physical Medicine and Rehabilitation at Queen’s University, with a clinical practice in acquired brain injury rehabilitation and electrodiagnosis. She is the Competency-Based Medical Education lead for the PM&R postgraduate training program at Queen’s University, and her scholarly interests include feedback in medical education, and the relationship between teachers and learners.
  • Dr. Karen Hauer is Associate Dean for Assessment and Professor of Medicine at the University of California, San Francisco (UCSF). She is an active researcher in medical education and a research mentor for fellows, residents and students, with a focus on new models of clinical learning in the workplace, competency-based medical education, learner assessment, coaching and remediation. In 2015, she completed a PhD in Medical Education through a joint program with UCSF and the University of Utrecht in the Netherlands. She directs the UCSF medical student coaching program. She is a practicing general internist, wife and mother of three!
  • Dr. Vanessa Thompson (@vthompsonmd) is a general internist and primary care provider at Zuckerberg San Francisco General Hospital. She also serves as the Academic Development Director for the Internal Medicine Residency Program and as a medical student coach at the University of California, San Francisco (UCSF).

• A summary of insights from the ALiEM community derived from the Twitter and blog discussions
• Freely downloadable PDF versions of the case and expert responses for use in continuing medical education activities

Expert Response 1: When Altruism Clashes With Reality (Dr. Nancy Dudek MD MEd FRCPC and Dr. Jessica Trier MD FRCPC CSCN Diplomate-EMG)

“The Case of the Failure to Fail” highlights a pervasive challenge facing many medical educators. This case details the experience of working with Trevor, a first-year surgical resident, who has recently rotated through the emergency department. Informal discussions amongst department members brought to light common concerns, including poor attention to detail, resistance to constructive feedback, “attitude,” and most worrisome, patient safety concerns.

Despite these major concerns, most of the group takes a rather passive, “it’s somebody else’s problem” approach, and ultimately decides to pass him on his final evaluation. Sadly, this experience rings true for a lot of medical educators, especially with off-service trainees.

Barriers to Failure

A recent BEME systematic review by Yepes-Rios et al. outlined several barriers to failing trainees.¹ Assessors may be concerned about the impacts of failing a trainee on their own professional lives, including extra time required for documentation, leading discussion and intervention, and potential negative repercussions on their professional standing and reappointment.¹ Duffy et al. also cite the fear of litigation as a significant contributor.²

Not only are professional considerations at stake, but assessors also report a sense of personal failure and guilt when deciding to fail a trainee.¹ In light of recent emphasis on the importance of learner wellness, failing a trainee creates an internal conflict knowing that the failure may have an impact on the learner’s psychological state or future career goals.¹ In addition, close relationships often develop between assessors and trainees, particularly in small programs. This further inflates the concept that failing a trainee can be filled with emotion or conflict.¹ In addition to personal and professional considerations, assessors cite concerns regarding how to deliver and document good quality feedback to trainees.¹

From an institutional standpoint, the culture may define whether there is support to fail a trainee, or whether there is pressure to pass an underperforming trainee out of concern for the institution’s reputation.¹ A lack of knowledge of the remediation process or a belief that the process will not help the trainee is also cited as a reason for not failing a trainee.^1,3

Factors That Support Failure

Despite all of the potential barriers, many assessors do appropriately fail trainees when their performance fails to meet expectations. Some literature suggests that this is because they are altruistic.¹ Yepes-Rios et al. report that assessors willing to appropriately fail trainees sense a duty to patients, to society, and to the profession.¹ They would not want Trevor taking care of their spouse, parent, or child which leads them to “do the right thing” and fail the trainee when warranted.

Another key component of appropriate trainee failure is institutional support, both for assessors and trainees. If assessors’ observations and judgements about trainees’ clinical performance are well supported and the institution provides effective guidance and resources for improvement to the trainee, the assessor is more likely to provide frank and honest negative evaluations because the remediation and support process is understood to be effective.¹

Off-Service Effect

Many trainees spend a substantial portion of their junior years training on off-service rotations. Failure to properly document concerns about trainee performance does a tremendous disservice to both the trainee and the residency program as it often leads to delays in recognizing trainees who are struggling. While the responsibility for addressing learning gaps ultimately rests with the home program, it is definitely the responsibility of off-service rotations to properly document and communicate their observations.

The Role of CBME

More frequent direct observation, better assessment tools, and accurate documentation are necessary to identify struggling learners in a timely fashion. Competency-based medical education (CBME) provides an assessment model that incorporates these important qualities through frequent, low-stakes assessments based on direct and indirect observation that are then reviewed by a competence committee to make decisions about trainee progression. If appropriately implemented, CBME should provide robust data on trainee performance to base decisions on and it shares the responsibility for making important decisions about trainee performance amongst a group of individuals. In addition, the CBME model for assessment is based on the demonstration of competence to independently perform the tasks of a physician. Therefore, when rating trainees the assessor does not need to consider whether the trainee performed a task above, at or below the expectations of a “first year surgical resident on their emergency medicine rotation”. Rather, they simply need to identify what the trainee can perform independently and what they need supervision for. This should facilitate and promote accurate resident assessment for clinician teachers and overcome some of the barriers that prevent accurate reporting of poor trainee performance.

Other Factors

Faculty who are reluctant to fail trainees cite fears of negative assessments of their own teaching skills as well as concerns regarding promotion and career development as barriers to documenting poor trainee performance.¹ Departments and institutions should make it clear to faculty and assessors that one or two poor assessments of a faculty member’s teaching skills will not negatively impact their chances at promotion or tenure. Similar to trainees, poor assessments for faculty should serve as a learning opportunity rather than a means to impede their career progression.

Summary

Culture change is needed to minimize or eliminate the problem of “failure to fail”. To promote effective culture change, departmental and institutional leaders need to find ways to help faculty overcome the barriers which prevent accurate trainee evaluations. Such culture change does not happen overnight, but institution of CBME into training programs is a step in the right direction. More faculty members should be encouraged to emulate Mark, who “called it like he saw it” when assessing and documenting Trevor’s performance. We hope that Mark has not lost faith in the process after having a negative experience. Mark did the right thing for Trevor, for the profession, and most importantly for the patients. Do the right thing, and be like Mark!

References

1. Yepes-Rios M, Dudek N, Duboyce R, Curtis J, Allard RJ, Varpio L. The failure to fail underperforming trainees in health professions education: A BEME systematic review: BEME Guide No. 42. Med Teach. 2016 Nov;38(11):1092-1099.
2. Duffy K. Weighing the balance: a grounded theory study of the factors that influence the decisions regarding the assessment of students’ competence in Practice. Thesis. 2006; Scotland, UK: Glasgow Caledonian University.
3. Dudek NL, Marks MB, Regher G. Failure to fail: the perspectives of clinical supervisors. Acad Med. 2005;80:S84-S87.
4. Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. Entrustability scales: outlining their usefulness for competency-based clinical assessment. Acad Med. 2016 Feb;91(2):186-90.

 

Expert Response 2: Failure to fail: Changing culture to support learners, faculty, and patients (Dr. Karen Hauer MD PhD,  Dr. Vanessa Thompson MD)

This vignette represents an all-too-common scenario in medical education and is a classic case of ‘failure to fail’. Supervising physicians recognize that a student or resident has performance problems, yet they document these concerns in muted language or not at all, and allow the learner to progress through training without receiving the corrective support needed. Trevor has multiple performance problems, consistent with the literature showing that severely struggling or failing learners typically have problems in multiple competency domains.^1,2 Mark describes deficiencies in Trevor’s medical knowledge and clinical reasoning, inappropriate patient management with threats to patient safety, concerning professional behavior with overconfidence and defensiveness, and poor insight into his own level of (dys)competence.

The physicians working with Trevor may have scored his sub-par performance at the level of ‘meets expectations’ for many reasons.³ A resident with multiple performance problems may leave an evaluator unsure about what the underlying problem is, or how to write an evaluation without seeming unduly harsh or unsympathetic. Some faculty may not have confidence in their own assessment skills and can default to the thinking that if the trainee made it to the post-graduate level of training they must be competent. An evaluator who has not taken the time to give in-person feedback may feel that it’s unfair to share new and negative news in a written evaluation. Unfortunately, some evaluators may find it easier to write a bland, passing evaluation than to take on the responsibility of documenting concerns and potentially facing resistance or retaliation from the learner, and follow up questions from program leadership. Educators may defer the accountability for a struggling learner, particularly as in Trevor’s case, where he ‘belongs to’ another department’s training program. These emergency medicine physicians worry that documenting Trevor’s performance problems will prompt more clinical time for him to remediate in their unit. The potentially onerous task of remediating a learner is just one of the many factors that can dissuade faculty from labeling performance as failing.

Mark finds himself in a challenging situation during this faculty meeting. He did a commendable job of clearly and succinctly communicating his greatest concerns regarding the impact of Trevor’s care of patients to start the discussion. He gave timely, specific, in-person feedback to Trevor and coached him to improve. Unfortunately, Mark’s colleagues took a more hands-off approach to their assessment and evaluation responsibilities. If there were written expectations or entrustable professional activities for the rotation, Mark could ask the group to discuss Trevor’s performance on these specific measures, soliciting comments only from faculty who had directly observed the learner. Mark’s colleagues appear to need more training about performance expectations, especially for residents rotating from other departments.

The multiple reasons faculty are hesitant to fail a struggling learner create a “failure to fail culture” in which the system itself doesn’t support identification and labeling of a learner as failing. Complexities inherent in modern training environments compound these factors. In the case presented here, the episodic nature of trainee supervision, as is common in the emergency department and other departments with frequent attending transitions, makes longitudinal relationships between learners and supervisors infrequent. Faculty want to give trainees the benefit of the doubt, and if a trainee underperforms they may be tempted to look for alternative explanations (e.g. a bad day, an unusual clinical scenario, etc.). In order to change the willingness of faculty to report poor learner performance, we need a systems-level approach to changing the culture in medical training.⁴

Competency-based assessment, which is increasingly being incorporated into medical training programs, offers opportunities to address the problem of failure to fail. In a competency-based model, all learners are expected to progress along a developmental continuum, with areas of strength and areas for improvement at any point through the journey of lifelong learning. Thus, faculty may be more willing to look at individual areas for development as opposed to only an overall pass-fail value judgment. To move from a pass-fail dichotomy to a learner-driven, competency-based developmental model, formative assessment must be emphasized more than is currently the norm. Medical schools and residency training programs need to set the expectation early for learners that formative assessment and feedback are valued, commonplace parts of their training. Learners should have the chance to receive feedback and apply it without feeling judged or graded with every observation. Faculty also should be supported to develop skills in assessing competencies and providing effective feedback. As with most culture shifts, change requires leadership from program directors and department chairs – to be explicit with expectations that faculty provide regular feedback to learners and to support faculty when that feedback includes areas for improvement. To overcome the fear of backlash from learners, transparency about how trainee assessments affect faculty promotion is important. Departments should develop a clear process for how they will discuss trainee development that would be scripted and separate from a regular faculty meeting. Finally, institutions need transparency around the process of supporting struggling learners. Faculty should not feel that, if they report trainee underperformance, they will then be responsible for remediation. Instead, there needs to be institutional investment in faculty time to do this important but time-consuming work.

In order to change the culture of failure to fail, the shift to competency based medical education offers a road forward, particularly when accompanied by strong emphasis on formative assessment and frequent feedback to learners. Framing assessment and feedback as an expected part of training that is key to both career success and improved patient care will destigmatize constructive feedback. Faculty should be supported in developing these assessment and feedback skills so that all learners, including those who are struggling, receive the individualized guidance they need for performance improvement. Faculty and department leaders can model openness to feedback and engagement in lifelong learning as they ask for and incorporate new ideas and skills into their own practice. This culture shift should be accompanied by a robust assessment system with frequent direct observations of trainee performance and structured opportunities for faculty to review and discuss trainee performance to determine whether it meets expectations. This system will enable identification and remediation of underperforming learners and enhance confidence in our graduates’ preparation for the next level of training or independent practice.

References

1. Guerrasio J, Garrity MJ, Aagaard EM. Learner Deficits and Academic Outcomes of Medical Students, Residents, Fellows, and Attending Physicians Referred to a Remediation Program, 2006–2012. Acad Med. 2014 89(2):352–8.
2. Nixon LJ, Gladding SP, Duffy BL. Describing Failure in a Clinical Clerkship: Implications for Identification, Assessment and Remediation for Struggling Learners. JGIM. J Gen Intern Med. 2016 31(10):1172–9.
3. Dudek NL, Marks MB, Regehr G. Failure to Fail: The Perspectives of Clinical Supervisors. Acad Med. 2005 80(10 suppl): S84–7.
4. Yepes-Rios M, Dudek N, Duboyce R, Curtis J, Allard RJ, Varpio L. The failure to fail underperforming trainees in health professions education: A BEME systematic review: BEME Guide No. 42. Med Teach. 2016 38:11: 1092-9.

 

Curated from the Community (Dr. Eve Purdy, MD FRCPC candidate)

This month’s case focused on the realities of failing medical trainees. Mark, an attending physician, was significantly concerned about the medical knowledge, safety, and attitudes of Trevor, an off-service resident. Mark was confident that his colleagues would voice similar concerns  around Trevor’s performance at their departmental meeting. Unfortunately, despite side conversations echoing Mark’s assessments, Mark was ultimately the only attending who had submitted a negative assessment. Trevor ended up passing the rotation despite legitimate patient safety concerns about his performance. The readers were left pondering why some staff provide struggling learners with satisfactory evaluations, why the culture of “failure to fail” exists, and what strategies could be implemented to overcome this seemingly ingrained culture in medicine.

Our online community of educators and learners provided valuable insight into this complex issue. Thank you to all who participated!

Culture of Feedback

The group identified that overall, medical practitioners are not great at providing objective, and sometimes difficult, feedback. As a profession we do not view failure with any degree of positivity. It is largely feared, and thus largely avoided. Our admissions processes select against those with experience in managing failure or those who dare to fail. In medical school we demand perfection, or risk humiliation. In resident education we sometimes avoid difficult conversations or have such ephemeral relationships that accurate feedback is challenging.

Culture is a learned set of values, beliefs and actions shared amongst a group of people. There are many aspects of medical culture that shape our approach to failure. Drs. Nadim Lalani and Krishan Yadav identified that most educators are “doves” not “hawks”. Many physicians dislike confrontation and so it is more within our comfort zone to give learners the benefit of the doubt than it is to have a difficult conversation with them. If educators are awkward around failure, then learners certainly will be.

Language is an absolutely integral component of any culture and the group discussed the negativity surrounding our language used. “Remediation” is a common term used for a program that learners who are struggling go through to satisfy learning objectives. Now, most licensing jurisdictions ask if learners have ever been through a period of remediation. The implication is that remediation may lead to licensing barriers, further perpetuating a fear of failure. The group felt that we must rethink our language, perceptions, and process around failure if we are to change the culture.

Barriers to Failing Residents

In addition to the nebulous but essential discussion around the culture of failure in medical education the group identified a number of practical barriers to failing learners.

The first challenge is gathering objective evidence on learners. Often it is not medical knowledge, but softer skills such as communication, that are the element of concern for a given learner. The methods that we have to detect, record, report, and remediate these skills are truly lacking. The lack of direct observation in our current educational model prevents the gathering of good, representative, non-biased, data on our learners.

The second challenge our participants identified is related to time and faculty burden. If we expect teaching faculty to provide valuable feedback, to engage in challenging conversations, to invest in remediation then we need to reward them for doing so. Dr. Lalani highlighted that the rate of burnout, or near burnout, in teaching faculty is high. There are many competing demands for the academic physician. This may lead to passing learners because it is easier, not because it is right. Dr. Swapnil Hiremath alluded to institutional barriers in failing learners. Exploration of these barriers, and solutions about how to empower teaching faculty to engage positively with struggling students must become a priority.

The case was purposefully designed to be centered around an off-service learner. The respondents identified that there may be less of a sense of ownership for these residents. However, the reality is that many residents, early in their training, spend a great deal of time off service. We are left considering what responsibility a non-home service has in addressing learners who are struggling and facilitating their remediation?

The Way Forward

In addition to discussing the shortcomings, the group channeled optimism for the future. They identified steps that individual educators/learners can take and larger system-based changes that will hopefully shape a healthier relationship between failure and medical education.

Dr. Lalani outlined an approach that he uses with his learners each shift. He suggested that an important part of providing feedback is setting expectations. He uses the “A,B,C’s” approach, an acronym that all emergency physicians should be comfortable with. On each shift he gives learners feedback on their Attitude, Behaviours, Clinical Skills, and Soft Skills. At the end of the shift they know this discussion is coming and usually view it as a positive opportunity to improve, even when critical feedback is included in the discussion. You could incorporate this model into your educational practice tomorrow!

The group suggested that there should be a set mechanism for “consulting” on lukewarm to poor evaluation of learners. These need to be further investigated by the rotation director to increase our sensitivity in detecting learners who might be struggling, since we know that our written evaluations at the end of shift are not always entirely forthcoming. Like any good screening test however, there must be a treatment available to change outcomes. A robust remediation program for learners that are struggling must be in place so that we can support them in their efforts to becoming the best version of themselves.

Finally, the group reflected on the change in attitude towards failure that may come with the transition to competency-based medical education (CBME). The move towards CBME should increase the amount of direct observation, objective evidence, meaningful feedback, and opportunity for the personalization of education. Much of this process will be guided by the learners themselves who will be actively seeking feedback and learning opportunities. Faculty will support them in their learning goals and hopefully help them identify where they are struggling. The opportunities and burdens that this new educational paradigm holds are yet to be determined but we are optimistic that it will improve our approach to detecting and supporting struggling learners!

Contributors

A Special thanks to the following participants in our discussion (in alphabetical order)

• Teresa Chan
• Swapnil Hiremath
• Nadim Lalani
• Tamara McColl
• Krishan Yadav

Case and Responses for Download

Click here (or on the picture below) to download the case and responses as a PDF (321 kb).

Author information

The post MEdIC Series: Case of the Failure to Fail – Expert Review and Curated Community Commentary appeared first on ALiEM.