STRENGTHS AND LIMITATIONS OF THIS STUDY
-
Evaluation of standardised case vignettes demonstrates that diagnostic reasoning by medical students is susceptible to the framing effect.
-
The results of subgroup analysis encourage targeted interventions based on academic standing.
-
Assessment of written case vignettes likely evokes different reasoning strategies than real-time interpretation of data in the emergency department.
-
Single-site data collection and limited sample size may limit statistical power and generalisability.
Introduction
Diagnostic error contributes to temporary harm, permanent harm and potentially death in approximately 18% of patient cases in hospitals.1 From the physician’s perspective, diagnostic error is increasingly costly, clinically and financially, given the tools available to track and prevent mistakes. As of 2013, diagnostic error was the leading category of medical malpractice claims and accounted for the highest proportion of total payments. Moreover, the economic pitfall and cost in quality or loss of life are now higher because more interventions are available when an illness is caught early.2
While error is unavoidable, physicians must reduce it wherever possible, and the clinical instructor’s duty is to teach medical learners mitigation strategies. Approximately 6.5% of autopsies of patients who were admitted to the intensive care unit (ICU) reveal a diagnostic error that could have prevented the patient’s death.3 Patients who had unplanned hospitalisations within 9 days of a primary care visit were misdiagnosed at a rate of 4.5% with certain conditions having higher rates such as spinal abscess (17.0%) and meningitis (10.9%).4 There is ample room for improvement in diagnostics, and diagnostic improvement will ultimately benefit patient outcomes.
More effective clinical reasoning is needed to address poor outcomes due to misdiagnosis. Clinical reasoning refers to the gathering and analysis of information to make informed and appropriate decisions in healthcare.5 6 An iterative process of data input, interpretation and problem formulation is required to fully understand clinical problems.7 Clinical decision-making in high-acuity settings such as the emergency department is often driven by a specific, defined strategy. The dual-process model of decision-making is currently the most robust option available to physicians.8 This model proposes two types of processes underlying decision-making: intuitive (Type 1) and analytical (Type 2). To meet the need for dynamic decision-making in a setting with time pressure, limited information, competing patient demands and the requirement to maximise department flow, emergency physicians default to Type 1 reasoning over Type 2. One example of Type 1 reasoning is the use of heuristics—shortcuts in cognition.8 9 Although intuition is critical in clinical decision-making, it can be dangerously swayed by cognitive bias.8
The framing effect is one type of cognitive bias, wherein identical information may not be evaluated identically given varying methods of presentation.9 Physicians in one study surrounding the framing effect were asked to consider the efficacy of a hypothetical therapeutic agent. Although they received the same information, conditions varied based on whether the information was framed in terms of survival (benefit) or mortality (cost). Ratings of the hypothetical treatment as more effective than the standard of care varied from a low of 52% to a high of 94% depending on the cognitive frame imposed.10 In a similar study, physicians were asked to provide the medical diagnosis between two versions of a pulmonary case vignette: one with a frame imposed to suggest a specific diagnosis, the other without. While the objective data presented in both vignettes were the same, cases framed non-specifically were associated with significantly greater rates of diagnostic error.11
Clinical reasoning is important not only to form the correct diagnosis but also to avoid associated harm to the patient and the healthcare system from diagnostic errors.12 However, little is known about the instruction of clinical reasoning in undergraduate medical education. Understanding these curricular gaps can better prepare medical students to enter the clinical sphere as early diagnosticians.
A traditional 4-year medical school education involves 2 years of non-clinical education (anatomy, physiology, etc.) coupled with 2 years of rigorous clinical education.13 To foster clinical reasoning skills during the non-clinical years, the current standard is a case-based learning integrated curriculum, in which students engage with a patient scenario to consolidate taught concepts. Case-based learning is typically conducted in a Socratic format; this collaborative environment, wherein students discuss the case with their peers and a facilitator, engages analytical reasoning. In this setup, clinical reasoning abilities are not explicitly taught, but more so ‘caught’.14
Case-based learning nonetheless remains only a small portion of the 4-year medical curriculum. Concepts of clinical reasoning are thus overlooked in favour of the content education required for board examinations, which is taught with emphasis on rote memorisation and completing case vignette multiple choice questions with ‘buzzwords’. During the clinical years, when students are immersed in patient care, clinical reasoning is the priority; however, few medical schools have formal teaching sessions for clinical reasoning in their clinical curriculum.15
While existing literature suggests that diagnostic reasoning among attending physicians may be shaped by the framing effect, it remains to be investigated whether susceptibility to framing bias in the diagnostic process varies longitudinally among a medical student cohort.
Goals of this investigation
The primary objective of this study is to assess whether diagnostic reasoning by medical students is influenced by the framing effect. The secondary aim is to investigate whether this effect is moderated by the level of clinical knowledge and experience. By exploring susceptibility within this demographic and cultivating an awareness of rational pitfalls, we aim to provide a deeper understanding to be considered in future educational interventions that will enhance clinical decision-making and improve patient outcomes, ultimately reducing rates of error for the benefit of patients.
Methods
Study design
This is a cross-sectional study conducted at a tertiary academic medical centre in an urban setting. The study was distributed electronically via the institutional Qualtrics account. Data were collected over a 2-month period from March to May 2022. Patients and the public were not involved in the design, conduct, reporting or dissemination plans of our research.
Study setting and population
Participants were recruited via email listservs for medical students at Sidney Kimmel Medical College in Philadelphia, Pennsylvania. Inclusion criteria included age ≥18 years and current enrolment at TJU, with at least 1 year of the Doctor of Medicine (MD) programme completed. Demographic information collected, including age range, gender, years of medical education completed and months of clinical rotations completed, was non-identifiable. Participants were enrolled via a single email and invited to complete the study at their convenience. As an incentive for participation, students were entered into a raffle for $10 prepaid debit cards.
Study protocol
To assess cognitive bias, we used a hypothetical case from a New Zealand study by Popovich et al11 that had been validated a priori. The case has two associated vignettes, which our team modified slightly with terms more familiar to American participants. The vignettes (online supplemental Appendix 1) are identical with regard to objective clinical information; each can be rearranged to produce the other. The difference lies in whether the information is organised to emphasise features consistent with pulmonary embolism (PE) or consistent with other diagnoses. In this way, we were able to operationalise and implement a framing effect. We refer to the former as a specific-frame vignette and the latter as a non-specific-frame vignette.
Participants were stratified by the number of completed years of medical education (one, two or three) and then randomly assigned to evaluate either the specific-frame or non-specific-frame vignette. Randomisation was performed on an ongoing and automated basis within Qualtrics with allocations concealed from researchers. This led to a 2×3 design. After reading the vignette, each participant was prompted with the following question: ‘Please provide your top three differential diagnoses’. This was succeeded by three free-text entry boxes.
Outcome measures
The outcome measure associated with each condition was the frequency with which participants identified an expected diagnosis (PE). We also considered the rank order of each participant’s differential diagnosis. For those differentials in which an expected diagnosis appeared, we also recorded whether it was the first line of investigation or if it was ranked at a lower likelihood.
Statistical analysis
Two researchers (AJM and XCZ) evaluated responses and recorded whether the disease suggested by the vignette appeared in the list of three diagnoses provided by each participant. A diagnosis was scored based on whether it included the main component of the diagnosis or a commonly accepted abbreviation or variation thereof (eg, ‘embolus’ vs ‘embolism’ vs ‘emboli’, ‘PE’). No missing data were identified.
Group differences in accuracy among responses to the specific-frame and non-specific-frame cases among each cohort of medical students were assessed using χ2 and Fisher’s exact tests in IBM SPSS (IBM Corp., Armonk, New York). We also performed point-biserial correlations for differences in accuracy between the specific-frame and non-specific-frame vignettes based on months of clinical rotations completed.
Results
Characteristics of study subjects
The study population included students enrolled in an MD programme at a major academic medical centre in a large, northeastern city. 2nd-year, 3rd-year and 4th-year students were eligible to participate (n=825). In total, 271 participants responded, 269 proceeded through informed consent and 183 completed the study in full.
All 183 participants provided some level of demographic information. The modal age range of participants was 18–25 years (n=99). The oldest participant fell into the 36–40-year cohort. The modal academic year was MS2 (n=101). 101 (55.2%) respondents identified as female and 77 (42.1%) as male. Two male participants identified as transgender. One participant identified as genderqueer/gender non-conforming, and two participants preferred to self-describe their gender identity as ‘masculine’ and ‘non-binary’ respectively. Participants had completed a mean (SD) of 6.5 (8.0) months of clinical rotations. Table 1 shows participant characteristics by study group.
Sample sizes of study groups by demographic characteristics
Main results
Counts of respondents who did or did not identify PE as a diagnosis of interest are reported below in table 2.
Counts of identification of PE by study group and medical student cohort
χ2 testing demonstrated that identification of PE differed depending on the cognitive frame to which participants were exposed (p<0.001, df=1, φ=0.392). This effect held on subgroup analysis of all class year cohorts, specifically among MS2s (p=0.001, df=1, φ=0.354); MS3s (p=0.014, df=1, φ=0.379) and MS4s (p=0.001, df=1, φ=0.548) after the Bonferroni correction. We further investigated whether a significant difference existed between class years with regard to how framing influenced the likelihood of identifying PE in one’s differential. Among participants in the specific-frame condition, a significant difference existed between academic years with regard to the likelihood of diagnosing PE (p=0.006, df=2, φ=0.344). As academic standing advanced, a greater proportion of respondents identified PE as a diagnosis of interest.
We also assessed whether the imposition of frame was related to the position that PE occupied in one’s differential, the data for which is described in table 3 below. A χ2 test including respondents whose differential included PE showed that participants in the specific-frame condition were significantly more likely to list PE as their first differential diagnosis than those in the non-specific-frame condition (p=0.016, df=1, φ=0.329). This effect did not hold in subgroup analysis by academic year, using Fisher’s exact tests (p=1.000, p=0.101 and p=0.082 for MS2, MS3 and MS4 cohorts, respectively).
Counts of identification of PE as leading diagnosis in differential by study group and medical student cohort
We conducted point-biserial correlations to measure the relationship between months of clinical rotations completed and the likelihood of identifying PE as a diagnosis of interest. For participants in both conditions, more time spent in clinical rotations was associated with a greater likelihood of identifying PE in the differential. The association was stronger in the specific-frame condition (r=0.353, p=0.001, n=87) than in the frame-away condition (r=0.247, p=0.015, n=96).
Discussion
The purpose of this study was to investigate the effects of the framing bias on diagnostic reasoning given varying levels of clinical knowledge among medical students. We hypothesised that the susceptibility of participants to the imposition of cognitive frames would be mediated by years of undergraduate medical education completed. The results of our study affirmed that diagnostic reasoning among medical students was influenced by how clinical information was presented.
We discovered that cognitive frame and years of medical education completed were associated with the order in which medical students identified a likely pathology in their top three differential diagnoses. Students were more likely to list PE as their first-line diagnosis when exposed to a framed case vignette. This tendency is vital when students are faced with a life-threatening, time-restricted case during their clinical rotations or later in residency, and quick thinking is required.
Advanced academic standing and a greater number of months of clinical rotations completed were associated with an increased likelihood of identifying PE when evaluating a framed case vignette. This effect is likely polyfactorial. During preclinical training, students are taught to keep an open mind to a broad range of differential diagnoses and pathologies as their knowledge base grows. Their clinical reasoning is, by necessity, deliberate and systematic. If a student has been exposed to fewer patients or case vignettes, it follows that they are more likely to leverage Type 2 reasoning (analytical) over Type 1 reasoning (intuitive) in the absence of cultivated illness scripts or other heuristics. These students may also require more time to analyse a case, allowing for metacognition that could stave off the influence of cognitive bias.
However, at the point in training where students enter the clinical environment, we speculate that they begin to adopt a problem-solving approach driven by pattern recognition and algorithmic thinking. These domains are reinforced by pressure to practise efficiently and by operating within the speciality-specific context of each clerkship. As students gain more experience with patients, particularly those with relatively common pathologies such as PE, their pattern of reasoning begins to shift from Type two to Type 1. We hypothesise that as familiarity with pathology grows and students gather an increasing repository of cases, they are more likely to reach a diagnostic conclusion that is influenced by cognitive bias.
Notably, medical licensing examinations rely to some extent on the ability to leverage framing as a test-taking tool. Students must learn to quickly identify pathology based on ‘buzzwords’ and other hallmarks in question stems. The scores of these high-stakes examinations are of particular importance in residency applications, where they are routinely used as a key selection metric (Green et al, 2009)16; students thus feel compelled to master this style of question. Paradoxically, this method of clinical reasoning may expose students to errors in diagnostic reasoning in the clinical setting, where more complex consideration of patient presentation becomes crucial.
The diagnostic landscape is only becoming more complicated as evolving medical technology allows for the discovery and management of novel pathologies. Moreover, as the number of uninsured individuals in the USA continues to increase,15 a physician’s capacity for critical thinking and ability to diagnose without additional cost remains critical. All physicians in training should be aware of susceptibility to cognitive bias.
We believe that the logical and actionable next step to address the influence of the framing effect is imparting awareness of cognitive bias among undergraduate medical learners. When students understand that their thought process around diagnosis may be steered by cognitive bias, they may be able to better execute a checkpoint in reasoning. This could look like intentionally broadening one’s thoughts or purposefully including more Type 2 reasoning at these junctures. Some curriculum interventions to consider are metacognitive training with structured reflection, error analysis tools such as the Misdiagnosis Tracker and cognitive debiasing techniques.17–19 Guided reflection can be through writing or the creation of concept or mechanism maps to help learners integrate new knowledge and experiences into their formed hierarchical structure.17 Error analysis tools, such as the Misdiagnosis Tracker, help learners recognise their own bias when atypical presentations arise.18 Some specific, rather simple, cognitive debiasing techniques to incorporate into the curriculum are the ‘consider the opposite’ strategy, diagnostic timeouts and system-based or problem-based differential diagnosis checklists.19 We hope that active awareness of framing bias and incorporation of some of these curriculum interventions would allow students to avoid premature closure on an incompletely considered first diagnosis. Our finding that differential levels of experience moderated susceptibility to the framing effect raises the question of whether educational interventions should be tailored between different medical school cohorts.
Limitations
The goal of this study was to assess the role of cognitive bias in diagnostic reasoning during patient care. Its primary limitation was that the cases were presented as hypothetical vignettes to be read and responded to. This format does not capture the demands of reasoning in the clinical environment, including information gathering, a careful physical examination, intra-team and inter-team communication and patient factors. A more realistic situation might involve a simulation in which participants gather data from a standardised patient and are challenged to interpret vital signs, examination findings and laboratory results in real-time and under limited time. It is unclear whether this difference in format would render participants more or less susceptible to an imposed clinical frame. A benefit of our methodology, however, is that the information and its presentation were standardised across participants, without the confounds encountered in the clinical environment or between standardised patients.
Beyond similarity to real practice, we also speculated about whether evaluation with case vignettes may have elicited the framing bias more reliably than a simulation. Because medical students and physicians grow accustomed to reading and responding to vignettes quickly as a necessary skill to succeed in licensing examinations, they may be especially primed for the framing effect given this method of presentation. The effects we found may or may not be observed in true clinical practice, limiting the ecological validity of this study.
Our protocol asked participants to provide their ‘top three differential diagnoses’. While students at our institution are trained to list differential diagnoses in order of likelihood, more explicit instructions would introduce more certainty of construct validity when evaluating for ordinal effects, as our study does. It is also difficult to tease out the effects of our imposed frame from individual cognitive phenomena that lead the frame to be effective. In our specific-frame case, for instance, elements of the history and physical that are salient to the diagnosis of PE are presented first, rather than later, as in our non-specific-frame case. It is possible that primacy alone is driving these findings rather than frame considered in gestalt. It may be worthwhile in future research to attempt to study frames in isolation from established ordinal effects.
This pilot study was conducted at a single site using funding from a small grant. A larger sample size may have bolstered existing findings or achieved significance where it was otherwise approached. A priori analyses of sample size would have bolstered our conclusions, particularly in comparisons between groups. Our response rate of 22% was suboptimal and raises concerns surrounding non-response bias. It is possible that the diagnostic reasoning of respondents and non-respondents is significantly different based on an underlying attribute—perhaps, for instance, more motivated or proficient students responded to this voluntary request to work through a clinical vignette. The study population was drawn from a single medical school and, barring minor year-over-year modifications, all subjects participated in the same curriculum. Future studies could investigate whether effects held or differed across multiple institutions based on undergraduate medical curricular structure.
Conclusions
At the core of this study is the principle that medical students may arrive at different diagnostic conclusions if objective data are arranged or delivered differently. A broad differential is critical in identifying a patient’s diagnosis for treatment, consultation or referral. If a physician reaches premature closure on an incorrect diagnosis due to frame, unwarranted diagnostic momentum is enacted, and time that is potentially critical to a patient’s outcome is wasted. For medical students, this awareness of the framing bias is crucial in the development of critical thinking skills during their curriculum. For educators, addressing the framing bias in curriculum by teaching metacognitive skills through active reflection, diagnostic timeouts and diagnostic checklists is imperative in shaping the next generation of physicians. We found that advanced academic standing and more months of clinical rotations were associated with a greater likelihood of identifying PE in a framed case vignette. This raises the question of whether curriculum interventions should be tailored to different medical school cohorts.
Data availability statement
Our dataset is available from the corresponding author upon reasonable request. Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involved human participants and was approved by the Thomas Jefferson University (TJU) Institutional Review Board (IRB) in Philadelphia, Pennsylvania (approval #21G.592). It was conducted in accordance with the principles of the Declaration of Helsinki. Informed consent was obtained from all individual participants before participation.
