Skip to Main Content
Follow us

Medical Education Research Grant — Past Recipients

2016

Joanna Bates, MDCM, CCFP, FCFP

Contextual Competence: exploring how residents recreate competent performance in new settings

Joanna Bates, MDCM, CCFP, FCFP
Joanna Bates, MDCM, CCFP, FCFP
Professor, Department of Family Practice, Scientist, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia (UBC)

Abstract

Postgraduate training in Canada is continuing to diversify in its settings. For residents and recent graduates, moving across these diverse settings can be challenging personally and professionally. Competent performance is embedded in context, so residents who move to a new context face a transition in which they must recreate their competent performance in the new context. The goal of this program of research is to understand and describe the process of recreating competence in a new context, in order to define a new dimension to competence we have termed contextual competence. We will take a qualitative approach (constructivist grounded theory). Through interviews with general internal medicine residents, fellows, and recently graduated clinicians from three Canadian sites (UBC; U Calgary; and Western), we will explore the process by which these physicians become aware of contextual differences, and how they learn to adapt their practice to the new setting. We will iteratively analyse interview data and adapt the interview framework to develop new insights. We will transcribe, code, and analyse interview data. We will use the analyzed data to build a theoretical model of the process by which trainees become aware of and adapt their practice and learning to a new setting. We anticipate that this work will enable postgraduate programs and trainees to better consider and support the development of skills, activities and attributes that make up contextual competence; and to enable trainees and recent graduates to conceptualize and work towards the development of contextual competence during their residency programs.

How this work will inform research in medical education

“All residents face multiple transitions into new contexts during training and beyond. With diversification of learning environments, these transitions are becoming more complex. While educational and health care systems can support such transitions, each individual trainee must have the ability to adapt to a new context. The objective of this research is to examine and describe what an individual trainee does or can do to recreate competent performance in a new context. Once we understand the process of adaption, then the goal is to deliberately strengthen this ability in trainees and physicians. We anticipate that this work will enable postgraduate programs and trainees to better consider and support the development of skills, activities and attributes that enable residents to perform competently in new contexts.”

Gisèle Bourgeois-Law, MD, FRCSC, MEd (PhD student, Maastricht University)

Barriers in addressing remediation for practicing physicians

Gisèle Bourgeois-Law, MD, FRCSC, MEd
Gisèle Bourgeois-Law, MD, FRCSC, MEd (PhD student, Maastricht University)
Liaison, Centre for Health Education Scholarship and Island Medical Program, University of British Columbia

Abstract

Objectives
The main objective of this study is to lay the foundation for a remediation framework for practising physicians. We will do this by a) using educational and sociocultural lenses to explore how different stakeholder groups conceptualize the remediation of practising physicians, b) considering how explicit and implicit conceptualizations of remediation affect both the perceived ability and the political will of various stakeholder groups to engage in remediation, and c) developing a suggested terminology for remediation in its various guises.

Design
This is a qualitative study, using constructivist grounded theory. We will conduct semi-structured interviews with a purposive sample across the broad range of stakeholders and the spectrum of remediation programs, including A) provincial regulatory authorities B) universities - Postgraduate and Continuing Professional Development Associate Deans C) medical associations D) individuals responsible for quality of care in health authorities and E) national organizations concerned with physician certification and professional integrity. Data collection will continue until theoretical saturation i.e. collecting new data does not yield new insights.

Setting
Interviews will take place in person or via distance technology such as Skype.

Analysis
Analysis will proceed apace with data collection, in an iterative process, with initial insights guiding future interviews. Themes will be compared within and across different stakeholder groups and analyzed to arrive at a theory (the goal of grounded theory methodology) as to how explicit and implicit conceptualizations of remediation affect both the perceived ability and the political will of various stakeholder groups to be involved in the remediation of practicing physicians.

Conclusion
Understanding how the different stakeholders conceptualize remediation is a necessary first step before we can focus future research efforts and develop sustainable and successful remediation programs on a broad scale.

How this work will inform research in medical education

“This work is important to medical education because identifying current definitions, conceptualizations and cultural values held by various stakeholders around the remediation of practicing physicians will facilitate organizational involvement and collaboration around this challenging component of continuing medical education.”

Kathy Boutis, MD, MSc, FRCPC

Performance-Based Competency in the Interpretation of Pediatric Musculoskeletal Radiographs

Kathy Boutis, MD, MSc, FRCPC
Kathy Boutis, MD, MSc, FRCPC
Staff emergency physician, Division of Emergency Medicine, Director of Research, Senior Associate Scientist, Research Insitute, Vice-chair, Research Ethics Board, The Hospital for Sick Children, Toronto

Abstract

Objectives
Traditional competency setting processes were designed primarily for "assessment of learning" paradigms. We have developed a validated education system that uses an "assessment for learning" cognitive simulation platform. It includes the deliberate practice with feedback of hundreds of pediatric musculoskeletal radiographs and continuous assessment. Before this can be launched as a standard learning-assessment tool in emergency medicine, a valid performance-based competency benchmark needs to be derived. Thus, as a new application of traditional standard setting methods, the main objective of this proposal is to determine a defensible competency threshold in pediatric musculoskeletal image interpretation using an "assessment for learning" education system.

Design
We will use the Ebel criterion-reference method of standard setting. This requires each item to be classified by difficulty and clinical significance. We will perform a prospective calibration cohort study with purposive sampling to generate item difficulty ratings. A five­ physician expert collaborative will determine item clinical significance. A unique expert collaborative with 20 participants that represent all key stake-holders will subsequently determine competency cut-scores for categories of items. A weighted mean of cut-scores/category will derive final performance-based competency score. Implementation will then occur at eight University affiliated emergency medicine training programs to establish feasibility.

Setting
Participant review of images will occur via an on-line education system. Expert collaboration will occur at two separate one-day conferences.

Statistical analysis
Descriptive statistics and power-law learning curves will summarize data. Item Response Theory will derive item difficulty calibration.

Conclusion
Evidence-based metrics of competency in musculoskeletal radiography will enable emergency training programs to objectively assess its trainees and ensure practice readiness for this commonly applied skill. The methods derived from this research may also be applied to other similarly assessed essential emergency medicine competencies that use an "assessment for learning" education platform (e.g. electrocardiograms, point of care ultrasound).

How this work will inform research in medical education

“This work is important to medical education because properly defining patient-relevant metrics of competency in musculoskeletal radiography will enable emergency medicine training programs to objectively assess its trainees in this core and commonly applied emergency medicine skill. Further, once the process for deriving metrics for the competency-based performance are defined for this application, one may extrapolate the same approach to other emergency medicine competencies that are amenable to cognitive simulation using an assessment for learning platform (e.g. electrocardiograms, point-of-care ultrasound, other diagnostic images, etc.).”

Kevin Eva, PhD

Tailoring data collection to improve the effectiveness of assessment for learning

Kevin Eva, PhD
Kevin Eva, PhD
Senior scientist and associate director, Centre for Health Education Scholarship, University of British Columbia

Abstract

Objectives
To improve the training of residents and the healthcare provided by physicians at all levels of training by researching ways in which feedback about practice can be provided most effectively.

Background
Ensuring that medical practice aligns with professional standards commonly involves the collection of performance data from multiple sources that are summarized for the reviewee to enable ‘assessment-for- learning’.” Each of the assessment technologies applied shares common features: (1) Considerable resources are spent to collect and feedback substantial amounts of data and, (2) despite that effort, the degree of improvement demonstrated varies greatly. In part this occurs because we do not yet fully understand how to present data in a way that optimizes receptivity to feedback.

Design
Double blinded and controlled experiment with stratified randomization. Participants will be assigned to participate in an MSF process that uses (a) a generic MSF survey based on the CanMEDS competencies, (b) a group-tailored MSF survey generated by a small panel of specialist physicians, or (c) an individually-tailored MSF survey consisting of items selected as relevant by individual physicians. Outcomes include credibility of the data provided, strength of learning plans generated, coverage of the CanMEDS roles, and inter-rater reliability of the resulting MSF scores.

Setting
Postgraduate medical education in the Department of Medicine at UBC and workplace-based assessment in various divisions of the Vancouver Coastal Health Authority.

Statistical Analysis
ANOVA, Chi-squared, Generalizability theory.

Conclusion
As the effectiveness of the Canadian healthcare system rests extensively on the performance of its practitioners it is crucial that we better understand why performance data is used so variably to determine how to intervene effectively to help physicians aspire for excellence. Examining the influence of tailoring assessment practices to the specific needs of the physician being examined will help maximize the positive impact of assessment-for-learning.

How this work will inform research in medical education

“This work will take us the next step towards quality improvement practices that meet the specific needs of the trainees and physicians involved to maximize relevance of the data collected. Specifically, we will use the Multisource Feedback processes implemented by many regulatory authorities to examine the extent to which meaningful and valuable data can be collected by surveying one’s peers and patients.”

Joanne Goldman, PhD

A case study of advanced post-licensure quality improvement and patient safety education

Joanne Goldman, PhD
Joanne Goldman, PhD
Postdoctoral Fellow, Centre for Quality Improvement and Patient Safety, Faculty of Medicine, University of Toronto

Abstract

Objectives
This study explores three different advanced post-licensure quality improvement and patient safety (QI/PS) education programs and seeks to enable an empirically based approach to the development of QI/PS leaders and experts. Study objectives are to: 1. Identify, compare and contrast education program elements of three advanced post-licensure QI/PS training programs. 2. Examine program participants, leaders, and other stakeholders’ perceptions and experiences of advanced post-licensure QI/PS training and reported outcomes. 3. Illuminate the social, cultural and political processes that shape: a) advanced post-licensure QI/PS education; b) physicians’ and other healthcare professionals’ participation in these programs; c) how program participants use QI/PS learning in future practice.

Design
An interpretive qualitative case study approach will be used since it allows for an in-depth exploration of education programs situated within complex social, cultural and political contexts. We will study three cases using observations of in-session teaching and learning, participant, curriculum director and key stakeholder interviews, and documentary analysis methods.

Setting
The cases centre on three units situated at the University of Toronto that offer advanced post-licensure QI/PS education: (i) The Institute of Health Policy Management and Evaluation (Masters-level) graduate degree program; (ii) The Centre for Quality Improvement and Patient Safety (university-based certificate course); and (iii) St. Michael’s Hospital (hospital-based fellowship program).

Analysis
The study will use conventional and directed content analysis. Directed content analysis will draw upon education and sociological theory on professional learning and socialization, and the relationship between individual actions and structural conditions.

Conclusion
This project will: 1. Inform QI/PS education planning to optimize capacity building and resources; 2. Enable education and institutional leaders to strategically support QI/PS learning initiatives based on insights of the social, cultural and political processes that inform QI/PS education and practice; 3. Support a greater alignment between QI/PS education, practice and policy.

How this work will inform research in medical education

“This qualitative case study of three advanced post-licensure quality improvement and patient safety (QI/PS) education programs will enable an empirically based approach to the development of QI/PS experts and leaders. In particular, the project will: 1. Inform the design, delivery and target participants of advanced post-licensure QI/PS education within and across institutional contexts; 2. Enable education and institutional leaders to strategically support QI/PS learning initiatives based on insights of the social, cultural and political processes that inform QI/PS education and practice; 3. Support a greater alignment between QI/PS education, practice and policy.”

Sandra Monteiro, PhD

In the Blink of an Eye: Very Rapid Visual Diagnosis in Medicine

Sandra Monteiro, PhD
Sandra Monteiro, PhD
Assistant professor, Clinical Epidemiology and Biostatistics, McMaster University

Abstract

Background
A core component required of CBME is demonstration of competence prior to unsupervised practice. Therefore there is a need to develop a new set of assessment instruments that determine competent practice in an authentic environment. Visual diagnosis is a complex task nearly universal to medicine. X-rays and (often ECGs) are part of internal medicine, surgery emergency medicine, pediatrics, radiology and related subspecialties.

Objective
To develop an assessment tool that establishes a benchmark of performance. This will allow us to categorize residents and faculty according to skill level at detecting abnormal and normal images.

Design
We will employ a speeded protocol in which participants view diagnostic images for very brief time windows. This protocol has been shown to discriminate between novices and experts when diagnosing mammograms, however this is the first attempt to use this protocol as a competency assessment tool. The authors include experts from Radiology and Cardiology who will help create separate assessment tools for each specialty/format of imaging.

Setting
Residents and staff physicians in Radiology, Cardiology, Internal and Emergency Medicine will be recruited as participants

Outcomes and Statistical Analyses
For each participant, we will compute a d’ statistic as a threshold independent, normally distributed measure of accuracy, at each exposure time.

Implications
While in-depth data gathering and additional diagnostic testing may help confirm a working diagnosis, physicians rely only on their own experience to determine if there is any underlying pathology present in a diagnostic image. The study is a novel approach to isolating the effect of direct experience on clinical expertise.

How this work will inform research in medical education

“This work is important to medical education because it applies cognitive theory to the development of competency-based assessment. Up to now there have been few studies linking rapid system 1 processing directly to diagnostic accuracy. Our pilot data indicate that we can use our paradigm to measure experiential knowledge and efficiency of processing. Critically this project will explore how well this paradigm succeeds at discriminating between residents and faculty and how well it works with other domains such as Cardiology.”

Dominique Piquette, MD, MSc, MEd, PhD

Programmatic Assessment in the Workplace: A Realist Exploration Of Contexts, Mechanisms, and Outcomes in Critical Care Medicine

Dominique Piquette, MD, MSc, MEd, PhD
Dominique Piquette, MD, MSc, MEd, PhD
Staff physician, Department of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto

Abstract

Objectives
Our overall research objective is to explore how a program of formative workplace-based assessments is designed, implemented, and modified in two Canadian critical care postgraduate training programs in order to achieve its intended goals: supporting learning (assessment for learning), and assessing competence (assessment of learning). More specifically, we will address the mechanisms and contexts affecting the outcomes resulting from programmatic assessment in terms of trainee learning and competence assessment.

Design
We will conduct a longitudinal, multiple case study of the implementation of a new program of workplace-based formative assessments in two postgraduate critical care training programs. We will draw on the principles of realist evaluation for data analysis. A pre-specified set of theoretical assumptions will guide the data collection that will occur concurrently with the implementation of the assessment program in each training program over 2 years. Multiple data sources will be triangulated, including semi-structured interviews, direct observations of assessment activities, collection of de- identified assessment reports, and administration of standardized questionnaires on trainee goal- orientation and learning climate.

Setting
Two critical care medicine postgraduate training programs (cases) in Toronto.

Setting
Residents and staff physicians in Radiology, Cardiology, Internal and Emergency Medicine will be recruited as participants

Statistical Analysis
Specific dataset will be linked to the initial theoretical assumptions. Preliminary findings will be reviewed at regular intervals, grouped according to specific theoretical assumptions, and synthetized in preliminary thematic summaries. Qualitative data and narrative summaries of quantitative data will then be coded as 'observed outcomes', 'contexts', or 'mechanisms'. Further analyses will lead to plausible patterns of contexts-mechanisms-outcomes. Findings from the two cases will be initially analyzed separately, and then compared between- and across- case. These analyses will explore the nuanced commonalities and contingencies of workplace-based CBME assessment between different contexts and of varied mechanisms.

Conclusion
We will produce a nuanced explanatory account that clarifies how programmatic assessment results in variable educational outcomes through varied mechanisms in different contexts.

How this work will inform research in medical education

"This work is important to medical education because competency-based medical education has the potential to affect trainee learning and patient care; however, the best ways to assess trainee clinical competencies are unknown. We need to better understand the different mechanisms and contextual factors that influence the educational outcomes resulting from a programmatic approach to formative assessment.”

Akshay (Shay) Seth, MD MAEd(c)

How Surgeons Make A Risk Assessment and Know When and How to Implement New Procedures

Akshay (Shay) Seth, MD MAEd(c)
Akshay (Shay) Seth, MD MAEd(c)
Resident, Orthopaedic Surgery, University of Ottawa

Abstract

Background
Surgeons regularly make changes in their practice to ensure they are providing high quality patient care. This includes the process of learning and safely integrating new skills, techniques and technologies into practice. Despite evidence that introducing a new surgical technique is associated with a learning curve during which there are reduced surgical and patient outcomes, there are no suggested protocols in place to support a surgeon in safely introducing a new procedure into an established practice. When faced with the challenge of integrating a new surgical procedure into practice, surgeons must determine when they are ready to overcome the associated risks.

Objective
To investigate how surgeons experience risk when learning and integrating a new procedure into practice.

Methods
This study uses a modified constructivist grounded theory approach with semi-structured interviews involving academic surgeons who have recently learned and integrated a new surgical skill into their practice. Their experience of learning and implementing the new procedure will be explored, including their perspectives on risk and managing their learning curve. Data from the interviews will be coded to identify emerging themes, which will be refined through the iterative process of data collection and analysis. The themes will serve as the foundation for the development of a conceptual framework to understand how surgeons perceive the risk associated with learning and integrating a new procedure into practice.

Significance
The framework that is produced from this research is expected to provide novel ways of understanding surgeon learning and risk assessment. This framework may lead to the future development of guidelines that would facilitate access to resources for surgeons looking to learn and implement new skills, techniques and technologies while maximizing patient safety.

How this work will inform research in medical education

"This work will explore how surgeons identify, perceive and mitigate the risks associated with learning and introducing a new surgical skill into practice. Despite evidence that introducing a new surgical technique is associated with a learning curve during which there are reduced surgical and patient outcomes, there are no suggested protocols in place to support a surgeon in safely introducing a new skill into an established practice. In this study, surgeons who have recently introduced a new skill into their practice will be interviewed to explore their approach to learning a new skill, assessing potential risks and managing their learning curve. A grounded theory approach will be used to develop a conceptual framework for assessing risk and maximizing patient safety that can be used to guide surgeons integrating new skills into practice.”

Kong Eric You-Ten, MD, PhD, FRCPC &
Sev Perelman, MD, CCFP-EM
Co-principal investigators

Role of A Difficult Airway and Oxygenation Cognitive Aid on Interprofessional Team Performance- A Randomized Controlled Study

Kong Eric You-Ten, MD, PhD, FRCPC
Kong Eric You-Ten, MD, PhD, FRCPC
Assistant professor, Department of Anesthesia, Mount Sinai Hospital, Toronto

Sev Perelman, MD, CCFP-EM
Sev Perelman, MD, CCFP-EM
Assistant professor, Medical director, SimSinai Simulation Centre, Department of Family Medicine-Emergency Medicine, Mount Sinai Hospital, Toronto

Abstract

Background
Teamwork is inadequate in 62% of airway-related death. Cognitive resources are limited during a crisis, leading to errors that may be mitigated with the use of cognitive aids. A cognitive aid would theoretically assist the clinicians through a sequence of complex steps and prevent the omission of key actions. Currently, the impact of an airway cognitive aid in teamwork is unclear. We aim to investigate the role of a newly designed airway cognitive aid on teamwork. We hypothesize that a Cognitive Aid in Difficult Airway and Oxygenation will improve interprofessional team performance.

Methodology
Following ethics approval, consented anesthesia and Emergency Medicine residents (PGY1-2), operating room and Emergency Medicine nurses and respiratory therapists at Mount Sinai Hospital (Toronto) will be recruited. In the pretest session, participants will be randomized into Cognitive Aid (n=25) and Control (n=25) teams. Each team will consist of an anesthesia or Emergency Medicine resident, nurse and respiratory therapist. All participants in both groups will receive a review of the ASA-Difficult Airway algorithm and watch a video on cricothyrotomy. Only the Cognitive Aid teams will be oriented and familiarized with the airway cognitive aid. Physician participants in both groups will then practice surgical cricothyrotomy. Two to four weeks later (Test Session) teams will manage a simulated “can’t intubate-can’t oxygenate” scenario. The airway cognitive aid will be present (poster size) for the Cognitive Aid teams, but absent for the Control teams.

Data analysis
Each team performance will be videotaped and rated by two independent raters. Primary outcomes are teamwork nontechnical skills, measured using the Mayo High Performance Teamwork Scale. Secondary outcomes are time(s) to perform critical key actions. Wilcoxon-test, students’ t-test and intra-class correlation coefficient, will analyze the primary and secondary outcomes and inter-rater reliability, respectively. Statistical significance is p<0.05.

Conclusions
The airway cognitive aid will improve interprofessional team performance in a simulated airway crisis.

How this work will inform research in medical education

“Human factor errors in poor team communication and lack of teamwork are identified as significant contributors to airway-related death and brain injury. Similar to a surgical checklist, airway cognitive aids have been shown to improve performance of the individual physician managing a crisis. Despite the perceived benefits to teamwork, evidence has not consistently demonstrated an improvement in team function with the use of cognitive aids. We have developed a highly visual difficult airway and oxygenation cognitive aid and is assessing it role in inter-professional team work among physician and non-physician care providers during a simulated airway crisis. Our study might provide evidence of a visual airway cognitive aid as an important training tool to improve outcomes of airway crisis in various acute care settings including the operating room, emergency department and paramedic; and to create an atmosphere of inter-professional communication such that any member of the team feels comfortable to make suggestions on management.”

Bin Zheng, MD, PhD

Kinesthetic Guidance for Enhancing Laparoscopic Proficiency

Bin Zheng, MD, PhD
Bin Zheng, MD, PhD
Assistant professor, Endowed Research Chair in Surgical Simulation, Department of Surgery, University of Alberta

Abstract

Despite its importance, our understanding of how surgical skills are acquired through kinesthetic feedback remains rudimentary. Unlike visual feedback that is easily observable, kinesthetic feedback is difficult to record or demonstrate to a learner in a controlled fashion. At the Surgical Simulation Research Lab (SSRL), the implementation of SensAble Phantoms into laparoscopic simulation allows us to record and “‘play back’ kinesthetic feedback generated from expert surgeons. In this study, we plan to examine the role of haptic feedback for teaching novices a laparoscopic transportation task. Twenty junior surgical residents (learners) will be recruited to practice laparoscopic navigation tasks under kinesthetic-guided and self-directed conditions. Over the course of 60 practice trials (30% of total trials), subjects in the kinesthetic- guided group will have a chance to feel and follow an expert surgeon’s performance via laparoscopic instruments interfaced with a haptic device (Phantoms). The learning outcomes for these individuals will be compared with subjects in the self-directed group in the absence of any kinesthetic feedback from experts. We also plan to investigate whether the skills learned with the aid of haptic guidance will be retained more effectively while performing similar tasks in a stressful environment. The knowledge obtained from this study will help generate evidence for a new training paradigm and explore how kinesthetic feedback can be used in the context of surgical education with simulation. We hope that a greater understanding of kinesthetic feedback will reflect in surgical practice by facilitating the development of more robust surgical skills.

How this work will inform research in medical education

“This work is important to medical education because we have developed a novel training platform based on kinesthetic feedback, which opens a new paradigm for teaching healthcare skills besides the conventional vision-motor channel. The system developed and the knowledge gained from this study will explore how kinesthetic feedback can be used in the context of surgical education with simulation. Once proved to be effective, the kinesthetic training platform can record an expert’s hand movement and deliver motion to multiple novice trainees when they start to learn a complex procedure. The system will enable us to significantly increase the opportunity of skill training without adding faculty hours. We also plan to investigate whether the skills learned with the aid of kinesthetic guidance will be retained more effectively while performing similar tasks in a stressful environment. The greater understanding of kinesthetic feedback will enhance surgical skill training and yield better patient outcome.”