Grant Recipients

Royal College Research on Competency-based Medical Education Grant Recipients

Tavis Apramian, MA, MSc, PhD

Tavis Apramian, MA, MSc, PhD

Centre Fellow, Centre for Education Research & Innovation, Schulich School of Medicine & Dentistry, Western University

Project Title: “Exploring the Influence of Surgical Variation on Competence Judgments in Postgraduate Surgical Education”

The assumption that experts hold the same idea of competence may lead workplace education and assessment astray. Our work explores how surgeons cultivate their own approaches to procedures. These individual variations, which many consider the ‘art of surgery,’ have important educational consequences. Learners must adjust to them. Learners must reconcile them with what they previously learned. And learners must anticipate the role of procedural variations in the workplace-based assessments against which they are measured. This program of research maps intraoperative variations and links to those variations to patterns in assessment. We plan to develop teaching tools that can help both surgeons and resident adapt to the complexity of procedural variations. Most importantly, though, we seek to continuously refine how workplace-based assessment functions to capture competence and entrustment in real time.

Project keywords: surgery, variation, workplace, assessment

Zia Bismilla, MD, FRCPC, MEd

Zia Bismilla, MD, FRCPC, MEd

Pediatrician, Division of Pediatric Medicine, Department of Pediatrics, Hospital for Sick Children

Assistant Professor, Department of Pediatrics, University of Toronto

Project Title: “Assessment of physician handover in the era of competency-based medical education”

Recognizing the relationship between competence and the quality and safety of patient care, our project aims to improve handover skills for residents and resultant patient care for Canadian children. We will gather validity and reliability evidence for use of a competency-based handover assessment tool in the workplace. Additionally, we will evaluate usability and applicability for everyday use. Our tool, which uses an entrustment-based scale, will provide a guide for programs to facilitate development of their trainees' handover skills, one of the important new competencies in CanMeds 2015. It will also provide educators with a competency framework for teaching handover skills and facilitate the provision of timely, meaningful and task-specific performance feedback to guide and support trainees’ learning of these critical skills.

Project keywords: competency-based assessment, team performance, communication, handover

Mitchell G. Goldenberg, MBBS

Mitchell G. Goldenberg, MBBS

Resident Physician, Division of Urology, Department of Surgery, University of Toronto
PhD Candidate, Institute of Medical Science, University of Toronto

Project Title: “Surgical Education - Setting Performance-Based Standards in Robotic Surgery”

Outcome-based assessments are a central component in the new paradigm of educational reform, across the continuum of medical education. Currently accepted methods of setting competency standards typically rely on the judgment or relative performance of content experts. We devised a method for setting standards in technical competency that aims to minimize variation in oncological, functional, and patient-centred outcomes, supported by recent literature demonstrating the role of procedural technical skills in predicting patient postoperative outcomes. This multi-surgeon, multi-centre study will provide important validity evidence toward this standard setting method, by analyzing the technical ability of trainees and faculty performing robotic-assisted radical prostatectomy (RARP). This procedure has had a sharp rise in prevalence across Canada and around the globe, despite no unified or formalized credentialing process in place. We believe this study will generate important hypotheses and discussion going forward around disruptive methods of standardizing surgical training and accreditation, across all procedural competencies.

Project keywords: technical skill assessment, standard setting, patient outcomes, credentialing

Stefanie Sebok-Syer, PhD

Stefanie Sebok-Syer, PhD

Postdoctoral Fellow, Centre for Education Research and Innovation, Schulich School of Medicine and Dentistry, Western University

Project Title: “Assessing residents’ independent and interdependent clinical performance using electronic health records”

While we are gaining more insight into the nature of interdependent performances in medical education, there exist limited measurement approaches to guide the assessment of clinical situations where interdependent performances are prominent. An ability to assess independent resident performance is a key premise of CBME; however, in clinical settings, performance outcomes can be difficult to link to one particular individual as performance is often interdependent with other team members, particularly faculty supervisors.

A currently untapped source of data to explore aspects of independent and interdependent clinical performance is the electronic health record (EHR). But, a key assumption with EHR data is that a specific data point can be attributed to an individual physician’s performance. This assumption might not hold true for residents. Our study investigates how data collected in the EHR might be meaningfully and appropriately used to assess residents’ independent and interdependent clinical performance.

Project keywords: assessment, electronic health records, collective competence, interdependence

Matthew Sibbald, MD MHPE PhD FRCPC

Matthew Sibbald, MD MHPE PhD FRCPC

Assistant Professor & Interventional Cardiologist, Faculty of Medicine, Division of Cardiology, McMaster University

Director, Centre for Simulation Based Learning, Faculty of Health Sciences, McMaster University

Adjunct Scientist, McMaster Faculty of Health Sciences Program in Education Research, Innovation and Theory

Project Title: “Exploring “Translational Activities” in Developing Assessment Programs for Competency Based Medical Education”

Building assessment programs for competency-based medical education requires translating outcome frameworks comprised of entrustable professional activities, milestones and competencies into assessment tools and programs to assess trainees’ progress in clinical work. The validity or trustworthiness of these assessment programs is shaped by this translation process. Specific residency programs may approach these translational activities in distinct ways. Despite their importance, these different translational activities are rarely made explicit, evaluated or their consequences elaborated. Given the foundational role of assessment plans and processes in informing and moderating decisions about competence made by assessment committees, further understanding this process of translation is essential to the success of competency based medical education.

Project keywords: assessment, evaluation, validity, competency based medical education

Ranil Sonnadara, Ph.D.

Ranil Sonnadara, Ph.D.

Director, Education Science & Associate Professor
Department of Surgery, McMaster University

Associate Professor, Department of Surgery, University of Toronto

Project Title: “Collective/group decision making process: How do competence committees make decisions?”

Competence committees have become the new standard for making decisions about trainee progression through competency-based medical education programs. However, since there is presently very little research on how competence committees make decisions, the role that contextual factors—for example, prior knowledge about a resident under review or the raters who completed their evaluations—will play in decision making is currently unknown. Our research will explore the role that context plays in competence committee decision-making processes through interviews, naturalistic observation, and an experimental study. This work will help maximize the effectiveness of competence committees, including their composition, access to, and interpretation of data to generate sound judgments of residents’ performance. We also anticipate that our findings will be useful with respect to member training, resource allocation, and ultimately, the creation of evidence-based guidelines for how competence committees should make decisions.

Project keywords: context, group decision making, competence committees, competency-based medical