Policy & Procedures for the Formal Review of Examinations and Practice Based Assessments

Date approved: November 2017
Subject: Royal College Formal Review Policy
Review Date: June 2020
Approved by: Royal College Assessment Committee


This document outlines the policy and procedures for the formal review of examinations (including marks) and practice based assessments of the Royal College of Physicians and Surgeons of Canada (the “Royal College”). It is intended for candidates, Chairs and members of Discipline Specific Examination Boards (including invigilators appointed by the Royal College), Examination and Assessment Committee members, and Royal College staff

Review of Marks

All examination scores are subjected to multiple levels of verification and quality assurance prior to being reported to candidates. This ensures that each candidate’s performance is appropriately assessed and their scores are accurately reported. The Royal College has implemented a thorough and rigorous process in this regard involving numerous steps and safeguards, including a dedicated data verification and quality assurance team that conducts the following activities:

  • Verification of Multiple Choice Question (MCQ) answer sheets and data transfer;
  • Verification of Short Answer Question (SAQ) scores and data; and,
  • Verification of Oral, OSCE and Practical scores and data.

Once candidate performances have been reported by the Royal College and the Summary of Performance has been issued, candidates who wish to request a further review of their marks may email their request to: formalreviews@royalcollege.ca. All such requests must be submitted within 30 days of the date appearing on the Summary of Performance and must be accompanied by a certified cheque or money order in the amount of $300 payable to the Royal College of Physicians and Surgeons of Canada. A request for a review of marks that does not include the required administration fee within the time set out above shall not be processed or considered.

Please note that any request for a review of marks by the Royal College will be strictly limited to a review for any potential mathematical or mechanical errors (e.g., marks mistabulated, answer sheets missed, etc.). A review of marks will NOT include any form of re-reading and/or re-grading of a candidate’s answers. In the event that a marking error is identified, the $300 administration fee will be reimbursed.

Formal Reviews

Policy

Formal reviews of examinations and practice-based assessments are conducted only on the basis of alleged significant procedural irregularities in the assessment process, not because of alleged errors in content. The exclusion of errors in content applies to allegations of errors in either the questions and structure of the examination, or the evaluation and content of the responses provided by the candidate.

The review process is not a re-grading or re-scoring exercise. This scope of review is designed to act as a safeguard for the validity of the examination and assessment process.

The distinction between content reviews and process reviews is of the utmost importance. Requests for reviews based on alleged errors in content will not be processed. A request for a formal review will only be considered in the event of a procedural irregularity serious enough to affect materially the candidate's performance.

Examples of potential issues that do NOT constitute process irregularities include:

  • Disagreements or concerns regarding the content and/or structure of the examination or practice based assessment;
  • Disagreements or concerns regarding the grading of the examination or practice based assessment;
  • Concerns regarding the extent of post-examination or post-practice based assessment feedback provided to candidates;
  • Requests for disclosure of Royal College documentation and/or records;
  • Mere fact of repeat examiners;
  • Requests for the Royal College to consider information not normally considered as part of the evaluation process.

Purpose

Formal reviews of examinations and practice based assessments are part of the evaluation and quality improvement system used by the Royal College to grant specialist certification. They provide a means to investigate the circumstances of the written, oral or other type of examination or practice based assessment administered to one or several candidates, and to ascertain whether the process was carried out under conditions appropriate to the conduct of an examination or practice based assessment as determined by the Discipline Specific Examination Boards and approved by the Examination Committee of the Royal College. Formal reviews of examinations and practice based assessments involve candidates, Royal College examiners, the Examination Committee, the Assessment Committee and the Office of Specialty Education in an effort to identify any significant irregularities in the conduct of the examination process and any procedural unfairness materially affecting one or several candidates.

Principles

Examiners in a given specialty are authorities on content. Their judgment relative to the correctness of a candidate's answers is not open to challenge, otherwise the review would become a content-oriented discussion between experts.

An examiner’s decision with respect to the correctness or completeness of a response and their assessment of a candidate’s knowledge and/or skills is final. While a candidate may disagree with their decision, such disagreement does not constitute grounds for review.

On occasion, deviations from the stated format or conduct of the examination may be unavoidable and irregularities may occur that do not result in any unfairness or significantly affect the performance of the candidate.

Procedures

There are three pathways to initiate a formal review:

Immediate notification to the local examination or practice based assessment authority by the candidate

If a candidate believes that one or several process irregularities has r occurred in the course of an examination or practice based assessment, he or she should immediately bring this to the attention of the responsible local authority (i.e., the Chair of the Discipline Specific Examination Board (or a delegate), an invigilator, or the local coordinator). Examiners and staff of the Office of Specialty Education are also expected to immediately report any material process irregularities. The person in charge may then contact the Director of Assessment, or his or her delegate, for immediate advice with respect to correcting the irregularity, whenever possible. In any such case, a written report must be sent as soon as possible to the Office of the Director of Assessment.

Reporting at the time of the incident affords an opportunity to resolve most concerns or irregularities. It also provides a record of events. This pathway does not preclude further requests for review on the part of the candidate concerned.

Reviews originating within the Office of Specialty Education

Where the Office of Specialty Education becomes aware of a material process irregularity, it may initiate the review mechanism on behalf of one or several candidates. In such a case, the procedure described in this document shall be adhered to, but no fee shall be required from the candidate(s) involved. The Office of Specialty Education may initiate a formal review at any time, even beyond the deadline stated for candidates. Where a candidate or candidates are materially affected by such a review, they will be notified by the Director of Assessment, or his or her delegate, at the most opportune time.

Where a review initiated by the Office of Specialty Education identifies a material process irregularity in the procedure, the Director of Assessment, or his or her delegate, will make a recommendation regarding appropriate remedial actions to the Chair of the Discipline Specific Examination Board. The recommendations may include allowing a repeat examination.

Reviews initiated by a candidate submitting a written request after the examination

A candidate wishing to request a formal review must notify the Office of the Director of Assessment of the Royal College in writing to formalreviews@royalcollege.ca within 30 days of the date appearing on the Summary of Performance or, in the event that a Review of Marks was previously requested, within 30 business days of receipt of the Royal College’s response in that regard.

Before submitting a request for a formal review, candidates must ensure that their concerns relate to one or more alleged irregularities in the evaluation process, not a content, grading or other non-process issue. As noted previously, candidates should also keep in mind that process irregularities are only considered relevant when serious enough to affect materially the candidate’s performance. Procedural irregularities of a minor nature or that are appropriately addressed and rectified (if necessary) at the time of the examination, and that do not significantly affect his or her performance, will NOT be considered material for the purposes of a formal review.

The request must explain, completely and in detail, all of the circumstances surrounding the alleged process irregularity and its effect on the candidate’s performance.

The request must include a certified cheque or money order in the amount of $750 payable to the Royal College of Physicians and Surgeons of Canada.

The Director of Assessment, or his or her delegate, shall acknowledge receipt of the request and deposit the administration fee.

A request for a formal review that does not include the required administration fee in the prescribed form within the time set out above shall not be processed or considered.

Procedure: Performing reviews initiated by candidates submitting written requests

Step I: Initial Review by the Director of Assessment, or his or her delegate

Where a candidate initiates a request for review, the request shall be reviewed by the Director of Assessment, or his or her delegate, to determine:

  • whether the request sets out in sufficient detail the grounds for review and the relief sought;
  • whether the request pertains to a process issue or a content issue; and
  • whether the request is scandalous, frivolous, vexatious, made in bad faith or otherwise an abuse of process.

If the Director of Assessment, or his or her delegate, determines that the request for review does not set out in sufficient detail the grounds for review and the relief sought, or that the request pertains exclusively to a content issue rather than a process issue, or that the request is scandalous, frivolous, vexatious, made in bad faith or otherwise an abuse of process, he or she shall advise the candidate in writing within 15 days of making the determination. In such a case, the candidate shall be provided a further 15 days from the date of notification by the Director of Assessment, or his or her delegate, in which to remedy any deficiency and/or to make written submissions outlining reasons why the request for a formal review should be permitted to proceed. Within 15 days of receipt of such additional submissions, the Director of Assessment, or his or her delegate, shall determine whether the request for a formal review should proceed. The decision of the Director of Assessment, or his or her delegate, is final and not subject to review.

If the Director of Assessment, or his or her delegate, determines that the request for a formal review should proceed, he or she shall, within 45 days of receipt of a complete and final request for a formal review, evaluate the request and conduct any enquiries that he or she may deem useful, and provide a written decision to the candidate as to whether or not the grounds for a formal review alleged by the candidate are founded and, if so, what remedy should be accorded, if any. The decision of the Director of Assessment, or his or her delegate, shall be final subject only to a formal review as set out below.

Where the Director of Assessment, or his or her delegate, decides the grounds for the formal review are founded, and to grant a repeat examination or practice based assessment at the next examination session or practice based assessment session of the specialty and a refund of the candidate’s examination or assessment fees, this concludes the formal review process and the candidate may not move the formal review process forward to an appeal panel.

Where a candidate disagrees, in whole or in part, with the decision of the Director of Assessment, or his or her delegate, the candidate may, within 15 days of the date appearing on the written decision, advise the Office of the Director of Assessment, in writing, that he or she wishes for the decision to be reviewed by a Formal Review Panel. The candidate shall specify whether he or she wishes for the Formal Review Panel to proceed by way of paper review or an oral review. Requests for a paper review by a Formal Review Panel must be accompanied with an administration fee in the amount of $1,500 by certified cheque or money order, payable to the Royal College of Physicians and Surgeons of Canada. Requests for an oral hearing by a Formal Review Panel must be accompanied with an administration fee in the amount of $1,750 by certified cheque or money order, payable to the Royal College of Physicians and Surgeons of Canada.

Step II: Review by a Formal Review Panel

If a candidate seeks a review of the decision of the Director of Assessment, or his or her delegate, whether by way of a paper review or an oral hearing, a Formal Review Panel consisting of three members of the Examination Committee and/or the Assessment Committee shall be appointed. The members of the Formal Review Panel may NOT be members of the Discipline Specific Examination Board in the candidate’s specialty, or faculty members in the candidate’s residency program, nor certified in the candidate’s specialty. One of the Formal Review Panel members will be designated as its Chair.

The Director of Assessment, or his or her delegate, in consultation with the Formal Review Panel, shall set a date for the deliberations of the Formal Review Panel (in the case of a paper review) or for the oral hearing by the Formal Review Panel (in the case of an oral hearing) within 90 days of receipt of a candidate’s request for a review by a Formal Review Panel.

At least 45 days prior to the date for the Formal Review Panel’s deliberations or oral hearing, as the case may be, the Office of the Director of Assessment shall provide to the candidate the documents and records relevant to the request for a formal review. The Office of the Director of Assessment may refuse to disclose certain information, records and/or documents that may, in the opinion of the Director of Assessment, or his or her delegate:

  • undermine the integrity of any aspect of the examinations and/or practice based assessments process, the formal review process, or any other function of the Royal College or other matter within its authority;
  • disclose financial or personal information or other matters of such a nature that the desirability of avoiding their disclosure in the interest of any person affected, or in the public interest, outweighs the desirability of adhering to the principle that reasonable disclosure be made;
  • be subject to privilege; or
  • for any other reason of public interest.

A candidate in receipt of documentary disclosure from the Royal College who wishes to address any new facts or issues raised by the disclosure may prepare and deliver to the Director of Assessment, or his or her delegate, within 30 days of the Formal Review Panel’s deliberations or oral hearing, as the case may be, further brief written submissions.

In the case of a paper review by the Formal Review Panel, the Director of Assessment, or his or her delegate, may reply in writing to any new submissions and/or evidence tendered by the candidate that was not already provided during the course of the review.

Within 21 days of the date for the Formal Review Panel’s deliberations or oral hearing, as the case may be, the Director of Assessment, or his or her delegate, shall provide to each member of the Formal Review Panel all materials, documents and evidence submitted by the parties.

In the case of a paper review, the Formal Review Panel’s deliberations shall be carried out under conditions of strict confidentiality. Deliberations amongst the members of the Formal Review Panel may be conducted in person or by electronic means (telephone conference, video conference, etc.), in whole or in part.

In the case of an oral hearing:

  • At least 20 days prior to the oral hearing, a candidate must notify the Director of Assessment, or his or her delegate, whether the candidate intends to appear in person or be represented by counsel, or both, and whether the candidate wishes for the oral hearing to proceed by way of an electronic hearing (telephone conference, video conference, etc.), in whole or in part.
  • A candidate’s request for an electronic hearing may be denied on the grounds that it would be unduly cumbersome, is likely to cause prejudice to any party, or is likely to prevent or limit the Formal Review Panel’s ability to apprehend the evidence and submissions to be made.
  • The Royal College may be represented by counsel at all stages of the formal review process and shall have full standing at the oral hearing, including the right to make representations, lead evidence and challenge the evidence of the candidate.
  • Where the Chair of the Formal Review Panel considers it appropriate, he or she may, prior to the oral hearing, request that independent legal counsel be retained for the Panel at the Royal College’s expense. Such counsel shall be appointed from a law firm other than that of counsel to the Royal College. In making such a determination, the Chair may consider the complexity and nature of the request for a formal review, the likelihood that significant legal or procedural issues will arise in the course of the review, or any other factors that may be relevant in the circumstances.
  • Requests for adjournments will not be granted as of right and will only be granted in exceptional circumstances. A request for adjournment must be made as soon as possible to the Formal Review Panel, and shall include the reasons for the request, and the amount of time needed for the adjournment. The Formal Review Panel may decline to accept the candidate’s convenience or that of its representative as a sufficient reason for adjournment.
  • The Formal Review Panel is the master of its own procedure in relation to the conduct of the oral hearing. However, in the ordinary course, the conduct of the oral hearing shall proceed as follows:
    • Absent permission from the panel, no new evidence is to be introduced at the hearing.
    • Counsel for the Royal College will provide a brief (5 minutes) opening statement outlining the material facts and its position in relation to the request for a formal review;
    • The candidate, or his or her counsel, will provide a brief (5 minutes) opening statement outlining the material facts and its position in relation to the request for a formal review;
    • The candidate will then be asked to present his or her case (40 minutes).
    • Counsel of the Royal College will then be asked to present its case (40 minutes).
    • The members of the Formal Review Panel may at any time ask any question of the candidate, counsel for the candidate, or counsel for the Royal College for further clarification.
    • The candidate, or his or her counsel, may then provide a closing statement (5 minutes);
    • Counsel for the Royal College will then provide a closing statement (5 minutes).
  • At the close of the oral hearing, the Formal Review Panel shall retire and deliberate.

Within 30 days of its deliberations, whether in the case of a paper review or oral hearing, the Chair of the Formal Review Panel shall issue in writing the Panel’s decision and any other relevant information or recommendation to the Office of the Director of Assessment. The available decision options are as follows:

  • No process irregularities occurred;
  • There was a process irregularity, but NOT of such a magnitude or impact that it could affect materially the candidate’s outcome of the examination or practice based assessment; or
  • There was a process irregularity of sufficient magnitude or impact to affect materially the candidate’s outcome of the examination or practice based assessment, and a repeat examination or practice based assessment is allowed at the next examination session or practice based assessment session of the specialty.

In cases when a repeat examination or practice based assessment is granted, the examination or practice based assessment subjected to review is to be considered null and must not be counted in determining the candidate's eligibility for certification, but such eligibility remains subject to confirmation by the Credentials Unit.

A repeat examination shall include all components of the original examination, in those specialties using comprehensive objective (COE) examinations, in which success on one component is not a prerequisite for passing the other component or components, nor for achieving certification.

Where a candidate is successful, in whole or in part, on review, the Formal Review Panel may, in its discretion, recommend a refund of the examination or practice based assessment fee, the formal review fee, or both fees where a repeat examination or practice based assessment is granted. The Royal College shall also waive the examination or practice based assessment fee for the repeat examination/assessment.

There is no provision for further review by the Royal College of the decisions of the formal review panel after a paper review or oral hearing, as the case may be, which are considered final.

The Director of Assessment, or his or her delegate, shall inform the candidate of the Formal Review Panel’s decision. The deliberations of the Formal Review Panel and the documents used in the formal review are strictly confidential.