The Netherlands 09/25/2018 08/09/2018 Final Review Special Report

U.S. Department of Education

Special Report for The Netherlands

Prepared July 2018


At its fall 1998 meeting, the National Committee on Foreign Medical Education and Accreditation (NCFMEA) initially determined that the standards and processes used by the Netherlands for approving medical education programs were comparable to the United States. The NCFMEA formally reaffirmed its prior determination that the standards and processes used by the Netherlands to accredit medical schools were comparable to those used in the United States in fall 2008. In fall 2011, the NCFMEA formally accepted the update report submitted by the Netherlands regarding its accrediting activities, to include its accreditation of medical schools in Saba. The country's accreditation activities are the subject of this report. NCFMEA reviewed the country's submission in 2017, but had a few remaining questions. This special report submitted for Fall 2018 is the subject of those remaining questions.

Summary of Findings


Staff Analysis

Outstanding Issues

United States Medical Licensing Exam (USMLE) pass rate and residency placement data for students

Country Narrative

1 SUSOM, SELF-EVALUATION REPORT, OCTOBER 2017 The program has an overall graduation rate of 80%. Of all students that successfully complete the program, 88% complete the program in four years. The most important overall indicators of achievement of the intended learning outcomes are the students’ performance on the national licensing examinations as well as number of graduates accepted into residency programs. 1.1 USMLE RESULTS Student Entry Cohort (Year of Matriculation) 2010 2011 2012 2013 2014 2015 USMLE Step 1 # Takers 202 184 227 204 162 57 # Pass First Attempt 198 183 224 203 161 57 First Time Pass Rate 98% 99% 99% 100% 99% 100% # Pass Subsequent Attempt - 1 2 - 1 - Total # Pass 198 184 226 203 162 57 Overall Pass Rate 98% 100% 100% 100% 100% 100% Student Entry Cohort (Year of Matriculation) 2007 2008 2009 2010 2011 2012 2013 USMLE Step 2CK # Takers 153 182 175 196 180 204 126 # Pass First Attempt 141 166 170 192 170 195 120 First Time Pass Rate 92% 91% 97% 98% 94% 96% 95% # Pass Subsequent Attempt 9 16 4 4 9 5 2 Total # Pass 150 182 174 196 179 200 122 Overall Pass Rate 98% 100% 99% 100% 99% 98% 97% USMLE Step 2CS # Takers 152 181 174 196 183 204 110 # Pass First Attempt 145 165 164 182 176 197 106 First Time Pass Rate 95% 91% 94% 93% 96% 97% 96% # Pass Subsequent Attempt 6 16 7 13 6 5 2 Total # Pass 151 181 171 195 182 202 108 Overall Pass Rate 99% 100% 98% 99% 99% 99% 98% 1.2 RESIDENCY PLACEMENT Number of SUSOM graduates 2011 2012 2013 2014 2015 2016 Graduates 151 161 154 192 135 189 Number of SUSOM students accepted in residency programmes 2011 2012 2013 2014 2015 2016 Residencies 146 152 152 184 128 168 In addition to the number of graduates accepted into residency programs, the residency programs continue to academically affiliated, and include a number of well-established hospitals, including: - Hospital of University of Pennsylvania (Anesthesiology) - Allegheny General Hospital/Drexel University College of Medicine (Diagnostic Radiology) - Case Western Reserve University (Family Medicine) - McGill University (Family Medicine) - Johns Hopkins University (Internal Medicine) - University of Toronto (Neurology) - Yale-New Haven Hospital (Preliminary Surgery) 1.3 DOCUMENTS See also Exhibit 0 SUSOM Results for the USMLE results and residency placement. A full list of the SUSOM residency placements for 2015, 2016 & 2017 is included as Exhibit 1a SUSOM Residency 2015, Exhibit 1b SUSOM Residency 2016 & Exhibit 1c SUSOM Residency 2017. 2 NVAO PANEL REPORT, 12 JANUARY 2018 Standard 11, Learning outcomes achieved: The programme demonstrates that the intended learning outcomes are achieved. Explanation: The level achieved is demonstrated by interim and final tests, final projects and the performance of graduates in actual practice or in subsequent programmes. 2.1 Findings Students must pass USLME Step 1 and must achieve approval for their RLRA paper prior to entering clinical rotations. Students must pass Step 2 Clinical Knowledge (CK) and Clinical Skills (CS) before being eligible to graduate. These exams are the same as the standardized exams students have to pass to be eligible for residencies (post-graduate taining programmes). The programme has an overall graduation rate of 80%. Of all students that successfully complete the programme, 88% complete the programme in four years. The most important overall indicators of achievement of the intended learning outcomes are the students’ performance on the national licensing examinations as well as number of graduates accepted into residency programmes. The panel reviewed the performance of SUSOM students and graduates in all these areas, as well as the list of individual SUSOM residency placements for 2015, 2016 and 2017 which were provided in the information dossier. 2.2 Considerations The panel considers the performance of the SUSOM students on the national licensing examinations to be successful. The panel is impressed that over the years the USLME Step 1 and Step 2 (CK & CS) passing rates from SUSOM students maintain on a very high level. The overall graduation rate of 80% and the figure of 88% of the students completing the programme in four years are positive indicators. After successfully passing USLME Step 1 students are not yet allowed to start with their clinical rotations. The 8-weeks course Research: Literature Review and Analysis course must be accomplished (RLRA paper) before students are allowed to start their rotations (cf Standard 3). The panel selected 15 RLRA papers and all panel members read five of these papers covering the range of grading (A to C). Overall the panel members agreed with the final grading given by the SUSOM faculty (cf Standard 2). An important performance indicator for measuring the achievement of the intended learning outcomes is the number of graduates that are accepted into residency programmes of academially affiliated programmes. The panel establishes that for the period 2010 – 2015 the average percentage of SUSOM graduates accepted into residencies is 94%. The panel considers this as a successful performance. The panel is convinced that the intended learning outcomes are achieved by students upon graduation. 2.3 Conclusion The panel assesses standard 11 as satisfactory. The full NVAO Panel Report is included as Exhibit 2 Assessment Report MD-program SUSOM_12Jan18.

Analyst Remarks to Narrative

In October 2015, the Secretary’s letter notified the Netherlands that based upon Saba's status of as a special municipality within the country of the Netherlands, that a formal action was being taken to remove Saba as a separate entity on the list of NCFMEA comparable countries. As a result, the request for this special report was made for the country to report on the status of the Accreditation Organization of the Netherlands and Flanders (NVAO)'s standards regarding the SABA University School of Medicine. The country provided an update to the NCFMEA in 2017 regarding its midterm report from the review of the SABA University School of Medicine that indicates the SABA University School of Medicine was reviewed in January 2016. This review was comprehensive. NVAO has provided documentation of USMLE, as well as information about residency placement of its students. The results state that for the period 2010 - 2015 the average percentage of SUSOM graduates accepted into residencies is 94%. The inclusion of this information addresses the questions that were raised by NCFMEA as it relates to understanding student outcomes.

The NVAO's process for the review of clinical sites, including how information is used to improve the program

Country Narrative

1 A group of external experts peer-reviewed the SUSOM program according to the NVAO standards. The expert panel assessed various quality aspects based on the self-assessment and a site visit. The emphasis in the panel review process and the report is on quality assurance (accountability) and quality enhancement (recommendations). 10 July 2017 NVAO decision on panel composition October 2017 SUSOM self-assessment 15 and 16 November 2017 Saba site visit 12 January 2018 Panel report Fall 2018 NVAO accreditation decisions for all 19 medical programs in the Netherlands, including Saba 2 NVAO panel: •Prof. dr. Ferry Breedveld (chair); professor emeritus internal medicine and rheumatology; former CEO Leiden University Medical Centre; •Prof. Alice Fornari Ed.D.R.D.; professor Science Education, Occupational Health and Family Medicine, Hofstra Northwell School of Medicine, New York; •Prof. dr. Dirk Ruiter; professor emeritus pathology, former dean and vice president Radboud University Medical Centre, Radboud University Nijmegen; •Dr. Susan Cox, MD; Executive Vice Dean for Academics, Dell Medical School at the University of Texas, Austin; •Joseph Wiley, MD (student member); resident physician - family medicine, Charleston Area Medical Centre. Both prof. Breedveld (Leiden) and Ruiter (Nijmegen) were also chair c.q. panel member assessing all 18 medical programs in the Netherlands. Dr. Cox did not take part in the Saba campus site visit. At the request of SUSOM and in agreement with NVAO, Dr. Cox acted since August 2017 as an external reviewer for the evaluation of SUSOM’s clinical teaching sites with residency training programmes (ACGME- approved). She started her site visits on 21 September 2017 at Greater Baltimore Medical Center and St. Agnes Health Care Ascension and she recently reviewed O’Connor Hospital, San Jose (13 October 2017). The focus of her review is to ensure the quality of teaching and the adequacy of the clinical experiences at each site as well as adequacy of supervision. She reviews submitted materials prior to arrival, then interviews leadership, faculty and staff, as well as clinical students to confirm experiences and standardization. Additionally, she inspects the physical facilities. Her predecessor visited 26 clinical sites during the period 2014-2017. Dr. Cox reviewed so far nine of the site visit reports done by her predecessor. In a letter dated November 13, 2017 the panel was informed of her findings. This letter is included as Exhibit 4 Letter Dr. Cox_13Nov17. 3 The panel established that SUSOM develops and maintains affiliations with ACGME-approved teaching hospitals in the United States and Canada to provide clinical rotations. ACGME-approval implies standards for the availability of a learning infrastructure for SUSOM’s students at all clinical sites. All sites make use of the same SUSOM core clinical curriculum. To validate the appropriateness of the intended learning outcomes as well as the required clinical encounters, SUSOM sent the core clerkship syllabi for external review before implementation. The panel finds that the revised Clinical Medicine curriculum contributes to an increased level of central oversight and control. The main principal features are: revised core clerkship assessments, NBME Clinical Subject Shelf Exams, revised clinical faculty evaluations, required patient notes & patient logs, and engaged learning experiences with reflections. The panel is convinced that a main objective of the revised clinical curriculum: “to provide a more comprehensive oversight into the progress of each student, as well as to ensure comparability of the educational programme between sites” has been achieved. Students are highly monitored in their progress in achieving clerkship objectives and are given formative assessment and feedback. To determine whether the curriculum is successful in helping students achieve the intended learning outcomes, students are required to take the national subject NBME exams as part of their clerkship assessment. Halfway through each clerkship students are required to discuss and document their progress with a clinical faculty member in order to provide them with formative feedback on their progression through the competencies. The panel considers the comprehensive Clinical Medicine Handbook a leading document and of vital importance for all clinical students, faculty and clinical sites. It contains for each core rotation a syllabus and the intended learning outcomes. In the past years, SUSOM further formalized its working relationship with its faculty at the clinical teaching sites. All clinical faculty members have been formally appointed as SUSOM teaching faculty in one of the three faculty ranks. As part of the external reviews of the ACGME-approved clinical teaching sites, there were interviews with the involved faculty and staff. All relevant documentation was provided in advance, including CV’s. These CV’s were reviewed. and all deemed satisfactory. The school also provides regular webinars targeted for faculty at clinical sites - available to all faculty members - that provide teaching techniques across a variety of clinical settings, as well as methods for both formative and summative assessment of student performance. The panel concludes that in the ACGME-approved clinical teaching sites there is a strong oversight over residency faculty. Feedback from students’ end of clerkship surveys is reviewed each term by by the Quality Assurance Committee, Examination Committee, and the Curriculum Committee. Core clinical clerkships are reviewed at least annually by the Clinical Department Chairs. In addition to reviewing curricular content, the clinical teaching sites are reviewed. To this end, SUSOM periodically hires external, independent reviewers to evaluate the sites. At the end of each core rotation preceptors complete the clinical evaluation form. After every core clinical rotation, students complete an evaluation of their clerkship that includes the quality of teaching at that site. These end-of-clerkship evaluations are reviewed by the clinical staff. If end-of-clerkship evaluations indicate issues with a particular site or clinical faculty member an intervention for remediation will be made. The panel establishes that the clinical teaching during the core rotations take part in ACGME-approved hospitals affiliated with SUSOM. The panel is confident that ACGME-approved residencies safeguard the quality of teaching and learning during the clerkships. The reports and the reviews conducted at the clinical training sites by the external reviewer confirm that teaching and supervision are consistent with the standards of SUSOM. Site visit reports are included as Exhibit 3 Binder Reports Clinical Site Reviews. 4 According to the European Standards and Guidelines on Quality Assurance (ESG) all quality assurance activities have the twin purposes of accountability and enhancement. NVAO Standard 9 on quality assurance stipulates that a programme should be evaluated on a regular basis, and that the outcomes of these evaluations constitute the basis for demonstrable measures for improvement. Detailed information on the topic in both the SUSOM self-assessment (October 2017) and the NVAO Panel Report (12 January 2018) is included in Exhibit 5 Quality Assurance. The document includes one recommendation that is directly related to quality assurance.

Analyst Remarks to Narrative

In the country's last review, it provided information stating that a review of the clerkship sites was conducted, but that the report of the visits is considered confidential. In response, NCFMEA asked the country to report back on the review of these sites. In response to the request, the country has provided documentation of its assessment report (exhibit 2) that explains the panel composition and the review of these sites. This information demonstrates a review of the curriculum, the faculty, the physical review of the clinical sites, and includes feedback from students. In addition, the country has provided a copy of the European Standards and Guidelines on Quality Assurance that also adds additional accountability for the country as it reviews clinical clerkship sites. The inclusion of this information resolves the concerns raised by the NCFMEA during the last review.