U.S. Department of Education Staff
Staff Analysis of the Report Submitted by CAAM-HP on behalf of Jamaica
Prepared September 2017
Jamaica submitted its initial application for a review of comparability for consideration at the Fall 2014 NCFMEA meeting. The Caribbean Accreditation Authority (Medicine and Other Health Professions) Act of 2006 formally authorizes the CAAM-HP to accredit programs of study in medicine and other health professions offered by institutions in Jamaica, and an Inter-Governmental Agreement with Jamaica establishes the CAAM-HP with the responsibility for the monitoring and continued certification/licensure of its medical schools. Medical schools wishing to operate in Jamaica must first attain CAAM-HP accreditation and its recommendation to the Minister of Education. At the Fall 2014 NCFMEA meeting, the country was asked to submit an update report to include information on the status of the development of any new standards and evaluation procedures being considered by CAAM-HP, specifically for the country of Jamaica. That report was deferred until the Fall 2017 meeting and is the subject of the current staff analysis.
Additional information is requested for the following questions. These issues are summarized below and discussed in detail under the Staff Analysis section.
-- The Committee may wish to request additional information from CAAM-HP regarding any actions that the agency has taken regarding UWI/Mona's declining first time graduation exam pass rates.
Please provide a review of your Fall 2016 meetings that includes information on the status of the development of any new standards and evaluation procedures being considered by the CAAM-HP, specifically for the country of Jamaica.
In keeping with its Procedures for revision of standards (Exhibit 1) the CAAM-HP Secretariat commissioned a sub-committee to review its accreditation standards for medicine over the period January 2016 and March 2017. Membership of the committee comprised:
• 1 representative from an accredited medical school
• 1 CAAM-HP member
• 1 representative of the Caribbean Association of Medical Councils (CAMC)
• 1 person from outside of the region with experience with the CAAM-HP accreditation standards and processes
• 1 experienced CAAM-HP site visitor
• The Executive Director as an ex-officio member
• maintained the current format of the CAAM-HP standards
• considered the existing standards in the light of the recently revised LCME standards
• compared the CAAM-HP standards against the revised LCME standards and identified discrepancies resulting in the addition of a number of new standards to the current CAAM-HP standards
• changed some standards from “should” to “must”
• amalgamated some standards where they overlapped
• renumbered the standards in the various sections to take account of the additions and amalgamations
• was mindful that the standards must be applicable both to regional and to “off shore” medical schools
In considering the standards the Committee unanimously agreed that the Internship and Continued Professional Education (CPE) sections did not fit well with the process of accrediting medical schools.
The Committee took into consideration the fact that Internship is not part of the undergraduate medical education programme nor is it the responsibility of the schools and hence should not be a part of the assessment process.
The question of quantitative standards has been controversial for CAAM-HP. Internationally, the trend seems to be that less is more in this area; it was the Committee’s view that it is more important to capture a small number of key, robust and verifiable data than to ask for vast amounts of data which are difficult to collect, verify and impossible to manage effectively.
The Committee felt that external examinations and residency rates could not be made compulsory standards and that students who choose not to sit the USMLE and apply for residency must not be put at a disadvantage. However, as many of the schools in the region are geared to successful USMLE and residency placements it was accepted that data should be collected and evaluated for individual schools and be included in the database. If a school has these as important goals for their programme they should form part of the accreditation judgement on that school. For schools where these are not major programme goals and where student rates of sitting these examinations are low these data would not be an important part of the assessment. External examinations are not necessary for graduates of the University of the West Indies (UWI) or the University of Guyana (UG) as the successful completion of the undergraduate course is followed by a period of internship which, when successfully completed, allows the doctor the privilege of Full Registration. There is no requirement for sitting further examinations. The vast majority of these graduates remain in the region and either enter postgraduate residency programmes or General Practice in the region
The differences between the regional and the “off shore” schools mean that this flexibility is required. Finally, since CAAM-HP does not insist that a proportion of students at all schools must obtain external examinations or residency placements the Committee determined that these cannot be requirements.
Should there be questions concerning the methods of assessment or standards of the assessment programme then it may be necessary to seek a benchmark examination and then require the school to mandate their students to sit this benchmark examination before graduating.
The draft revised standards were presented to CAAM-HP’s Annual General Meeting held in Jamaica, July 27-29, 2016. With respect to quantitative standards the meeting agreed with the Committee’s recommendation to use only objective quantitative data such as examination pass rates, attrition, and placement in postgraduate training.
The meeting also discussed the issue of including measures for student performance outcomes in the draft revised standards. The NCFMEA had suggested that CAAM-HP should base its accreditation decisions, in part, on the effective use of data in evaluating the performance of students after graduation from the medical school. CAAM-HP should, therefore, establish student performance outcome standards such as acceptable numbers of graduates from the school passing a licensing examination and an acceptable percentage of all students graduated during the preceding year that obtained placement in an accredited US postgraduate medical training programme to determine whether to grant accreditation.
While acknowledging that such data would be useful, the meeting expressed concern about including performance outcome data in the standards and noted that this was not a requirement of the LCME. The meeting agreed that this matter should be considered carefully at the 2017 Annual General Meeting.
While there was general agreement in principle with the draft revised accreditation standards members were given two weeks to comment further following which any suggestions/amendments would be forwarded to the Committee for consideration.
The majority of the members responded indicating agreement with the proposed new standards, strongly supported the removal of the Internship and Continued Professional Education standards as they are not a required part of the undergraduate medical education programme. Members were also pleased that they would not have to adopt additional quantitative standards as these would have been not only difficult to provide but also somewhat useless without internationally accepted benchmarks.
CAAM-HP hosted what has been described as very successful and well received Capacity Building Workshop in Barbados, March 19-22, 2017. The over 100 participants in attendance represented 24 medical schools in the region. The main purpose of the Workshop was to introduce the draft revised accreditation standards for medicine, their interpretation, application and assessment. This was followed by Group deliberations using case studies which helped participants to identify the relevant accreditation standards. The programme and presentations can be accessed on the CAAM-HP web site at: http://www.caam-hp.org/workshop.html.
The two areas of significant concern during the extended discussion of the new standards were: the examination results data particularly from the schools whose graduates do not sit external examinations and the dropping of the Internship standards. However, a number of schools wanted to be able to supply some data on the participation of their graduates in Internship and residency programmes. Consequently, changes were made to Medical Education Database-Section III-Educational Programme Part A and Standard ED-46, Exhibit 3C.
The proposed new standards are appended and are highlighted in bold and underlined for ease of identification, Exhibits 2 and 3 and will be presented for final approval to the CAAM-HP Annual General Meeting scheduled for July 2017 to come into effect on August 1, 2018.
Analyst Remarks to Narrative
At the Fall 2014 NCFMEA meeting, the country was granted an initial determination of comparability, but asked to submit an update report to include information on the status of the development of any new standards and evaluation procedures being considered by CAAM-HP, specifically for the country of Jamaica.
In its current report, the country states that CAAM-HP formed a subcommittee comprised of five members, plus the CAAM-HP Director as an ex officio member, to review CAAM-HP standards over a period of 14 months in 2016-2017. The subcommittee developed revised draft standards, which were presented for discussion by the member countries, including Jamaica, at CAAM-HP's general meeting in July 2016. Concerns were raised in some areas, which were then scheduled for further deliberation at the July 2017 general meeting. The members were also given two additional weeks following the 2016 meeting to submit written suggestions to the subcommittee for its consideration in further revising the standards. CAAM-HP subsequently hosted a workshop focused on the revised standards in March 2017 that was attended by 100 participants from 24 medical schools in the region. The proposed standards are scheduled for final approval by the member countries at the CAAM-HP general meeting in July 2017 and will go into effect as of August 1, 2018.
The country provided a copy of CAAM-HP's procedures manual, which specifies that the agency's standards will be reviewed on a five-year cycle with input from appropriate constituent groups (Ex. 1, pp. 10, 37). It appears that the agency followed its published procedures, giving extensive opportunities for stakeholders to provide feedback on the draft standards. [Note: ED staff found that page 10 of the procedures manual references Appendix E as providing information on the process for changing standards, but that the change process is actually described in Appendix F. The agency may wish to make this correction to the text of its procedures manual on page 10.]
The country provided a copy of the draft revised CAAM-HP standards that were developed in 2016-2017 (Ex. 2). Changes to the standards were highlighted and appear to address numerous concerns raised by the NCFMEA in the past with regard to schools that are accredited by CAAM-HP. However, the narrative notes that some concerns were raised at the July 2016 general meeting regarding the use of outcomes data related to USMLE pass rates and residency placement rates, and that the standards were then scheduled for additional discussion at the July 2017 CAAM-HP general meeting. It would appear from the narrative that concerns were also raised at the March 2017 workshops regarding the removal of standards related to internships.
The narrative states that changes were made to the agency's database documents, which serve as the guidelines institutions use in preparing their self-studies, to reflect the revisions to the CAAM-HP standards (Exs. 3, 3A 3B, 3C, 3D, and 3E). In reviewing these exhibits, it appears that the lack of consensus among the countries as to the types of outcomes data that should be provided has been circumvented by giving institutions an option to provide information related to various types of outcomes without requiring which types of data should be used and without establishing any thresholds that the institutions must meet regarding various types of data that may (or may not) be reported (Exs. 3A, p. 3 and 3C, pp. 2-3, 13, 23). It was not clear to ED staff whether any changes/deletions were made regarding the internship standards, which were reportedly also an area of concern for some member countries.
ED staff have three concerns related to the agency's draft revised standards. First, it is unclear whether the draft standards that have been provided in Exhibit 2 will be the final standards that are adopted in July 2017, given the decision made at the July 2016 general meeting to continue the standards' discussion at the 2017 meeting. Second, it does not appear that the revised standards will establish any thresholds or benchmarks related to outcomes data, which has been an ongoing concern that the NCFMEA has raised with CAAM-HP previously. Third, it is not clear how the standards and databases were altered and if requirements related to internships were deleted from either document, or whether the agency's previous requirements remain in place.
The country is requested to provide additional information as to whether further changes, including benchmarks, were made to the draft revised CAAM-HP standards at the CAAM-HP general meeting in July 2017. It is also requested to clarify whether any standards or database requirements related to internships were deleted as part of the review process.
It is unclear whether the draft standards that have been provided in Exhibit 2 will be the final standards that are adopted in July 2017 given the decision made at the 2016 annual general meeting to continue the standards’ discussion at the 2017 meeting
At the CAAM-HP Annual General Meeting held in Barbados, July 20-22, 2017 the draft standards as submitted to the NCFMEA on June 26, 2017 were presented to the meeting for approval, Exhibit 4. There were no changes following either the re-submission to the ad hoc committee charged with the responsibility of revising the standards following the 2016 meeting and the additional two seeks given to members to make any further comment. See Exhibit 5 Presentation to the 2017 AGM.
The meeting focused on Standard ED-46 Medical schools must evaluate the performance of their students and graduates in the framework of national and international norms of accomplishment and performance within the health care system. There were concerns with respect to the relevance and accessibility with respect to some of the supporting examination data to be provided on the performance of graduates. The meeting agreed to expand the examination data requested to include USMLE Steps 1 and 2 and MCCQE Part 1. These were subsequently included in the Medical Education Database Section III ED-46 (1) Examination Results. Exhibit 8C. The meeting then approved the revised Standards and databases. See extract from minutes of the meeting, Exhibit 6.
Further discussion centered around the transition to the 2018 date since there were schools already preparing for site visits early in 2018. The meeting decided to seek the advice of CAAM-HP’s counsel who advised as follows:
I suggest that the revised accreditation standards become effective on August 1, 2017 and that all applications that have already been received and in the pipeline be guided by the old standards.
Anything received AFTER August 1, 2017 should as such be guided by the new standards and notice of these and the effective date should be published as stated.
This advice was communicated electronically to all members who responded promptly giving their approval. All Caribbean medical schools have since been advised and copies of the revised standards and databases along with the effective date have been sent to each school, Exhibit 7. This has also been published on the CAAM-HP website.
Standards and databases related to Internship have been deleted due to the fact that Internship is not part of the undergraduate medical education programme. Exhibit 8.
Establishment of thresholds or benchmarks related to outcomes data
It is important to emphasise that CAAM-HP is in not attempting to circumvent the NCFMEA’s requirement to establish thresholds that institutions must meet regarding the various types of data that may or may not be reported.
The reality of the Caribbean medical education landscape is that it is not homogenous which explains why the ad hoc committee was of the view that external examinations and residency rates could not be made compulsory for all schools in the Caribbean.
There is the regional University of the West Indies (UWI) whose primary mission is to recruit and train students as health care professionals to meet the needs of and improve the health care systems of the Caribbean population that it serves. Its history and administrative structure separates it from the other newer medical schools in the region whose students primarily come from and return to countries outside of the region. While some of UWI graduates do sit the USMLE examinations after graduation it is not a graduation requirement for students of the UWI.
At the UWI uniformity and maintenance of standards are facilitated by a number of factors including the important common final examination for students from all campuses at the end of the fifth year of training. Uniformity of standards is assured by cross-campus internal examiners and by the continued use of experienced senior external examiners from institutions outside of the region.
This examination is accepted by the national Medical Councils as the basis for registration and licence to practice in the region. There is no requirement for a licensing examination as the successful completion of the undergraduate course is followed by a period of internship which when successfully completed allows the doctor the privilege of Full Registration. All (100%) graduates obtain internship positions in the local/regional hospitals. Following graduation some graduates do enter postgraduate training at the UWI and some go overseas but the vast majority remains in the region fulfilling the mission of the UWI.
Over the past almost 70 years of its existence the Faculty of Medical Sciences has produced generations of highly skilled and able medical doctors and other health care professionals who now constitute the backbone of the region’s health care system at every level. Many graduates of the UWI’s medical programmes have also excelled internationally. Exhibit 9 shows the Final MB BS results of first time takers of the May/June graduation examinations, 2012-2016.
On the other hand, there are the new, for-profit medical schools geared towards success in the USMLE examinations and residency placements and to which external examinations and residency rates are important. Therefore these data have been included in the database and form part of the schools’ evaluation as recommended by the ad hoc committee which took the view that it is more important to capture a small amount of key, robust and verifiable data than ask for vast amounts of data which are difficult to collect, verify and impossible to manage effectively.
Differences between the regional and the ‘off shore’ schools have meant that some flexibility is necessary.
It is not clear how the standards and databases were altered and if requirements related to internships were deleted from either document or whether the agency’s previous requirements remain in place.
Exhibits 8 and 8A-8E represent the approved accreditation standards and databases for medicine. The additions/changes to these documents have been highlighted and underlined throughout the documents for ease of identification.
The only change which has been made to the draft document is the additional examination requirements made to ED-46.
Standards and database related to the Internship have been removed from the review process, Exhibits 8-8E.
Analyst Remarks to Response
In the draft staff analysis, CAAM-HP was requested to: 1) clarify whether the draft standards that had been provided as an exhibit were adopted at a meeting scheduled for July 2017; 2) provide more information related to the establishment of thresholds/benchmarks; and 3) clarify whether requirements related to internships were deleted from the agency's standards or databases.
1) In response to the draft staff analysis, CAAM-HP reports that its draft standards were approved at the July 2017 CAAM-HP meeting. The only change to the draft was with respect to Standard ED-46 regarding examination data to be provided on the performance of graduates. The examination data was expanded to include USMLE Steps 1 and 2 and MCCQE (Medical Council of Canada Qualifying Examination) Part 1. The revisions were accepted, and notification of the acceptance of the revised standards was sent to the member countries (Exs. 4, 5, 6, and 7).
2) CAAM-HP notes that while it is not attempting to circumvent the NCFMEA's requirements related to the establishment of thresholds/benchmarks, it accredits schools with various missions, and those missions do not lend themselves to the establishment of uniform benchmarks. For instance, some of the agency's schools educate primarily students from other countries who will not remain in the Caribbean to practice medicine, whereas other schools, such as the University of the West Indies in Mona, Jamaica (UWI/Mona), educate local students who largely remain in their home countries to practice. In at attempt to accommodate those differences, the agency is hesitant to set compulsory requirements regarding external examinations and residency rates for all of its schools. The agency notes that some UWI graduates do sit for the USMLE examinations after graduation, but it is not a graduation requirement for students of the UWI. The agency instead accepts first time graduation exam results as an indicator of program success. There is no requirement for a separate licensing exam. CAAM-HP provided the exam results for the UWI/Mona for the years 2012-2016 (Ex. 9). The results indicate that first time pass rates have declined from 98% to 82% during that time. It is unclear whether CAAM-HP has established any thresholds for the UWI/Mona's pass rates, or what actions have been taken in response to the declining pass rates. The Committee may therefore wish to request additional information from CAAM-HP regarding any actions that the agency has taken regarding UWI/Mona's declining pass rates.
3) CAAM-HP clarified that the requirements in its standards and databases related to the review of internships were deleted (Exs. 8-8E).
Staff Conclusion: Additional Information requested