The VHP – Ethiopia Nurse Midwife Program:
An Analysis of Test Scores, 2021 – 2023
(with implications for mentoring)
by Peter Van Arsdale, VHP
(Executive Summary, December, 2023)
Introduction:
As noted in the Village Health Partnership (VHP) October, 2021, trip report, a number of nurse midwives (NMWs) in the remote Suri region of s.w. Ethiopia’s West Omo Zone (WOZ) were not passing the most basic skill checks. Others also were struggling. Five from Suri were sent to the Mizan Tepi University Teaching Hospital (MTUTH) for training, but failed the certification test. At that time, in consultation with our MTUTH colleague and in-service specialist, Yared Deyas, a new initiative was proposed: Implementation of hospital- and health center-specific visiting teams which would be part of a systematic NMW assessment, training, and mentorship program in the WOZ.
The initial concept, rather than classroom training, was to travel to selected hospitals and health centers in the WOZ 4 times annually with a 2-person team led by Yared; 2 NMWs would be assessed on each visit. NMW knowledge of procedures in the obstetric and neonatal arenas would be assessed, relevant teaching offered, and personal relationships built. Mentoring opportunities would ideally emerge. A skills-oriented database would be developed and used for longitudinal tracking and comparative purposes. Yared noted that Tum – Maji health center-hospital connections and Dizi – Suri tribal conflicts (including over scarce resources) helped frame the practical thinking and logistics. After its first year, this new initiative would be evaluated.
Operational Definitions and Protocols:
This initiative began in October, 2021. VHP had been providing a three-week training annually since 2017. It relies upon the established Johns Hopkins/JHPIEGO BEMONC Training Program curriculum, enabling the testing of NMWs on 14 specific core obstetric and neonatal factors/skills, and was instituted for the current program’s on-site assessments. These are as follows:
Obstetric Factors/Skills (8) Neonatal Factors/Skills (6)
6.1 Module 1 Active management of third stage of labor 6.2 Module 1 Knowledge check
6.1 Module 2 Atony 6.2 Module 2 HBB bag & mask skill check
6.1 Module 3 Retained placenta & manual removal 6.2 Module 3 HBB evaluation A, with role play
6.1 Module 4 Non-pneumatic anti-shock garment (NASG) 6.2 Module 4 HBB evaluation B, with role play
6.1 Module 5 Vacuum extraction 6.2 Module 5 Essential Care for Every Baby (OSCE A)
6.1 Module 6 Breach delivery 6.2 Module 6 Essential Care for Every Baby (OSCE B)
6.1 Module 7 Pre-eclampsia/eclampsia [Note: These two sets of factors should be cross-
6.1 Module 8 Post-abortion care referenced while reviewing the analyses presented here.]
HBB = Helping Babies Breathe ECEB = Essential Care for Every Baby OSCE = Objective Structured Clinical Examination
The scores for each NMW, for each factor, are converted into percentage correct, ranging from 0 to 100. Average scores also are presented as percentages. NMWs must score 81 percent or above to pass each module and the overall exam. New IPad applications, complementing VHP’s SafeClinic Toolkit app, now allow rapid data entry, storage, and retrieval on the above 14 factors. These protocols are a first for s.w. Ethiopia’s health care system. (Note that the term “aggregate” is used when considering all the factors, together, in either the obstetric or neonatal category. For example, an “aggregate average” in obstetrics refers to the average of the 8 factors/skills scores, each represented as a module and presented as a percentage, earned by a NMW in a single test session.)
Training, education, and mentoring are related concepts. For NMWs who are struggling, our Ethiopian and American personnel believe that in-class education and on-site training, complemented by mentoring, can be beneficial. Mentoring allows systematic follow up in an evolving relationship of trust between mentor and mentee. Ultimately, higher-level knowledge and resource-savvy expertise for the mentee are possible. The mentors themselves must be knowledgeable, interpersonally strong, and readily available. Cell phone and in-person connections are utilized.
What VHP and Our Colleagues Hope to Learn:
The present analysis and report benefited greatly from the input of Tim Wellman and Jenny Mattern. As this program has emerged, generous funders have come on board. One organization of particular importance is Together Women Rise (TWR). Twenty years old, with several hundred chapters across the U.S. (including one in Denver where two of VHP’s board members recently spoke), it supports innovative and empowering initiatives. TWR members are asking insightful questions of this program, including:
-In rural Ethiopia, who employs the NMWs, how much are they paid, and how are they recruited?
-What is the “women’s solidarity component,” including broad-ranging educational and empowerment opportunities?
-How is mentoring conducted to benefit NMWs?
-In what ways are women engaged as leaders?
-What are the on-site assessments yielding data-wise?
This report emphasizes the last of these questions. Others have been partially addressed in other VHP memos, briefs, and trip reports. Most of the analyses in this report look at the obstetric and neonatal data separately.
NMW data were collected at 3 hospitals and one affiliated health center: Aman (MTUTH, the regional referral hospital for s.w. Ethiopia located in the large city of Mizan-Aman), Bachuma, Maji, and Siz (affiliated with the Siz Hospital). Data also were collected at 7 remote health centers: Chebera, Chiruharoot, Jomu, Kibbish, Kuju, Tulagit, and Tum. For a majority of these, data from 8 sets of individual tests, administered to 2 NMWs per site visit, over the three-year period were available. Security concerns caused VHP to drop, for now, work at Kibbish and Tulagit. Now including both the Siz Hospital (where NMW testing was not initially conducted) and the affiliated Siz health center, VHP’s work is proceeding in the WOZ with 10 facilities.
Summary of Key Findings:
Data were collected on the 14 assessment factors/skills noted earlier; 8 cover obstetrics and 6 cover neonatal. My “first analytic cut” was to evaluate the data in these two sets overall. My “second analytic cut” was to evaluate the data for each of the 11 facilities. My “third analytic cut” was to evaluate the data for each NMW. Over the period from October, 2021, through April, 2023, 57 different individuals were tested. The scores for one person proved invalid. Seven of these individuals were tested twice and one was tested three times. In sum, we have 65 valid test scores.
Scores on the 8 obstetric and 6 neonatal factors/skills (presented as modules) that were assessed were transformed into percentages; 81 percent was the passing threshold for a NMW on any single module. By contrast, the two charts [see attachments] summarize each facility’s score for each module for all those tested during the three-year period. The score and date for each facility’s high scoring individual also is noted for the three-year period.
Reviewing the obstetric data writ large, combining the 8 modules, the average for the 11 facilities was 61, with the hospitals proving much stronger (67) than the health centers (52). (For analytic purposes, the Siz health center was grouped with the three hospitals.) Of those NMWs who were strongest, 27 scored between 70 and 99 when averaging the 8 modules. Yet, a similar number proved to be weak, with 26 scoring between 20 and 49. The highest aggregate score for one person was 96, the lowest 25. For those few individuals tested more than once, their scores generally improved. For the module “breach delivery,” 30 NMWs passing at 81 percent or better was the highest total; for the module “non-pneumatic anti-shock garment,” 13 NMWs passing at 81 percent or better was the lowest total. (Such garments are not readily available.) Only one person met or exceeded the 81 percent threshold on all 8 modules.
Reviewing the neonatal data writ large, combining the 6 modules, the average for the 11 facilities was 60, with the hospitals again proving much stronger (72) than the health centers (53). Of those NMWs who were strongest, 25 scored between 70 and 99 when averaging the 6 modules. Of those NMWs who were weakest, 15 scored between 20 and 49. The highest aggregate score for one person was 97, the lowest 24. Again, for those few individuals tested more than once, their scores generally improved. For three modules, all related to Helping Baby Breathe, 19 NMWs passing at 81 percent or better was the highest total; for the module “Essential Care for Every Baby (OSCE B),” 8 NMWs passing at 81 percent or better was the lowest total. Only two people met or exceeded the 81 percent threshold on all 6 modules.
Among the 3 hospitals and Siz affiliated health center, for both obstetrics and neonatal Aman is the strongest. It is the regional hospital. Among the 7 remote health centers, for both obstetrics and neonatal Jomu is the strongest. However, in all cases more NMWs did not reach the 81 percent threshold on a majority of the modules for which they were tested, than those who did.
The obstetric arena is more comprehensive than the neonatal arena. Yet both are essential. With every obstetric and neonatal module, a minority of those tested excelled. Yet, with every module, a majority of those tested did not excel. The module-by-module scores for a single person often spanned a range of 50 or more percentage points, and more so in the obstetric than neonatal category, a stark reminder of the skills/knowledge consistency each person still needs to attain.
Recommendations:
These data and interpretations will allow specific facilities, specific NMWs, and specific factors/skills (as represented by the 14 modules) to be targeted for further training, education, and mentoring. In 2024 20 NMWs will receive BEMONC training and will be tested and tracked through time.
In 2021, at baseline, all NMWs tested in the WOZ failed their obstetric and neonatal skill checks. With implementation of the program, scores have improved over time. Repeat testers show improvement and average scores have increased. Unfortunately, approximately 2/3 of the NMWs continue to test poorly. The results of testing indicate that we have made progress but that we have more work to do.
Testing has revealed areas of both clinical strength and weakness. There is also wide variability in NMW scores depending on where they work, and, for any single person across the 14 modules. In general, those NMWs who work in the hospitals tend to score better than those who work in the small rural health centers. Health centers are located in remote areas where staff turnover is high and providers have little support.
Averaging the scores for the 8 obstetric modules, 13 NMWs met or exceeded the 81 percent threshold. Averaging the scores for the 6 neonatal modules, 15 NMWs met or exceeded the 81 percent threshold. Perhaps these individuals could systematically assist, or co-mentor, other NMWs in their respective facilities. We have their names, several of the strongest of whom are listed in the full report.
Going forward, in concert with our Ethiopian partners we are going to focus our testing and training efforts on 10 facilities (Aman/MTUTH, the Siz, Maji and Bachuma district hospitals and the Tum, Chiruharoot, Kuju, Jomu, Chebera and Siz health centers). We are not going to work in Tuligit and Kibbish for now because of security concerns. That said, we will work to get the 5 Suri students from that area (noted at the beginning of this report) certified as NMWs. Once certified, they will return to work in Tuligit and Kibbish.
We plan to continue testing, training and mentoring 2 NMWs in each of these 10 facilities 4 times per year. We will work with health facility and government leadership to target those who score poorly and enroll them in the three-week BEMONC intensive training that is held once a year. We will also work to procure critical equipment including vacuum delivery sets, non-pneumatic anti-shock garments, and neonatal resuscitation equipment that are key to a NMW’s ability to perform basic obstetric and neonatal tasks. Finally, we will set up an intensive HBB, ECEB and Essential Care of the Small Baby (ECSB) training for all medical providers on labor and delivery, in the neonatal intensive care unit, and in the maternal operating room theatre at the MTUTH in October of 2024. We will continue to collect data – including that through the WASH dashboard [second chart] that will inform all of our activities.
Please go the VHP website, on-line, to review the full report and appendix.
