Introduction The hypertensive disorders of pregnancy (HDP), and in particular pre-eclampsia and eclampsia, remain one of the top three causes of maternal mortality and morbidity, globally [1]–[4]. Pre-eclampsia also increases fetal risks, having been found to be associated with increased risk of stillbirth, neonatal death, intrauterine growth restriction, and preterm birth [4]. The majority of deaths associated with HDP occur in the low- and middle- income countries (LMICs) in the absence of a trained health professional [5],[6]. The increased burden of adverse outcomes in LMICs is believed to be due primarily to delays in triage (identification of who is, or may become, severely ill and should seek a higher level of care), transport (getting women to appropriate care), and treatment (provision of appropriate treatment such as magnesium sulphate, antihypertensives, and timed delivery) [7]–[9]. A major contributing factor to the morbidity and mortality associated with pre-eclampsia is the shortage of health workers adequately trained in the detection and triage of suspected cases [9]. One method suggested for enhancing outcomes in LMICs is task-shifting aspects of antenatal care to existing cadres of mid-level health workers [5],[10]. To do this effectively, these health workers require simple, evidence-based tools for monitoring pregnant women and accurately identifying who is at greatest risk of severe complications. By identifying those women at highest risk of adverse maternal outcomes well before that outcome occurs, transportation and treatment can be targeted to those women most in need. Our group has previously developed the Pre-eclampsia Integrated Estimate of RiSk (fullPIERS) clinical prediction model, which predicts adverse maternal outcomes among women with pre-eclampsia on the basis of a woman's gestational age at diagnosis, the symptom complex of chest pain and/or dyspnoea, oxygen saturation by pulse oximetry, and laboratory results of platelet count, serum creatinine, and aspartate transaminase. The fullPIERS model, validated in a high-income tertiary hospital setting, has excellent discriminatory ability with an area under the receiver operating characteristic curve (AUC ROC) of 0·88 (95% CI 0·84–0·92) [11]. However, due to the inclusion of laboratory tests, the fullPIERS model may not be suitable for all settings, particularly primary care settings in LMICs. The objective of the miniPIERS study was to develop and validate a simplified clinical prediction model for adverse maternal outcomes among women with HDP for use in community and primary health care facilities in LMICs. Methods Study Design and Population The miniPIERS model was developed and validated on a prospective, multicentre cohort of women admitted to a participating centre with an HDP. Participating institutions were: the Colonial War Memorial Hospital, Suva, Fiji; Mulago Hospital, Kampala, Uganda; Tygerberg Hospital, Cape Town, South Africa; Maternidade Escola de Vila Nova Cachoeirinha, São Paulo, Brazil; Aga Khan University Hospital and its secondary level hospitals at Garden, Karimabad and Kharadar and Jinnah Post-graduate Medical College, Karachi, Pakistan; and Aga Khan Maternity & Child Care Centre, and Liaqat University of Medical Sciences, Hyderabad, Pakistan. Ethics approval for this study was obtained from each participating institution's research ethics board as well as the clinical research ethics board at the University of British Columbia. All participating institutions had a hospital policy of expectant management for women with pre-eclampsia remote from term, and similar guidelines for treatment of women with regard to magnesium sulphate and antihypertensive agents. Institutions were chosen to participate on the basis of the consistency of these guidelines in order to achieve some level of homogeneity within the cohort and to reduce systematic bias that could result from differences in disease-modifying practices between institutions. Women were admitted to the study with any HDP defined as follows: pre-eclampsia, defined as (i) blood pressure (BP) ≥140/90 mmHg (at least one component, twice, ≥4 and up to 24 hours apart, after 20 weeks) and either proteinuria (of ≥2+ by dipstick, ≥300 mg/d by 24 hour collection, or ≥30 g/mol by urinary protein:creatinine ratio) or hyperuricaemia (greater than local upper limit of local non-pregnancy normal range); (ii) haemolysis, elevated liver enzymes, and low platelets (HELLP) syndrome even in the absence of hypertension or proteinuria [1]; or (iii) superimposed pre-eclampsia (clinician-defined rapid increase in requirement for antihypertensives, systolic BP [sBP] ≥170 mmHg or diastolic BP [dBP] ≥120 mmHg, new proteinuria, or new hyperuricaemia in a woman with chronic hypertension); or an “other” HDP defined as: (i) gestational hypertension (BP≥140/90 mmHg [at least one component, twice, ≥4 hours apart, ≥20+0 weeks] without significant proteinuria); (ii) chronic hypertension (BP≥140/90 mmHg before 20+0 weeks' gestation); or (iii) partial HELLP (i.e., haemolysis and low platelets OR low platelets and elevated liver enzymes). All women participating in the study gave informed consent according to local ethics board requirements. Women were excluded from the study if they were admitted in spontaneous labour, experienced any component of the adverse maternal outcome before eligibility or collection of predictor variables, or had confirmed positive HIV/AIDS status with CD4 count 0.5) they were re-coded as a combined indicator variable. Model building Stepwise backward elimination was used to build the most parsimonious model with a stopping rule of p 10); moderately informative (LR 0·1–0·2 or 5–10); and non-informative (LR 0·2–5). A risk stratification table was generated to assess the extent to which the model's predictions divided the population into clinically distinct risk categories [20]. Model validation Internal validation of the model was assessed using 500 iterations each of Efron's enhanced bootstrap method [21]. Details of this approach have been described previously [11],[14]. The bootstrapping procedure involved (i) sampling with replacement from the original cohort to generate a bootstrap dataset of 2,081 women; (ii) redevelopment of the model including all model development steps; variable coding (transformations and categorizations), variable selection, and parameter estimation in the bootstrapped sample; (iii) estimation of the AUC ROC for the model in the bootstrap sample; (iv) application of this new model to the original dataset and estimation of AUC ROC. Model optimism is then calculated as the average difference between model performance in the bootstrap sample and the original dataset after 500 iterations of this procedure. The choice was made to use 500 iterations because previous studies have shown no benefit is achieved when using a higher number of repetitions [16]. A final assessment of calibration was performed using the Hosmer-Lemeshow goodness-of-fit test. A final assessment of model validity was performed by applying the miniPIERS model to the fullPIERS dataset and estimating the AUC ROC. Due to the marked difference in underlying rate of outcomes in the fullPIERS population (6.5% in fullPIERS versus 12.5% in miniPIERS), the model intercept (i.e., the baseline rate) was adjusted before estimating predictive performance [14]. This difference in outcome rate between the two cohorts is due to the difference in setting in which the data was collected, as noted in the description of the cohorts above, fullPIERS was completed in high-income country facilities only. Sensitivity analyses were performed to assess the generalizability of the model in various subsets of study data. In addition, sensitivity analyses were performed excluding the most common components of the adverse maternal outcome to ensure that model discriminatory ability was maintained. Generalizability of the model across study regions was further assessed based on the AUC ROC calculated for the model when applied to each region's subset of the total miniPIERS cohort. All statistical analyses were performed using STATA v11·0 (StataCorp). Results From 1 July 2008 to 31 March 2012, 2,133 women were recruited to the miniPIERS cohort. Fifty-two of these women were excluded prior to analysis after review of their medical record revealed that they were ineligible. Medical chart review was able to resolve all instances of missing predictor variables in the total cohort. Data relating to the remaining 2,081 women were included in the model development and internal validation process. Compared with women who did not have an adverse outcome, women who had an adverse outcome were more likely to be nuliparous, to be admitted earlier in gestation, to be admitted with a diagnosis of pre-eclampsia, to have worse clinical measures in the first 24 hours of admission, and to have received corticosteroids and magnesium sulphate, but less likely to have been delivered by cesarean section (Table 1). 10.1371/journal.pmed.1001589.t001 Table 1 Demographics of women in the total cohort comparing women with and without adverse maternal outcomes (N = 2,081). Characteristic Women with Adverse Outcomes (n = 401 women) Women without Adverse Outcomes (n = 1,680 women) p-Value* Demographics (within 48 h of eligibility) Maternal age at EDD (years)mean (±SD) 27·9 (±5·9) 28·5 (±6·2) 0·17 Parity ≥1n (%) 183 (45·6%) 939 (55·9%) 1 h 5 7 Intubation (other than for cesarean section) 14 25 Pulmonary oedema 37 51 Haematological Transfusion of any blood product 129 174 Platelets 0·5) between the symptoms of chest pain and dyspnoea, and headache and visual disturbances. Therefore, these symptoms were re-coded as combined indicator variables and entered accordingly into the multivariate model. As expected, systolic and diastolic blood pressure were highly correlated. Systolic blood pressure was selected for final model development because it is easier for minimally trained health care providers to measure by radial artery palpation than detection of Korotokoff sounds and it has been shown to be reflective of stroke risk in women with pre-eclampsia [22]. Systolic blood pressure measurements were log transformed for final model development as was gestational age at admission due to the highly skewed distribution of both variables. Table 3 presents results of the univariate and multivariate analysis of miniPIERS predictors. The final miniPIERS equation was: logit (logarithm of the odds)(pi) = −5.77+[−2.98×10−1×indicator for multiparity]+[(−1.07)×log gestational age at admission]+[1·34×log systolic blood pressure]+[(−2·18×10−1)×indicator for 2 + dipstick proteinuria]+[(4·24×10−1)×indicator for 3 + dipstick proteinuria]+[(5.12×10−1)×indicator for 4 + dipstick proteinuria]+[1·18×indicator for occurrence of vaginal bleeding with abdominal pain]+[(4.22×10−1)×indicator for headache and/or visual changes]+[8.47×10−1×indicator for chest pain and/or dyspnoea]. 10.1371/journal.pmed.1001589.t003 Table 3 Univariate and multivariate analysis of candidate predictors in the miniPIERS cohort. Candidate Predictor Univariate OR [95% CI] Multivariate OR [95% CI] Demographics Maternal age (years) 0.99 [0.97–1.01] n/a Gestational age at admission (wk) 0.95 [0.92–0.98] 0.34 [0.11–1.11]a Parity (multiparous versus primiparous) 0.73 [0.57–0.95] 0.74 [0.56–0.99] Signs Systolic BP (mmHg) 1.02 [1.01–1.02] 3.89 [1.19–12.66]a Diastolic BP (mmHg) 1.03 [1.02–1.03] n/a Dipstick proteinuria 2+ 1.44 [0.99–2.09] 0.80 [0.51–1.27] 3+ 2.88 [2.07–4.00] 1.53 [0.99–2.37] 4+ 3.23 [2.18–4.85] 1.67 [0.96–2.88] Symptoms Headache 3.42 [2.58–4.52] 1.53 [1.07–2.17] Visual disturbances 2.63 [2.00–3.45] Chest pain 6.42 [3.62–11.37] 2.33 [1.38–3.94] Dyspnoea 6.35 [4.08–9.89] Epigastric/right upper quadrant pain 3.93 [2.96–5.21] n/a Nausea/vomiting 3.40 [2.53–4.57] n/a Abdominal pain with vaginal bleeding 6.03 [4.25–8.57] 3.24 [2.13–4.94] Variables presented as part of the multivariate analysis are those that were retained after model development and backward selection. a Log transformed. OR, odds ratio. The model appeared well-calibrated, as shown in the calibration plot (Figure 1). In all deciles except for the highest the 95% confidence interval around the observed outcome rate crossed the diagonal fitted line. The AUC ROC for this model was 0·768 (95% CI 0·735–0·801) (Figure 2) with an average optimism estimated to be 0.037. Using a cut-off of predicted probability of 25% to define a positive test resulted in a LR of 5.09 [4.12–6.29] and classified women with 85.5% accuracy (sensitivity 41.4%; specificity 91.9%). The stratification capacity of the model was good, as shown by the 784 (37.7%) and 256 (12.3%) women in the lowest and highest risk groups, respectively (Table 4). 10.1371/journal.pmed.1001589.g001 Figure 1 Calibration plot of the miniPIERS model applied 2,081 women in the cohort (H–L goodness of fit p = 0.1616). Green line represents line of perfect fit between observed and predicted outcomes and orange line is a smoothed fit line between predicted probability and mean observed probability in each range. 10.1371/journal.pmed.1001589.g002 Figure 2 Receiver operating characteristic curve of the miniPIERS model developed in 2,081 women in the miniPIERS cohort. AUC 0.768 (95% CI 0.735–0.801). 10.1371/journal.pmed.1001589.t004 Table 4 Risk stratification table to assess the miniPIERS prediction model. Predicted Probability n Event/n in Range Percent Sens Percent Spec Percent PPV Percent NPV LR [95% CI]a 0–5·5% 33/784 — — — — 0.31 [0.22–0.42] 5·6–8·0% 18/286 87.4 41.3 17.6 95.8 0.47 [0.29–0.74] 8·1–15·0% 46/456 80.5 56.0 20.8 95.2 0.78 [0.59–1.03] 15.1–24.9% 56/299 62.8 56.6 29.5 93.6 1.61 [1.24–2.08] ≥25% 108/256 41.4 91.9 42.2 91.6 5.09 [4.12–6.29] Upper limit of predicted probability range used to define a positive test for sensitivity (Sens), specificity (Spec), positive predictive value (PPV), and negative predictive value (NPV). a LR for each category calculated using the method described by Deeks et al. [19]. Data from 1,300 women in the fullPIERS cohort were used for external validation of the developed miniPIERS model. Table 5 presents the results of a comparison of demographics and clinical characteristics of women in fullPIERS compared to miniPIERS. The cohorts differed significantly with respect to demographics, interventions, and pregnancy outcomes. When the miniPIERS model was applied to the fullPIERS dataset the AUC ROC was 0.713 (95% CI 0.658–0.768) after adjusting the model intercept to account for differences in the outcome rate between the fullPIERS and miniPIERS populations (Figure 3). 10.1371/journal.pmed.1001589.g003 Figure 3 Receiver operating characteristic curve of the miniPIERS model applied to the fullPIERS (11) external validation cohort. AUC 0.713 (95% CI 0.658–0.768). 10.1371/journal.pmed.1001589.t005 Table 5 Demographic table comparing characteristics of women in the development and validation cohorts. Characteristic miniPIERS Cohort (n = 2,081 Women) fullPIERS Cohort (n = 1,300 Women) p-Value* Demographics (within 48 h of eligibility) Maternal age at EDD (years)mean (±SD) 28.4 (±6.2) 31.7 (±6.0) 34+6 wk 167/1,503 0.767 [0.723–0.807] 49/973 0.729 [0.636–0.822] Including only women admitted ≥37+0 wk GA 108/997 0.780 [0.731–0.829] n/a n/a a Other hypertensive disorders excluded: chronic hypertension, gestational hypertension without proteinuria, or other adverse conditions, partial HELLP. GA, gestational age. 10.1371/journal.pmed.1001589.t007 Table 7 Performance of the model in each study site region as a predictor of combined adverse maternal outcome occurring within 48 Region Contribution of Cases to Total miniPIERS Cohort (%) Outcome Incidence in Cohort Used (n/N) AUC ROC (95% CI) Brazil 9.0 13/187 0.685 [0.524–0.826] Fiji 6.1 5/127 0.721 [0.489–0.953] Pakistan 50.7 157/1,056 0.758 [0.713–0.804] South Africa 16.8 67/349 0.762 [0.702–0.821] Uganda 17.4 19/362 0.656 [0.513–0.799] Table 6 also presents sensitivity analyses performed using the fullPIERS cohort. Due to the smaller number of events in this cohort not all analyses could be meaningfully repeated but where performed, model performance appeared to be maintained. Discussion Using data from a prospectively collected cohort of 2,081 women with HDP admitted to a hospital in five LMICs, we have developed and internally validated the miniPIERS model. The final miniPIERS model includes only demographics, symptoms, and signs that can be measured in primary health care facilities in low-resourced settings. Data for the study were collected by nurses and research staff with basic training to ensure the feasibility of replication of the measurements by comparable workers. For example, gestational age can be estimated from clinical information when ultrasound in unavailable, symptoms can be ascertained with simple questions, systolic blood pressure can be estimated easily using the radial pulse, and dipstick proteinuria can be estimated by assessing the opacity of boiled urine when dipsticks are not available [23]. By confining ourselves to these simple measures, the miniPIERS model has potential for use by mid-level health workers in low-resourced settings. To add to the ease of use of this model, miniPIERS is being converted to a mobile health application that will be useable on any mobile device so that health care workers are not required to calculate risk directly. Overall, the miniPIERS model performed well on the basis of accuracy and discrimination ability (i.e., the AUC ROC). There was a slight underestimation of risk in the highest decile of predicted probability, but because the model was designed to be used as a categorical decision rule, this error in calibration is not thought to be clinically relevant. This model attains similar stratification, calibration, and classification accuracy as other established risk scores used in adult and reproductive medicine [24],[25]. To our knowledge, the miniPIERS model is the only clinical prediction model developed and validated for use with pregnant women in LMICs. The miniPIERS model was used to designate women as being high-risk if their predicted probability of adverse outcome was ≥25%. The LR associated with this threshold showed potential utility as a rule-in test for adverse maternal outcome. By improving the ability of care providers to identify women at high risk of adverse outcomes, our specific aim was to reduce triage delays for women with any HDP in LMICs. What may be most useful is to set one threshold of predicted probability of adverse outcome, such as >15%, to initiate increased surveillance and use the higher threshold of ≥25% to initiate transport to a facility where emergency obstetric care is available. The positive predictive value of the 25% threshold was approximately 40% in all datasets with a corresponding 85% classification accuracy. These modest results highlight the fact that demographics, symptoms, and signs alone will not identify all women with severe disease but still have the potential to significantly improve care in resource limited areas and community settings where no or minimal monitoring of women with the HDP currently occurs. There are several limitations to this study. The first is the use of a combined adverse maternal outcome comprising events of unequal severity. The Delphi consensus group determined that all components of the outcome were important enough on their own to warrant avoidance. The sensitivity analyses performed using a restricted definition of the adverse maternal outcome demonstrated that the model maintained its performance even when the more common and less-severe outcomes were excluded. A second limitation of the study is the use of broad inclusion criteria that included women with any HDP. This decision was made to make the model maximally useful for women who present with HDP, and for whom the exact diagnosis may not (or cannot) be determined at the time of clinical presentation. Reassuringly, when we restricted the cohort to only those women who were admitted with classically defined pre-eclampsia (hypertension and proteinuria), model performance was maintained. A third limitation is the use of a backward elimination method for final variable selection in the model. Automated variable selection methods for model development have been shown to be sensitive to minor changes in the data and are not easily reproducible [26]. Ultimately, we felt that creating a simpler model with only those few variables that were most predictive of the outcome was important to make application of the model by minimally trained care providers easier. A fourth limitation is the use of the fullPIERS dataset for external validation of the model. Although the data were collected for both fullPIERS and miniPIERS using the same definitions and protocols, the populations between the two studies differed significantly, as did the care received. Ideally the model should be validated in another cohort of data from low-resourced settings collected by mid-level care providers as part of routine care. This is planned and would address the possible concern for a reduction in model performance should these health workers be unable to maintain the level of measurement accuracy achieved in the facility data we have used for this study. In the interim, it was reassuring that there was consistency of results between fullPIERS and miniPIERS models. miniPIERS model performance was maintained in the fullPIERS cohort and more importantly coefficients were similar in overlapping predictors between the fullPIERS and miniPIERS models. This gives us confidence that this is a well-defined and stable model. A final limitation is the inclusion of clinically defined gestational age within the miniPIERS model, usually based on last menstrual period dates. As in fullPIERS, increasing gestational age was associated with diminishing risk [11]. This inverse relation was maintained in this study despite the inaccuracy inherent in clinically based gestational age assessment. Despite these limitations we were able to achieve accurate predictions from the miniPIERS model. A major strength of this study is the high quality of data collected in a standardized manner. We were able to ensure that complete data were collected in five different LMICs through careful study monitoring and training of research staff. A second strength of this study is the generalizability of the resulting model. By combining high quality data from multiple international sites we are able to generate a model that should be applicable to any LMIC setting. The generalizability of the model is further supported by the results of the region-specific analysis of model performance. It is likely that we would have had greater predictive power had we developed the model using a more homogeneous population from one geographic region, but this would have resulted in a less generalizable model. By trading some predictive ability for generalizability, we believe we will have achieved greater impact on global public health. A final strength of the study is the use of clinically important timeframes for assessment and prediction. The miniPIERS model predicted adverse maternal outcomes occurring within 48 hours of assessment using data from within 24 hours of assessment; such timeframes represent clinically useful time periods in which transportation or disease-modifying interventions such as magnesium sulphate, antihypertensive agents, and delivery can be initiated. When Thaddeus and Maine first proposed the three delay framework for explaining maternal mortality, they characterised the first delay as a “delay in deciding to seek care on the part of the individual, the family, or both” [9]. Factors that have been identified to influence this decision are the mother's level of education and health knowledge, perceived severity of the complication that is occurring, antenatal care attendance, and distance to facility [8],[9]. An additional barrier to women receiving quality care for HDP is the global crisis for human resources for health [6]. We believe that the miniPIERS model represents a significant step towards overcoming many of these barriers by providing evidence-based information on disease severity and allowing task-shifting of monitoring for complications related to HDP to mid-level health workers. The potential implications of introduction of this model into routine antenatal care for LMICs are 2-fold: first, at the individual level women would not suffer the cost and time away from their families for unnecessary referrals when safe, increased community surveillance would be appropriate. Secondly, at the health systems level, evidence-based monitoring and primary triage for HDPs (especially pre-eclampsia) is moved from the tertiary facilities alone into lower level or primary health clinics, thereby increasing the potential for broad population-based screening, as well as making more efficient use of already burdened acute care facilities. We believe that this clinical prediction tool is an important contribution as it offers the potential to improve health outcomes of women for a condition that is at the root of a large amount of morbidity and mortality in the developing world. Nevertheless, as with any prediction model, its ultimate value will only be demonstrated with an implementation project that is able to demonstrate that its potential can be translated to real health systems change and clinical improvements; such a project, called the Community Level Interventions for Pre-eclampsia (CLIP) study (clinicaltrials.gov ID NCT01911494), is presently underway. For more information on the CLIP study please see http://pre-empt.cfri.ca/OBJECTIVES/CLIPTrial.aspx). Until that study is complete, the miniPIERS model can be used as a basis of a community education programme to increase women's, families', and community-based health workers' knowledge of warning symptoms and signs associated with the HDP. Supporting Information Dataset S1 Data file containing predicted probability calculated by the miniPIERS model and observed outcome for all cases in the miniPIERS and fullPIERS cohorts. (XLSX) Click here for additional data file. Table S1 Table of full definitions of maternal adverse outcomes used in the miniPIERS study. (DOCX) Click here for additional data file.
