INTRODUCTION
Universal access to antiretroviral therapy (ART) and increased levels of HIV testing have created hope that HIV infection can be controlled globally. Approximately 9.7 million people now receive ART in low- and middle-income countries, representing a 32-fold increase over the last decade [1].
Effective ART reduces viral load (VL) to undetectable levels and dramatically reduces associated mortality and morbidity [2–4]. As a public health intervention, ART is at the core of a treatment-as-prevention strategy, as reducing community viral load reduces HIV transmissions [4].
CD4+ cells counts and measures of VL are surrogate biomarkers of disease progression that help to stage, initiate and monitor treatment. CD4+ cell counts provide an immunological measure of HIV progression; these counts are utilized in the care of HIV+ patients for staging infections and in assessing patients for ART eligibility [5–8]. Specialized laboratories use highly trained personnel and sophisticated flow cytometry techniques to perform CD4+ cell counts, as this is the current gold standard technique [6, 9].
When ART is available, rapid staging, continual monitoring, and retention of individuals on ART become crucial to controlling HIV infection, but the availability of quality patient monitoring in all field settings remains a challenge [10]. In global settings, patients are often required to travel long distances to clinics with specialized central laboratories to be staged and initiated on ART [11, 12]. However, the nature of a centralized laboratory demands that patients attend on separate days for testing and treatment, which strains patients’ time and resources [11, 12]. As a consequence, estimates from studies suggest that only about 60% (range 35–88%) of individuals who receive an HIV diagnosis in sub-Saharan Africa receive a CD4+ count, meaning many remain oblivious to the need for treatment initiation or switching [3]. Furthermore, 25% are initiated on ART at CD4+ counts below 100 cells/μL, by which point the virus has already inflicted considerable damage to the immune system [1, 14]. In addition, once initiated on ART about 30% are lost to follow up later on in the cascade of care, reflecting the inefficiencies in delivery mechanisms [9, 11, 13, 15–18]. Centralized laboratories must transport specimens to and from the laboratory and they cannot always deal with the demand for CD4+ cell counts [9, 16, 19]. Decentralized and linked point-of-care (POC) systems will help to plug some of the inefficiencies in the monitoring and delivery systems, preventing loss to follow up of patients.
CD4 and VL POC tests that are less reliant on infrastructure, and can operate in the absence of a continuous electricity supply and without laboratory equipment, specialist personnel or sample transport systems [9, 15, 16]. Some CD4 POCs aim to provide individual CD4+ cell counts in as little as 8 minutes at the POC contact with a finger stick specimen. These assays could expedite staging and allow for ART to be initiated at the same site and within the same clinic visit, saving time and money for providers and health systems with reduced burden on patients. This should translate into greater adherence to treatment, better compliance with prescribed regimens, and a reduction in loss to follow up [16]. Indeed, in a recent study, when a POC CD4 device was successfully introduced in the primary health sites in Mozambique, the total computed lost to follow up prior to ART initiation dropped substantially from 64% to 33% [15].
POC CD4 devices that are now being marketed or developed use different underlying technologies. The PIMA (Alere, USA) device uses dual fluorescence image analysis. In turn, the MiniPOC (Partec, GmbH), the PointCareNOW (PointCare, USA), and the HumaCount (Human Diagnostics, GmbH) use miniaturized flow cytometry, whereas the Daktari CD4 (Daktari Diagnostic, USA) device is based on microfluidics and the MBio CD4 analyzer (MBio Diagnostics, Inc., USA) uses optical technology. Two smaller devices are also in development: the Zyomyx’s CD4 test (Zyomyx, Inc., USA) and the VISITECT CD4 (Omega Diagnostics group, UK) [10, 16, 20].
Although the performance of some of these devices has been evaluated against the gold standard (flow cytometry), a comparative evaluation of their performance in finger stick and venous blood specimens has not been done. In view of the likely increased use of POC CD4 assays across the sub-Saharan and Asian settings, there is a need for a meta-analysis that is independent-of-industry evaluations. This will help answer a number of questions: (1) which device performs best, (2) how do they compare with the gold standard, and (3) are capillary blood specimens better than venous specimens. We therefore performed a meta-analysis to present combined estimates of these parameters.
METHODS
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were used to report this review [21]. Two reviewers (SW and TC) completed the search and reviewed the literature. The flow chart is presented in Figure 1. Search strategy was designed by a librarian (BN). Literature was searched for the period 1 January 2000 to 1 January 2013, with a Medline update to 24 April 2014; the databases included in the search are detailed in Appendix 2 and search in Appendix 1. Bibliographies and citing articles of our eligible studies (and selected reviews) were retrieved using Web of Science and Scopus. Authors were contacted for additional data where necessary, and one study team provided additional information for the meta-analysis [22]. Manufacturers of the devices were also contacted for data. In the first review for eligibility, we independently screened the titles and citations for inclusion (SW and TC). In the second review, we retrieved the full texts of selected articles to determine eligibility; discrepancies were resolved through discussion and in consultation with a third reviewer (NPP).
Working definitions of POC were based upon criteria established by our group in 2012 [23]. We included studies of devices that provided CD4 counts for HIV+ adult populations. CD4 counts were obtained, either for pre-ART staging of infection in those newly diagnosed with HIV or for regular follow-up monitoring of treatment response. We excluded studies where HIV– specimens were used, as well as studies including children as they constitute a separate group where there is a huge variability.
DATA ABSTRACTION
A standardized data form was created and used. Data were abstracted on study setting, participant characteristics, sample size, reference test, index tests, Bland–Altman (BA) mean bias, sensitivity, specificity, and raw data (true positives, etc.). Reviewer one (SW) abstracted all data from the eligible articles, reviewer 2 (TC) abstracted data from 50% of the articles to check for consistency.
STATISTICAL ANALYSIS
Bland–Altman mean bias
To investigate our questions, our primary outcome measure was the BA mean bias. The BA mean bias is an average of the difference between the result from the index test and the reference. It is a popular method for assessing the agreement between two continuous measures and more appropriate than correlation coefficients for comparing continuous diagnostic tests to a gold standard [24]. In the context of our review, the BA mean bias represented the average number of cells by which the result from the POC device differed from the reference test [24].
In order to understand whether performance differed between the types of specimen used, we performed a subgroup analyses by specimen (capillary and venous). The PIMA device was heavily represented in the dataset (80%, 32/40) and we were only able to meta-analyze the data for the PIMA (venous and capillary) and the PointCareNOW (venous only). Data for the MBio CD4 System and CyFlow miniPOC were limited (two and one data points, respectively), therefore we were unable to meta-analyze Bland–Altman data for these devices. Instead, the performance of these devices was explored graphically, using forest plots created in R (R 3.0.2 GUI 1.62 Snow Leopard build). In pooling our data in meta-analysis both operators and settings were assumed to be equivalent as POC devices should be robust to these factors in order to perform well in the field. We used a Bayesian hierarchical normal model to combine the BA results across studies. At the first level of the hierarchical model, the BA mean bias from each study is assumed to follow a normal density, with study specific mean and study specific standard deviation. At the second level of the hierarchy, the means across studies are assumed to follow a normal density, with an overall mean and an overall variance across studies. We converted the standard deviations within each study from standard errors by multiplying by the square root of the sample size. We assumed that the logarithms of the study specific variances then followed a normal hierarchical density. We used diffuse priors across all parameters so that the data drive the final inferences.
Sensitivity and sensitivity
Due to lack of data, use of varying CD4+ cell count cutoffs, and sporadic use inclusion ranges, we were only able to complete a meta-analysis for a CD4 cutoff of 350 cells/mL. The remaining data were explored visually using forest plots.
We used a Bayesian hierarchical logistic regression model for meta-analysis of the sensitities and specificities, each modeled separately. All sensitivity and specificity values were first converted to a logit scale. On this scale, we assumed that sensitivities and specificities from different studies followed a normal density. We used a uniform prior for the SD with an upper limit of 3 on the logit scale, and wide normal priors for the mean parameters. Once meta-analyses were completed, all results were converted back to the probability scale by an inverse logit transform.
Repeatability
Although we did abstract data on repeatability, due to lack of data, we presented the available information in Appendix 3. Regarding quality, both reviewers independently assessed each study using the QUADAS-2 criteria [25]. Disagreements were resolved by consensus or by consulting a third reviewer (NPP).
RESULTS
For the period January 2000 to April 2014, about 19 databases were systematically searched, 6619 citations retrieved, and 25 articles were selected. Diagnostic performance was compared across devices (i.e., PIMA, CyFlow, miniPOC, MBioCD4 System) and across specimens (i.e., capillary blood vs. venous blood).
Figure 1 details the study selection process. In total, of the 6619 papers that we screened, 25 studies were included in our analysis, summarized in Appendix 2.
By assessing each study using the QUADAS-2 criteria [25] we found that included studies were of moderate quality, Figure 2.
Bland–Altman
For the PIMA meta-analysis by specimens, 13 studies contributed 15 data points to the venous meta-analysis and 13 studies contributed 17 data points to the capillary meta-analysis (32 data points in total).
With regard to our first objective, which device performs the best, we found that the PIMA device was superior. Our findings highlight a severe lack of evidence for all other devices.
We found that the PIMA device was comparable to flow cytometry. The results from the meta-analysis for the PIMA device indicates that, when used with capillary specimens, the device will agree well with the reference flow cytometry method.
The PointCareNOW device did not appear to agree well with the reference standard, and the device appears to overestimate the CD4 cell count. The data for the PointCareNOW device were contributed by just two studies, and the results from these two studies appears to substantially differ: the more recent study indicated that the device performed better, perhaps indicating that the PointCareNOW device is still under development.
The other devices in our review, the MBio and MiniPOC appear to be promising; however data are severely lacking for these devices at this time.
We also found that capillary specimens yielded results that were as accurate as the venous specimens (Figure 3). The results of the meta-analysis for the PIMA data indicate that on average the PIMA device estimates capillary specimens as well as venous specimens. The capillary specimens had a small BA mean bias point estimate of 2.2 cells/μL; 95% CrI: –19.32 to 23.6, vs. 9.5 cells/μL; 95% CrI: –37.71 to 18.27 for venous specimens.
Sensitivity and specificity
Our meta-analysis of the results of diagnostic accuracy (i.e. sensitivity and specificity) data may be a more easily understood analysis, in that it provides direct information about the ability of CD4 devices to correctly identify patients for treatment. We found that of the 26 included articles, only 12 reported accuracy information, refer Figure 4. Of these, a majority (10) reported at the 350 cell/μL cutoff; the cutoff used in other papers varied widely (three reported on the 200 cell/μL cutoff, one used 250 cell/μL, another one used 300 cell/μL, and two chose 500 cell/μL as the cutoff). Much of this variation is probably due to the World Health Organization’s (WHO) steadily increasing CD4 cutoff guidelines for ART initiation over the last 5 years [7, 26] Again, the PIMA device dominated the data, only one study contributed data for the accuracy of the PointCareNOW device.
We found that at the 350 cells/μL cutoff the PIMA device identifies patients for treatment with a sensitivity of 91.58 (CrI: 84.65–95.46) and specificity of 94.79 (CrI: 90.08–97.28).
Repeatability
Data relating to the repeatability of CD4 POC devices were difficult to summarize due to a wide range of measures used, inter-rater, inter-assay, and quality control runs. The data appear to indicate good levels of repeatability (Appendix 3).
DISCUSSION
Summary of evidence
Our review suggests that, overall, the POC CD4 device, the PIMA was comparable in performance to conventional flow cytometry. In comparison, the PointCareNOW device does not appear to perform as well, and further development of the technology should be completed prior to widespread use of this device. For other devices, we found that the data are still limited; more POC CD4 devices evaluations are urgently needed to respond to the global demand for HIV patient monitoring.
In interpreting our findings for the PIMA, it is vital that the mean bias results are not considered in isolation; the credible intervals (CrIs, the Bayesian equivalent term for the frequentist 95% confidence intervals) are more informative and provide a complete picture of overall performance, the bias, and the range of variability. The CrIs estimate the likely range of the difference between the CD4+ cell counts of the device and that of the reference. Given this, the PIMA device performed well for capillary specimens as the range of expected agreement was –19.32 to 23.60 cells/μL, so we can be confident that the PIMA can be used interchangeably with flow cytometry. For venous specimens, the CrIs are wider, indicating more variability in the agreement –37.71 to 18.27 cells/μL. Thus, we concluded that despite a small number of data points, the PIMA device for now offers the potential to be scaled up and operationalized in decentralized settings for the monitoring and staging of HIV-infected individuals.
Our secondary findings indicate that PIMA devices work well for capillary specimens (compared to venous); it appears that most devices are best optimized for capillary specimens. This finding reflects that capillary blood could be safely used to monitor CD4 counts, which is important for all international settings. Oftentimes, phlebotomists are not easily available on site to procure venous specimens, but capillary specimens, can be obtained easily, by clinic staff with minimal training and certification in testing.
Implication of our findings for practice
In addition to provide a staging and treatment initiation solution at one clinic, portable CD4 POC devices will help solve a number of barriers to accessing care (i.e., number of clinic visits, long waiting time in clinics, physical distances and location of clinics). These barriers impact on timely access to services by rural poor communities. If portable CD4 devices can also be taken to the rural village clinic, then high-quality HIV care could be delivered at the very place it is most needed, in a convenient and patient friendly way. Additionally, current services available in clinics could be expanded to include both POC CD4 counts and VL counts, allowing for faster triage of the sickest individuals and patients could be tracked for follow-up.
Furthermore, these technologies are developing fast and many manufacturers are equipping their devices with data storage and global positioning system capabilities, which can allow data to be transferred to online storage; encouraging technology-assisted quality control systems. For example a service in Mozambique is using general packet radio service-enabled PIMA devices to make use of the already available cellular communication networks [27]. Such capabilities allow for regional- and national-level supervision of performance of laboratories, tracking test results, of skilled personnel, and with tracking a rapid resolution of problems. These modern, high-quality integrated care systems are sensitive to patient needs with data storage and quality capability.
For global use, the absolute CD4 count has been shown to be unreliable in pediatric populations, making many CD4 POC devices unsuitable for staging in this patient group; this issue does not apply to adult HIV-infected populations [28]. New generations of POC CD4 devices will need to incorporate the ability to provide CD4% for the less sensitive pediatric populations [7]. Currently, only two devices (PointCareNOW and the miniPOC) offer that option, so further improvements and evaluations of these devices would be welcomed.
Strengths of this review include a comprehensive search, a synthesis of current evidence relating to the precision of POC CD4 devices, and with advanced analytical techniques; in addition, this is the first Bayesian meta-analysis that compares POC tests using a Bland–Altman bias.
Limitations
As is always the case with the pooling of published data, our meta-analysis is vulnerable to bias. Because of the restricted data available, we were only able to fully pool data for the PIMA and the PointCareNOW devices. More evidence must exist for the full range of devices that are on the market, but these data have not yet been published. We were unable to find any published data relating to the performance of some high-profile POC devices (i.e., Zyomyx, VISITECT, Daktari). We also noted that a majority of the BA plots were funnel shaped, perhaps indicating some interference. The use of the absolute mean bias can make the BA plots appear to be funnel shaped, when in fact this is due to a large range of the CD4 count. Because the ranges in the CD4 counts are wide, the mean bias is relative to the CD4 count, and should be reported as such. However, in our sample only two studies [29, 30] reported the relative mean bias [27, 30]. Future evaluations should report the relative mean bias, and more useful for comparison in future meta-analyses.
Implications for future research
More evaluations of newer CD4 POC devices such as handheld devices will always be useful for evidence. Cost-effectiveness analysis will help understand the incremental benefit of use in dollar amounts, prior to their large-scale implementation. The reliability of the devices will need to be considered in the future; for this analysis insufficient and inconsistent data were available.
CONCLUSIONS
To conclude, our meta-analysis suggests that, for one, the PIMA device performed comparably to conventional flow cytometry and although the data for the PointCareNOW device are limited, the average mean bias was higher than the PIMA. Secondly, capillary specimens were as accurately counted as venous specimens. We recommend that PIMA CD4 POC devices could be operationalized in decentralized settings with capillary blood, and more field research, data, and improvements in technology are needed for other CD4 POC devices.