Since 2015, there has been a vigorous debate about Rwanda’s official statistics in this journal and in the associated blog. While the National Institute of Statistics of Rwanda (NISR 2015, 2018) and the World Bank (2018) have claimed that poverty has decreased sharply over the past decade, estimates published by independent researchers have contradicted this claim (Roape.net 2017b; Desiere 2017; Roape.net 2019c). Furthermore, qualitative researchers have pointed out that the official statistics do not tally with their own findings regarding the impact of recent agricultural reforms on Rwanda’s farmers (Ansoms et al. 2017; Dawson, Martin, and Sikor 2016; Huggins 2017).
Beyond the specific issue of knowing whether poverty has gone up or down in Rwanda in recent years, this debate raises fundamental questions about the reliability of development statistics in undemocratic countries (Gladstein 2018), and about the complex relation between those governments and international institutions that are tasked to both monitor the evolution of poverty and take credit for possible successes in reducing this same poverty. Rwanda has long been held up by donors as a development success story, a proof of the correctness of the development prescriptions imposed on developing countries by international institutions. Rwanda has also become a model for the growing forces in Africa, and elsewhere, who are attracted to the apparent efficiency of authoritarian rule, and see in Rwanda the proof that restrictions on civil and political rights are necessary prices to pay for development.
Over the years, as the Rwandan poverty debate has progressed and different issues have been gradually clarified, many different estimates have been produced by different researchers involved in the discussion about Rwanda’s poverty statistics. These methodological evolutions mean that it is difficult to get a coherent picture of how poverty has evolved over time in Rwanda. The many different estimates and the fog of ongoing research can also lead to confusion among external observers, and to scepticism, as different researchers seem to be able to generate estimates that fit their preferred narrative by choosing convenient assumptions. While there is significant room for error and disagreement in poverty estimation, it is important to understand what the differences mean in order to be able to clearly establish the facts.
In view of these considerations, the objective of this briefing is to provide a consistent overview of the Rwandan poverty debate to date and try to explain the differences between the various estimates generated by different contributors to this discussion.
Background
In September 2015, NISR (2015) published the official poverty profile based on the country’s fourth Integrated Household Living Conditions Survey (known as EICV4, by its French abbreviation), claiming that poverty had decreased by six percentage points between 2010 and 2014.
Shortly thereafter, Reyntjens (2015) published a piece arguing that NISR’s poverty decrease had been engineered by a change in the poverty line that had been introduced by NISR in EICV4 and that poverty actually increased by six percentage points over the same period.
Following Reyntjens’ critique, NISR published a second poverty profile using a completely different methodology (NISR 2016). The second poverty profile did not use the new poverty line that had been calculated by NISR in 2015. Instead, it claimed to use a cost of living index to update the old poverty line, which had been computed by NISR in 2001. In principle, both methods are valid and should yield similar results, if the data used are of good quality and the calculations are accurate. The new method, NISR claimed, confirmed the initial findings, namely that poverty had decreased by over six percentage points between 2011 and 2014.
However, following NISR’s release of the EICV4 micro-data, independent researchers were unable to replicate its results. Instead, a blog posted in May 2017 (Roape.net 2017b), claimed to have replicated Reyntjen’s results, thus confirming that poverty had increased (not decreased) by six percentage points between EICV3 and EICV4. Crucially, the post also included syntax and data files allowing other researchers to independently verify the findings.
Consumption estimates
In any poverty estimation, there are a large number of small and large assumptions and corrections that have to be made to arrive at a final poverty estimate. Only a small fraction of these can be explicitly stated in the published reports or papers. Since neither Reyntjens nor NISR had released the syntax files they used when calculating their estimates, the researchers had to guess which assumptions they had used by seeing which set of plausible assumptions generated results that most closely replicated the findings of Reyntjens and NISR.
One of the key assumptions made in the initial Roape.net blog post was that yearly food consumption for a given food item could be estimated by multiplying the monthly food consumption by the number of months in which the household declared having consumed that item. This assumption was chosen (a) because the information was available in the survey, and it was therefore assumed that it was meant to be used in the calculations, and (b) it allowed us to replicate both the six-percentage-point increase in poverty when using Reyntjen’s poverty line, and the six-percentage-point decrease in poverty when using NISR’s poverty lines.
The theoretical rationale for using this assumption, rather than multiplying the monthly consumption by 12 months to obtain the yearly value, was that some items may only be consumed during the harvest season. Multiplying monthly consumption by 12 could lead to an overestimation of consumption for those items. During times of economic hardship, items may be consumed during a shorter period, something that could not be picked up if monthly consumption was uniformly multiplied by 12.
The counter-argument to this would be that households may consume different items at different times of the year. Consequently, in the months that they do not consume, say, maize, they may instead consume wheat or rice. In order to be complete, the survey would therefore need to ask what other items are consumed in the months that the household does not consume the reported item. Since EICV does not ask this complementary question, the multiplication by the declared consumption months would underestimate consumption.
After dropping this controversial assumption, we were able to closely replicate NISR’s food baskets. This suggests that this new 12-months assumption more closely resembles those used by NISR.
Inflation and price indices
In June 2017, a second blog was posted on Roape.net by Sam Desiere, confirming the increase in poverty by a slim margin (1% increase). Even though this finding was presented as a confirmation of the findings of the first Roape.net blogpost, we would argue that the differences are too large for Desiere’s findings to be seen as confirmation of the initial Roape.net blogpost findings.
After investigation, we have found that the key assumption driving Desiere’s result is that Desiere included own non-food consumption in his food consumption aggregate. Own non-food consumption doubled between 2010 and 2014, and then decreased sharply between 2014 and 2017. Including it in the consumption aggregates therefore significantly affects the results. NISR did not include this element in its own consumption aggregates, which suggests that this component might have had a problem that makes it unreliable.
When we include own non-food in our food consumption aggregate, we are able to replicate Desiere’s findings to within less than one-percentage-point difference, which falls within the acceptable statistical margin of error (see Figure 1).
Cumulative distribution of food expenditure per adult equivalent for EICV3 and EICV4 including and excluding own non-food consumption (assuming 30% inflation). Source: author’s own calculations.
Desiere’s contribution was extremely valuable in identifying that the key issue that needed to be addressed in order to resolve the Rwandan poverty debate was the question of which inflation rate the poor had faced between 2010 and 2014. When Desiere used NISR’s inflation rate (based on consumer price index [CPI] data) to update the poverty line, he obtained the same sharp decrease in poverty that NISR had claimed, but when he used the higher inflation rate calculated from the EICV and e-Soko1 price datasets, he obtained an increase in poverty.
A blog posted in April 2019 (Roape.net 2019b) showed that EICV and e-Soko inflation rates were almost identical at all points from 2001 to 2017, thus confirming Sam Desiere’s conclusion that EICV provided an adequate price data source for updating the poverty line. Furthermore, the analysis showed that the CPI food inflation rate was significantly lower at all points from 2001 to 2017 than e-Soko/EICV inflations, thus confirming NISR’s (2012) own assessment that CPI prices are inadequate for updating the poverty line.
In August 2019, the Financial Times published an article by their investigative journalism and data analyst teams, claiming to have replicated the findings of the ROAPE blogs using NISR’s publicly available micro-data (Wilson and Blood 2019). The article confirmed that poverty was most likely to have increased by around six percentage points between EICV3 and EICV4.
The World Bank’s role
The most unexpected contribution to this debate came in 2018, when the World Bank published a paper that backed NISR’s results and claimed to provide theoretical and empirical evidence (in addition to the World Bank’s name and reputation) to settle the debate (World Bank 2018). However, the key piece of empirical evidence presented by the World Bank was presented with a serious mistake: the World Bank claimed that the ‘real food share’ had decreased in Rwanda, which according to them provided solid evidence of improving living standards in Rwanda, in accordance with Engel’s law.
The ‘real food share’ was supposed to represent the proportion of food consumption in total household consumption, deflated for price changes between the two surveys. The problem is that the World Bank decided to deflate food consumption using the food inflation rate, while they deflated total consumption using the much lower total national inflation figure. This invalid adjustment alone accounted for their claimed decrease in the ‘real food share’ between EICV3 and EICV4. In reality, there is no such thing as a ‘real food share’ in Engel’s law, and standard indicators (i.e. the ‘nominal’ food share) showed a sharp increase in the proportion of food consumption between 2010 and 2014, thus providing strong independent support for the increase in poverty hypothesis (see Roape.net 2018).
In the absence of any valid empirical evidence, the World Bank’s contribution was thus limited to providing a theoretical framework describing the conditions under which NISR’s results could theoretically have been valid. This theoretical effort had the merit of clarifying once and for all (although not stating explicitly) that NISR’s (2015) initial change in poverty line between EICV3 and EICV4 was invalid and rendered trend comparisons impossible. The paper also showed that NISR’s second trend estimate (NISR 2016) could theoretically have been valid, so long as NISR’s assumptions held and the new price data introduced in 2016 were accurate. Unfortunately, the World Bank did not explore whether either of these two conditions held true.
A more detailed review of the World Bank’s paper in March 2019 (Roape.net 2019b) revealed some more disturbing facts about the World Bank’s role in this affair. Indeed, even when using NISR’s own price index, NISR’s own consumption aggregates, NISR’s own poverty line, and even the lowest available inflation rate (i.e. the total national CPI inflation rate, which is not designed to measure poverty), we were still not able to replicate NISR’s nor the World Bank’s results. Instead, we found a sharp increase in poverty between 2011 and 2014, and a net increase in poverty between 2011 and 2017.
The poverty results published by NISR (2016), and endorsed by the World Bank (2018), did not correspond to the national CPI inflation rate of 23%, which the World Bank had used to endorse NISR’s results, nor even to the unsourced 14% inflation rate that NISR claimed to have used to generate its results.2 This finding provided the first direct evidence of statistical manipulation, or at least statistical misreporting, as NISR’s claimed poverty trends did not match the inflation rate that they claimed to have used (and inflation rate that was, itself, unsourced and far lower than all existing inflation rates).
Most shocking of all, however, is the fact that the careful review of the World Bank’s paper showed that the World Bank was aware of this discrepancy, as they noted in footnote 10, p. 13, of their own paper (World Bank 2018) that the NISR (2016) deflator used an implicit inflation rate of 4.71% for the period 2011–14. Furthermore, the World Bank’s own paper included a graph that showed that NISR’s deflator assumed negative to nil inflation between October 2011 and October 2013 in all of Rwanda’s provinces except Kigali (see Roape.net 2019c). Despite these facts, the World Bank proceeded to endorse NISR’s results as if they had been generated using the official CPI inflation rate.
The Financial Times investigation (Wilson and Blood 2019) obtained confidential communication from World Bank staff, which confirmed that the World Bank was aware of the discrepancies in Rwanda’s official statistics from as early on as December 2015. At that point, World Bank staff had warned the top management in the Bank about the reputational risks incurred by continuing to back Rwanda’s flawed official statistics.
Despite these revelations, and despite the publication of data and syntax files providing verifiable and incontrovertible proof of the mistakes in the World Bank’s calculations, the World Bank has continued to back Rwanda’s official statistics until now. So far, they have issued one working paper and two press statements confirming their endorsement of Rwanda’s official poverty statistics, despite not providing any evidence to counter the allegations published in the various ROAPE blogs and in the Financial Times article (Ibid.).
2018 . poverty estimates
In December 2018, NISR published its fifth EICV survey (NISR 2018), claiming that poverty had decreased by a further 0.9 percentage point since 2014. Surprisingly, and extremely unusually, NISR’s report did not contain any methodological details whatsoever describing how the poverty line had been updated between EICV4 and EICV5, nor which inflation rate had been used.
A month later, a new blog appeared on Roape.net, showing that poverty had continued to increase since 2014, when using the price data contained in the EICV survey itself. This result implied that poverty would now be over 64% in Rwanda, which is significantly higher than it was at the time of the first EICV in 2001 (58%), using a methodologically consistent and comparable poverty line (Roape.net 2019a). However, these initial poverty estimates were too high, as they failed to take into account non-food inflation due to lack of data.
In March 2019, the EICV5 poverty estimates were refined, using alternative methods to estimate non-food inflation, by looking at the changing share of non-food consumption in total household consumption. The method provides a conservative estimate of poverty, since the food share could decrease due to deteriorating living standards in accordance with Engel’s law, even in the presence of high non-food inflation.
These new estimates confirmed the main findings of the previous paper, namely: (a) there has been a two-digit increase in poverty between 2011 and 2018; and (b) total poverty is now higher than it was when NISR started to measure poverty back in 2001.
The findings were shown to be valid both when using price data contained in the EICV survey itself, and when using price data contained in the more detailed e-Soko survey, which monitors food prices in rural areas.
Conclusion
This briefing set out to take stock of the Rwandan poverty debate and explain differences between the estimates generated by various researchers. There is still room for improvement in these estimates. However, with each iteration, the unexplained differences are getting smaller, and the results are becoming more robust.
The most reliable estimate, using the NISR’s 2014 four-step, non-normative poverty line, indicates that poverty has increased by over 15 percentage points in the last eight years and that the current poverty rate in Rwanda is between 61% and 64% (with and without non-food inflation, respectively). This is higher than when NISR started to measure poverty in 2001. As far as we are aware, the only country in the world, for which data are available, that has had a sharper increase in poverty in the past decade is South Sudan. Despite this, Rwanda continues to be presented as a development success story by international donors and financial institutions. As recently as November 2019, the IMF decided to revise their GDP growth estimate for Rwanda up from 7.8% to 8.5% for 2019 (IMF 2019), based on new data provided by NISR, notwithstanding the fact that NISR’s GDP figures have in the past been shown to contain major inconsistencies, for which neither NISR nor the IMF have so far provided any explanations (Roape.net 2017a).
The series of blogs and papers published on the issue of Rwandan poverty have helped to establish beyond any reasonable doubt that poverty has increased sharply in Rwanda since 2010. But in settling the debate on Rwanda’s poverty record, they have raised new and fundamental questions about the development sector as a whole that extend well beyond Rwanda: how could poverty have been increasing so sharply for over eight years without any donors or international organisations apparently noticing? How could it take almost four years from the time when the first concerns were raised until these facts were thoroughly investigated and checked – and then not by the people who are mandated to monitor these issues? How could an organisation that claims to exist for the sole purpose of reducing poverty, endorse NISR’s figures publicly on three separate occasions despite clear and verifiable evidence from several different and credible sources showing that those figures were incorrect? And, most of all, how could one of the sharpest non-conflict-related increases in poverty in recent times have been passed off as an economic miracle for so long, and lauded as a model for the developing world?
These questions will need to be examined by political economists and political scientists for many years to come and are likely to haunt the organisations involved for many more decades.