THE COST OF LATE PAYMENTS IN PUBLIC PROCUREMENT

​: The objective of this paper is to show how detailed open data on public procurement payment processes can be used to enable cost savings for the public administration. We do this by using the duration of public procurement payments to estimate the cost that delays in the payment process have on suppliers and to identify variables that can affect these delays. Our analysis is based on detailed open data from the National Treasury Department from Paraguay to analyze payments of public institutions from 2011 to 2017. We use a descriptive analysis and a financial cost estimation to calculate the cost of late payments on suppliers. In addition, we model the duration of payments using survival analysis to identify which variables have a role in delaying payments. The preliminary findings show that the duration from the moment an invoice is issued to when the payment occurs can be of approximately 55 days on average, for each payment. In comparison, international practice considers 30 days an acceptable payment period. Our analysis on the historical data shows that late payments have an accumulated cost of $142.29 million in the analyzed time frame. Furthermore, roughly 48% of these costs could be cut down in Paraguay if some of the steps in the payment process are analyzed and an appropriate corrective normative framework is created. This analysis shows the impact that open data can have in cost saving when properly implemented and analyzed. Keywords​: open data, public procurement, open contracting, late payments, financial cost, public payment delays


INTRODUCTION
Open public procurement data has gained attention in the past years as a way of increasing transparency and helping governments improve their procurement practices. According to the latest report by Open Contracting, in 2018 over 30 countries are publishing data following the open contracting data standard (OCDS). Our analysis is a case study of how open data of public procurement, when analyzed and re-used, can yield useful inputs to identify inefficiencies in the procurement process, recommend structural reforms to improve the procedures and drive cost savings for the public administration.
For this paper, we focus on the last stage, particularly the financial implementation which relates to the process of payment to suppliers. In this phase, governments sometimes face the problem of not paying suppliers in time, which causes inefficiencies and financial costs. Failure to pay invoices promptly, generates a negative effect on firms that provide goods, services and public works to the government, causing short-term liquidity problems. This problem might force businesses to turn to the financial market to cover their obligations, use their savings, or go out of business (World Bank 2017). If money is lent from the financial market, providers incur in extra costs due to interest rates, which is included in the cost structure of the provider, implying an extra cost for the public administration.
In addition, this practice can ultimately affect economic growth (Checherita-Westphal, Klemm and Viefers, 2015). Moreover, if this procedure becomes the norm, suppliers may decline to do business with the government, reducing competition, and thus the possibility of obtaining better value for money for the purchasing entity (World Bank 2017).
Previous work on public procurement payments has focused on measuring arrears in national accounts (Diamond and Schiller 1993) and on how governments can prevent and manage payment delays (Flynn and Pessoa 2014). There are also specific country reports that analyze public procurement payment systems (World Bank, 2008Giussani, Guardiola & Ospina, 2016). On the other hand, Checherita-Westphal, Klemm and Viefers (2015) use annual data from 17 European countries and a proxy for government arrears to calculate the economic impact; and Connell (2014) estimates the cost of late payments on Government to business transactions for 26 European countries. Other works have studied inefficiencies in other stages of the procurement process (Balaeva, 2017;Fiordelisi, Franco et al., 2012) or analyzed late payments in the private sector (Smirnov, 2016). This paper estimates the financial cost of the delays in public procurement payments and identifies variables that can affect the delays. Contrary to previous work that used proxy variables to calculate the cost, we analyzed 599.354 detailed payments of 59 public institutions from 2011 to 2017, released using open data formats from the National Treasury of Paraguay (datos.hacienda.gov.py). We used a set of methodologies for cost analysis readily available in the literature and adapted them to the data availability to show how this can be applied in different countries, using the Paraguayan data as a case study. Our calculation, follows Connell's approach, to estimate the cost of late payments on firms and on the public administration but focusing on a single country (Paraguay) for a seven-year period given the rich information about the payment process released at datos.hacienda.gov.py . In addition, we 3 use survival analysis, a technique that allows us to model payment duration and to identify which variables have an impact in delaying payments. The preliminary findings show that the median duration from the moment an invoice is issued to when the payment occurs via a bank transfer to the supplier's bank account can be of approximately 55 days, for each payment. In comparison, international practice considers 30 days an acceptable payment period (Flynn and Pessoa, 2014). Moreover, our analysis on the historical data shows that the cost of the late payments is of approximately $142.29 million in the analyzed time frame, and that roughly 42% of the costs ($68.6 million) could be cut down in Paraguay if some of the steps in the payment process are analyzed and the appropriate normative framework is created.
Our work aims to encourage the publication of open public procurement payment data, by showing how this information can be analyzed to reduce costs in the public procurement process. Our methodology could be replicated in other countries that have already published payment data in open formats.
This paper is structured as follows. Section 2 describes the state of the art relating to our problem of study. Section 3 explains the data used and methods used for the analysis, transformations made to the variables and limitations of the study. Section 4 shows the results of the analysis and section 5 presents the conclusions and future works.

II. STATE OF THE ART
The current state of the art on public procurement payments, has mostly focused on discussing the issue of government arrears, on measuring the economic impact and financial cost of late payments for several countries and on analyzing other stages of the procurement process.

A. The problem of arrears
The International Monetary Fund has addressed the issue of late payments in different reports. Flynn and Pessoa (2014) analyzed how can governments prevent and manage expenditure arrears, which includes payments to private contractors. According to the report, one of the biggest issues on managing arrears, is that what is considered as a delay may vary between countries and is dependent of the maturity of the payment system. They also list the impact chronic arrears may have at an aggregated level, which includes reduced economic growth, increased cost of service provision, reduced or interrupted public delivery and increased interest rates, among others. However, these effects are discussed theoretically and not measured using other techniques.
On the other hand, Ramos (1998) focused on how can governments address the arrears problem with securitization in order to provide temporary relief from debt service obligations and increase government credibility.
For the specific case of Paraguay, the World Bank pointed out in a 2008 fiduciary assessment that improvements were needed in the public procurement (PR) payment management system, since a relevant percentage of contracts were in arrears. "The delay in effecting payment negatively impacts the willingness to participate in public PR processes or is reflected in quoting higher prices, discounting the financial costs associated to these delays, thus producing inefficiencies to the system" (p. 62). In 2016, another assessment (Giussani, Guardiola & Ospina) pointed out that the volume of arrears (of all government expenditure) was around 9.68% of total expenditure, when the recommended percentage is of 2%.
In addition, they cited a report by the Public Comptroller, which had estimated a 20-day average delay.
More recently, a World Bank's public procurement report (2017) presented comparable data on public procurement regulations across 180 economies; they divided the analysis in eight pillars, one of which was the payment of suppliers. In this category, Paraguay received a score 48 out of 100 -the lowest score in all the indicators measured-, since the actual time for suppliers to receive payment was between 31 and 90 days. In addition, there were no automatic penalties paid to the suppliers in case of delay and the time to process the payment did not start from the submission of invoice, which can slow down the payment arbitrarily.

B. Cost and impact of late payments
To the best of our knowledge, the cost of late payments in the public sector hasn't been explored yet in Latin America, and recent studies have measured the economic impact of late payments in European Countries. For instance, Checherita-Westphal, Klemm and Viefers (2015) use annual data from 17 European countries and a proxy for government arrears to calculate its economic impact. They discovered that payment delays reduce economic growth, increase the likelihood of bankruptcies and reduce profits. For instance, a one standard deviation change in delayed payments reduces the growth rate by 0.8 to 1.5 percentage points, and reduces profit growth by 1.5 to 3.4 percentage points. Additionally, they found that the larger the delayed payments, the higher the probability of default among private companies. Because of the cross-country and short time period data, they used dynamic panel models and a Bayesian VAR to estimate these effects.
Additionally, Connell (2014) estimates the cost of late payments on government to business transactions at an aggregated level, also in the European context. The study estimates the short-term financial cost, applying annual interest rates to the claims against the public administration (which serves as a proxy for the payment delays), times the average delay.
This approach to calculate aggregated costs is useful for our estimation, with the difference that we focus on a single country. Furthermore, Connell calculates the impact these delays may have on the exit rate of firms, using the ratio between the number of deaths of enterprises and the total number of firms in a given country, across several years. The result shows that a 1 point reduction in the delay ratio leads to a decrease in exit rates of about 1.7 percentage points.
A limitation, found in these studies is the lack of full information regarding government payments, so the estimation is made using a proxy. On the contrary, we have micro-level data about each payment, which allow us to calculate the aggregate cost more precisely.
Furthermore, Fiordelisi, Franco et al. (2012) calculate the cost for the Italian economic system resulting in the delay of trade loans by the public administration. The study explores different scenarios according to different payment times, and estimates the financial cost using the interest rate, times the delay and volume of credits towards the public administration. In addition, they estimate the aggregate social cost comparing this value with the expenditure the government would have faced to pay on time, using the rate on Treasury bills.
On the other hand, (Valcani Vicari Associati et al., 2015) evaluated if the 2011 European Union Late Payment Directive had accomplished its achievements of reducing payment times in 28 countries in Europe. They found that after four years of implementation, the average duration had fallen from 65 days in average in 2011 to 58 days in 2014, but stayed beyond the 30-day optimal deadline. "Rather than legislation, national business culture, economic conditions and power imbalances are the driving factors for payment behaviour" (p. 68). This is a relevant result for our investigation, since it is important to consider other factors besides legislation in order to drive change in the payment culture. They also found that firms that had the government as a main client were more likely to have difficulties paying to their suppliers, which indicates that late public payments can have an impact on a larger supply chain (p. 53).

C. Inefficiencies in public procurement
The literature on public procurement also includes several studies which analyze the cost and efficiency of the public procurement process, and not exclusively of the payment stage. Some of these works use Data Envelopment Analysis (DEA), a method that is generally used to estimate a production cost function, with minimal assumptions. This methodology calculates an efficiency frontier for a set of units, using different inputs and outputs, and then gives an efficiency score (it considers efficiency as the lowest input amount to produce one unit of output). Guccio et al. (2012) used this method to investigate the performance of Italian public contracts in terms of time of completion of works and cost overruns.
We find more useful the approach used by PricewaterhouseCoopers (2011) and Yakovlev and Balaeva (2017). They analyze the cost and effectiveness of public procurement in the European Union and Russia using estimates of person-days spend in the procurement process, and then they applied data of employee remuneration to calculate the labor costs and the total costs of the procurement. In their regard, shorter procedure times indicate higher efficiency. This methodology can be adapted to analyze exclusively the payment stage of the process. However, we could not implement this methodology due to the lack of data.
Finally, another set of studies have analyzed delays in other stages of the public procurement process. For instance, Gori et al. (2017) explored the variables that affect the duration of public works in Italy using survival analysis and found that the lack of experience of local governments results in a higher delay probability in the execution of the contract.
On the other hand, Smirnov (2016) proves survival analysis is a good approach to model late invoice payment times in the private sector. Given the similarity of the transaction, this approach can be used to model government to business payments, our case of study.

A. Data description
The public procurement payment process in Paraguay consists of three different stages. The first one, comprises the period from when the invoice is issued until it is approved by the procuring entity (Invoice stage). In the second stage, a transfer request (TR stage) is generated, and it is then sent to the Treasury. In the final stage, the Treasury creates a transfer order (TO stage) and then makes the payment to the contractor, via a bank transfer. The first two stages happen inside the procuring entity, while the last stage is in charge of the Treasury. There are also intermediate stages in each of the steps, explained in Figure  1. Invoice creation (receipt of invoice) (obl_fecha elaboracion ) 4 2. Invoice approval (obl_fecha_aprobacion) 3. TR creation (str_fecha_ingreso) 4. TR approval (str_fecha_aprobacion) 5. TR transfer to Treasury (str_fecha_recepcion_tesoro) 4 In parenthesis, we add the variable name derived from the dataset. All the variables are explained in Table 1 6. TO creation (ot_fecha_generacion) 7. Payment (ot_fecha_deposito) The process is as follows: The provider presents the invoice to the procurement entity (1) and then has to wait until the institution creates an obligation for the invoice and approves it (2). For one or more obligations, a transfer request (TR) is created (3), it is then approved (4) and finally the procurement entity sends it to the Ministry of Finance (Treasury) for payment (5). There, for one or more TR's the Treasury creates a transfer order (TO) (6) and then executes the payment (7).
According to the IMF, "international practice on what is an acceptable delay between receipt and payment of the invoices varies from anywhere between 30 to 120 days" (2014, p. 4). In addition, the World Bank (2017) reported that most suppliers in high-income economies receive payments in less that 30 days.
In Paraguay, there is not a single deadline in laws or regulations, from receipt of invoice to payment, and the specific deadline is stipulated in each contract. However, according to the executive order 8452/2018, the TR stage has a 30-day deadline. In addition, the DNCP order 1024/11 about General Conditions of the Contract (Pliego estándar de contratación) states that the contracting party will execute the payments as soon as possible, but in no case, it may exceed sixty (60) days after receipt of invoice or request of payment, and after the contracting party has accepted the request. The acceptance or rejection of the invoice must be given no later than fifteen days (15) after its presentation . These 15 days are before the process starts and the invoice is created. This means, that 5 the whole payment process can last 60 days, if not specified otherwise in the contract. In addition, according to the Law 2051, and the General Conditions of the Contract, if the payment is delayed, the contractor has to recognize financial interests to the provider, which implies an additional cost for the public administration.
According to this legislation, the regulation of payment times is not clear, which can contribute to long costly delays. For our analysis, we consider the following delays and deadlines: Time between the issue of the invoice and its approval in the procuring entity.
(2) -(1) 6 No specific deadline TR stage Time between the issue of the TR and its dispatch to the Treasury. • List of obligations (Listado de obligaciones, available at https://datos.hacienda.gov.py/data/obligacion) , a dataset with information about each invoice.
• Transfer request (Solicitud de transferencia de recursos, available at https://datos.hacienda.gov.py/data/str) , a dataset with all the request that are generated in the entity for one or several payment obligations to the provider. Each transfer request (TR) can have one or more invoices.
• List of transfer orders (Listado de órdenes de transferencia, available at https://datos.hacienda.gov.py/data/orden-transferencia) , which contained the detail of each transfer order, an instrument through which the delivery of funds to the beneficiary is made. Each transfer order can have one or more transfer requests.
The above datasets were combined using a unique identifier, where the observation unit was given by each invoice. The dataset had information about each invoice, the contracting procedure and the provider. We also used two other datasets from the Treasury's open data portal to obtain two variables about the institutional budget execution and the number of officials working in each entity . 7 8 These served as proxies for the institution size and its efficiency using public resources.
The initial database had 599.354 observations, however we found several errors, which we validated with the Treasury officials. In the cases where these errors could not be corrected we eliminated the observations from the dataset. We made the following transformations to clean the data : 9 • We eliminated the observations with empty date values and those in which the date of the invoice was before the date of the contract. • After creating variables for the delays, we obtained several negative values in the duration.
We eliminated these observations, since we could not correct the error. • We deleted outliers found in each of the three stages of the process.
• We eliminated observations where the duration of the payment was less than 10 days, an unlikely scenario; and observations where the sum of the invoice was less than 50.000 guaranies ($9).
In addition, we identified outliers in each of the stages of the process and segmented the database in two: We decided to do a separate analysis of the two datasets, to avoid the effect of the outliers in the larger sample. After the transformations, both the databases had and the following variables:

B. Survival analysis
We modeled late invoice payments using survival analysis, to identify which variables have a role in delaying payments. Survival analysis is a statistical method used to analyze and model the data when the outcome variable is the time until the occurrence of a specific event. In our case, the event of interest is the date of payment of an invoice. These methods have been previously used by researchers to study the duration of public works in Italy (Gori, Lattarulo and Mariani 2017) and to model late invoice payments times in the private sector (Smirnov 2016).
The idea of this approach is that subjects are followed during a time period until the event occurs. The event is called a failure . In this case, t=0 is the date the invoice is created and the failure will occur when the payment happens. In the context of survival analysis survival will mean the invoice hasn't been paid yet.
The duration of the state (payment period) is a non-negative random variable called T , with a cumulative distribution F(t) (Cameron and Trivedi 2005). The probability that duration of the episode is less than t is: (1) s rob The probability that the event equals or exceeds t , or in our case the probability that the invoice is not paid before time t , is given by the survival function: We also estimate d the hazard function h(t) , which is the probability of leaving a state conditional on survival time t . That is to say, the rate of success at time T=t given that the invoice has not been paid for up to time t . (3) We used the Kaplan-Meier method to estimate the survival function, project the survival curves and estimate the differences between groups. The estimator S(t) is given by: where is the number of survivors at time and is the number of events that happened until Finally, to estimate the effect of covariates in survival time, we calculated a parametric model with a Weibull distribution, after testing the fit of different distributions on the data . We included the 12 following variables as predictors: • sum of each invoice (log transform) • contract sum, (log transform) • institution , in this case this will be the debtor • number of officials, size of the institution payroll • budget execution, average budget execution of the entity (contractor) • type of funds , the financial source used to pay the invoice (categorical variable) • type of purchase, segmented in services, goods and materials, investment and others (categorical variable)

C. Financial aggregated cost estimation
To estimate the short-term aggregated financial cost of late payments for firms, we followed Connell's (2014) approach on calculating this effect on Government to Business transactions in Europe.
The idea behind this calculation is that private providers need to compensate the lack of liquidity generated by payment delays. Our assumption, is that failure to pay invoices promptly, generates a negative effect on providers, causing short-term liquidity problems for firms, and forcing them to turn to the financial market to cover their obligations. In addition, this can be seen as an opportunity cost for firms that, even if they have enough liquidity to face the delays, cannot invest and have to use the money to face the costs of the delay. Moreover, national legislation indicates that providers are entitled to receive interest due to delays, so we assume that if the payment was delayed the public administration had to pay this extra cost.
To obtain the financial cost, we used annual the average lending rate, which is the bank rate that meets the short and medium-term financing needs of the private sector, to the amount of the overdue payment, times the delay expressed as a fraction of a year: C=P*i*d (5) where: C= estimated cost for firms P=amount of each payment in guaranies i= annual average lending rate d= delay expressed as a fraction of a year The cost was then adjusted by inflation using the GDP deflator. Both the data of the average lending rate and the GDP deflator were obtained in the International Financial Statistics website . The 13 amounts were converted to USD using the average 2017 exchange rate (5,618.933) from the World Bank open data portal . This approach can be applied to other countries, using these same sources of 14 financial information and local procurement data. 12 See http://rpubs.com/camilamila/pagos_paraguay 13 Dataset available in: http://data.imf.org/?sk=4C514D48-B6BA-49ED-8AB9-52B0C1A0179B&sId=1409151240976 14 https://data.worldbank.org/indicator/PA.NUS.FCRF?locations=PY We calculated the number of invoices that surpassed the mentioned deadlines and then estimated the aggregated cost of this delay. In addition, we calculated the cost of the total delay, even if the payment was executed before the above deadlines. Finally, we also considered different scenarios, with different payment times (30, 45 and 60 days), to analyze the possible savings the public administration could have if it modifies its payment regulations.

1) Lack of exact invoice due date
We didn't have the exact due date for each of the invoices, since this data is specified in the contracts. This impedes to calculate the exact due date on the invoices and thus a potential cost that the administration had to recognize in interests to providers. A recommendation to public authorities would be to include this information as part of the public procurement open data.

2) Information about the providers
Not having information about the characteristics of the firm (size, sector) can result in an underestimation of the cost, since the interest rates can be different. For simplicity, we used the average lending rate for the private sector. Moreover, having open data about beneficial ownership could help expand the analysis to determine if there are clusters of providers that accumulate contracts.

3) Institutional data availability
Even though we have a robust dataset with payment information of 59 institutions that pay through the Treasury payment system, this is not a complete sample of all the public institutions, thus the cost is underestimated. The study could be expanded and recalculated when more institutions provide payment information in open data formats. Finally, we had to discard a lot of observations due to errors on the original dataset. This shows there is still an opportunity to improve the collection and transformation of procurement data in open data formats, and implement other validation techniques in order to minimize the errors.

A. Descriptive and survival analysis 1) Payment duration by stages and funds
From Table 3 it is determined that only 22% of the invoices issued for public contracts between 2011 and 2017 in Paraguay, were paid in 30 days or less . The 30-day deadline is considered an ideal 15 threshold in which payments to providers should be executed. This means that 78% of payments in the analyzed period were delayed. On average, public institutions took 55 days to pay invoices. However, considering the 60-day deadline that payments can have according to regulations, if not specified otherwise on the contract, 47% of payments where delayed, using the complete sample. This shows, that even though the payment deadline in Paraguay is twice the period of what is considered an optimal payment time, a quarter of all public procurement payments in the analyzed period surpassed that limit (see Figure 2). This practice affects providers and the public administration, since this cost might be internalized in the provider's cost structure, and thus the final price of the good, service or public work is higher than expected. When segmented by year, we found that the median duration spiked after 2012 (See Table 3), it increased from 37 days to more than 52 for the subsequent years. Between 2011 and 2012, there were pay rises approved for public servants, which increased the spending in wages 44% between 2011 and 2017. In contrast, the spending in the budget items related to public contracts only grew 11% on the analyzed period. This caused liquidity problems from the public administration, who had more trouble assigning funds to pay contracts. This change is well illustrated in Figure 3, were we see an increase in payment times in the first and last stages of the payment process. While the invoice stage has the longest median duration of 22 days, the TO stage increased from a median of 5 days in 2011 to between 13 to 20 days in the subsequent years. In this last stage, the Treasury has to execute the payments according to the available resources, which were more limited after 2012.  Moreover, it is clear that the first stage of the process is the one that takes longer, regardless of the institution (see Figure 5). According to Treasury officials the main cause for this delay is that invoices cannot be loaded into the system and then approved if there is not a cash plan assigned by the Treasury, to pay those obligations. The main problem at this stage is that in practice there is a difference between the projected spending budget and the actual income budget that is financed through tax revenues, thus, the entities might not receive all their projected resources for a certain period. However, since the obligations are not recorded into the system, the Treasury has no way of knowing how to prioritize the assignment of resources to institutions that have a large number of bills to pay, which contributes to the delay. Our recommendation would be to improve cash planning and management, to reduce timing problems between payments coming due and the availability of funds to pay them. For instance, if the system allowed to load the invoices prior to having a cash plan, the Treasury could identify how much resources to distribute according to the needs of each entity.

Figure 5: Average payment duration by institution
Moreover, the duration is also affected by the funds used to pay the invoices. It takes longer to pay invoices that depend on Treasury funds (55 days), than those using institutional resources (36 days).
To explore this relationship further we used Kaplan-Meier curves. As it follows in Figure 6, the curves do indicate a difference between the groups: the invoices that use Treasury funds, take longer to be paid (higher survival probability), than those using institutional funds. As shown in Figure 7, using Treasury funds delays two stages of process: the invoice stage, since institutions cannot load the invoices into the system without a cash plan (as explained above), and the TO stage, since the Treasury cannot pay for the obligations if there are no available resources. On the contrary, bills that use public credit funds and institutional resources, do not need a cash plan to create obligations in the first stage, and once the transfer order arrives to the Treasury, the funds are available to execute the payment. However, for these two types of funding, the invoice stage has a median duration of more than 22 days, which could be a sign of inefficiencies inside the procurement agencies and not a result of budget constraints.

2) Duration by institution
We divided the institutions by their size, according to their current payroll, to explore if the size of the entity could indicate a difference in payment times. Our hypothesis is that larger institutions (with more than a 10,000 employees) have a larger volume of invoices to pay and might have more complicated bureaucratic processes, than smaller entities (less than 1,000 employees). As shown in Figure 8, the Kaplan-Meier curves show a significant difference between the groups. Large institutions, have a higher survival probability (meaning they take longer to pay invoices) than smaller entities. While 50% of the invoices in small and median institutions are paid in less than 50 days, larger entities take longer. When calculating the duration by institutions (in general), there seems to be important differences of payment times, which can be a sign of different practices between the entities. The Ministry of Health (the second in size) has a median payment duration of 73 days (without outliers), while the Foreign Ministry (median size) pays invoices with a median of 22 days. Only 6 of 41 institutions analyzed 16 have a median duration of less than 30 days. Moreover, besides having the longest median duration, the Ministry of Health is the second with the highest spending in payments, with an accumulated real spending of approximately $772 million (18% of the total), in the analyzed period. Nevertheless, the delays in the Ministry of Public Works can be the costliest, since this institution accounts for 45% of the total spending in public procurement payments. The difference in the duration of this two institutions could be the funds used to pay the invoices: while 68% of the Ministry of Health's invoices were paid using Treasury funds, the Ministry of Public Works paid 60.7% of its bills with funds from the Public Credit. As explained above, using Treasury funds causes delays in two stages of the process, however, Figures 9 and 10 show that both entities are inefficient loading the invoices regardless of the type of funds, which contributes to extend the duration. Thus, it is necessary to evaluate the payment practices in these entities to find the cause for the delays. 3) Providers and type of purchase On the other hand, out of the 4,742 providers that received payments in our sample, 80% of them had more than half their invoices paid after 30 days. Only 8% of the contractors were paid without a delay. Thus, payment delays are common and affect must of the participants in the procurement processes. In addition, for the larger sample, providers that receive transfers from the government suffer from the longest median duration of 70 days (see Table 6). This is a special case of the Ministry of Education that uses transfers to buy food for school. In addition, goods and materials suffer have median duration of 56 days, while invoices related to services or infrastructure are paid faster, with a median of 47 and 43 days, respectively. For the extreme values sample, the median duration of goods and services, of 177 days is similar to the one of investment works, of 176. We would expect that smaller invoices and simpler contracts, like the ones related to goods and materials are easier to pay than works of infrastructure that tend to be costlier, however this does not seem to be the norm in Paraguay's public procurement. This can be due to the funds used to pay the invoices (as explained above) or that some providers might have more negotiation power when executing contracts and demanding payments. Nonetheless, the lack of information about the companies regarding their size, sector and financial characteristics, impedes a more detailed analysis and this could be explored in future works.

4) Seasonal trends
We also observed seasonal trends and found that payments tend to increase in the last quarter of the year, before the end of the fiscal year in December (See Figure 3). In 2013, 2014 and 2015, there is also a peak in May, that might be caused by the time it takes to execute payments. Thus, the invoices issued at the beginning of the year start to be paid at the end of the first quarter. In addition, there were no payments registered in January, in any of the years analyzed, which can be due to lack of budget or liquidity in public institutions. In comparison, the issue of invoices, peeks for most of the years between July and September. In general, there is a lag between the issue of invoices and the payments, which confirms there are delays. Providers might be aware of this problem, and thus issue their invoices at particular times in order to guarantee payments before the end of the year. We also observed that invoices are accumulated before the Treasury pays them. As shown in Figure  12, the stock of unpaid bills in higher in the second and third quarters and decreases at the end of the year. This means, that providers that present invoices at the beginning of the year face longer delays and thus higher costs, than those with payments in the last months. In fact, the median duration of payments for bills issued in the first quarter is of 66 days, while it reduces to 34 days for invoices presented after October; in the case of invoices presented in the second and third quarters, the median duration is of 49 days. By taking into account the seasonal trends institutions can plan better their procurement processes, in order to schedule payments in periods where there is enough liquidity to pay promptly; or if possible distribute the stock of invoices through the year, instead of concentrating them in a particular period.  Table 6. Statistical significance *** p<0.01 We found that a one percent increase in the invoice amount shortens survival time by 0.97 times, this means that larger invoices are paid faster, under the assumption that all variables are held constant. However, the same increase in the amount of the contract extends payment time (survival) by 1.03 times, so the effect might seem contradictory. A possible explanation could be that for bigger contracts the payments are segmented and thus the invoices have a smaller amount.
Using Treasury funds to pay the invoice, increases survival time (payment takes longer) by 1.31 times in comparison to using institutional or Public credit funds. Moreover, institutions with more than 10,000 employees extends survival time by 1.43 times in comparison to smaller entities with a payroll of less than 1,000. As explained with our descriptive results, larger institutions have a higher volume of invoices and the payment process can be more complex. Also, an increase in the institutional budget execution reduces survival time, showing that entities that are more efficient executing public resources tend to pay faster to providers.
Finally, invoices related to services purchases are paid faster than purchases for investment or goods and materials. For instance, investment contracts extend payment time by 0.03 times, in comparison to services. Other previous work (Gori et al., 2017) has found that infrastructure contracts are associated with longer delays, and thus this could affect payments. However, the invoices related to goods and materials have a larger effect on payment time than investment, as seen on the descriptive results.
These results show areas where institutions might be able to improve in order to reduce delays in payments.

B. Cost estimation
The cost of late payments was calculated using different scenarios, according to the deadlines, explained in Table 1: • 30 days (ideal case consistent with international recommendations) • 45 days • 60 days (current maximum deadline in local legislation) • 75 days • 15 days (only for the invoice stage) • total delay (we estimate the total cost of the payment duration) According to our estimates, between 2011 and 2017, the total cost of late payments was of $142.29 million (0.48% of 2017 nominal GDP). Considering a deadline of 30 days, the cost of the delay is reduced to $81.07 million (0.28% of 2017 nominal GDP). This means that if this new deadline is established and the institutions pay on time, costs could be cut down 56.9%. For the payments overdue in more than 45 days the cost was of $61.12 million (0.21% of GDP), in the case of payments in 60 days it accounted for $47.11 million (0.16% of GDP) and for the 75-day deadline $36.82 million guaranies (0.13% of GDP).
Finally, the cost of delaying the invoice stage in more than 15 days costed $68.63 (0.23% of GDP). As explained in the descriptive results, the invoice stage is the one that accounts for most of the delay, and thus has the biggest cost of the different stages of the process. The complete results are shown in Table 6. The results can be interpreted as the costs private providers had to incur in order to face the liquidity problem caused by payment delays. Having longer deadlines, such as the 60-day in national regulations, could benefit public institutions, since they have to recognize less interests to providers if the payments are delayed and claims are presented. However, this has a negative effect on contractors and at an aggregated level, since they have to find resources to cover their financial needs for the whole payment period, not only after the 60-day deadline. Moreover, since providers know there are delays in payment times, they might internalize the financial cost in the cost structure of the contract, which translates into higher prices for the public administration.
A possible solution to this problem could be to reduce payment deadlines in regulations, to force institutions to accelerate their payment processes and thus reduce the aggregated financial cost. For instance, if the new deadline is set in 30 or 45 days, costs could be reduced by 56% and 42%, respectively. Nevertheless, this must be accompanied by better, and more efficient practices inside the procuring entities, since if only the legislation is changed, but not the internal processes, this may imply that the institutions might have to recognize more interests for late payments to suppliers. For instance, an evaluation of the European Union Late Payment Directive in 2014, concluded that a new regulation that forced institutions to pay invoices in 30 days or less had not reduced greatly payment times in most of the European countries, but stakeholders argued that legislation had obliged governments to act and improve their payment practices (Valcani Vicari Associati et al. 2015).
When analyzed by year, the cost has risen since 2011, but remained relatively constant after 2015.
The huge difference between 2011 and the other years, is a result of the change in duration in the different stages of the process, as explained in the descriptive results. As a percentage of GDP, the total cost was of 0.02% of GDP in 2011, then rose to 0.09% of GDP in 2014 and stayed the same in the following years ( Figure 14). In comparison to previous studies about the aggregate cost of late payments in other countries in Europe, Paraguay has a similar cost to countries like Italy or Spain, with a cost close to 1% of annual GDP (Connell, 2014). Moreover, the cost is higher in the first quarter of the year and reduces significantly in the last quarter, showing that seasonal trends in payments can increase the costs at specific periods of the year. As expected, the larger cost of the duration of the payment is concentrated in the first stage of the process, which accounts for $95.8 million or 67% of the total cost. Changes can be made in this stage in order to improve the process. For instance, if a 15-day deadline is established in this step to encourage institutions to accelerate their payments, the costs could be reduced by 48%. Figure 15: Total cost by stages Finally, three entities concentrate 73.9% of the total cost. The Ministry of Public Works accumulates 37% of the total cost, the Ministry of Health 30% and the Ministry of Education 5.9%This means that procuring payment practices must be improved in these institutions in order to obtain the biggest savings. Figure 16. Cost of delayed payments by institution

V. CONCLUSIONS
This work shows how to calculate the cost of the duration and delay of payments, and the variables that affect payment time, using detailed open contracting data. We found that the total cost of the payment duration in Paraguay, between 2011 and 2017, was of $142.29 million, equivalent to 0.48% of Paraguay's 2017 Gross Domestic Product. In general, procuring entities take 55 days on average to pay invoices, while the international optimal deadlines are stablished in 30 days. For some institutions, the duration can extend for more than a year, which shows there are inefficiencies in the procurement payment process.
Moreover, two ministries, Health and Public Works, concentrate 67% of the total cost, so the corrective efforts should concentrate in these entities. There is also evidence that the funding source, the type of purchase, the size of the institution and its budget execution affect payment duration times. Besides, the first step of the payment procedure (invoice stage), which is not regulated, is the one that takes longer. Our recommendation would be to revise the procurement practices at this stage in order to shorten the delays. If a deadline of 15 days is met, costs could be reduced by 48%. Moreover, if the total payment deadline is established in 45 or 30 days since the invoice issuance, the cost could be reduced by 42% or 52%, respectively.
We conclude that there are two main areas of improvement in the payment process. First, regulate the first stage and stablish a deadline, to force institutions to modify their practices to load invoices into the system more quickly. This is particularly important for invoices paid with institutional and public credit funds, since there is no justification to delay the process if there are resources available to pay. In the case of bills paid with Treasury funds, our recommendation would be to modify the steps in the process, so that invoices can be loaded into the system without a cash plan. This can help the Treasury identify which payments are coming due and distribute the funds accordingly.
Moreover, this work demonstrates how public procurement open data can be analyzed to generate high value insights for the public administration, in order to improve contracting practices and save public funds. Our methodology can be implemented in any country that publishes payment and contracting data in open formats, and aims to serve as an example of how institutional efforts to publish detailed open data about all the steps of the procurement process, can pay off when valuable insights are derived from its analysis.
Our study faced some limitations regarding mainly the availability of data. It would be useful to have more disaggregated data about the providers in order to determine whether the delays are affecting more small and medium enterprises or big companies. Even though the cost can be higher to bigger providers that obtain larger contracts, for smaller enterprises it could be harder to access credit or to cover the lack of liquidity caused by the delay. In addition, not having information about the characteristics of the firm (size, sector) can result in an underestimation of the cost, since the interest rates can be different. Having open data about beneficial ownership could help expand the analysis to determine if there are clusters of providers that are more affected by the delays, and could also be useful to estimate the effect of late payments on the exit rate of firms. In addition, the analysis could be extended if payment information of other institutions in the public sector published payment data in open formats.
Finally, this investigation could be extended with a dataset of the cash plan, to compare the availability of resources with the payment information, in order to determine more efficient ways to redistribute the resources and thus reduce payment duration.