A Comparison of Three Model Checkers Applied to a Distributed Database Problem

Increasingly model checking is being applied to more abstract problem domains than the traditional protocol analysis. The extent to which such an approach is able to provide useful insight into the problem domain depends to a large degree on the nature of the tool used. This paper reports the results of a study using three different model checkers which differ widely in their specification language internal implementation and facilities for specifying correctness properties. 
 
An abstract model of an industrial distributed database application has been studied using the three tools. A detailed model of the application using each of the tools is presented and the extent to which each tool allows us to investigate interesting properties of the problem domain is compared. Some conclusions are drawn regarding the usefulness of model checking at an abstract level and the importance of selecting a tool appropriate to the nature of the problem.


Abstract
Increasingly, model checking is being applied to more abstract problem domains than the traditional protocol analysis.The extent to which such an approach is able to provide useful insight into the problem domain depends to a large degree on the nature of the tool used.This paper reports the results of a study using three di erent model checkers, which di er widely in their speci cation language, internal implementation, and facilities for specifying correctness properties.
An abstract model of an industrial distributed database application has been studied using the three tools.A detailed model of the application using each of the tools is presented, and the extent to which each tool allows us to investigate interesting properties of the problem domain is compared.Some conclusions are drawn regarding the usefulness of model checking at an abstract level and the importance of selecting a tool appropriate to the nature of the problem.

Introduction
Model checking is now widely accepted as a valuable tool for the analysis of many problems which can be abstracted by a nite state model.Most papers have either reported on the application of a speci c model checker to a speci c problem area, or have focussed on techniques for model checking ever larger problems.In contrast, in this paper we look at the application of three popular and widely used model checkers to the same abstract problem, and focus on their expressive power and usability rather than on problem size.
The problem considered is based on a real industrial application supplied by one of our industrial partners 4].A distributed data base system is used to serve customers such that each service (a payment) can only be provided to one particular customer, and only once.The system has the following components: A single Centre, where most of the data is held A n umberofO ces, where relevant parts of the data are held Many Customers.Each customer has a home o ce which will normally hold the data pertaining to that customer.A foreign o ce should be able to provide a customer with the same services as the home o ce, after consultation with the Centre.
Cust. 3 The basic model of this system was kept as simple as possible.It is shown in graphical form in Figure 1.This identi es the centre, two o ces and three customers.Data for Customers 1 and 2 is held at their respective home o ces 1 and 2. Customer 3 does not have a home o ce (in this model) their data is held by the centre.
All three customers will attempt to obtain payment from one or other of the o ces.Customer 1 will be successful when contacting o ce 1, but unsuccessful when contacting o ce 2 similarly for customer 2.An o ce which does not hold the appropriate data passes the request on to the centre.Customer 3 should be successful in obtaining payment regardless of which o ce they visit.Once a customer has been served, successfully or not, they are satis ed and do not request further service.
Formal models of this system were built using three di erent formalisms, as detailed below.In each case, various properties both of the basic model and a numb e r o f v ariants were investigated.The properties considered were as follows: 1.The correctness of the basic model: (a) All customers should eventually be served.(b) Customer 3 will always be served successfully.(c) No customer can visit two o ces at the same time.

The result of introducing an error cor-
responding to missing functionality|the model of the centre was modi ed so that it failed to reply to a query if it had no corresponding customer record in its database.This leads to deadlock, and property 1(a) above no longer holds.3. A variant o f the basic model was produced by changing the behavior of the customers so that they would repeatedly try to obtain service until they were successful.This can lead to livelock/starvation, and properties 1(a) and 1(b) no longer hold.4. A re nement of the model in 3 above c a n b e constructed by replacing the synchronous interactions between customer and o ce and between o ce and centre with queues.Although not free of livelock (a customer can repeatedly go to the wrong o ce indenitely), provided the queues are long enough it is free of starvation, and properties 1(a) and 1(b) hold.
(Note: In the Mur model, as in this discussion, the customers are numbered from 1 to 3 however in the other two models they are numbered from 0 to 2).

The Tools
The three model checkers used were Spin 7, 1 2 ], Mur 2, 8] and FDR2 9, 3].All three were used on a Pentium processor running Linux and X-Windows.

Spin
Spin is a model checker originally designed for validating communications protocols 6].Its modelling language, Promela, supports dynamic creation of concurrent processes and both synchronous and asynchronous message passing, as well as shared variables.Facilities for data abstraction are very limited, only bits, bytes, ints and arrays being provided.The syntax of Promela is loosely based on that of C, with nondeterministic choice and iteration constructs similar to those in Dijkstra's guarded command language 1].
Spin allows speci cation of correctness properties in a number of di erent forms.Assertions may b e embedded in the model, and properties may be speci ed by means of a linear-time temporal logic (LTL) formula.In addition Spin will check for invalid end states and acceptance or non-progress cycles, with or without a weak fairness assumption.
Spin can perform random or interactive simulations, and can perform an exhaustive state space veri cation of both safety and liveness properties.Xspin is a user-friendly graphical front-end to Spin, which greatly enhances its usability.This provides a window-based, menudriven interface to Spin and allows animation of simulations and error traces in a variety of di erent w ays.

Mur
Mur is another model checker, also originally designed for communications protocols.A Mur description consists of a set of transition rules comprising a condition and an action the execution model involves repeatedly making a nondeterministic selection of a rule whose guard is enabled and executing the corresponding action.All interactions take place by means of shared variables.Its syntax is based on that of Pascal, and it shares Pascal's fairly good (if rather lowlevel) data structuring facilities.
Mur can only check safety properties, as there is no way of specifying liveness properties.Assertions may be associated with speci c rules or de ned as invariants.Invalid end states are also detected.
Mur can perform simulation or exhaustive veri cation, although only deadlock and assertion violations are detected.Error reporting is by means of a textual trace of rules red which must be interpreted by hand.

FDR2
FDR2 is a model checker for CSP 5].Its modelling language is a machine-readable form of CSP 11] augmented by a rich set of data abstraction facilities.Sets, sequences, tuples and compound types are all provided, together with a p o werful pattern-matching facility.
FDR2 is unusual in that its method of establishing whether a property holds of a given system is based on the notion of re nement: a specication which captures the property o f i n terest is written, in the same notation as is used to de ne the system, and the tool checks that the system is a re nement (in the CSP sense 10]) of the speci cation.It is also possible to check a system for determinism and for possible deadlock o r livelock.No conventional simulation facilities are provided.
FDR2 is a window-based program which a l l o ws the user to select a process (or pair of processes in the case of the re nement c heck) and a check to be performed.If the check fails, there is a sophisticated debugger which allows the system structure to be navigated in a hierarchical manner and the behavior which led to the error to be observed.

The Spin Model
Below is the basic model written in Promela, the modelling language used by Spin.Firstly the database is modelled as a pair of arrays indexed by customer number -a boolean to record whether or not that customer's information is present, and an integer specifying the amount due.The variables pmc, pm and ac represent t h e centre's database, the o ces' databases and the customers' personal accounts respectively.All are initialized to zero the interesting values are set up by the init process (below).The centre is modelled as a server process Centre, which sits in an in nite loop waiting for a request on its input channel s.This loop is marked with the label end to indicate that it is a v alid end state i.e. that it is not an error for the process to remain blocked at this point indefinitely.
The request (from one of the o ces) consists of a customer number b and a reply channel r.If information about customer b is present i n t h e centre's database, the amount due to that customer is sent o n t h e reply channel and the customer is removed from the database.If the customer is not known to the centre the value zero is returned.

The O ces
Each o ce is modelled by a server process (Office below) in the same way as the centre.An individual o ce is distinguished by its identi er n, and uses the channels r and s to receive requests from customers and to send queries to the centre, respectively.The request consists of a customer numberb and a reply channel c.If the o ce knows about customer b, it sends the specied amount on the reply channel and removes the customer from its database otherwise it queries the centre, and passes on the reply from the centre to the customer, using an unbu ered channel t to communicate with the centre.
proctype Office (byte n chan s, r) { byte b, x chan c chan t = 0] of {byte} end: do :: r?b,c if

The Customers
Each customer is modelled by a process (Customer below) whose parameters are an identi cation numbern and two c hannels to allow i t to communicate with either of the two o ces.The customer makes a nondeterministic choice between the o ces, engages in a transaction with the chosen o ce by sending its customer number n and the name of a reply channel c on the appropriate channel, and waits for a reply.The amount received is stored in the customer's account ( s i mply to make it globally visible during simulation), and the customer process then terminates.

Initialization of the processes
The init process is the rst one to be run, and sets up the model.First it initializes the databases so that o ce 0 knows about customer 0, o ce 1 knows about customer 1, and the centre knows about customer 2. It then starts six concurrent processes representing the centre, two o ces and three customers in the con guration shown in Figure 1.Using Xspin makes Spin very easy to use.The Promela source code is displayed in a window, and can be edited, checked and debugged in a nicely integrated way.The various options and facilities are selected by means of menus and check b o xes.The actual analysis is done by i nvoking Spin (a command-line program) as a background process, which analyzes the Promela description and generates a C program.This is then compiled and executed to perform the state-space search.If an error is found an error trace is produced, which can then be used to guide a simulation to reconstruct the execution sequence that led to the error.All this is handled by Xspin, and the user need not be aware of the detailed operation of the tool.
The model was investigated as described in Section 2 above the results were as follows:

The Mur model
Below is the basic model in Mur .The description consists of a series of constant, type and variable declarations, followed by a set of transition rules which de ne the behavior of the system.Each rule is a guarded command, consisting of a condition (a Boolean expression on the global variables) and an action.The execution model is to repeatedly evaluate the conditions, choose (nondeterministically) one rule whose condition evaluates to true, and execute the corresponding action.The execution of the actions is atomic | all interleaving of executions is speci ed explicitly by breaking a process down into a set of rules.
The rst section declares some constants, types and variables.The database is modelled as an array of records, where each record has two e l d s : a boolean to record whether or not that customer's information is present, and an integer specifying the amount due.The variables pmc and pm represent the centre's database and the o ces' databases respectively.CustStat, OffStat and CentreStat are state variables for the customers, o ces and centre respectively.The remaining variables are used to model the interactions between customers and o ces, and between the ofces and the centre.

The O ces
Each o ce is modelled by a set of four rules and a four-valued state variable: status is either Free, Serving (engaged in a transaction with a customer), Querying (the centre), or Awaiting (a reply from the centre).The ruleset construction replicates the set of rules for each possible value of the parameter off (in this case 1 or 2).

The Customers
Each customer is modelled by a pair of rules and a three-valued state variable: CustStat c] is either Pennyless (the initial value), Waiting (engaged in a transaction with an o ce) or Done (having completed a transaction).Once a customer's status becomes Done it can no longer engage in any transactions (i.e. it e ectively terminates).Mur assumes that system execution is in nite it does not distinguish between valid and invalid end states.It therefore always reports an error in a client/server model where all the clients have terminated, and is unable to di erentiate this case from a genuine deadlock.To circumvent t h i s problem, we introduce an additional rule which detects the valid end state, and avoids deadlock by repeatedly changing the value of a dummy variable.(Mur 's de nition of deadlock is that the current state has no successor other than itself.)This allows genuine deadlocks to be detected and reported correctly.We also use this rule to introduce an assertion regarding a property of the system -customer 3, whose details are held at the centre, will successfully complete his transaction regardless of which o ce he goes to.

Correctness properties
In addition to associating assertions with particular rules, it is possible to specify properties which are globally true, i.e. invariants.As an example, we include a speci cation that no customer can be at more than one o ce at the same time.Mur is a text-only, command-line program, but straightforward to use.Running Mur on the le containing the system description causes a C++ program to be generated, which then has to be compiled and run in order to do the analysis.The command line arguments given to this program determine whether a simulation or an exhaustive state-space search is performed, and control the amount of debugging information produced.
Running the Mur veri er on the above model allows us to establish the safety properties, as follows: 1. (a) All customers should eventually be served: this is established by the fact that the valid end state is always reached.(b) Customer 3 will always be served successfully: this is established by the assertion in the end state.(c) No customer can visit two o ces at the same time: this is established by the invariant speci ed as part of the model.
2. Missing functionality in the Centre.If we i ntroduce the missing centre functionality e rror, an error trace is produced, allowing the user to determine (albeit with some diculty) where the error occurred.The trace consists of the sequence of rules that were red to reach the state in which the error was detected.For each rule, the values of the state variables that changed as a consequence of executing that rule are printed, except for the initial and nal states, for which the values of all the state variables are printed.By working backwards through the trace the sequence of events which led to this situation can be determined, and hence the nature of the error can be deduced.Although this method of debugging is perfectly feasible, it is di cult and time-consuming, especially in more complex cases.
As Mur can only determine safety properties and has no way of reasoning about liveness, nothing was learned from investigating the variants of the basic model described in Section 2.

The FDR2 model
Below is the basic model written in the machinereadable form of CSP used by FDR2.The database is modelled as a set of pairs mapping customers to amounts.The sets pmc, pm0 and pm1 represent the databases of the centre and the two o ces respectively.Interaction between customers, o ces and the centre is modelled by CSP compound events, declared in FDR2 as channels of a particular type.The auxiliary function amount(r) extracts the amount from a set containing a (customer, amount) pair using FDR2's pattern matching facility.The centre is modelled as a process which w aits for an event on the channel centrein, corresponding to an enquiry from o ce off concerning customer cust.It then extracts the record corresponding to cust from its database if such a record is present it noti es the corresponding amount t o the enquiring o ce and removes the corresponding record from its database otherwise it sends zero.

The O ces
Each o ce waits for an event on the appropriate tooffice channel, corresponding to a request from a customer.If a record for this customer is present in its database, it sends the corresponding amount to the customer and removes the record from the database otherwise it queries the centre, waits for a reply on the appropriate centreout channel, and passes the reply on to the customer.8. 4 The System and the Speci cation The system model is a parallel composition of centre, o ces and customers.Two OFFICEs in parallel interact with the CENTRE via the channels centrein and centreout This subsystem in turn interacts with the CUSTOMER processes via tooffice and getpayment.The internal communications on centrein, centreout, tooffice and getpayment are hidden.The two speci cations SPEC and SPEC2 encapsulate behaviors that we w ant to check t h a t SYSTEM exhibits.The rst speci es that each customer will eventually receive a response from the o ce, while the second adds the additional constraint that customer 2 will always be successful in obtaining payment.Running FDR2 is fairly straightforward, although it takes longer to learn to use it e ectively than for Xspin.Although the CSP source code is not displayed by FDR2, it is possible to invoke an editor from within FDR2 which can be used to make c hanges, and then to reload the changed le.As properties of the system are speci ed in FDR2 by writing a CSP speci cation that captures the property in question, this has to be done quite frequently.The debugging facilities in the event of an error being discovered are powerful but not as intuitive and easy to use as those of Xspin.
The model was investigated as described in Section 2 above the results were as follows:

Conclusions
Applying the three tools to the same, fairly simple, problem has produced some interesting results.The three models described in detail above are quite di erent from one another, and each required di erent skills and thought processes during development.This was even more pronounced when variants of the basic model were constructed.Similarly, e ective use of the tools entailed a di erent approach and philosophy in each case.Writing the descriptions in both CSP and Promela was natural and fairly straightforward, because of the nature of the distributed database/ client-server problem.In contrast the Mur description was much longer and more difcult to write, because the nature of the problem does not map well onto the modelling paradigm provided by Mur .The Mur model is far removed from the user's mental model of the problem it is necessary to make explicit the points at which non-deterministic choice and interactions between processes occur, and to provide explicit handshaking for each s y n c hronous interaction.As a result, every sort of structure in the mental model has to be unraveled in order to express it in Mur , and reported errors have rst to be tracked down in terms of the model, then related back to the original problem.This is very much a characteristic of the particular application studied it is likely that for problems which are naturally described by a set of rules, Mur would be natural and appropriate.
Comparing CSP and Promela, CSP is more elegant and has a much more rigorously de ned semantics, and the data abstraction facilities provided by FDR2 are very much better than those of Promela.However Promela is richer and strictly more expressive (e.g.asynchronous communication is supported, and channels are rst class objects in Promela but not in CSP).Furthermore, Promela's C-like s y n tax makes it more accessible to non-experts.
None of the three tools was particularly dicult to use.Both Spin and FDR2 bene t greatly from having a graphical front end, as the facilities they provide are complex.Mur is much simpler, so the textual command-line-based approach is quite adequate.The fact remains though that debugging using the Mur error trace is quite hard work.Xspin is probably the easiest to use, being both powerful and intuitive.FDR2 is initially rather harder to understand, but once the learning curve has been overcome it provides an equally powerful and productive debugging environment.
As previously discussed, Mur is signi cantly less powerful than the other two tools as it cannot deal with liveness properties.The contrast between the re nement approach to specifying system properties of FDR2 and the more traditional temporal logic approach of Spin is particularly interesting.It is possible using temporal logic formulae to specify concisely many properties which cannot easily (or in some cases at all) be expressed as a re nement relationship.On the other hand, using re nement allows a number of properties to be captured and checked simultaneously and more importantly, it allows the stepwise construction of more and more detailed models while ensuring that essential properties are preserved.
The overall conclusion is that any model checker can be of value in investigating an abstract model of a problem but selecting one which is appropriate to the nature of the problem and the properties of interest can greatly increase its e ectiveness.

Figure 1 :
Figure 1: Graphical representation of the model, showing the centre, two o ces and three customers.