Extending the Test Template Framework

The Test Template Framework (TTF) is a formal, abstract model of testing, used to derive a hierarchy of test information, including test inputs and outputs, from a model-based formal specification. In this paper we propose two modifications to the framework: first, that testing information derived from state components be factored out from operation-specific information, and second, that focusing on the input space of an operation under test be deferred for as long as possible. The first modification facilitates reuse of derived information in the context of abstract data types (ADTs) and classes to minimise duplication. The second modification supports testing strategies based on operation outputs, and provides a basis for further extensions for testing derived from behaviour-based notations.


Introduction
The Test Template Framework (TTF) is a formal, abstract model of testing, used to derive a hierarchy of test information, including test inputs and outputs, from a modelbased formal speci cation.The TTF has been presented by way of case studies deriving testing information from formal speci cations, including: the text formatter; the UNIX le Read operation; a symbol table Update operation; the dependency management system CanAdd operation; the triangle problem speci cation written in a transformational style, i.e., without persistent state; and a topological sort speci cation, also written in a transformational style 2, 14, 23,22,24].
The TTF provides a systematic way to derive and record test cases from a model-based speci cation for individual operations.The original work focused on speci cations written in Z 20, 26] and recent work has extended the framework to speci cations written in Object- Z 17, 18].The existing TTF is applied to operations rather than the entire state space of the speci cation, and de nes testing information in terms of the input spaces of the operations.
In this paper we propose two modi cations to the framework: factoring out testing information derived from state components and deferring focusing on the input space of an operation.The rst modi cation facilities reuse of derived testing information.The second modi cation supports use of testing strategies considering operation inputs and outputs.
By convention, in Z 20, 26] a state schema describes the state space of a system, and operation schemas describe changes to the state space.The variables of an operation are either incorporated by inclusion of a state schema or directly declared in the operation.We refer to these categories of variables as state and operation variables respectively.By convention, state variables are further classi ed as before-state (unprimed) and after-state (primed) variables, and operation variables are further classi ed into inputs (distinguished by trailing `?') and outputs (distinguished by trailing `!').
In the TTF, testing information is derived by constraining the variables of an operation schema.Previous case studies have not classi ed the variables of an operation either because there was no state schema (transformational speci cations) or because testing information was only derived for a single operation.The potential for factoring testing information derived from state components as a means to minimise duplication has therefore not been apparent.In this paper we propose separate derivation of testing information for the state variables to facilitate testing multiple operations with state variables in common.We use the Z schema calculus to achieve this for Z speci cations.The starting point in using the TTF for a particular operation is to determine the valid inputs to the operation which, in Z, is the precondition of the operation.Previous studies have used output and after-state variables for simpli cations and for testing strategies such as cause-e ect analysis, even though these variables are not accessible.Also, to extend the TTF to encompass behaviour-based notations, we must consider sequences of operation invocations with the output of one invocation establishing the input of another.
In this paper we propose deferring the focus on input and before-state variables for as long as possible.
We present our modi cations by revisiting the dependency management system (DMS) case study 22].The next section contains a Z speci cation for relevant sections of the DMS.Section 3 outlines the derivation of testing information using the unmodi ed TTF while Section 4 presents the corresponding derivation using our modi cations.Section 5 discusses related work and Section 6 concludes.

DMS speci cation
The dependency management system (DMS) case study was a testbed for a project exploring software development methodologies 13].The DMS is a critical component of a theorem proving tool, tracking dependencies between theorems and assertions in a proof, thus preventing circular reasoning.However, the concept of dependency management generalises to other problems (such as revision control systems), so the DMS is a reasonably generic component.
The DMS tracks dependencies between nodes.The basic DMS maintains a set of nodes, direct dependencies between nodes, and inferred transitive dependencies between nodes.
The DMS provides operations for manipulating a dependency graph, from adding nodes and dependencies to calculating all the nodes dependent on some other node.In this paper we consider only features associated with the set of nodes to simplify our examples.We take as given the set of nodes (X ) In the next section we derive testing information for Nodes using the unmodi ed TTF and in Section 4 we present the corresponding derivation using our modi ed TTF.The material in the next section is not new but provides necessary background for our modications.

TTF testing
The Test Template Framework (TTF) 14,22,24] is used to derive a hierarchy of test information from a formal, model-based speci cation.Abstract test data can be instantiated as test inputs and test oracles can be generated.
generic parameter as a given set for typographic clarity.
3rd Northern Formal Methods Workshop, 1998 The TTF formally de nes test templates as the nodes of the hierarchy.The relationships of the test templates to each other and to the system speci cation are also de ned.A variety of functional testing strategies can be used to derive the templates; using multiple strategies is advocated.Test templates are used to derive instance templates and oracle templates.Instance templates specify a unique choice of test data de ning a speci c test input.An oracle is a means of determining whether or not a test is successful.Oracle templates are derived for test templates using the input-output relationship of the formal speci cation.In Section 3.1 we de ne the TTF structures required for our case study, in Section 3.2 we derive test templates, in Sections 3.3 and 3.4 we de ne instance and oracle templates, and in Section 3.5 we identify problems with the existing approach.

De nitions
The input space (IS) of an operation is the space from which input can be drawn.The input spaces for the Nodes operations are given by Z schemas with signatures restricted to the before-state variables (xs) and the input variables (x?), with no predicate part2;3 .
Input to an operation may be type-compatible, i.e., fall within the input space, and still be invalid.The valid input space (VIS) is the subset of the input space for which an operation is de ned, and is equivalent to the precondition 4  The output space (OS) and valid output space (VOS) of an operation are de ned over the output signature of the operation similarly to the de nitions of the IS and VIS.The output space is used in oracle derivation.The basic unit for de ning data in the framework is a test template (TT) which is expressed as a Z schema.A test template is a formally constrained data space.Test templates are generic, abstract, instantiable, and derivable from a formal speci cation.
A test template for an operation is a subset of the valid input space for the operation.We de ne Z types for the test templates of the operations.
Test templates are derived from some other test template using a testing strategy.We introduce the set of all testing strategies without further speci cation.

Strategy]
Test templates are organised into a test template hierarchy (TTH).The root of the hierarchy for an operation is the valid input space of that operation.The test template hierarchy for an operation is a digraph with test templates as nodes and testing strategies as arcs.The hierarchy is usually a tree and each derived test template is typically a subset of its parent in the hierarchy.

Test templates
We derive testing information for the operations using two testing strategies: type-based selection and cause-e ect partitioning.A common, intuitive testing strategy is 0-1-many, based on the cardinality of a container type.We refer to this as (one example of) type-based selection.In contrast, cause-e ect analysis involves partitioning the input space based on equivalence classes of the output space.We identify type-based selection (TB) and cause-e ect analysis (CE) as particular testing strategies.
TB; CE : Strategy 3rd Northern Formal Methods Workshop, 1998 For operation NoNodes, we apply cause-e ect analysis to derive two test templates and we establish their places in the hierarchy.The equivalence classes are based on the two possible values of the output variable result!reporting whether xs is empty or not.We distinguish the new test templates by sequentially numbering the subscript.The parent of the test template being derived from (here VIS NN ) is incorporated by schema inclusion and the new constraint is included as the predicate part of the test template schema.3rd Northern Formal Methods Workshop, 1998 We derive three test templates from RemoveNode using type-based selection over xs 0 .Note that we are exploring values of xs in the output space (0-1-many) and translating these to the input space (1-2-many).
TTH RN (VIS RN ; TB) = fTT RN:1 ; TT RN:2 ; TT RN:3 g We choose to stop the derivations at this point, a decision that is left to the individual human tester under the TTF. Figure 1

Instances
We de ne instance templates for the leaf test templates of the test template hierarchy.
The instance templates specify a choice of test data on the assumption that each element of a given test template is equivalent for testing purposes.We de ne an individual member of X to use in instance templates.

Discussion
In this paper we propose two modi cations to the framework: factoring out testing information derived from state components and deferring focusing on the input space of an operation.The rst modi cation facilities reuse of derived testing information.
The second modi cation supports use of testing strategies considering operation inputs and outputs.We applied type-based selection over the value of xs to each of the operations of Nodes; four times in the input space of an operation (twice for IsNode) and once, for RemoveNode, in the output space.Deriving these templates once and reusing them will reduce the derivation e ort, particularly if simpli cations or contradictions arise due to the state invariant (not the case here).
The existing test template approach has an early focus on the input space of the operation under test, resulting in problems for strategy application and simpli cation.One of the rst steps in using the test template framework is to calculate the input space and the valid input space.The valid input space is then used as the root of the test template hierarchy.As a result, testing strategies that require the output space, such as cause-e ect analysis, cannot be easily formalised since the information required for their application is not immediately available.For example, the application of cause-e ect analysis for NoNodes and IsNode uses the value of result!which is not available in VIS NN and VIS IN .Similarly, testing strategies that could make use of the output space, such as the application of type-based selection for RemoveNodes, do not have the relevant variables available (in this case xs 0 ).Also, simpli cation of testing information typically uses operation results in addition to precondition constraints.Also, the initial step of calculating the valid input space involves existentially quantifying output and after-state variables over the predicate of the operation under test.Simpli cation of the quanti ed predicate may not be straightforward, possibly resulting in a complex valid input space.The existing approach also results in problems for oracles.Oracle templates, which are constrained subsets of the output space, are calculated by reversing the initial focus on 7 Schema projection S T hides all components of S except those that are shared by T.
3rd Northern Formal Methods Workshop, 1998 the input space (by conjoining the operation with the test or instance template).It has proved di cult to formalise oracle templates due to limitations on Z type rules, principally because schemas are not well integrated in the Z notation, as noted in 25].In `ZRM' Z 20], schema expressions are distinguished from other expressions and our problems arise in mixing the two.We have not investigated using the Z Standard 26] to deal with these problems.In this section we derived testing information for the Nodes operations using the existing TTF approach.In the next section we demonstrate our modi ed approach by deriving testing information for the same operations.

Modi ed TTF testing
We propose to modify the TTF derivation approach by factoring out state components to facilitate their reuse in other operations and by delaying the focus on input and beforestate variables for as long as possible.We achieve the rst aim by deriving hierarchies for the state components of operations.We achieve the second aim by making the operation the root of the test template hierarchy, rather than its precondition.In Section 4.1 we derive testing information for the state components and in Section 4.2 we derive testing information for the operations using our modi ed approach.

State
In this section we derive hierarchies for the state components of operations.The testing information contained in such hierarchies is factored out to facilitate sharing between the operations.We use the Z schema calculus to combine the shared testing information with operation-speci c tests.In conventional Z, the state space is represented by a schema.Operation schemas specify relationship between input and output variables and a pair of states, the before-and after-states.The name of the state schema is used to represent the before-state, and a prime ( 0 ) is added to represent the after-state.A further convention is used to construct pairs of states for operations.A schema name with prepended is implicitly de ned as the unprimed and primed schemas with no additional constraints.This signi es making all components available for change.A schema name with prepended is implicitly de ned as the unprimed and primed schemas with each unprimed component equated to the corresponding primed component.This signi es make all components available for reference.We are only interested in deriving templates in either the input space or the output space in any particular derivation step so we restrict our attention to the unprimed state schema and signify the output space by primed templates.We derive a test template hierarchy for Nodes (which as an after-state would be Nodes 0 ).The modi ed approach considers the entire state space of a component rather than focusing on the input (output) and valid input (output) spaces.The state space (SS) of a component is the declaration part (signature) of the state schema.

SS Nodes b = xs : X ]
3rd Northern Formal Methods Workshop, 1998 In the modi ed TTF approach, we de ne a type for test templates as a subset of the component or operation under test, in contrast to the original TTF approach of de ning a type for test templates as a subset of the valid input space of an operation.We use the name NewTT to distinguish templates de ned over the entire state space from TT s which are de ned over the input space 8 .
NewTT Nodes == Nodes NewTTH Nodes : NewTT Nodes Strategy NewTT Nodes We derive three test templates for Nodes using type-based selection over xs.These can be transferred to the output space by referring to the primed template name (e.g., NewTT 0 Nodes:1 ).NewTT Nodes: NewTT Nodes:3 b = Nodes j #xs > 1] NewTTH Nodes (Nodes; TB) = fNewTT Nodes:1 ; NewTT Nodes:2 ; NewTT Nodes:3 g It is technically possible to derive instance and oracle templates from NewTT s derived over the state schema.This is not particularly useful because these NewTT s will be combined with operation-speci c information which will impose additional constraints on instance and oracle templates.

De nitions
In the original TTF approach, a type for test templates is de ned as a subset of the valid input space.In the modi ed approach, we de ne types for test templates based on the relation denoted by the operation rather than the valid input space, and we de ne corresponding test template hierarchies.NewTT  For operation NoNodes, we apply cause-e ect analysis to derive two test templates and we establish their places in the hierarchy.The e ect, which is the source of the test, is explicit in the template, and the cause can be derived.NewTTH RN (RemoveNode; TB) = fNewTT RN:1 ; NewTT RN:2 ; NewTT RN:3 g The graphical representation of the test template hierarchies derived in this section is identical to Figure 1, i.e., we derive the same templates by di erent means.

Rei ed templates
The usual step at this point is to de ne instance templates that specify a unique choice of test data de ning a speci c test input.We have deferred the focus on input so we de ne rei ed templates (RT) that specify a unique choice of test data de ning both test input and output.

Inputs and outputs
The step from rei ed test data to speci c test inputs and outputs is straightforward.We de ne rei ed input and output templates (RIT and ROT respectively ) using the Z precondition operation (pre), and a corresponding postcondition operation (post) 10   10 We use a symmetric de nition of post, i.e., remove input and before-state variables from the schema signature and existentially quantify them over the schema predicate.

Discussion
We have proposed modifying the TTF by factoring state-speci c testing information and deferring focusing on the input space.The rst modi cation is achieved by deriving testing information from the state schema and incorporating the resulting templates by name rather than as predicates included on an operation by operation basis.The second modi cation is achieved by deriving testing information from the operation schema rather than its precondition, deferring the move to the input space until concrete test data has been derived.Although these modi achieve our aims of facilitating reuse and supporting testing strategies that use operation outputs, there is some cost.Previous case studies have observed that tool support is essential to manage bookkeeping tasks associated with any but the most trivial TTF derivations.Reusing state-based templates by name will permit more complex strategy applications but will strengthen the requirement for tool support to enable expansion of included schemas.Also, in a sense we have exchanged reuse of template derivation for reuse of precondition simpli cation.In the existing TTF, the precondition is calculated and simpli ed once for each operation.In the modi ed TTF, the precondition is calculated for each rei ed template.Storage and application of simpli cation steps as tactics (in the theorem prover sense) is an obvious area for tool support.There is a problem with our second modi cation that may a ect other approaches that support testing based on outputs.The problem can arise when tests are based on outputs in the context of non-determinism.Consider an abstract non-deterministic square root operation with tests de ned over the output space.Performing the test corresponding to RT Asqrt:2 on such an implementation will return a false failure.We are still considering solutions to this problem.Hall and Hierons 7,8] also describe the generation of a nite state machine from a Z speci cation.The input domain is partitioned using the category-partition method of Ostrand and Balcer 19].The Test Template Framework, in contrast, permits use of a variety of testing strategies, including category-partition.The states for the nite state machine are calculated by rewriting the speci cation as disjoined input and output predicates (e ectively preconditions and postconditions).The transitions are determined by pairwise consideration of states as operation pre-and postconditions.The e ect is similar to the machine constructed by Dick et al.Hierons also notes previous work on test control and test sequencing from nite state machines but does not pursue these aspects.Donat 5] also converts speci cations to predicate logic but uses conjoined implications of the form (S 1 ) R 1 ) ^: : : where S and R are stimulus and response expressions respectively.A stimulus is a predicate that refers only to input and before-state variables, and a response is a predicate containing at least one reference to an output or after-state variable.The modi ed TTF input-output relationship is disjoined conjunctions as used by Dick et al. and Hall and Hierons.
Stepney 21] describes a method for formally specifying testing information based on applying abstraction to a speci cation.The method uses the ZEST object-oriented variant of Z 3], and involves systematically introducing non-determinism into the speci cation.The method includes three techniques: conjunctive partitioning, simpli cation by hypothesis introduction, and weakening of after-state constraints.A ring condition interpretation of precondition is used, the method is supported by the Zylva tool, and testing information is structured as an inheritance tree of speci cations.The inheritance tree corresponds to the TTF test template hierarchy but the nodes are related by abstraction rather than strategy application.Input and output spaces are not treated symmetrically due to di ering abstraction (re nement) relationships.Stepney's method is more more closely aligned with formal development than the TTF, but the TTF is more rmly grounded in software testing.In related work of our own, we are extending the TTF to accommodate speci cations written in Object-Z 6].In 18], we adapt the TTF to object-oriented formal speci cations, deriving testing information for operations (test inputs and expected outputs) from Object-Z speci cations, and inheriting testing information from parent classes.In 17], we focus on testing the class as a whole, achieved by deriving a class' nite state machine from its speci cation, which allows us to formulate sequences of the class' operations to conduct testing.In 15], we present a case study demonstrating a complete process for object-oriented speci cation-based testing, including integration with the Classbench test execution framework 9, 10].The latter studies suggested some of the modi cations presented in this paper.6

Conclusion
In this paper we proposed two modi cations to the Test Template Framework: rst, that testing information derived from state components be distinguished from operationspeci c information, and second, that focusing on the input space of an operation under test be deferred for as long as possible.The rst modi cation facilitates reuse of derived testing information and the second supports testing strategies that use operation outputs.As mentioned above, we are extending the TTF to accommodate speci cations written in Object-Z.We are also extending the TTF to accommodate multiparadigm speci cations of interactive systems, involving model-based and behavioural sub-speci cations.The symmetrical treatment of input and output spaces proposed in this paper is an important step in that work, enabling consideration of sequences of operation invocations with the output of one invocation establishing the input of another.

Figure 1 :
Figure 1: Graphical representation of TTHs x : X For example, the fully expanded form of TT IN:1:2 is TT IN:1:2 b = xs : X ; x? : X j x? 2 xs ^#xs = 1] A possible instance template derived from TT IN:1:2 is IT IN:1:2 b = xs : X ; x? : X j xs = fxg ^x? = x] 3rd Northern Formal Methods Workshop, 1998 3.4 Oracles An oracle is some means of determining the result of a test.An oracle template is calculated by conjoining a test or instance template with the operation under test and then projecting the result onto the output space (OS) of the operation 7 .The general expression for an oracle template for a test template TT of operation Op is OT TT b = (Op ^TT) OS Op The output and valid output spaces of IsNode are OS IN b = xs 0 : X ; result!: B ] VOS IN b = xs 0 : X ; result!: B ] and the oracle template of instance template IT IN:1:2 is OT IN:1:2 b = xs 0 : X ; result!: B j xs 0 = fxg ^result!= True]

=
Asqrt j o! < 0] NewTTH Asqrt (Asqrt; TB) = fNewTT Asqrt:1 ; NewTT Asqrt:2 gWe could reasonably de ne symmetrical rei ed templates from these tests with the obvious input and output templates.RT Asqrt:1 b = i? : ; o! : j i? = 1 ^o! = 1] RT Asqrt:2 b = i? : ; o! : j i? = 1 ^o! = ?1]However,an implementation of Asqrt can reduce the non-determinism in the output and could, for example, perform as follows (note that o! is of type rather than ).Csqrt b = i? : ; o! : j i? = o! o!] FalseOperations NoNodes and IsNode can query, but not change, the state (denoted by inclusion of Nodes).NoNodes returns true if and only if xs is empty.IsNode returns true if and only if the input node x? is a member of xs.
1and de ne the state as a nite set of nodes, B ::= True j Operations AddNode and RemoveNode can change the state (denoted by inclusion of Nodes).Node x? is added to xs by AddNode.AddNode is unde ned if x? is already in xs, since, in Z, an operation gives an unde ned result (interpreted as a is a graphical representation of the test template hierarchies derived in this section. 5rd Northern Formal Methods Workshop, 19985Related workThe modi ed TTF presented in this paper has much in common with the approaches of Dick et al., Hall and Hierons, and Stepney.Also, H orcher et al. extend the work of Dick et al. to Z, and Donat derives testing information using an input-output relationship based on implication rather than conjunction.Dick et al. 1, 4] present techniques for using a model-based speci cation (written in VDM) to generate test cases and sequence tests.They derive test cases, termed suboperations, by reducing operations to a canonical disjunctive normal form (DNF). Test sequencing is achieved by constructing a nite state automaton, using the sub-operations as transitions and the disjoined sub-operation before-and after-states (input and output spaces) as nodes.The modi ed TTF similarly provides a symmetrical treatment of input and output spaces, but permits use of a variety of testing strategies, including DNF partitioning.H orcher et al. 12, 11, 16] use DNF partitioning to derive test inputs from Z speci cations and have developed a Z predicate compiler to automate test result evaluation.