Formal Specification of Interconnection Networks

Interconnection networks are an important and well-studied topic in parallel computing and architecture, but a homogeneous and general method for defining and classifying the topologies and behaviors of interconnection networks is lacking. Topologies are usually specified informally by picture or more formally by permutations of wire enumerations. This paper presents an improved method for specifying multistage networks via permutations, along with two styles of formal functional specification of the entire network, using both a standardmultistage organization and a generalized fat tree organization. This method is applied to two specific indirect multistage switch networks: the baseline and the butterfly. The functional specification emphasizes the similarities between the networks, and also captures the functionality provided by general-purpose network nodes.


Introduction
Indirect interconnection networks play an important role in the design of general purpose parallel architectures.A network can be seen as a black box realizing either communications between the processors and the memory (for shared memory systems) or interprocessor communications (for distributed systems).The design depends on performance criteria like latency, bandwidth, connectivity and functionality.A particularly important class is the multistage switch network comprising a number of stages, each consisting of primitive switching elements.Many such networks have a regular structure with the same number of switches in each stage, and where all switches have identical designs; examples include the omega, banyan, baseline and butterfly networks [1].The input and output ports of consecutive stages are connected in order to construct the entire network.Many such networks have been shown to be topologically equivalent [9,8].Research issues include network structures, the relationships between different networks, and network communication algorithms [4,6].
Although the field of interconnection networks is well studied, a concise and coherent way of defining them is lacking.In the case of multistage networks with regular stages and constant degree of wires, the stages are often defined by permutations of the enumerations of the wires, where the enumeration is usually based on bit manipulation of binary numbers.This approach may hide the relationship between networks, and it considers only network topology-it does not take account of the general processing capabilities of the switching nodes.
In this paper we study a systematic and coherent method addressing these two problems, and apply it to the baseline and butterfly networks.In common with the standard approach, we also use enumerations of the nodes to specify wires, but (unlike the standard method) we subdivide each stage into disjoint sets of nodes which are enumerated independently.This has three advantages: it simplifies the stage-combining functions, it produces a recursive network decomposition that clarifies the relationship between multistage networks and tree-structured networks (the tree and the fat-tree [5]), and it leads directly to a short and clear functional definition of the stage-combining functions, now seen as functions on lists.
Much past research on networks has focused exclusively on topology and message routing.However, it is also interesting to consider networks where the basic switch elements can compute an arbitrary function, rather than being restricted to simple routing.This provides additional functionality that has been exploited in real machines, e.g. in the emulation of shared memory machines with a butterfly network [7] where the fast access to the memory is realized by multiprefix operations [3], and in the NYU Ultracomputer [2].Our functional specification provides a good foundation for studying this richer class of network.In the standard multistage network specifications, only local wirings are specified directly; the global network is specified implicitly.In contrast, our functional specification is global, in that it defines a general communication/computation function describing the global behavior of the entire network as a complete set of communications takes place.The behavior of the network is a higher order function, and one of its arguments is a function defining the behavior of an individual node.This is more general than the usual approach, where it is assumed that nodes can only execute a fixed communication algorithm.

Formal Specification of Interconnection Networks
The functional specifications in this paper are expressed in Haskell.These are executable programs that formally express the connectivities of the networks and simulate their behaviors.
The remainder of the paper is organized as follows.Section 2 introduces terminology and gives the standard definitions of the baseline and banyan networks.Sections 3 and 4 then present these networks in more detail and give our new specifications for the intra-stage wiring.Section 5 defines the global network and completes the functional specifications.Section 6 compares our approach with the standard specifications and discusses the role of the functional specifications.

Multistage interconnection networks
This background section gives the standard semi-formal definitions of the baseline and butterfly networks.Alternative specifications will be developed in the following three sections.
A multistage interconnection network is an acyclic directed graph consisting of a number of subgraphs called stages.Each stage consists of a permutation graph (containing only wires) followed by a row of nodes, where each node takes two inputs and produces two outputs.The size of the network is characterized by the number of stages, which is called the depth d.The stages are enumerated 1; : : : d .The dimensionality is d 1.A network with d = 0 has no stages, and consists only of a wire connecting a singleton input to a singleton output.Figure 1 shows the smallest nontrivial network, where d = 1 , along with the corresponding tree.
The width of each stage is n = 2 d 1 nodes.Since each node has two inputs and two outputs, the entire network takes 2 d input signals and produces 2 d output signals; the inputs and outputs are organized as a list of n signal pairs.For example, the two networks in Figure 2 each have depth d = 4 , with stages 1; 2; 3; 4, each stage contains 2 d 1 = 8 nodes, and the entire network has 2 d = 1 6 input signals and output signals.
In Section 5 we will characterize the behavior of the network by two additional parameters: the interconnection function c and the node behavior function g.
The standard interconnection functions are defined by specifying explicitly which nodes are connected.(Section 5 will present a different approach.)To achieve this, each node is identified uniquely by its position within a stage (given as a binary number) and the number of that stage.More precisely, the node (b; s) is located in stage s 2 f 1 ; : : : ; d g and within the stage at position b 2 f 0 ; : : : ; n 1 g , represented as a binary number.For regular interconnection networks, the stage interconnection function C :: (Position; Stage) !(Position; Stage) is specified as a relation on nodes.The following definitions of C baseline and C butterfly for the baseline and the butterfly network are based on the binary enumeration (see also [4]).

Baseline network:
The stage connecting function C baseline of the baseline network is given by: C baseline (b; s) = ( b 0 ; s + 1) where b 0 is the string obtained by performing a left cyclic shift on the rightmost s bits of b , or b 0 is the string obtained by performing a left cyclic shift on the rightmost s bits of b and then changing the rightmost bit of b (see Figure 2).Both definitions depend explicitly on the number of stages and, especially in the case of the baseline network, the stage number is strongly involved in computing the value C baseline (b; s).This kind of function definition is not suitable as a starting point for a more abstract specification of interconnection function using recursion and lists of nodes (of one stage).

The baseline network
We now present an alternative specification of the baseline network which is a more appropriate starting point for the derivation of a global specification of the network in the functional style.This new specification is based on a subdivision of each stage into several independent substages, leading to much simpler interconnection functions.
Stage s is organized into n=2 s 1 substages, for s = 1 ; : : : ; d .The nodes within each substage are enumerated with natural numbers starting with 0, independently from the nodes in other substages.The substages of stage s can be enumerated 0; : : : ; 2 s 1 1 .
The division into substages emphasize that the connections between consecutive stages fall into groups of connections.Every substage (except in the top level) has connections to exactly one substage of the subsequent stage, and every substage (except those in the bottom level) has connections to exactly two substages of the level below.This results in a structure of substages organized like a binary tree (see Figure 3).
The enumeration leads to a very clear connection function which we call the local baseline connection.The local baseline connection function connects the outgoing edges of two substages X and Y of equal size with the ingoing edges of a third substage Z with double the size, using the indices of the enumeration.

Functional Programming, Glasgow 1995
Formal Specification of Interconnection Networks Definition 3.1 (Local baseline connection) Let X = ( X 0 ; : : : ; X n 1 ) and Y = ( Y 0 ; : : : ; Y n 1 ) be two stages with the same number of nodes n, and let Z = (Z 0 ; : : : ; Z 2 n 1 ) be a stage of nodes of size 2n.Each node of X , Y , and Z has degree two, i.e. two ingoing and two outgoing egdes.The nodes in each stage are numbered consecutively starting with 0.

The local baseline connectivity is a relation bas with exactly two function values for each argument. It is defined as
bas : f0; : : : ; n 1 g !f 0 ; : : : ; 2 n 1 g The edges connecting the substages X (or Y ) with Z are (X i ; Z bas(i) ) (or (Y i ; Z bas(i) ), i.e. outgoing edges of X i (or Y i ) are connected with ingoing edges of Z 2i and Z 2i+1 .
The connection between two substages X; Y and one subsequent substage Z is identical to the direct shuffle connection (in Definition 3.1 defined with natural numbers).Thus, the baseline network can be interpreted as a structured combination of shuffle functions.In contrast to the omega network, there are several shuffle functions on each level.

Messages
The edges (X i ; Z j ) between nodes X i and Z j represent wires for transmitting messages between the two nodes connected by an edge.This means X i (or Y i ) sends two messages, one to the node Z 2i and one to the node Z 2i+1 .In order to make messages explicit, we express the local baseline connectivity for pairs of messages (or wires) which are assigned to nodes as their pairs of input or output messages.Let i ) be the output of node X i ; i = 0 ; : : : ; n 1 y i = ( y 0 i ; y 1 i ) be the output of node Y i ; i = 0 ; : : : ; n 1 z k = ( z 0 k ; z 1 k ) be the input of node Z k ; k = 0 ; : : : ; 2 n 1 : The local baseline connectivity is fulfilled if i ; y 0 i ) for k = 2 i; i = 0 ; : : : ; n 1 ( x 1 i ; y 1 i ) for k = 2 i + 1 ; i = 0 ; : : : ; n 1 i.e. the input (z 0 k ; z 1 k ) of Z k is identical to a pair composed of messages coming from X and Y .Equation 2 follows immediately from Equation 1 by fixing an order for the pairs of messages.Because of the fixed order the specification is less general than the connectivity described in Definition 3.1.Message pairs of opposite order would also realize the more general baseline definition.
Functional programming provides a way to give a global network specification based on the list representation of nodes or message pairs, without relying on indices.The global description of the baseline connectivity uses the entire lists of input and output pairs, taking the lists of output signals xs = [ x 0 ; : : : ; x n 1 ] and ys = [ y 0 ; : : : ; y n 1 ] as arguments, and producing the input list of signal pairs zs = [ z 0 ; : : : ; z 2 n 1 ] as a result, while ensuring that Equation 2 holds.
The function baseline connect performs the global connection of substages.It takes two lists of message pairs as inputs and creates one new list of message pairs; the list may have arbitrary size but the input lists must have the same size.The function expandpairs takes a list of message pairs of a list of nodes and produces the plain list of messages, i.e. the assignment of messages to nodes is released.using k = 2 i + j; j = 0 ; 1 i = 1 ; : : : ; n 1 in the last derivation step.Lemma 3.1 shows the equivalence of local and global specification.Thus, as well as Equation ( 2), the functional specification is a specific realization of the original Definition 3.1.

The butterfly network
Althought the interconnection pattern is quite different, the butterfly network can be specified in a similar manner to the baseline.This works because both networks have similar global recursive decompositions.This section summarizes the local butterfly connectivity, defines the message relation, gives the global functional specification, and proves the equivalence of the local and global definitions.

Definition 4.1 (Local butterfly connection)
Let X = ( X 0 ; : : : ; X n 1 ) and Y = ( Y 0 ; : : : ; Y n 1 ) be two stages with the same number of nodes n and let Z = (Z 0 ; : : : ; Z 2 n 1 ) be a stage of nodes of size 2n.Each node of X , Y , and Z has the degree two, i.e. two ingoing and two outgoing egdes.The nodes in each stage are numbered consecutively starting with 0.

The local butterfly connectivity is a relation bly with exactly two function values for each argument. It is defined as:
bf y : f0; : : : ; n 1 g !f 0 ; : : : ; 2 n 1 g bf y(i) = i; and bf y(i) = i + n The edges connecting the substages X (or Y ) with Z are (X i ; Z bfy(i) ), (or (Y i ; Z bfy(i) ) i.e. outgoing edges of X i (or Y i ) are connected with ingoing edges of Z i and Z i+n , i = 0 ; : : : ; n 1 .

Messages
We consider lists of message pairs xs,ys, and zs , as introduced in the last section.The local butterfly connectivity on the level of messages with a fixed order of messages is expressed by the following equation: Functional Programming, Glasgow 1995

Formal Specification of Interconnection Networks
Functional specification We consider substages and incoming or outgoing messages of substages in their list representation.The local butterfly connectivity as well as the relation between messages in Equation ( 4) contain a dependence on the length n of the arguments X and Y (or xs and ys) which is a dependence on the specific stage of X and Y .This dependence is released when defining the global functional definition.The definition is based on message pairs which also for this network has turned out to be the most intuitive and clear definition.The global butterfly connectivity on message pairs is described by the function butterfly.The function separates emitted message pairs of xs and ys and produces new message pairs by combining first items of xs and ys and by combining second items of xs and ys.where the order of indices is the reverse, i.e. j and then i.Thus, the derivation of the functional specification of stage connectivities for both the baseline and the butterfly network results in a simple characteristics of the difference between the networks: the difference is captured in the order of indices of the list representation of messages.

Unified functional specifications
Sections 3 and 4 specified the interconnections between substages of two adjacent stages.This section defines the entire multistage network in two different styles: a nonrecursive definition similar to the standard definition (although more precise), and a recursive definition in the form of an abstract fat tree.
The definitions of the stage and networks are higher order, taking the interconnection function c and the node behavior function g as arguments.This enables a large class of multistage networks, including the baseline and butterfly, to be defined simply by varying the interconnection function.

Nonrecursive network definition
The nonrecursive definition views a network as the composition of d > 0 stages, each n = 2 d 1 units wide (suggesting a rectangular layout for the network).This definition formalizes the standard network specification, and it corresponds to the definition given in Section 2.

Functional Programming, Glasgow 1995
First a stage is defined as the composition of an interconnection function c and a row of nodes with behavior g.The interconnection function operates on a sequence of substages each containing 2 s 1 nodes, where s 2 f 1 ; : : : ; d g is the stage number.
Thus each group of inputs of width w consists of a pair (l;r) where l and r are lists of length w=2, and each element of these lists is a pair of input signals.The function is defined as an iteration over xs.: group w (drop w xs) where w' = w 'div' 2 Each group has type ([(S; S)]; [(S; S)]), where S is the signal type.After the inputs are grouped, the connection function c is mapped over the list of groups, thus applying it independently to each group of inputs.The application of connection function c to one group-pair corresponds to the connectivity of two substages to one substage in the next stage described in Sections 3 and 4.
The resulting outputs from the interconnections are then concatenated into a flat list of signal pairs; finally the node function g is mapped over these to produce the outputs from the stage.The complete network can now be defined as the composition of a sequence of stages: The most complicated aspect of the network is the splitting of inputs into groups of different sizes within different stages; since that has already been accomplished by stage, the definition of multistage is a simple iteration:

Equivalence of network definitions
The following theorem establishes that multistage and gfattree define identical networks, although their differing forms emphasize that they have different origins, a multistage network and a fat tree.Base case (d = 1 ) .multistage An interesting point in this proof is that concat appears only in a trivial way; therefore it might appear that it could be omitted from the definition of stage.However, this proof relies on the Corollary and Lemma, which make nontrivial use of the concat.