Increasing Visibility of IEC 104 Communication in the Smart Grid

Energy systems like smart grids are part of critical infrastructure and their interruption or blackout may have fatal consequences on energy production, distribution, and eventually the life of individual people. In order to secure communication in Industrial Control Systems (ICS) and detect cyber attacks on smart grids, we need to increase visibility of ICS communication so that an operator can see what commands are sent between ICS devices. Security monitoring of ICS transmission requires (i) retrieving monitoring data from ICS packets, (ii) processing and analyzing extracted data, (iii) visualizing the ongoing communication to the operator. The proposed work presents a concept of ICS flow monitoring system that extracts meta data from ICS packet headers and creates ICS flow records similarly to Netflow/IPFIX system. ICS flows represent communication in the smart grid network that is further visualized using dashboard and communication charts. Unlike traditional monitoring approach that works with network and transport layer data only, we extend flow monitoring to application layer with focus on ICS protocols. The proposed approach is demonstrated on monitoring IEC 60870-5-104 communication.


INTRODUCTION
With the progress of digitization and industrial automation, Industrial Control System (ICS) plays an essential role in monitoring and controlling industrial devices, processes and events. The main function of ICS is to gather real-time data from industrial devices, realize device automation, and supervise the system, see Knapp and Langill (2015). An ICS concept covers a variety of communication systems including distributed control systems (DCS), supervisory control and data acquisition (SCADA) systems, and many others.
ICS communication was originally designed for serial communication that was physically separated from external networks. Recently, it has been adopted to operate over standard Ethernet link layer and the Internet Protocol (IP) with UDP and TCP transport on top of IP. This solution made possible to interconnect ICS networks over wide-area networks (WANs) and provide remote control and monitoring. Figure 1 shows various industrial protocols (MMS, Goose, Modbus, SV, IEC 104) operating within a smart grid substation and interconnecting Intelligent Electronic Devices (IEDs) controls power system equipment such as circuit breakers, bay controllers, or relays. Interconnection of ICS systems with IP networks uncovered lack of security of industrial protocols. Vulnerability of ICS communication to external attacks was revealed by the cyber attacks against Ukraine's power grid in December 2015, see Lee et al. (2016), and 2016, see analysis in Assante et al. (2017) and Cherepanov (2017). An ICS system of Ukrainian power grid was infected by a malware that was installed on operator's station without noticing it. The malware called Industroyer by ESET or CrashOverride by Dragos, Inc., see Dragos (2017), masqueraded as a legitimate process for communication with Remote Terminal Units (RTUs). First, the malware scanned the internal network for RTUs and learnt their addresses and functions. Following that it tested ability to switch on and off RTU devices. Since the malware was communicating from inside the network, its behavior became unnoticed by an IDS system and a firewall that were located on the edge of the network. The case also unveiled missing visibility of ICS transmissions within the power grid as mentioned in Assante and Lee (2015). Without monitoring of ICS communication, it is hard to detect common cyber attacks like scanning, command injection, data spoofing, etc.
The issue of missing monitoring of ICS networks was highlighted by the Report of European Union Agency for Network and Information Security (ENISA), see ENISA (2016), which states that without active network monitoring, it is very difficult to detect suspicious activity, identify potential threats, and quickly react to cyber attacks. Firewalls, anti-virus software or IDS systems can detect security threats against ICS systems only partially.
For monitoring ICS communication, we can get inspiration from IP networks where security monitoring is well established. Monitoring techniques that are widely deployed in IP networks include SNMP monitoring, see Presuhn et al. (2002), IP flow monitoring, see Claise (2004), and system logging, see Gerhards (2009). These approaches can be applied on ICS systems to a certain extent. Since IP monitoring relies on IP layer, it cannot be applied on ICS protocols that run directly over link layer, e.g., Modbus or GOOSE. Also, due to the restricted hardware and firmware of RTUs and IEDs, it is not easy to implement SNMP agent or Syslog client on these devices. However, passive network monitoring based on IPFIX protocol, see , seems to be a viable option for ICS networks. IPFIX systems include an IPFIX probe or exporter that observes a traffic and creates IPFIX records with statistics data about ongoing network flows. IPFIX records are then exported to the IPFIX collector where are stored and visualized to network operator. Since IPFIX supports flexible definition of monitoring data, it is possible to extend IPFIX records so that they also include ICS monitoring data.
The proposed approach is demonstrated on passive monitoring of IEC 60870-5-104 (aka 104) communication, see IEC (2006) and Matoušek (2017). We show how retrieved monitoring data are obtained from IEC 104 packets and sent to the IPFIX collector. By analyzing and visualizing ICS monitoring data, we are able to get network statistics about the communication in a ICS network (number of connected hosts, transmitted data, etc.), detect cyber security incidents (unauthorized access, scanning, DoS attack), or system operation (detection of malfunctioning devices, misconfiguration, etc.).

Structure of the text
The paper is structured as follows. Section 2 gives an overview of published work related to the monitoring and security of IEC 104 communication.

Contribution
The main contribution of the paper is a design of IEC 104 monitoring system that provides application visibility of IEC 104 communication. The system is based on IPFIX flow monitoring extended by IEC 104 operations obtained from IEC 104 packet headers. Using IEC 104 flow records we can increase visibility of smart grid control communication that helps to detect cyber attacks and anomaly behavior of the system. The proposed system was implemented as a plugin to the IPFIX monitoring probe and evaluated on available datasets.

STATE-OF-THE-ART
Protection of industrial control systems in smart grids is an important task especially with relation to the recent cyber attacks, see Lu et al. (2010) and Miller and Rowe (2012). NIST Guide to ICS Security, see Stouffer et al. (2015), gives an overview of past attacks on ICS systems with recommendation how to secure ICS architecture using network segregation, firewall rules, NAT translation and other techniques.
Security of ICS/SCADA networks is often arranged by proprietary IDS systems with deep-packet inspection (DPI) focused on selected ICS protocols. Generally, IDS systems parse ICS packets and extract selected data from ICS payload that are further subject to signature-based or behavior-based analysis. If a suspicious communication is detected, an alert is risen and the traffic is filtered out. IDS systems with DPI provide active scanning and analysis of incoming ICS traffic which puts huge

Increasing Visibility of IEC 104 Communication in the Smart Grid
Matoušek • Ryšavý • Grégr requirements on processing power and memory. Because of DPI analysis, a pre-processor (parser) is to be implemented for each ICS protocol that is part of IDS system, see Horkan (2015). IDS systems are usually located at the edge of ICS networks which limits the protection to external threats only. The proposed system of monitoring ICS visibility can be deployed both on the edge of the network or inside the network which covers both external and internal threat scenarios.
Similar monitoring system for protecting SCADA communication in power grid was proposed by Jarmakiewicz et al. (2017) where the authors implemented SCADA probes observing IEC 104 and IEC 61850 communication. Protection includes two phases: training and detection. In the first phase, the SCADA probe learns profiles of authorized control messages transmitted between SCADA nodes (white list). Then the probe observes active SCADA communication and compares it against learnt profiles. Unlike SCADA probes, our monitoring approach uses IPFIX standard which is supported by many Security Information and Event Management (SIEM) systems, so that ICS monitoring can be incorporated into common monitoring systems. Obtained IPFIX monitoring data can be used for whitelisting or anomaly detection similar to SCADA probes.
Another IDS system for IEC 104 with Snort rules defining various types of attacks was proposed in Yang et al. (2013). As discussed above, IDS systems are useful for protecting the SCADA network for external threats, however they do not provide visibility of internal communication which is our main focus.
Common attacks on IEC 104 communication including replay attack, man-in-the-middle attack, command modification and command injection are described in Maynard et al. (2014) where the authors simulate these attacks in order to show how attacker can compromise the integrity of SCADA communication. Communication patterns of these attacks can be revealed in IPFIX records of IEC 104 communication as proposed in this paper.
Recent papers discussing ICS communication also focus on advanced protocol analysis and anomaly detection. For example, Lin and Nadjm-Tehrani (2018) observe IEC 104 traffic patterns such as inter-arrival times of spontaneous events and classify communication into previously observed five different groups. This helps them to identify anomaly in communication. Unlike our approach, they do not observe relations between IEC 104 commands. Our approach is more general because the monitoring system observes any IEC 104 communication (not only spontaneous events) and the obtained data can be used for time-based analysis but also for detecting communication exchange patterns, statistics, etc.

MONITORING IEC 104 COMMUNICATION
IEC 104 protocol is a part of IEC Telecontrol Equipment and Systems standard IEC 60870-5 that provides a communication profile for sending basic telecontrol messages between two systems in electrical engineering and power system automation. IEC 104 operates over TCP using client-server communication model and delivers supervisory data and acquisition request for controlling power grids.
IEC 104 messages are exchanged between the controlled and the controlling station. Controlled station (also called outstation, RTU slave) is monitored or commanded by a master station. Controlling station (typically a PC with SCADA system, RTU master) performs control of outstations. IEC 104 communication is delivered in the monitoring direction, i.e., from controlled station to the controlling station, or in the control direction, i.e., sent by a SCADA system towards the controlled station (RTU), see Figure 2.  (2006) is implemented on application layer (Layer 7) of TCP/IP stack using Application Protocol Data Unit (APDU) and Application Service Data Unit (ASDU), see Figure  3. Based on APDU Control field 1, three APDU formats are defined: I-frames for transmitting data, S-frames for numbered supervisory operations, and U-frames transmitting unnumbered control functions (test frame, start transfer, stop transfer).
The most important APDUs for security monitoring are I-frames that transmit ASDUs. The ASDU includes two sections: fixed-length ASDU header and a variable-length list of information objects. The header includes ASDU type (e.g., single point of information, measured value, regulating step command, read command), number of transmitted objects, cause of transmission (COT, e.g., periodic, spontaneous, activation, interrogation) and ASDU address (station address).  Each information object is addressed by Information Object Address (IOA) that identifies particular data on the given node. For each ASDU type, the IEC 104 standard defines the format of information object, that includes information elements which form the object and structure of the data. For example, information object in ASDU with type 36 (Measured value, scaled value with time tag) and cause of transmission 20 (interrogation) contains two information elements: SVA (Scaled value) and QDS (Quality descriptor), see Figure 4.  Table 1 that shows selected IEC 104 transactions (transactions no. 2,6,8,10,14) exchanged between the IEC 104 master and slave in both directions: control and monitoring. Each transaction contains at least one ASDU with an object addressed by the Information Object Address (IOA), Cause of Transmission (COT) and transmitted value or command. You can notice that one ASDU may transmit several IOA objects with different IOA addresses, COTs and data. The proposed monitoring system retrieves interesting data from IEC 104 packets in order to create IEC 104 flows for ICS monitoring.   and . An IP flow is a sequence of packets going through an observation point and having the same properties. The key five properties that defines the IP flow and are obtained from packet headers are source and destination IP addresses, source and destination ports, and IP protocol. All packets having the same key properties form one IP flow. IPFIX monitoring probe observes ongoing IP flows and collects statistical data about them, e.g., timestamp of the first packet, number of transmitted packets and bytes of the flow, flow duration, etc. Statistical data are written into a IPFIX flow record that is later sent to the IPFIX monitoring center (collector).
The IP flow uses Layer 3 and 4 data for monitoring only. In order to increase IEC 104 visibility, we have to redefine the flow and add new fields extracted from IEC 104 header. For our purposes, IEC 104 flow will include all five key properties of the IP flow extended by the following IEC 104 values: • APDU frame type In addition to standard IP flows, we have to deal with several ASDUs encapsulated in one TCP packet. Using traditional IP flow monitoring these ASDUs would be aggregated into one IP flow record. In such case, IEC 104 visibility would be lost since the flow record cannot keep several different values within one field, e.g., COT. To deal with this issue, we need   The fields with NIL value in entries no. 1,2,6 and 9 indicate APDUs without the ASDU payload, i.e., Uframes (frame type=2) or S-frame (frame type=1). Only I-frames (with frame type=0) transmit ASDUs as showed in Figure 3.
By observing IEC 104 flow, we can see that there are two controlling stations with originator addresses ORG=0 or 1 running on the same IP address 192.168.1.113. Flow no. 3 describes an APDU with an I-frame (type=0) which was sent by a controlling station with ORG=1 to the controlled station with ASDU address 37133. Type of this ASDU is 100 (interrogation command) and COT = 6 (activation). The ASDU transmitted one information object.
The above described IEC 104 flows were obtained by an IEC 104-enabled IPFIX probe that monitored IEC 104 communication. The parser for IEC 104 was developed in frame of IRONSTONE research project and is freely available at project web site 1 .

TOWARDS IEC 104 VISIBILITY
Network visibility provides an insight into day-today activities of the monitored network. The intrinsic characteristics of industrial networks, namely stable communications patterns, static topologies, periodic behavior and a limited number of applications and protocols, can be used to implement effective network visibility solution. IEC 104 communication is strict with the use of a limited number of fixed communication patterns. By collecting monitoring data, it is possible to identify whether the endpoint plays the role of controlling or controlled station. Due to the static topology of IEC 104 networks, the number and function of connected devices does not change so much.
Visibility can be detected at different levels of details. Communication visibility provides aggregated network information that can be represented, for example, by the statistical profile of the entire network communication. At a lower level, the visibility characterizes the aggregated communication between two devices. The lowest level enables to view and analyze a single data transaction with transmitted object values. Different techniques can be adopted for network monitoring providing the required information.
• Packet-based monitoring analyzes every single packet and offers the most abundant data source for network visibility. Decoded information from packet traces usually needs to be aggregated to provide adequate information for network visibility.
• Flow-based monitoring deals only with the selected data from the communication that should, however, provide enough information on important events in the network as well as on the overall network state. Flexible format of flow records using IPFIX allows users to implement new user-defined fields, and thus to provide enough data for network visibility at multiple levels.
Following text demonstrates how IEC 104 flows increase IEC 104 visibility on typical scenarios.

Observing IEC 104 Activity
The main goal of ICS visibility is to discover all communicating ICS nodes on the network and all transmission initiated by them. By observing IEC 104 flows we get a list of active IEC 104 hosts, see   We can also identify IEC 104 nodes sending data in control direction. Table 5 shows communication of an RTU master sending data to a RTU slave in control direction. Following commands are sent by the master RTU: interrogation commands (type=100), single commands (type=45,58), double commands (type=59), or set-point commands (type=50).  Figure 5 shows IEC 104 communication between two IEC 104 nodes within 3 days. The green line represents ASDUs sent with monitoring data to the RTU with COT=3 (spontaneous event), the blue line depicts file transfers between these nodes (COT=13, data transmission).

Activation of an IEC 104 device
Considering time sequences of IEC 104 communication, we can observe typical communication patterns, for example, activation of a IEC 104 node. Table 6 shows IEC 104 flows related to the activation of node 10.33.232.121 by node 10.33.232.120 with the timestamp. The same communication can be visualized using message flow chart, see Figure 6. Having IEC

Requesting Unknown Resources
IEC 104 flow monitoring data also reveals a scenario when an IEC 104 node requests unknown resources. This may indicate either misconfiguration of the sending device or the scanning attack.  The flows contain a set-point command (ASDU type=50) that is used to control the information object, e.g., to set the value of the object. Packets in this scenario were sent with a wrong ASDU address, a wrong object address IOA and an unknown COT value. Such ASDUs were refused by the receiver node by response ASDUs with COT=46 (unknown ASDU address), COT=47 (unknown object ID), or COT=45 (unknown Cause of Transmission) in IEC 104 flow database. When such IEC 104 responses are detected, the monitoring center may raise an alert to inform the system operator about the unknown request. The IEC 104 flow record contains an ASDU that was requested and identification of the original sender. For example, the second entry in Table 7 says that a node with IP address 10.211.55.2 and ASDU originator address 2 requested data from IEC node with IP address 10.211.55.5 and ASDU address 65281. The request was refused because of unknown destination ASDU address.

Identifying Cyber Attacks
By observing long-term communication patterns of IEC traffic we are able to identify unusual behavior that deviates from typical communication profiles. Figure 7 shows a case when an attacker tries to manipulate with IED node by sending Act and ActTerm ASDUs. The blue line represents the number of Act commands (a), resp. ActTerm commands (b) sent during 10 minutes period of normal traffic. The green line shows the number of Act and ActTerm commands during the attack. During normal communication only 1 or 2 activation commands were sent within 10 minutes while during the attack 12 activations were detected within 2 minutes. By comparing the number of sent Act ASDUs with the number of received ActTerm ASDUs we can notice, that that the number of Acts and ActTerms is the same for the normal communication which means that all Act ASDUs were confirmed by ActTerm ASDUs. However, during the attack, some Act commands were ignored, i.e., the number of received ActTerm ASDUs is lower than the number of requested Act commands. This presented attack simulated behavior of the Industroyer malware Dragos (2017) when an IED device was continuously switched on and off for a few seconds. Using IEC 104 flow monitoring we are able to detect such behavior in flow statistics. The question is what level of details of IEC 104 communication is sufficient for security monitoring? Naturally, with the increased number of details, more processing power is required by the probe and more storage is needed to save monitoring data. Let us look at attack described in Section 4.4 and see how the granularity of flow records impacts the visibility of ICS communication. Table 8 shows repeated patterns of ASDUs with COT 6 (Activation), 7 (Activation Confirmation), and 10 (Activation Termination).

IP flows
The above mentioned approach can be compared to the traditional IP flow monitoring where only Layer 3 and Layer 4 data are observed by an IPFIX monitoring probe. Table 9 shows IP flows related to the attack and obtained by the Silk analyzer 2 . We can see that each packet with an ASDU content forms an individual flow, however without processing ASDU layer, we cannot see much details about ICS behavior. However, by analyzing flow statistics, we can at least build similar baseline like in the case of IEC 104 flows. As we could see, Silk analyzer forms IP flows from each packet which is not typical for IP flow monitoring. Table 10 shows IP flows obtained by softflowd 3 . The whole attack is aggregated into two IP flows exchanged between IEC 104 nodes. Using this flow monitoring, we cannot detect or reconstruct attacks against ICS communication.

Extended IEC 104 flows
The level of details of monitoring ICS communication depends on the ICS processor in the IPFIX probe. Table 11 adds IOA address that shows which information objects is a target of the attack.
By analyzing information elements forming the information object, as described in Section 3.1, we can also see what operations were applied on the information object with IOA=11272301, see

Summary
As seen above, the proposed model of ICS visibility using IPFIX flows extended by ICS values is flexible. Granularity of visibility of ICS communication depends on ICS pre-processor (plugin) of ICSenabled probe and can be enhanced according to the operational requirements. Figure 8  Traditional IP flows offer only statistics about source and destination on Layer 3 and 4 which does not provide sufficient visibility of ICS transmission, as demonstrated on scenario in Table 10.
Splitting an IP flow into IEC104-based flows as shown in Table 9 unveils detailed statistics about individual IEC 104 packets but further analysis relies on Layer 3 and 4 data only and no real ICS visibility is provided. However, some types of attacks can be detected on this level.
The third level requires Layer 7 processing on the monitoring probe. Monitoring data reveals details about ASDUs such as type, cause of The most detailed level is obtained when processing individual IEC 104 information objects and information elements. This corresponds to full packet capturing. However, Layer 7 processing is CPU and memory intensive. This level of details is implemented by IDS and IPS systems which are able to process much lower traffic bandwidth (around 100 Mb/s) comparing to IPFIX probes on level 3 which processes data on tens of Gb/s. As demonstrated on previous scenarios, level 3 is sufficient to give visibility of active IEC 104 communication.

CONCLUSIONS
In recent years, several vulnerabilities have been exploited in ICS systems demonstrating the need to improve security of these critical systems. Many attacks became unnoticed by security devices that protect ICS network perimeter but do not prevent attacks from the inside of the system. This paper presented the monitoring system of ICS communication based on extended IPFIX probes that captures the selected application protocol data. In our study we focused on smart grid communication protocol IEC 104. The presented approach describes how IPFIX can be extended for IEC 104 protocol pre-processing which increases the visibility of smart grid control communication. Since it is passive monitoring, it does not affect perfomance of the monitored system.
We demonstrated that IEC 104 flows can be used to detect cyber attacks and as a source data for analysis of system behavior. Various levels of details can be implemented in the ICS-enabled IPFIX probe as presented in Section 5.4. In real-world deployment level 3, i.e., monitoring flows per ASDU packet, is sufficient to visualize important details of ICS transmission and detect common cyber attacks such as network scanning and manipulation with IEC 104 resources. By applying statistical analysis, we can also reveal unusual behavior as shown in Section 4.4.
Further work will be focused on detailed analysis of IPFIX flow data aiming to provide methods for attack detection and diagnostics. We plan to evaluate and adapt existing methods proposed for anomaly detection of ICS communication. Because of lack of available long-term datasets of ICS communications, we also aim to create a corpus of IEC 104 communication samples that would help other researchers who work with network monitoring and anomaly detection of ICS systems.