*3.3. Applicability*

Our proposed method can be applied to optimize the detection of the most known network steganography techniques shown in Figure 2. The spectrum of detected steganographic techniques relies on network steganography detection methods utilized by the presented multilayer detection method. In Table 1, we outline the potential advantages and disadvantages of applying our multilayer network steganography detection method to each group of techniques.



Based on the above findings, we sugges<sup>t</sup> limiting the use of our method to stream modification and hybrid network steganography detection.

#### **4. Case Study**

#### *4.1. Experiment Scope and Methodology*

To measure the crucial features of the proposed method, we decided to perform an experiment by applying the method to a chosen network steganographic technique. The main need was to evaluate steganalysis time and its characteristics. To perform accurate measurements, we needed to choose a steganographic technique that has the following features:


The above set of features ensures that the proposed method application is best utilized and operates on at least three layers. In our opinion, applying the proposed method to any steganographic technique satisfying the requirements above should provide performance gains, depending on the chosen steganalysis methods on each layer. Given the requirements, we chose to apply our method to RSTEG (retransmission steganography) [5,13,24]. The application to RSTEG detection provides us a set of steganalysis methods, presented in the literature, that can operate on aggregated metrics as well as raw data.

The main idea of RSTEG is to not acknowledge a successfully received packet in order to intentionally invoke retransmission. The retransmitted packet carries a steganogram instead of user data in the payload field [5]. Although RSTEG is intended for a broad class of protocols that utilize retransmission mechanisms, we chose to conduct the experiment on hidden communication detection in TCP/IP networks.

The objective of our case study is to document the performance of network steganography detection utilizing steganalysis method(s) individually and in the multilayer approach presented in this paper. Various RSTEG steganalysis methods can be implemented using a passive warden [25] in the architecture we describe in Section 4.2. We proposed detection methods and assigned them to particular layers.

We measured packet processing time to determine the effectiveness of the method. We divided the experiment into two parts:


Processing time was measured between the times the warden started and finished analyzing captured traffic. All measurements were performed on ~100 MB chunks of ~5 GB of captured network traffic on a virtual machine with a single CPU and 2 GB of RAM. Each measurement was repeated 10 times to provide average results.

#### *4.2. RSTEG Steganalysis Methods*

The most effective methods for RSTEG communication in TCP/IP networks are based either on payload comparison or anomaly detection in derived stream metrics, i.e.:


#### 4.2.1. Comparison of the Retransmitted and Original Payload

The method of detection based on a comparison of retransmitted and original payload operates on the assumption that every retransmitted TCP segmen<sup>t</sup> should have a similar payload to the original one. Any outliers can be safely assumed to be carrying steganograms.

Processing and memory requirements for this method are excessive [5] and limit the method's application to selected network connections only. Required resources scale with the amount of transmitted data and the number of network connections.

Based on the above description, we assign this method to the "Raw Data Steganalysis" layer.

4.2.2. Anomaly Detection in a Number of Retransmissions for an Individual Connection

Anomaly detection in a number of retransmissions for an individual connection requires the following operations to be performed:


Based on the fact that all of the above steps operate on a packet's extracted features and aggregated metrics, we assign this method to the first layer.

4.2.3. Anomaly DETECTION in a number of Retransmissions for an Individual Device

The method of anomaly detection in a number of retransmissions for an individual device is similar to the method presented above but operates in a broader scope. In this approach, the retransmission ratio for all network device traffic is determined, and outliers are detected.

Based on the fact that this method operates in a higher layer of aggregated metrics, we assign this method to the second layer.
