IoT IP Overlay Network Security Performance Analysis with Open Source Infrastructure Deployment

Abstract

Some of the most deployed infrastructures nowadays are Overlay Networks (ONs). They consist of hardware and software components designed to establish private and secure communication channels, typically over the Internet. ONs are among the most reliable technologies for achieving this objective and represent the next-generation solution for secure communication. In this paper, we analyze important network performance metrics (RTT, bandwidth) while varying the type of Overlay Network used for interconnecting traffic between two or more hosts (within the same data center, in different data centers in the same building, or over the Internet). These networks establish connections between KVM (Kernel-based Virtual Machine) instances rather than the typical Docker/LXC/Podman containers. The first analysis will assess network performance as it is, without any overlay channels. The second will establish various types of channels without encryption, and the final one will encapsulate overlay traffic via IPsec (Transport mode), where encrypted channels like VTI are not already available for use. The obtained performance is demonstrated through a comprehensive set of traffic-simulation campaigns.

1. Introduction

Modern IT systems often cover extensive geographical areas, requiring secure and reliable infrastructures that are cost-effective in both space and time. Virtual Private Networks (VPNs) and Overlay Networks (ONs) are robust technologies that meet these needs, effectively operating over both traditional Public Switched Telephone Networks (PSTNs) and modern 4G/5G architectures. To clarify, a VPN establishes a secure and encrypted connection over a public network, such as the Internet, allowing users to transmit data securely as if their devices were connected to a private network. VPNs are commonly used for privacy and security while browsing the Internet, especially on public Wi-Fi networks, and to bypass geographical restrictions by hiding the user’s IP address and location. In contrast, an Overlay Network (ON) is a virtual network built on top of an existing physical network infrastructure. It involves creating additional logical connections, protocols, or network segments to provide specific functions or services without altering the underlying network. Overlay Networks are widely used in applications such as Content Delivery Networks (CDNs) and Peer-to-Peer (P2P) networks. They offer flexibility and scalability, enabling the easy deployment of new services or enhancement of existing ones without disturbing the base network. The main differences between an Overlay Network and a VPN lie in their approach to security, architecture, and data-flow management over a public network. In particular:

• Architecture: ONs are built on top of existing networks with added abstraction, allowing for flexibility in hardware and software. VPNs are implemented on existing IP networks, using specific protocols for secure connections.
• Data Flow Management: ONs manage data flow in a distributed manner, with each node responsible for its data transmission, allowing multiple paths based on network topology. VPNs manage data flow centrally, routing data through a secure tunnel between clients and a central server.
• Approach to Security: Overlay Networks (ONs) use end-to-end encryption, ensuring data security even over untrusted networks. VPNs use tunneling, encapsulating data in encrypted packets to create a secure pathway through public networks.
• Scalability: ONs are often more scalable, adapting to network changes and supporting more nodes without performance loss. VPNs may have scalability issues due to centralized tunnel management and device capacity limitations for encrypted traffic.

It is evident that both Overlay Networks (ONs) and Virtual Private Networks (VPNs) provide solutions for creating private and secure networks over public infrastructures. However, they differ in their approach to security, architecture, and data-flow management. The choice between them depends on the specific requirements of the network environment and applications. The primary goal of this work is to offer a comprehensive performance comparison of ONs from various perspectives. The study focuses on analyzing network performance (RTT, bandwidth) as different types of Overlay Networks are used to interconnect traffic between two VMware Virtual Machines. Future studies will include hardware-constrained devices such as Raspberry Pi 4/5 (Raspberry Pi Foundation, Caldecote, England). The networks connect KVM (Kernel-based Virtual Machine) machines rather than the usual Docker/LXC/Podman containers. In the MACsec case, two network namespaces are used directly: one on the first physical endpoint and the other on the second endpoint. Both virtual machines reside on the same physical host, which has the following specifications: 12th Gen Intel(R) Core(TM) i7-1280P 2.00 GHz processor, 32 GB LPDDR5-SDRAM at 4800 MHz, and a 1 TB SSD. This study considers various ON tunnels, including GRE, GREtap, FOU, GUE, IPIP, GENEVE, VXLAN, and MACVLAN, as well as different IPsec approaches: IKEv1/IKEv2 Tunnel Mode, IKEv1/IKEv2 Transport Mode, IKEv2 VTI Route-Based, and IKEv2 XFRMi Route-Based. Additionally, a TLS/NOISE-based VPN is evaluated, Nebula Overlay VPN. For completeness, it is noted that these networks can also be configured to create Site-to-Site, Road Warriors, Mesh, and Hub and Spoke topologies. Not all networks support all features, as they are designed for specific purposes. This paper is divided as follows: Section 2 shows related works on overlay implementations in the literature; Section 3 shows the components of an overlay architecture; Section 4 shows deployed scenarios in the testbeds; Section 5 gives the experimental results in the implemented scenarios; finally, Section 6 summarizes the paper.

2. Related Works

Many works in the literature propose security systems at different protocol stack layers and in different network typologies, showing cryptography performance analysis. This section is dedicated to an overview of some of the most important contributions related to VPNs and ONs.

In Ref. [1], a novel experimental WAN solution, Software-Defined Wide Area Network (SD-WAN), is proposed, entirely based on secure tunnels, implemented by the Generic Routing Encapsulation (GRE) tunneling protocol [2]. The effectiveness of the proposal is validated through two SD-WAN testbeds, covering a very long distance (from northeast to south Italy). The authors demonstrated the SD-WAN can ensure rapid recovery and resilience in the event of network failures, by leveraging an innovative Berkeley Packet Filtering (BPF)-based monitoring technique. The authors demonstrated also that SD-WAN has several advantages in terms of recovery time and high performance with respect to other solutions (e.g., Multi-Protocol Label Switching—MPLS).

Reference [3] assesses a network layer-based VPN, which facilitates secure encrypted communication between remote LANs worldwide by employing Internet Protocol (IP) tunnels over a shared medium, like the Internet. The primary focus of the authors lies in demonstrating the efficiency of various VPNs, backed by several studies on contemporary practices, followed by an analysis of the Wireguard protocol.

The authors of [4] provide a comprehensive analysis of MACVLAN and IPvlan modes, focusing on their architecture characteristics and their utility in enabling container communication across diverse network segments within complex application scenarios. The proposed work details the setup of container Overlay Networks in both single-machine and multi-machine environments, offering insights into the intricacies of these modes. Through the evaluation of key network indicators such as round-trip time (RTT), bandwidth, and jitter, using tools like Ping and Iperf, the research assesses the end-to-end communication performance of containers. The experimental results indicate that IPvlan mode demonstrates superior network performance compared to MACVLAN, presenting a more extensive and adaptable application scenario for container Overlay Networks. This finding highlights the significance of selecting the appropriate network mode based on specific performance and scalability requirements within containerized environments.

The detailed contribution given in [5] presents a comprehensive approach to studying VPNs in general. It emphasizes that understanding VPN technology necessitates consideration of the values and motivations of individuals within organizations. A significant discovery highlights the divergent perceptions and interpretations of the terms VPN, security, and privacy among corporate employees.

In Ref. [6], attention is focused on the widespread utilization of Secure Socket Layer (SSL)-VPN to ensure both authentication and data confidentiality between the client (typically a web browser) and the server (SSL server).

In Refs. [7,8], VPN technology is extensively examined. Specifically, these works delve into the vulnerabilities and suboptimal performance experienced under heavy loads by web browser-based SSL VPNs, which are limited to supporting only Windows operating systems.

The authors of [9] justify the selection of the IPsec-IKEv2 suite over other potential implementations such as SSL/TLS VPN or SSH Tunnel by emphasizing the security of the IP communication layer. This choice is motivated by the desire for transparency for layers above IP within IPsec-IKEv2, as well as the ability to promptly update encryption in the event of a compromise in the data channel.

In the study detailed in [10], the use of IPsec for creating VPNs tailored to e-business requirements is evaluated. A Linksys router, operating both as an access point and a VPN concentrator, is employed, with OpenSWAN serving as a system daemon. This router supports the IKEv1-L2TP tunnel type, where authentication and traffic are handled via L2TP, while encryption is ensured by IPsec.

The article [11] proposed an approach to deploy VPN technologies for securely encrypting traffic in untrusted networks. This includes environments like bars, lounges, conferences, free hotspots, and airport Wi-Fi, as well as more broadly during any travel scenario.

Generally speaking, the IPsec stack comprises three key subcomponents: Encapsulating Security Payload (ESP), Authentication Header (AH), and Internet Key Exchange protocol (IKE). In the previously overviewed work [12], the AH protocol is discussed, focusing on the security and Quality of Service (QoS) aspects of VPNs. The IKE protocol facilitates the generation of session keys for secure communications among various VPN endpoints through a series of message exchanges. Currently, IKE exists in two versions: IKEv1 [13,14] and IKEv2 [6]. IKEv1 offers high flexibility with different configuration options but is burdened by architectural complexity, making deployment increasingly challenging in modern networks. On the other hand, IKEv2 builds upon the legacy of its predecessor while addressing its limitations, and it provides more modern and efficient encryption systems for network traffic. A significant advantage of IKEv2 is its native support across contemporary platforms such as OS X 10.11+, iOS 9.1+, Linux 3.x+, and Windows 8+. Moreover, IKEv2 enjoys robust support on mobile devices, being compatible with native clients and third-party apps on iOS, Android, Blackberry, and Windows platforms.

In Ref. [15], a comparative analysis of various VPN technologies including L2TP, PPTP, OpenVPN, Ethernet over IP (EoIP), and MPLS is proposed to determine the best fit for specific business requirements. This evaluation considers both performance metrics and packet transit characteristics for each VPN technology item.

In Ref. [8], the focus is on the authentication parameters required for VPN server access. While some implementations may only necessitate the traditional username/password combination, other deployments like IKEv1-XAUTH may require additional parameters.

Refs. [16,17] proposed Microsoft’s Secure Socket Tunneling Protocol (SSTP) for channeling PPP device traffic through an encrypted VPN tunnel via HTTP using SSL/TLS transport, handling key negotiation, channel encryption, and traffic integrity.

Quick UDP Internet Connections (QUICs) is highlighted as suitable for IoT devices with limited resources [18], with implementation guidance in general transport services (TAPSs) [7] discussed in [9].

VPNaaS refers to a simplified approach for configuring network security, consolidating configurations into a single product to address various corporate security needs, and accessing virtualized networks on the cloud. Ref. [19] proposes the implementation of VPNaaS using WireGuard with a WAN backbone based on 5G transport.

In Ref. [12], experimental analysis on a Debian Linux environment is conducted by implementing the IPsec tunneling protocol with different encryption algorithms. The study concludes that IPsec AES-sha1 offers reasonable performance comparable to IPsec 3DES-sha1, noting the resource-intensive nature of UDP VPN encryption/decryption leading to performance degradation due to CPU and memory usage.

The article [20] examines the network performance of virtual and native nodes on ARM-based single-board computers (SBCs). It uses KVM (Kernel-based Virtual Machine) as the hypervisor for virtualization. The aim is to compare the network performance between virtualized and non-virtualized environments, analyzing parameters such as latency, bandwidth, and throughput capacity. The main focus is on the efficiency of resources and the scalability of virtual solutions on ARM hardware, which are known for being energy-efficient and cost-effective. The article [21] discusses using reinforcement learning to optimize the selection of Overlay Networks in SD-WAN environments. It focuses on improving network performance by allowing edge devices to choose the best paths dynamically. Networked agents learn from changing conditions to enhance metrics like latency, bandwidth, and reliability, ensuring efficient and reliable data transmission.

The article [22] examines the impact of imperfections in successive interference cancellation (SIC) on the security of downlink cognitive radio networks using Non-Orthogonal Multiple Access (NOMA). It investigates how these imperfections affect the system’s ability to protect data from eavesdropping and explores factors like channel conditions and power allocation to optimize secure communication despite these challenges. The article [23] explores how imperfections in successive interference cancellation (SIC) affect the security of downlink cognitive radio networks using Non-Orthogonal Multiple Access (NOMA). It assesses the impact of these imperfections on maintaining data secrecy and examines strategies to optimize secure communication despite the challenges posed by SIC flaws. The article [24] explores an AI-driven SDN (Software-Defined Networking) intrusion-detection system for identifying and mitigating DDoS (Distributed Denial of Service) attacks. It proposes an innovative approach to improve the effectiveness and efficiency in detecting such attacks. The article [25] addresses security measures for MQTT (Message Queuing Telemetry Transport) communications in Machine-to-Machine (M2M) systems within the food retail distribution sector. It proposes strategies to protect data transmission and ensure the integrity and confidentiality of communications in this context. The article [26] discusses how 6G mobile networks can enhance AI applications through an intelligent user plane, which optimizes network resources and supports advanced AI-driven services. It explores the integration of AI with network infrastructure to improve performance and enable new functionalities in future mobile networks. The article [27] examines how SDN (Software-Defined Networking) can be used for the dynamic deployment of Intrusion-Detection Systems (IDSs) with load balancing to support drones during emergency scenarios. It focuses on optimizing network management and enhancing security for drone operations in critical situations.

3. Components of Overlay Architecture

This section introduces the currently most popular overlay implementations of Linux virtual interfaces, optionally using IPSEC for cryptography. In particular, the analyzed tunnels are:

• IPIP: The IPIP tunnel is a method of tunneling that allows IP packets to be transmitted within another IP packet. This type of tunnel is primarily used to connect two internal IPv4 subnets across the public IPv4 Internet. Due to its minimal overhead, it is optimal for this application, although it only supports IPv4 unicast traffic and not multicast. Furthermore, the IPIP tunnel supports both IP over IP and MPLS over IP.
• VTI: The Virtual Tunnel Interface (VTI) on Linux resembles Cisco’s VTI and Juniper’s secure tunnel implementation (st. xx). This tunneling driver facilitates IP encapsulations, compatible with XFRMi, to establish secure tunnels, enabling kernel routing atop. VTI tunnels function similarly to IPIP or SIT tunnels, with the addition of fwmark and IPsec encapsulation/decapsulation capabilities.
• GRE and GREtap: Generic Routing Encapsulation (GRE), defined in RFC 2784, involves inserting a GRE header between the inner and outer IP headers. Unlike IPIP, which is limited to encapsulating IP, GRE theoretically supports encapsulating any Layer 3 protocol with a valid Ethernet type. GRE tunnels can transport multicast traffic and IPv6. While GRE operates at OSI Layer 3, GREtap functions at OSI Layer 2, meaning there is an Ethernet header in the inner header.
• FOU: Tunneling operates across different layers of the networking stack, with IPIP or GRE working at the IP level, and FOU (Foo Over UDP) functioning at the UDP level. Leveraging UDP tunneling offers numerous advantages, tapping into existing hardware infrastructure like RSS in NICs, ECMP in switches, and checksum offload. Performance boosts have been evidenced for IPIP protocols through developer patch sets. Currently, the FOU tunnel accommodates encapsulation protocols such as IPIP and GRE, with an example FOU header provided. Configuring an FOU receive port for IPIP entails setting it to port 5555, while for GRE, setting IP proto to 47 is required. Another command establishes a new IPIP virtual interface (FOU1) configured for FOU encapsulation, with the destination port set to 5555.
• GUE: Generic UDP Encapsulation (GUE) is a form of UDP tunneling distinct from FOU. Unlike FOU, GUE features its encapsulation header, which includes protocol information and additional data. Setting up a GUE receive port for IPIP bound to 5555 involves configuring an IPIP tunnel for GUE encapsulation. Currently, the GUE tunnel supports inner IPIP and GRE encapsulation. An example GUE header is provided.
• GENEVE: Generic Network Virtualization Encapsulation (GENEVE) consolidates the functionalities of VXLAN, NVGRE, and STT, aiming to address their perceived limitations. Many anticipate that GENEVE could ultimately supplant these earlier formats entirely. The GENEVE tunnel header closely resembles VXLAN, with the key distinction lying in its flexibility. The GENEVE header allows for easy integration of new features through extension with a new Type-Length-Value (TLV) field.
• VXLAN: VXLAN (virtual extensible local area network) is a tunneling protocol designed to address the limitation of VLAN IDs (4096) in IEEE 802.1q. As described in IETF RFC 7348, VXLAN introduces a 24-bit segment ID, known as VXLAN Network Identifier (VNI), allowing for up to $2^{24}$ (16,777,216) virtual LANs. This capacity is 4096 times that of traditional VLANs. VXLAN finds common deployment in data centers across virtualized hosts, often spanning multiple racks. It encapsulates Layer 2 frames with a VXLAN header into a UDP-IP packet.
• IP/MACVLAN: MACVLAN enables multiple MAC and IP addresses on a single physical interface through MACVLAN sub-interfaces, unlike VLANs where sub-interfaces share the same MAC address. Each MACVLAN sub-interface has a unique MAC and IP address, directly integrated into the underlay network. Typically employed in virtualization, MACVLAN interfaces allow containers or VMs to obtain DHCP addresses directly, easing integration into existing networks. MACVLAN offers four types, with MACVLAN bridge being common, enabling local communication without external routing. External connectivity utilizes the underlay network, as illustrated by two containers communicating via the MACVLAN bridge.
• MACsec: MACsec operates at Layer 2, ensuring protection (integrity and/or encryption) transparently within the network. Unlike IPsec, which can pose performance challenges, MACsec is designed to run at line rate, typically in hardware’s ASIC, though not universally supported across hardware. The protected MACsec frame utilizes an Ethertype of 0x88e5. IPvlan is akin to MACVLAN but with a key distinction: the endpoints share the same MAC address. Supporting both L2 and L3 modes, IPvlan offers flexibility in networking configurations. In L2 mode, each endpoint retains the same MAC address but receives a different IP address. Conversely, L3 mode facilitates packet routing between endpoints, enhancing scalability. While the Ethernet Header and SecTag are sent in plaintext, they are always integrity-protected by ICV. The default cryptographic algorithm is AES-GCM-128. Additionally, MACsec supports optional replay protection with a configurable replay window.

Figure 1 illustrates the generic concept of stacking and encapsulation, fundamental to understanding the functioning of Overlay Networks, as well as VPNs.

They are at the forefront of current research in the sector and serve as a foundation for new developments. These protocols support commercial platforms and open-source solutions. In the preceding section, we outline the protocols for implementing Overlay Networks on Linux operating systems, covering proprietary and open hardware platforms.

3.1. IPsec

IPsec, or Internet Protocol Security, is used for managing encrypted tunnels at the OSI Layer 3 level. It is part of a suite of protocols standardized by the Internet Engineering Task Force (IETF), the organization responsible for advancing Internet technologies. Originally developed for IPv6, IPsec has been adapted for IPv4. It offers three primary functional implementations: Databases: Security Policy Database (SPD) and Security Association Database (SAD). Encryption Process Management: Internet Key Exchange (IKE) and Internet Security Association and Key Management Protocol (ISAKMP). Transfer Protocols: Encapsulating Security Payload (ESP) and Authentication Header (AH). IPsec uses the AH and ESP protocols to ensure the integrity and authenticity of transmitted data. The AH protocol authenticates the data source and protects against packet modification during transmission, including a sequence number in the header to prevent replay attacks. The ESP protocol verifies data integrity and identity while also encrypting the payload data. However, ESP authentication excludes the outermost IP header, requiring additional encapsulation for correct delivery, especially across networks using Network Address Translation (NAT), such as private xDSL networks. IPsec operates in two modes: tunnel and transport. In transport mode, the IP packet header remains unchanged, with the transport protocol inserted between the header and the data area, providing security between the source and destination addresses. This mode is suitable for host-to-host or host-to-router connections. In tunnel mode, a new complete IP header is added to the data packets, concealing the original addresses and data. This encapsulated packet is forwarded between encrypted endpoints defined by the new outermost IP header. Tunnel mode is commonly used for site-to-site, host-to-site, and host-to-host data-transmission scenarios.

3.2. The Noise Protocol Framework

The Noise Framework is a versatile and efficient cryptographic library designed to offer various cryptographic tools for secure communication over unreliable networks, such as the Internet. Within the Nebula Overlay, the Noise Framework provides a solid foundation to ensure the security and privacy of communications between network nodes. Here is a technical overview of how the Noise Framework operates within Nebula Overlay: Supported Cryptographic Primitives: The Noise Framework includes a range of cryptographic tools, such as symmetric encryption, Diffie–Hellman key exchange, key authentication, digital signatures, and cryptographic hashes. These tools can be flexibly combined to meet specific security requirements. Noise Protocol Patterns: The Noise Framework employs protocol patterns to outline the phases and operations involved in secure communication. Protocol patterns, like the Noise IK pattern, detail a sequence of cryptographic actions during message exchanges between parties. These patterns provide a standardized method for designing secure and scalable cryptographic protocols. Selection of Protocol Pattern: In Nebula Overlay, the Noise Framework allows for choosing the most suitable protocol pattern based on the application’s and network’s specific needs. For instance, the Noise IKpsk2 pattern might be used for pre-shared key exchange with authentication, while the Noise NN pattern might be suitable for anonymous communication. Configuration of Cryptographic Parameters: The Noise Framework allows the configuration of cryptographic parameters, such as the choice of encryption algorithm, the use of digital signatures, and the setting of Diffie–Hellman parameters. This flexibility enables adjustments to the security level and communication performance based on the application’s and network’s specific requirements. Integration with Nebula Overlay: In Nebula Overlay, the Noise Framework is used to establish secure and authenticated communication channels between network nodes. Noise-based communication protocols ensure that nodes can exchange data securely, maintaining the confidentiality, integrity, and authenticity of the transmitted information. In summary, the Noise Framework is crucial within Nebula Overlay for ensuring secure communication over unreliable networks. By using flexible protocol patterns and a comprehensive set of cryptographic tools, the Noise Framework provides secure and scalable infrastructure for distributed communication over the Internet.

3.3. Linux Daemons

On Linux systems, setting up a VPN using IPsec involves implementing the IKE, AH, and ESP protocols through software, utilizing appropriate kernel modules. The IKE protocol is used to authenticate VPN peers (using methods like pre-shared keys, public key typing, or freeradius), dynamically generate keys, and share them between peers. These keys are also used in the second phase of IPsec from IKE. LibreSwan implements the IKE protocol through its bar program. The ESP protocol, which defines the actual policy agreed upon by peers, is implemented within the Linux kernel’s IPsec stack (NETEY/XFRMi).

3.4. LibreSwan

LibreSwan [28] and StrongSwan [29] are open-source implementations of the IPsec protocol. They are derived from the FreeSwan project and are available as pre-packaged solutions on Linux distributions like RedHat. However, detailed instructions for compiling the software are provided within the project’s code for use on non-packaged Linux platforms. Following installation and proper configuration, an IPsec VPN gateway is established to safeguard data transmission among network members. LibreSwan [28] and StrongSwan [29] are open-source implementations of the IPsec protocol, derived from the FreeSwan project, and are available as pre-packaged solutions on Linux distributions like RedHat. For non-packaged Linux platforms, detailed instructions for compiling the software are included within the project’s code. After installation and proper configuration, an IPsec VPN gateway is established to secure data transmission among network members. Both StrongSwan and LibreSwan support policy-based and route-based VPN configurations. However, in the former case, they are used exclusively unless transitioning from the traditional IPsec.conf configuration to the newer swanctl.conf format. In the latter case, both policies are supported by default, with specific flags available to manage them. In this article, LibreSwan will be referenced.

Table 1. LibreSwan features table.

Feature	LibreSwan
Pre-shared key authentication	Yes
Public-key authentication	Yes
IKEv1 key exchange	Yes
IKEv2 key exchange	Yes
AH support	Yes
NSS compatibility	Yes
DnsSec/XAUTH	Yes
Network Manager compatibility	Yes
VIP (Virtual IP Pools)	Yes
NAT Traversal	Yes
MOBIKE	Yes
Route-based VPN	Yes
Policy-based VPN	Yes
Native {Policy/Route}–based VPN	Yes
HA (High Availability)	Yes
Legacy cipher suites backwards compatibility	No

provides a list of features of the LibreSwan suite. Although both use the same format for the IPsec.conf configuration file, LibreSwan’s configuration files are not compatible with StrongSwan’s due to differences in supported architectures and options. For example, LibreSwan does not support L2TP IPsec for Android because it relies on the DH2 version (MODP1024), which is considered too weak and is disabled during compilation.

OpenWrt

OpenWrt [30] is a distribution that significantly boosts these devices’ functionality beyond the default firmware provided by manufacturers. It includes a user-writable filesystem, which allows for the installation of third-party applications, thus expanding the device’s capabilities. By incorporating the latest routing software, OpenWrt enhances security and reduces software bugs compared to the original manufacturer software, especially on older devices that no longer receive support. Furthermore, OpenWrt can be deployed on custom hardware, including specific boards. For x86-64 bits platforms, it supports virtualization, enabling the setup of small container networks for on-demand services and ensuring network access for all devices connected to the LAN.

4. Description of Possible Deployment for Overlay Topologies

To create an Overlay Network, there are different solutions, both HW and SW, for example, by implementing the appropriate stack on a Debian server (DEBIAN OS https://www.debian.org/index.it.html (accessed on 21 August 2024)). In particular, we will focus on three macro application scenarios:

• A site to site scenario (where two remote offices communicate securely).
• A site to multi-site scenario (where multiple remote offices communicate securely).
• A road warriors scenario (in which multiple users from remote offices communicate securely with a particular local office).

These solutions ensure encryption and data integrity during transit by utilizing encryption algorithms ranging from AES to elliptic curve implementations (e.g., ecp192). In this work, we wanted to make a field assessment of the current landscape at the level of devices that it is possible to connect (NUC, mini PC, Raspberry Pi, Virtual Machines), trying to ensure the largest spectrum of overlay technologies while maintaining the same security.

Table 2. A summary of the Overlay Network implemented testbeds.

Overlay Network Managed	Overlay Deploy	Overlay Implemented	Operative Systems	Platforms
IPsec LibreSwan 4.15	Site to Site	IKEv2 PSK TUNNEL IKEv2 PSK TRANSPORT	Linux DEBIAN 12 WINDOWS 10/11 (client) DEBIAN 11/12 (client) ANDROID 11 (client) iOS 16 (client) MAC OS X 14 (client) RASPBERRY Pi 2/3/4 OpenWrt 23.x	armv7 x86 x86-64 ARM64 ARM MIPSBE MMIPS SMIPS PPC
IPsec LibreSwan 4.15	Site to Site	IKEv2 XFRMi/VTI ROUTE BASED	Same as above	Same as above
FOU	Site to Site		Same as above	armv7 x86 x86-64
GUE	Site to Site		Same as above	armv7 x86 x86-64
GRE	Site to Site		Same as above	armv7 x86 x86-64
GRETAP	Site to Site		Same as above	armv7 x86 x86-64
IPIP	Site to Site		Same as above	armv7 x86 x86-64
GENEVE	Site to Site		Same as above	armv7 x86 x86-64
VXLAN	Site to Site		Same as above	armv7 x86 x86-64
MACSEC	Site to Site		Same as above	armv7 x86 x86-64
Nebula Overlay VPN 1.6.1	Site to Site Host to Host		Same as above	armv7 x86 x86-64 ARM64 ARM MIPSBE MMIPS SMIPS PPC

provides a summary of the implemented testbeds: from a description of the network infrastructure to the software used.

Many of the proposed architectures have been implemented during the mentioned projects and are still in use. For example, Vxlan and Ipip tunnels are part of the COGITO project (Progetto COGITO, https://www.icar.cnr.it/progetti/cogito-sistema-dinamico-e-cognitivo-per-consentire-agli-edifici-di-apprendere-ed-adattarsi/ (accessed on 6 August 2024)) to interconnect the cloud and local offices. With the exponential growth of IoT technologies and the development of communication paradigms such as Fog Computing or cloud-oriented approaches (like AMAZON AWS), or multi-site clusters (like those implemented through Kubernetes), whether on fiber/PSTN networks or 4G/5G networks, it is necessary to adequately manage the increasing volume of data, also in compliance with GDPR [31,32].

The use of the OpenWrt operating system allows for the implementation of technologies on various routers and devices, supporting protocols such as LoRa, ZigBee, and TCP/IP. The x86-64 versions support virtualization, facilitating the creation of Fog networks of microservices and multi-protocol IoT with reduced costs and secure connections. An example is the installation of sensor networks in areas covered by 4G/5G WAN networks, manageable locally through edge computing.

5. Implemented Scenarios and Experimental Results

We evaluate the performance of different VPN protocols on different operating systems and hardware as represented in Table 2.

The choice to use guest KVM stems from the ability to nest container networks, possibly using a MACVLAN approach instead of a classical bridge, directly on individual guest networks. This would enable harnessing the capabilities of both virtual machines, such as resilience, and containers, such as lightweightness and portability.

Both Iperf v2.2 and Atop v 2.11 software are used to record the activities of the server’s operating system during the various VPN sessions. Network topology nodes are shown in

Table 3. Network topology HW components.

Hardware	Quantity
Workstation with 12th Gen Intel(R) Core(TM) i7-1280P-2.00 GHz, 32GB RAM	1
VMWare VM Debian 12 x86-64 virtualized	2
VMWare VM Alpine Linux Latest x86-64 virtualized	1
VMWare VM OpenWrt 23.x x86-64 virtualized	1

.

Listings 1–3 (bash scripts) depict the script executed by Ansible to activate a tunnel (in this case of FOU type) and automatically configure the network settings and kernel firewall parameters.

Listing 1. Bash script for a FOU interface.

Listing 2. Bash script for an FOU interface Iptables Rule.

Listing 3. Bash script for an FOU interface Nftables Rule.

The nodes are connected to a gigabit VMWARE Ethernet switch with 1000 Mbps links. The complete topology consists of a subnet representing the fake WAN (with public IP assigned) and two private subnets (the KVM networks) for each workstation, as shown in Figure 2. In the Site to Site topology, two VPN servers/Overlay Endpoints act as software routers and endpoints of the tunnel. They are connected to local workstations where traffic is generated. In the Site to Multisite Site topology, the servers may become three (the number of nodes can be increased); they maintain the function of the software router and endpoint of the VPN tunnels and are connected to local workstations on which traffic is generated. To generate the network traffic, the Iperf, Netperf, and ping tools are used for measuring productivity, jitter, bandwidth, and round-trip time (RTT) while several counters of the same operating system measure the CPU usage.

To run the emulations necessary to collect the data, Ansible (Ansible Project, https://www.ansible.com/ (accessed on 21 August 2024)) was used on the network endpoint machines. An Ansible playbook to manage network tunnels on Linux works by defining a series of steps to be executed on one or more Linux systems to configure the desired network tunnels. These steps typically include:

• Installing necessary packages to support network tunnels, such as OpenVPN, ipsec, or in this case use the scripts to deploy needed configurations;
• Configuring the specific tunnel configuration files, such as OpenVPN (‘*.conf’) or LibreSwan (‘*.conf’) configuration files;
• Enabling and starting the necessary services to manage the network tunnels;
• Configuring firewall rules, if necessary, to allow traffic through the network tunnels;
• Verifying the status of network tunnels to ensure they are active and functioning correctly.

In essence, the Ansible playbook defines the actions to be taken to configure and manage network tunnels on the target Linux system, allowing administrators to automate the process of managing network tunnels through a series of declaratively defined steps in the YAML playbook file.

5.1. Considered Overlay Network Deployed Topology

Some deployments described in this paper were also used in the context of the COGITO (COGITO project, https://www.icar.cnr.it/progetti/cogito-sistema-dinamico-e-cognitivo-per-consentire-agli-edifici-di-apprendere-ed-adattarsi/ (accessed on 21 August 2024)) and can be reapplied in any environment. The reason for the choice of the HW considered is given by the need to measure the performance of components already put into operation in the research project referred to above and also to ensure maximum backward compatibility. The basic network topology with an example IP address range is depicted in Figure 2. From time to time, the right endpoint is replaced in the site to site scenario, and the connections whose data are shown in the figures are established. For Wi-Fi-only devices, acting as a VPN client, the connection is established with the left endpoint. The Wi-Fi network these devices are originally connected to is the dummy WAN.

The dummy WAN is a simple network with a router (IP 192.168.21.254) on the segment 192.168.21.0/24, which simulates a public network (ISP sites 1, 2 and 3) and allows communication between VPN endpoints (connected wired/wireless clients to the router or switch). In particular, two private IPs of the fictitious WAN simulate the public IP of the gateways of two LANs located at a lower level and used to test site to site topology, respectively, with internal networks 192.168.142.0/24 (Site one) and 192.168.143.0/24 (Site two). The Wi-Fi clients are connected directly to the access point of the primary router. All the RTT tests were carried out by reaching the local network IP of the OpenWrt router, which acts as a VPN centralizer from the client on duty, as shown in Figure 2.

The same approach to obtain throughput results as in reference [33] was used.

5.2. System Tools for Performance Analysis: Iperf and Netperf

This approach allowed us to produce the following graphs:

Figure 3 shows the trend of the average RTT for different connections based on pure IP and tunneling (for a Maximum Transfer Unit (MTU) equal to 1500 B on the left and 9000 B on the right). Connections were sorted in increasing order and it can be noticed that the performances are quite acceptable for all connections (always under 5 ms). In both sides of the figures, some nearly comparable values are observed so, to better understand how the RTT evaluations are related in function of time, their pdfs were considered.

Figure 4 shows how the instantaneous RTT is distributed over 10 min of simulations for the first seven tunnels of Figure 3. Their distributions can be perfectly comparable in terms of both means and standard deviations.

Although there is a slightly higher difference in the mean values, another cluster can be identified as in Figure 5, for the middle set of tunnels illustrated in Figure 3. Also, in this case, the mean and standard deviations are mostly comparable.

The same considerations of Figure 4 and Figure 5 can be applied for Figure 6, where the last tunnels of Figure 3 (right side) have been clustered.

At this point, we can state that for a Clean (no tunneling), FOU, GENEVE, GENEVE IPsec, GREtap, GREtap IPsec, and IPsec (Cluster 1) the average RTT with MTU = 9000 B is ${\mu }_{RTT}$ = 1.4231 s, with a standard deviation of ${\sigma }_{RTT}$ = 0.4821 s. For GRE IPSec, IP in IP, IP in IP IPsec, MAC VLAN, VTI IPsec, VXLAN, VXLAN IPsec, and XFRMi (Cluster 2), the average RTT with MTU=9000B is ${\mu }_{RTT}$ = 2.2439 s, with ${\sigma }_{RTT}$ = 0.6458 s. For Open Connect, Open VPN, Tinc, and WG (Cluster 3), the average RTT with MTU = 9000 B is ${\mu }_{RTT}$ = 4.4304 s, and ${\sigma }_{RTT}$ = 0.9753 s.

As regards the MTU = 1500 B, we found that for Clean, FOU, GENEVE, GENEVE IPsec, GRE IPsec, GREtap, GREtap IPsec, GUE, IPsec, MACVLAN, MACsec, VTI IPsec, xvLAN, and XFRMi (Cluster 1) ${\mu }_{RTT}$ = 1.5247 s and ${\sigma }_{RTT}$ = 0.43023 s, for IP in IP, IP in IP IPsec, tinc, VXLAN IPsec, WireGuard, and MACsec (Cluster 2), ${\mu }_{RTT}$ = 2.12041 s and ${\sigma }_{RTT}$ = 0.4868 s and for Nebula, OpenConnect and OpenVPN (Cluster 3), ${\mu }_{RTT}$ = 3.22462 s and ${\sigma }_{RTT}$ = 0.52786 s. Bold fonts are used for the connections that belong to different clusters in the function of the MTU. From the data illustrated above, we can conclude that, independently from the chosen MTU, in terms of RTT, Clean (obvious), FOU, GENEVE, GENEVE IPsec, GREtap, and GREtap IPsec offer the best performance (the needed header length is low and their reaction time is the minimum one for the MTU of both 1500 B and 9000 B). Out of cluster 1, the RTT becomes worse in the function of the needed header dimension.

Let us examine how MTU sizes affect network performance in overlay channels and VPNs. Using larger MTU sizes enables the operating system to send fewer but larger packets while maintaining the same network speed. This results in a significant reduction in the processing required by the operating system provided that the workload allows for the sending of larger messages. However, if the workload primarily consists of small messages, larger MTU sizes will not have a significant impact. It is advisable to use the largest MTU size supported by both the adapter and the network. For example, on ATM adapters, the default MTU size of 9180 bytes is much more efficient than using an MTU size of 1500 bytes, which is commonly used by LAN Emulation.

In Gigabit and 10 Gigabit Ethernet networks, if all machines have Gigabit Ethernet adapters and no 10/100 adapters are present, it is best to use jumbo frame mode. For instance, connections from server to server within a computer lab can typically be optimized using jumbo frames. MTU, or maximum transmission unit, is also known as link MTU. Measured in bytes, it is the maximum packet size that can be sent in one piece over a hop without being fragmented. It differs from path MTU (PMTU) which is the minimum MTU of the hops in a path. In the presence of packet loss, the path MTU discovery process can generate an underestimation for the MTU. One way to verify the discovered MTU is to re-run the test a couple of times to see if the result is consistent. That said, the fact that the discovered MTU is associated with a known standard suggests that it is not an accidental underestimation. Common values for MTU are shown in

Table 4. Common MTU values by link type.

MTU Type	MTU Size (bytes)
IP over SONET	4470
Ethernet Jumbo Frames	9000
IP over ATM Ethernet Jumbo Frames	9180
Classic Ethernet	1500

.

The overhead introduced by various tunneling protocols varies depending on the specific protocol and configuration. Focus on overlay links deployed is shown in

Table 5. Size of the headers of the analyzed tunnel protocols and types of headers used.

Tunneling Protocol	Header Type	Header Size (bytes)	Max Header Size (bytes)
IPIP	IP	20	20
VTI	IP + VTI	20 + 4	24
GRE	GRE	4 + [0…34]	38
GREtap	Ethernet Frame + GRE	14 + 4	18
FOU	UDP + IP	8 + 20	28
GUE	UDP + GRE	8 + 4 + [0…34]	46
GENEVE	UDP + IP + GENEVE	8 + 20 + 10	38
VXLAN	UDP + IP + VXLAN	8 + 20 + 8 + [0…16]	52
MACVLAN	Ethernet Frame + MACVLAN	14 + 4	18
MACsec	Ethernet Frame + MACsec	14 + [8…30]	44

. These overheads should be considered when designing and implementing network tunnels, as they affect the overall payload size and network performance.

When using a tunnel interface over IPsec, the overhead consists of the IPsec headers, the outer IP header for IPsec, and the additional IP header for tunnel encapsulation. The IPsec header includes the IPsec Security Association (SA) information, encryption and authentication headers (if used), and any additional IPsec-related metadata. The IPIP encapsulation adds another IP header, which includes the source and destination IP addresses of the tunnel endpoints, as well as any additional IP options. The total overhead for a tunnel over IPsec is the sum of the IPsec and tunnel type headers, along with any additional IPsec overhead. This overhead varies depending on the specific IPsec configuration (encryption, authentication algorithms, etc.) and the tunnel type configuration. To provide a specific overhead, you would need to analyze the IPsec and tunnel configurations in use.

Generally, IPsec overhead can range from tens to hundreds of bytes, while the IPIP, for example, header adds 20 bytes. In LibreSwan, the default MSS (Maximum Segment Size) for both tunnel mode and transport mode is typically calculated based on the MTU. For tunnel mode, where the default MTU is typically 1400 bytes, the MSS would be set accordingly. MSS is typically calculated as MTU minus the size of the IP and TCP headers. In IPv4, the IP header is typically 20 bytes and the TCP header is also 20 bytes.

Therefore, for a 1400-byte MTU, the MSS would be 1360 bytes For transport mode, where the default MTU is often the interface’s default MTU (usually 1500 bytes), the MSS would again be calculated similarly and is 1460 bytes. These calculations are based on typical IPv4 header and TCP header sizes. The MSS value can be influenced by factors such as any additional headers, options, or extensions present in the packet, but these calculations provide a general guideline for determining the MSS in LibreSwan based on the default MTU values.

Each protocol has its strengths and trade-offs, making them suitable for various network configurations and requirements.

First of all, it can be noticed that Tinc shows very bad performance and following are some of the reasons why Tinc VPN may experience low performance:

• An MTU that is too high or too low can cause packet fragmentation or inefficiency in packet transfer. A value of 1400 can improve performance.
• If compression is enabled, it can cause extra CPU load.
• Suboptimal TCP/IP parameters can affect performance.
• Using outdated versions of Tinc can lead to performance issues, and Tinc has not been updated since 2021.

In both cases, Tinc connections are completely unsuitable. In the UDP cases, there are no differences in terms of bandwidth, while for Layer 4 TCP sockets, OpenVPN, OpenConnect, VTI, Nebula, XFRMi are unsuitable (very low reachable bandwidth), and WG, FOU, GUE, IP in IP IPsec, VXLAN IPsec and GREtap IPsec performance are comparable to the performance obtained with a Clean connection. The other tunnels outperform the previous ones.

5.3. KVM Vs Containers, Strengths and Weaknesses

The work focuses on analyzing OVERLAY technologies within open source platforms, specifically using LibreSwan for implementing VPNs in a user-transparent manner. It plans to include a comparative analysis with VPNs in future research and deviates from traditional Docker networks to use KVM for creating more robust VM ecosystems. This approach leverages the power of modern hardware to address needs where containers alone may fall short.

KVM (Kernel-based Virtual Machine) offers robust isolation and security by running full operating systems in VMs, which minimizes the risk of issues spreading between them. It supports a wide range of applications and tools, and provides flexible resource configuration and compatibility with modern hardware virtualization technologies. However, KVM VMs experience higher overhead and slower startup times due to the need to run a full guest operating system. Additionally, resource management with KVM is less dynamic compared to containers, potentially leading to inefficiencies in environments with fluctuating workloads.

Docker containers are resource-efficient and start up quickly since they share the host operating system’s kernel and do not require a full OS. They offer portability, making it easy to deploy applications across different environments, and provide good process-level isolation. However, containers have limited isolation compared to VMs, as they share the host kernel, which can pose security risks. Additionally, containers are constrained by the host kernel, limiting the ability to use different operating system configurations and complicating security management.

Table 6. KVM vs. Docker Strengths and Weaknesses.

KVM vs. Docker Strengths and Weaknesses
KVM		DOCKER
Strengths	Weaknesses	Strengths	Weaknesses
Isolation and Security	Higher Overhead	Resource Efficiency	Limited Isolation
Full Operating System Support	Startup Speed	Startup Speed	Host Kernel Dependency
Resource Flexibility	Resource Management	Portability	Security Management
Compatibility and Stability		Process-level Isolation

provides a summary of the comparison between the strengths and weaknesses of KVM and Docker platforms.

6. Conclusions and Future Works

In this work, we gave several details about VPN and overlay connections in TCP/IP networks. In particular, we focused our attention on the analysis of several tunnels, from different points of view, implementing them into virtualized environments and giving different information on how they can be implemented. The second part of the work was dedicated to the analysis of the obtainable results, in terms of RTT, maximum bandwidth and throughput. One of the main results consists in the possibility of clustering the tunnels in the function of the reachable RTT, thus having a wide variety of choices that respect a given RTT distribution, depending on the desired results. The second analysis that has been illustrated regards the performance of tunneling for TCP and UDP connection: we have shown how, for UDP, all the connections saturate, while for TCP there are some preferred tunnels, with respect to some other ones. The same trend was found for the throughput analysis.

A future study will certainly involve translating this study onto the Proxmox platform to evaluate the performance achievable through systems managed through this hypervisor. Proxmox VE optimizes both the network and CPU management layers to deliver high performance and efficiency in virtualized environments. There is an emphasis on these optimizations: Proxmox VE supports a wide array of advanced networking technologies, which are crucial for creating flexible and high-performance virtual networks. These include:

• FOU (Foo Over UDP): Efficient encapsulation of IP in UDP, reducing overhead.
• GUE (Generic UDP Encapsulation): This offers flexibility for different protocols with minimal latency.
• GRE (Generic Routing Encapsulation) and GREtap: This ensures robust tunneling of various network protocols and Ethernet frames, respectively, optimizing traffic flow.
• IPIP (IP-in-IP): This simplifies tunneling with minimal processing overhead.
• GENEVE and VXLAN: These provide scalable solutions for network virtualization, enhancing performance in Overlay Networks.
• MACsec: This secures data link layer communications, maintaining high throughput with encryption.
• OpenVPN and WireGuard: These deliver secure and efficient VPN solutions, with WireGuard specifically known for its minimal CPU usage and high performance.
• IPsec (VTI, XFRMi, Classic): This ensures secure communications with optimized encryption and tunneling techniques.
• Tinc and Nebula: These offer mesh networking solutions that scale efficiently while maintaining security.

Proxmox VE enhances CPU management through various techniques to ensure that virtual machines (VMs) and containers run efficiently:

• KVM (Kernel-based Virtual Machine): This utilizes hardware virtualization extensions (Intel VT-x and AMD-V) to ensure near-native performance for VMs.
• CPU Pinning: This allows binding of VMs to specific CPU cores, reducing context switching and optimizing performance.
• NUMA (Non-Uniform Memory Access) Awareness: Proxmox can optimize memory and CPU allocation across NUMA nodes, improving access times and overall VM performance.
• Resource Limits and Quotas: This enables setting limits and quotas for CPU usage, ensuring fair distribution of resources and preventing any single VM from monopolizing CPU resources.
• Dynamic Resource Allocation: Proxmox dynamically adjusts resources based on current workload demands, optimizing CPU and memory usage.
• CPU Hotplug: This allows one to add CPUs to running VMs without downtime, providing flexibility and scalability.
• Integration with cgroups and namespaces: This ensures fine-grained control over resource allocation and isolation in containers, optimizing CPU usage.

Proxmox VE’s optimizations in both network and CPU management layers ensure nowadays that virtualized environments operate with high efficiency and performance. The comprehensive support for advanced networking technologies combined with robust CPU management techniques makes Proxmox VE a powerful platform for modern data centers and cloud environments.