Detection of TCP and MQTT-Based DoS/DDoS Attacks on MUD IoT Networks

Abstract

Mitigating cyberattacks on IoT networks is critical and remains a significant challenge, as such attacks can cause severe damage to the network systems and services. Moreover, the large volume of devices in IoT networks presents another challenge in managing security to reduce the risk of attacks. The Manufacturer Usage Description (MUD) is a standard for limiting attack risks on IoT networks. However, MUD has limitations, as it relies solely on pre-defined access control list (ACL) rules to allow permitted traffic and block unknown traffic. This can lead to false-negative filtering, where malicious traffic may still be allowed by MUD, compromising an entire IoT network. This study presents the implementation of a network behaviour analysis (NBA) system for DoS/DDoS attack detection in MUD-based IoT networks. We designed a set of algorithms to enhance the effectiveness of malicious traffic detection compared to using MUD alone. The NBA system groups related traffic and detects a variety of DoS/DDoS attacks that utilise TCP and MQTT protocols. Our evaluation demonstrates that the NBA system achieves high detection accuracy, effectively identifying attacks that MUD alone would not be able to detect, thereby enhancing the effectiveness of attack detection in MUD-based IoT networks.

1. Introduction

Preventing the risk of cyberattacks on IoT networks remains a significant challenge due to limited security measures and the difficulty of managing large volumes of IoT devices. One of the most common attacks on IoT networks is the denial of service (DoS) or distributed denial of service (DDoS) attack [1]. These attacks exploit security vulnerabilities in IoT devices, infecting them with malware to turn them into botnets, which are then used to launch attacks against target victims [2].

To mitigate the attacks, the Internet Engineering Task Force (IETF) approved the Manufacturer Usage Description (MUD) under Request for Comments (RFC) 8520 [3]. MUD is designed to reduce cyberattacks by defining the intended network behaviour of IoT devices. For example, a smart lightbulb is intended to light a room and be controlled by a smartphone application. It should not be connected to unauthorised devices or external services. These intended behaviours are used to create a device behaviour profile, which is then published and retrieved to enforce communication restrictions on the local network. This ensures that only authorised traffic is allowed while blocking unknown connections, reducing attack risks.

Despite the benefits of MUD, it still has limitations that must be addressed to enhance the effectiveness of attack mitigation. MUD enforcement primarily relies on pre-defined access-control list (ACL) rules to filter unknown network traffic; however, these do not analyse network behaviour to detect and mitigate potential threats. Take the previous example of a smart lightbulb. Now, suppose that it attempts to connect to a server for software updates. If a malicious actor has compromised the server, MUD may fail to identify the traffic as malicious, allowing the device to establish a connection with the compromised server. This limitation could lead to false-negative filtering, potentially compromising the network.

1.1. Objectives

The existing limitations of MUD could lead to false-negative filtering, decreasing the effectiveness of network enforcement and putting the entire network at risk. To address this limitation, we proposed and designed a set of algorithms for detecting DoS/DDoS attacks on MUD-based IoT networks. This work extends our previous work, where we proposed a network behaviour analysis (NBA) system and enhanced profiling assurance (EPA) architecture [4], the background of which is presented in Section 2.

The objectives of this study are as follows:

• To propose a system that would allow MUD to consider additional aspects of network traffic besides considering each packet based on pre-defined ACL rules in MUD.
• To design and implement a set of algorithms to support the grouping of network traffic and identify malicious traffic.
• To evaluate the detection accuracy of the proposed algorithms.

1.2. Scope

This study aimed to address the limitations of MUD by proposing an alternative approach that employs a non-ML-based technique to enhance attack detection in small-scale IP-based MUD networks. We focus specifically on TCP- and MQTT-basTCP- and MQTT-based DoS/DDoS attack detection, of which current research remains limited. This is achieved by analysing network traffic from packet capture files to evaluate the effectiveness of the NBA system.

Our study does not focus on large-scale MUD network deployment, as we emphasise small-scale implementations. Additionally, UDP-based attack detection is beyond our scope, as this study focuses on TCP-based protocols, which exhibit more complex behaviour. Furthermore, non-IP-based IoT network deployments are excluded, as MUD is primarily designed to filter unwanted traffic in IP-based networks.

1.3. Research Method

Our study began by examining MUD to understand its purpose and how it mitigates attack risk in IoT networks. We then investigated the existing limitations of MUD in detecting and mitigating DoS/DDoS to identify research gaps. After identifying these gaps, we designed a set of attack detection algorithms to be integrated into the NBA system to enhance security in MUD-based IoT networks. We started by setting up a network testbed to collect network traffic data, including benign and malicious traffic, to study TCP- and MQTT-basTCP- and MQTT-based attack patterns. Using this dataset, we analysed attack patterns and designed detection algorithms accordingly. The algorithms were then refined and optimised to ensure their detection accuracy before being implemented into the NBA system for attack detection evaluation.

Finally, we evaluated the detection accuracy of our designed algorithms using the collected network packets to assess their effectiveness in attack detection. The evaluation process consists of two stages: TCP-based attack detection and MQTT-based attack detection. To measure detection accuracy, we compared the number of detected attack packets generated by our algorithm against the original dataset. We then calculated the miss rate and detection accuracy to demonstrate the effectiveness of our proposed algorithms for TCP- and MQTT-based attack detection in MUD-based IoT networks.

1.4. Contribution

MUD has been proposed to reduce attacks on IoT networks, both from external threats and compromised IoT devices. Our contribution is creating a system to detect the attacks that MUD may be unable to detect for small-scale IoT network deployment. Our focus is the algorithm design for detecting TCP- and MQTT-basTCP- and MQTT-based DoS/DDoS attacks. At the time of writing, limited studies were available focusing on protocol-specific attacks, particularly an MQTT protocol, a key protocol in IoT networks. Furthermore, current studies on attack detection in MUD-based IoT networks focus primarily on large-scale network deployment, with limited studies on small-scale network deployment. This study may be of interest to security professionals, network administrators, and others interested in reducing attack risks in IoT networks. While this study primarily focused on TCP and MQTT, our system can be easily extended to support other network protocols, such as UDP-based communication.

1.5. Paper Structure

The remaining structure of this paper is as follows. Section 2 provides an overview of the TCP protocol, MQTT protocols, and DoS/DDoS attacks. Section 3 reviews the existing literature on attack detection and mitigation in MUD-based IoT networks to identify the research gaps. Section 4 proposes and describes the algorithms for detecting and identifying TCP- and MQTT-basTCP- and MQTT-based DoS/DDoS attacks on MUD-based IoT networks. Section 5 evaluates the detection accuracy of our designed algorithms in detecting TCP- and MQTT-basTCP- and MQTT-based DoS/DDoS attacks. Section 6 concludes this paper, discusses the limitations, and outlines future work.

2. Background

2.1. Manufacturer Usage Description (MUD)

2.1.1. Overview of MUD

According to the NIST definition, MUD is a component-based architecture specified in RFC 8520 to provide a means for end devices to signal to the network what sort of access and network functionality they require to properly function [5]. MUD specifies intended network interactions, such as allowed protocols, IP addresses, and port numbers. It allows device manufacturers to define the intended behaviour as a MUD profile and publish it on the MUD file server to be retrieved and translated into ACL rules for enforcing communication on the IoT network. Many major organisations and industry bodies support MUD, including NIST [6], Cisco [7,8,9], and Huawei Technology [10].

To achieve this objective, the MUD architecture has been defined for communication enforcement based on defined intended behaviours, as illustrated in Figure 1.

When an IoT device connects to MUD networks for the first time, it transmits a MUD URL embedded in the device by the manufacturer through the firewall (1) and forwards this to the MUD manager (2). The MUD manager then uses the received MUD URL (the location of the MUD profile) to obtain the MUD profile hosted on the MUD file server (3). The MUD profile is then retrieved from the MUD file server (4) and translated into ACL rules to enforce them on the firewall for traffic filtering, allowing intended traffic (MUD-allowed traffic) and denying unwanted traffic (MUD-rejected traffic) (5).

The MUD architecture uses a MUD profile to enforce communication policies by allowing only the intended traffic, as defined by the device manufacturer. The MUD profile is structured using a yet-another-next-generation (YANG) module [11], which enables the device behaviour profile to be translated into ACL rules for enforcement at the firewall. The MUD profile contains the following behaviour information for communication enforcement:

• Name: The name of the ACL and the associated access control entity (ACE).
• Type: Specifies the IP address version (IPv4 or IPv6).
• Protocol: Indicates the protocol usage, represented by its protocol number.
• Destination: The address of the destination host to be connected.
• Direction-initiate: Specifies the direction of connection initiation for TCP-based communication (e.g., from device or to device).
• Forwarding: Indicates whether a specific communication behaviour is allowed or denied (accept or deny).

An example of a MUD profile is shown in Figure 2.

The snippet in Figure 2 shows the intended behaviour profile of the ACE named ‘from-ipv4-amazonecho-0’. It defines the communication behaviour of outbound TCP-based traffic from the device to the host address ‘dcape-na.amazon.com’ using an HTTPS protocol (represented by port number 443). This communication is permitted by MUD, as indicated by the ‘forwarding’ field set to ‘accept’. Any other communication not explicitly defined in the MUD profile will be blocked by the MUD enforcement mechanism.

The ability to enforce communication based on the intended behaviour profile provided by device manufacturers forms the foundation for securing IoT networks through the use of MUD. MUD can also complement other standards that focus on a secure-by-design cyber–physical system (CPS). For example, the IEC 62443 standard [12] provides a framework for securing CPS through risk assessment and security level zone assignment based on potential risks. A notable application of IEC 62443 is the integration into renewable energy communities (RECs), as proposed by Gaggero et al. [13]. In this context, MUD can serve as a complementary mechanism within the architecture to enhance the security of REC networks by filtering network traffic to prevent unauthorised or malicious connections.

2.1.2. Limitations of MUD for Attack Mitigation

While MUD can offer a degree of protection against cyberattacks, it is limited in its effectiveness. These limitations include the following.

• MUD relies solely on pre-defined ACL rules with a scope limited to certain layer 3 and layer 4 header fields.
• Behaviours and trends are not identified. The ACL rules that are implemented in the firewall from the MUD profile simply block or allow traffic on a packet-by-packet basis.
• Traffic that is otherwise allowed by MUD may still be malicious. For example, the MUD profile may allow communication to occur between a device and a server for the purposes of software updates. If that server were to be compromised by a threat actor, MUD would be unable to identify the traffic as problematic, as it still may comply with the ACL rules.

Research to improve the effectiveness of MUD is ongoing. Heeb et al. [14] studied the impact of MUD usage on IoT network security and concluded that MUD can reduce attacks on layer 3 and layer 4. However, the authors overlooked the limitation of lacking network behaviour analysis in MUD-based IoT networks to reduce attack risks.

The limitations found in the MUD-based IoT network highlight the need for solutions to improve the effectiveness of attack mitigation. It is crucial to maintain a secure IoT environment to prevent it from being compromised by cyberattacks. This study was motivated by the need to improve the effectiveness of attack mitigation by integrating the NBA system into MUD-based IoT networks to prevent DoS/DDoS attacks.

2.2. Enhance Profiling Assurance (EPA) Architecture and Network Behaviour Analysis (NBA) System

In our previous work, we introduced a newly designed network architecture called enhanced profiling assurance (EPA), aimed at improving the effectiveness of attack detection and mitigation in MUD-based IoT networks. The primary goal was to address the limitation of MUD that can cause false-negative filtering due to the reliance on pre-defined ACL rules. The design of our previously proposed EPA architecture is shown in Figure 3.

The EPA architecture contains the network behaviour analysis (NBA) system, which serves as the core component for detecting attacks that MUD may be unable to identify due to its reliance on pre-defined ACL rules. In this context, the NBA system acts as a second layer of defence by analysing network traffic to detect potential attacks, while MUD functions as the first layer, allowing authorised traffic and filtering unwanted traffic based on pre-defined intended behaviour.

Our previous proposal for the EPA architecture introduced only the conceptual framework for improving the effectiveness of attack detection. In this study, we built upon that architectural foundation by designing a set of attack detection algorithms aimed at identifying attacks occurring within MUD traffic. Currently, our focus is on detecting TCP- and MQTT-basTCP- and MQTT-based DoS/DDoS attacks.

2.3. Transport Control Protocol (TCP)

The transport control protocol (TCP), an essential protocol in internet protocol (IP) networks, is defined in RFC 793 for client–server communication [15]. TCP is a ‘connection-oriented’ protocol, which establishes a session for the purpose of data exchange [16]. To ensure reliability, TCP sends an acknowledgement message within a specific time frame to confirm that it has been successfully received [17]. TCP establishes a connection between two endpoints using a three-way handshake process. Once established, data exchange occurs, which is associated with application-layer protocols operating over TCP, such as MQTT. After data exchange, both endpoints terminate the connection using a four-way handshake. However, a connection may terminate abruptly without completing this process due to various factors, such as intermittent connection. In such cases, TCP sends a reset packet to the other endpoint, aborting the current session and re-establishing a new one.

2.4. Message Query Telemetry Transport (MQTT)

Message query telemetry transport (MQTT) is a real-time machine-to-machine communication protocol developed by IBM [18]. It is considered a key protocol in IoT network deployment, as it was designed to support low-bandwidth communication and energy efficiency for small sensor devices. MQTT was publicly released in 2013 as an OASIS standard [19] and later became an ISO recommendation (ISO/IEC 20922) [20]. MQTT operates over the TCP protocol for data transport. MQTT uses a publish–subscribe model to facilitate one-to-one or one-to-many communication under specific topics. It relies on an ‘MQTT broker’ to manage the interaction between applications or devices. According to the Internet Assigned Numbers Authority (IANA), two ports are assigned for MQTT: 1883 for standard communication and 8883 for secure MQTT communication [21].

2.5. DoS/DDoS Attacks

DoS/DDoS attacks are malicious attempts to disrupt the availability of target devices or network infrastructure [22]. These attacks exploit network protocols by overwhelming targets with excessive data transmission from malicious devices. Since they leverage network protocol mechanisms, multiple protocols across different layers can be manipulated to facilitate the attack. This study aimed to detect attacks on IoT networks to reduce the risk of compromise by such attacks. We focused on five TCP-based attacks: SYN, FIN, FIN-ACK, ACK, and RST. For MQTT-based attacks, we emphasise MQTT CONNECT and PUBLISH attacks.

2.5.1. SYN Flood

A SYN flood attack exploits the three-way handshake process to overwhelm the target device. The attacker sends a large volume of SYN packets without completing the three-way handshake process. A victim tends to acknowledge by replies with SYN-ACK packets. However, an attacker will not acknowledge the victim, causing an incomplete three-way handshake and denying other services from legitimate connections.

2.5.2. FIN Flood

A FIN flood attack takes advantage of the TCP connection termination process by flooding a victim with an unusual volume of FIN packets. The attack causes the connection to close or deny service as the target device looks for the corresponding connection to FIN flood packets.

2.5.3. FIN-ACK Flood

A FIN-ACK flood is another TCP traffic amplification-based attack that exploits the four-way handshake termination process. Aside from usual resource exhaustion, this attack can cause the connection session to close mistakenly due to the confused response to the FIN-ACK flood attack.

2.5.4. ACK Flood

ACK flood attacks are used with the intent of overwhelming the target with ACK packets that do not belong to the existing connection, causing a burden on the target device as the operating system must locate the related established service, leading to it becoming unresponsive.

2.5.5. RST Flood

An RST flood is usually used to signal a connection restart to the other end for various reasons. An RST flood uses this advantage to disrupt the service and traffic by flooding RST packets to either one or more target devices.

2.5.6. MQTT CONNECT Attack

An MQTT CONNECT flood is conducted by sending a large volume of request connect packets to overwhelm the target during the MQTT broker connection process. The purpose is to disrupt the service at an MQTT broker, causing the MQTT network to become inoperable. As MQTT is an application layer protocol, the attack is conducted after completing a three-way handshake process.

2.5.7. MQTT PUBLISH Attack

An MQTT PUBLISH flood occurs during data exchange by sending enormous ‘PUBLISH’ packets after connecting with an MQTT broker. This attack disrupts the central broker and can affect other clients depending on the MQTT topic used to launch the attack.

3. Related Work

We categorised and reviewed the literature on attack detection and mitigation in MUD-based IoT networks. We have categorised these studies into two broad groups:

• Machine learning (ML)-based methods.
• Non-ML-based methods.

3.1. Machine Learning-Based Methods

Machine learning (ML) has gained increasing attention as a high-performance solution for attack detection and mitigation. Many studies have integrated ML into MUD-based IoT networks to improve the effectiveness of attack detection and mitigation. Our study categorised ML-based methods into two groups: centralised ML and distributed ML.

Centralised ML-based methods refer to traditional ML approaches deployed on a single high-performance device for attack detection. Several studies have adopted this approach for high-performance attack detection. Zangrandi et al. [23] introduced MUDScope to detect attacks on MUD-rejected traffic using HBDSCAN clustering and signature matching. Their approach was evaluated using IoT network intrusion [24] and MUDScope [25] datasets, demonstrating successful detection of anomaly behaviours that deviate from MUD profiles. However, their study heavily relied on pre-defined MUD policies and only emphasised attack detection in MUD-rejected traffic.

In contrast, Singh et al. [26] incorporated a learning-based approach to dynamically update MUD profiles using hierarchical clustering to identify normal behaviour patterns. They evaluated their approach with a LoRaWAN testbed and successfully achieved dynamic MUD profile updates. However, their study did not evaluate the effectiveness of attack detection or mitigation, as it was not their focus.

Other studies have explored integrating centralised ML-based methods with software-defined networking (SDN) MUD-based networks. Hamza et al. [27,28] proposed a two-stage anomaly detection model using one-class classification (OCC) and X-means clustering, later extending it by integrating microflow rules to detect low-volume DoS/DDoS attacks. Their studies successfully detected both volumetric and low-volume attacks, which were evaluated with their generated dataset. However, they did not consider MQTT-based threats and were limited to detecting only SYN attacks for TCP-based attack detection.

Similarly, Krishnan et al. [29] focused on SDN MUD-based network deployment by implementing a hybrid deep learning (DL)–ML-based approach with digital twinning for detecting anomalies in Industrial IoT (IIoT) networks. They evaluated their approach with Bot-IoT [30], CICIDS2017 [31], Hogzilla [32], and Kitnet datasets [33], achieving a high rate of Mirai botnet detection with a very low false-positive rate. However, hybrid ML–DL requires significant computational cost. Additionally, their TCP-based attack detection was limited to SYN attacks, and MQTT-based attack detection was not evaluated.

S and M [34] also adopted a hybrid DL–ML-based method, utilising MUD profiles to train their system to detect anomalies in smart building networks with digital twinning. They evaluated their approach using CICIDS2017, Bot-IoT, Hogzilla, and MUDGEE-cap [35] datasets, achieving high accuracy in detecting DDoS and botnet attacks. However, their approach relied on pre-defined ACL rules to train the ML model and did not focus on evaluating MQTT attack detection or mitigation.

Decentralised ML-based paradigms, such as federated learning (FL), have gained attention due to their lower hardware requirements and ability to mitigate single points of failure. Feraudo et al. [36] introduced CoLearn, an FL-based MUD architecture for attack detection and mitigation using MUD profiles and the PySyft FL framework. They evaluated their proposal using a testbed setup with the Bot-IoT [30] dataset. However, their study did not assess the effectiveness of attack detection or mitigation.

To address this, Matheu et al. [37] integrated MUD and FL to enhance attack detection and mitigation in IoT-enabled smart cities. The approach was evaluated using a simulated test bed with the attack traces dataset [27], and achieved 90% accuracy in attack detection. However, their study only covered a limited range of TCP-based attacks and did not address MQTT-based attack detection or mitigation. Similarly, Datta et al. [38] introduced iDAM, an FL-based attack mitigation system for smart home networks. Their approach integrated MUD for device authorisation, one-class support vector machine (OC-SVM) for anomaly detection at IoT gateways, and FL to aggregate models across gateways. They evaluated their proposal using a testbed setup with real captured traffic. However, like Matheu et al., their study did not assess MQTT-based attack detection, and their TCP-based attack was limited to SYN and ACK attacks.

The centralised ML approach provides the advantage of a high-performance process on a single centralised device for attack detection and mitigation. However, existing studies have yet to analyse network behaviour on specific protocols. Recent research has explored attack detection in MUD-rejected traffic [23], dynamic MUD profile updates [26], and SDN-based traffic monitoring [27,28,29,34]. These studies limited TCP-based attack detection to SYN attacks and overlooked potential threats from MQTT-based attacks. Moreover, ML-based methods require significant processing power, increasing deployment costs and making them unsuitable for small-scale network deployment. [39]. Additionally, ML-based methods rely on feature extraction and reduction processes to maintain detection accuracy, which can be time-consuming [40].

The decentralised ML-based method provides an alternative approach to a centralised ML-based approach by incorporating multiple devices to train an ML model. Despite the promising potential of this method, existing research has yet to focus on protocol-specific attacks. One study proposed an FL-based MUD architecture, but did not evaluate its effectiveness in attack detection [36]. Other studies focused on attack detection in smart city networks [37], and smart home networks [38]. However, both primarily addressed SYN attack detection for TCP-based threats and lacked coverage of MQTT-based attack detection and mitigation. Training an ML model in a decentralised setting requires multiple participating devices, making it less feasible for small-scale IoT network deployment. Additionally, as an ML-based method, decentralised ML also relies on the feature engineering aspect to ensure attack detection accuracy.

Considering the limitations of the ML-based method, such as high deployment cost and reliance on a time-consuming feature engineering process, our study chose not to pursue the ML-based method for attack detection, as it is unsuitable for small-scale network deployment. Instead, our study proposes an alternative approach by adopting a non-ML-based approach for detecting attacks on small-scale MUD-based IoT networks focusing on TCP- and MQTT-based attack detection.

3.2. Non-ML-Based Method

Non-ML-based attack detection relies on classical algorithms, making it lightweight compared to ML-based approaches due to lower computational requirements [2]. Additionally, non-ML-based methods also offer better explainability, making it easier to understand the process. In the context of attack detection and mitigation in MUD-based networks, our study categorises non-ML-based methods into MUD profile extension and network behaviour analysis.

MUD profile extension allows manufacturers to augment their MUD profile, increasing the flexibility of MUD enforcement beyond its default capability. Feraudo et al. [41] extended MUD profiles to include rate-limiting policies, integrating them with an extended Berkeley packet filter (eBPF) for high-speed packet filtering. They evaluated their approach in a virtualised testbed with the ToN-IoT dataset [42], and demonstrated low latency and more effective traffic filtering compared to iptables. However, their method primarily relies on pre-defined ACL rules and does not analyse network behaviour to effectively detect attacks.

Similarly, Andalibi et al. [43] extended MUD profiles to support rate-limiting policies in fog computing-based MUD networks. They evaluated their proposal in a virtualised environment, successfully reducing excessive traffic and demonstrating the feasibility of MUD deployment in fog-based networks. However, like Feraudo et al. [41], their study lacked network behaviour analysis for attack detection mitigation.

Hanes et al. [44] proposed an extension of MUD profiles by incorporating layer 2 (datalink layer) support to improve security and functionality for IoT devices with multiple network interfaces. However, their approach only focused on security enforcement and did not include an attack detection mechanism. Additionally, no evaluation was conducted.

Houben et al. [45] integrated the MUD framework into Thread networks to demonstrate its applicability in low-power IoT networks. They extended the MUD functionality to support MUD profile requests through a Thread-specific mesh link establishment (MLE) message. Their testbed implementation successfully detected and blocked unauthorised telnet and ICMP traffic, demonstrating effectiveness in mitigating unwanted connections. However, TCP- and MQTT-based DoS/DDoS attack detection has was not covered in this study.

García et al. [46] proposed an enhancement of the MUD profile for better security assessment in a CPS. They extend the MUD profile to describe potential security risks in CPS environments and to enable notification to users when such risks are detected, allowing for timely mitigation or patching. The authors evaluated their proposal using the smart grid information and communication technology (ICT) gateway, demonstrating the generation of a MUD profile based on their proposed extension for risk assessment. However, their study did not evaluate the detection of TCP- or MQTT-based attacks, which may leave the system vulnerable to such threats.

Although limited, some non-ML-based methods have explored network behaviour analysis for attack detection and mitigation. Datta et al. [47] introduced DNSguard, a system for monitoring and mitigating attacks on a Domain Name System (DNS) server. They evaluated their approach using a Raspberry Pi-based testbed with real captured traffic, successfully mitigating DNS attacks and achieving a 67.2% reduction in response time during attacks. However, their study was limited to DNS traffic and did not cover any TCP-based protocols. Hadi et al. [48] also analysed network behaviour by integrating Snort [49] to detect attacks within MUD-rejected traffic. Their approach was evaluated with Mirai botnet attack detection and successfully detected attacks with high accuracy. However, their method only focused on MUD-rejected traffic and, like Datta et al., did not evaluate TCP-based or MQTT-based attack detection.

Rao and Kulkarni [50] introduced a MUD profile extension that included a dynamic trust attribute called ‘trust-status’ for attack mitigation. Their approach analysed network behaviour to continuously adjust device trust levels for security enforcement. However, their approach did not focus on attack detection, and no evaluation was conducted.

The extension of MUD profiles allows new capabilities to be added to MUD enforcement, such as rate-limiting [41,42] for reducing the risk of volumetric attacks, layer 2 integration for managing multiple network interfaces [44], support for low-power IoT network deployments [45], and the ability to describe potential risks in a CPS [46]. However, the MUD profile extension method relies on pre-defined ACL rules for traffic filtering, which can still cause false-negative filtering. As a result, this method cannot detect or mitigate potential threats that occur in MUD traffic.

Network behaviour analysis-based attack detection offers the advantage of providing better granularity in attack detection compared to solely relying on MUD enforcement. However, existing studies remain limited, focusing on DNS traffic monitoring [47], botnet detection in MUD-rejected traffic [48], or dynamic trust level adjustment [50]. None of these approaches fully address TCP-based or MQTT-based attack detection in MUD-allowed traffic, leaving networks vulnerable to such attacks.

The non-ML-based method is more lightweight than the ML-based one, making it suitable for deployment in small-scale networks. Considering this advantage, we chose to adopt the non-ML-based method for attack detection in small-scale MUD-based IoT networks. However, existing research on non-ML-based methods only covers limited TCP-based attacks and has yet to address the detection of MQTT-based attacks. To bridge this gap, our study proposes a system for analysing network traffic behaviours in MUD-based IoT networks to detect TCP- and MQTT-basTCP- and MQTT-based attacks.

3.3. Summary of Related Studies on Attack Detection and Mitigation in MUD-Based IoT Networks

A summary of related studies on attack detection and mitigation in MUD-based IoT networks is shown in

Table 1. Summary of related works on attack detection and mitigation in MUD-based IoT networks.

Reference	Advantages	Limitations	Description
Zangrandi et al. [23]	High detection accuracy, few false positives	Only analyses MUD-rejected traffic, no MQTT attack detection	Development of a tool for attack detection in MUD-rejected traffic using HBDSCAN clustering and signature matching.
Singh et al. [26]	Real-time behavioural learning for MUD profile update	Not focused on attack detection	A learning-based approach to dynamically update MUD profiles based on extracted normal behaviour data using hierarchical clustering.
Hamza et al. [27]	Scalable, reduced false negative filtering	No MQTT-based attack detection, TCP-based attack limit to SYN, high computational resources	ML-based multistage anomaly detection with MUD for SDN-based network using One-class classification, X-means clustering.
Hamza et al. [28]	Scalable, finer granularity on attack detection	No MQTT-based attack detection, TCP-based attack limit to SYN, high computational resources	Extension of [27] to include microflow rules for inspecting packet header in the flow if an attack has been detected.
Krishnan et al. [29]	Scalable, high accuracy, fast attack detection	High computational resources, no MQTT-based attack detection	SDN-based MUD architecture for attack detection with digital twinning for updating MUD policies using non-symmetric deep autoencoder, random forest, and hierarchical clustering.
S and M [34]	Scalable, high-accuracy, fast attack detection	Relies on pre-defined ACL rules, no MQTT-based attack detection	MUD-enabled deep learning framework with digital twinning for updating MUD policies using non-symmetric deep autoencoder, random forest.
Feraudo et al. [36]	Reduces single-point failure, can be deployed in edge devices	Not focused on attack detection or mitigation	Proposal of FL-based MUD network architecture deployment using PySyft FL framework, deep neural network.
Matheu et al. [37]	Reduces single-point failure, high-accuracy attack detection	No MQTT-specific analysis, high computational overhead	Proposal of FL-based approach for attack mitigation in MUD-based smart city networks using FL, multilayer perceptron (MLP).
Datta et al. [38]	Reduces single-point failure, multilayered protection	No MQTT-specific analysis, high computational overhead, risk of model poisoning	FL-based multilayer MUD architecture for volumetric attack mitigation using FL, OC-SVM.
Feraudo et al. [41]	Lightweight, high-speed packet filtering	Relies solely on pre-defined ACL rules, no MQTT attack detection	Proposal for a rate-limit extension for MUD profile and integration with eBPF for high-speed traffic filtering.
Andalibi et al. [43]	Lightweight DDoS mitigation, secure fog layer	Relies solely on pre-defined ACL rules, no TCP- or MQTT-based attack detection	Proposal for a rate-limit extension in MUD profiles for fog computing-based MUD deployment.
Hanes et al. [44]	Ability to manage devices with multiple interfaces	Relies solely on pre-defined ACL rules, does not directly detect or mitigate attack	Extension of MUD profile to include layer 2 details for managing multiple interfaces.
Houben et al. [45]	Ability to deploy in low-power IoT networks	Relies solely on pre-defined ACL rules, no TCP- or MQTT-based attack detection	Extension of MUD functionality for deployment on a low-power IoT network.
García et al. [46]	Security and risk assessment for threat notification to users	Relies solely on pre-defined ACL rules, no TCP- or MQTT-based attack detection	Extension of MUD profile for describing potential vulnerabilities to assess the risk in CPS.
Datta et al. [47]	Lightweight, low DNS response time during attack	No TCP or MQTT attack detection	Proposal of Raspberry Pi-based MUD-enabled DDoS attack mitigation system on DNS server.
Hadi et al. [48]	Lightweight, real-time attack detection, high detection accuracy	Limited to MUD-rejected traffic, no protocol-specific analysis	MUD deployment with Snort IDS for botnet attack detection in MUD-rejected traffic.
Rao and Kulkarni [50]	Real-time trust adjustment	Not focused on attack detection	Proposal of dynamic ‘trust-status’ extension in MUD profile based on network behaviour analysis to automatically adjust trust level.

.

4. Proposal of the Implementation of the NBA System

4.1. Overview of the NBA System

An overview of the NBA system for detecting TCP- and MQTT-basTCP- and MQTT-based DoS/DDoS attacks on MUD-based IoT networks is shown in Figure 4.

The NBA system comprises different processes to analyse network traffic in MUD-based IoT networks. The focus is to detect potential TCP- and MQTT-based DoS/DDoS attacks, which MUD may be unable to detect as it only filters unwanted traffic. To achieve this, we have designed a set of algorithms to analyse network traffic and detect attacks on MUD networks.

The attack detection process of the NBA system begins by obtaining a network packet capture (pcap) file and filtering only TCP-based traffic, excluding non-TCP-based protocols. The NBA system then performs TCP session grouping to group packets into sessions (Algorithm 1). During this process, the NBA system tracks the three-way handshake status of each session. If a session lacks a complete three-way handshake, the TCP attack detection is initiated (Algorithm 2), as it is considered malicious. In contrast, the NBA system checks for MQTT protocol usage if a session completes the three-way handshake. Non-MQTT sessions are excluded from further analysis. Then, the NBA system performs MQTT attack detection to identify potential threats during data exchange (Algorithms 3 and 4).

An expected normal behaviour should exhibit this sequence of the following characteristics:

• A complete TCP three-way handshake process.
• MQTT broker connection.
• MQTT data publish.
• MQTT broker disconnection
• A TCP session termination (via a complete four-way handshake or reset).

If a session does not exhibit the expected normal behaviour, the NBA system conducts further analysis to detect potential attacks. The analysis can result in two possible outcomes: abnormal behaviour (indicating TCP or MQTT attack) or uncertain behaviour (requiring continuous monitoring).

4.2. Session Grouping Algorithm (Algorithm 1)

Session grouping organises related network packets into sessions based on connection attributes such as IP address, ports, and protocol. This process is crucial, as DoS/DDoS attacks exploit session behaviour rather than individual packets. By grouping packets into sessions, the NBA system can identify abnormal session patterns, such as incomplete three-way handshakes and excessive reconnections that indicate potential DoS/DDoS attacks. This process results in categorised sessions for further TCP- or MQTT-based attack detection analysis.

The TCP session grouping process can be outlined in three stages, as follows:

• Initialisation.
• Session grouping.
• Session categorisation.

At the initialisation stage, three variables are created to track session status during grouping. Details and functions of these variables are as follows.

three_way: Tracks whether a complete three-way handshake has been successfully established in a session. This variable is crucial for categorising sessions for further analysis of attack detection on either TCP or MQTT.

four_way: Records the completion of a four-way handshake, signifying normal session termination to identify normal session termination. This variable is used to end sessions that terminate properly.

terminate: Flag sessions that are abruptly terminated by detecting RST or RST-ACK packets. This is used to end sessions that do not follow the standard four-way handshake process.

The session grouping process begins by receiving network packets from a packet capture file (pcap) and detecting SYN packets, which initiates a new session (newsession). Then, subsequent packets that belong to the same session are added to the session until termination occurs, either by a complete four-way handshake or an abrupt termination with RST or RST-ACK packets. Session termination status is tracked using the ‘four_way’ and ‘terminate’ variables.

Once session grouping is finished, the NBA system moves to the session categorisation stage by examining the ‘three_way’ value of each session to categorise grouped sessions into two groups based on the three-way handshake status:

• Complete three-way handshake session: Sessions that completed the handshake process (three_way = True).
• Incomplete three-way handshake session: Session that did not complete the handshake process (three_way = False).

Sessions that contain a completed three-way handshake are stored in the ‘complete3way’ list for further analysis by the MQTT-based attack detection algorithms. Otherwise, they are put into the ‘incomplete3way’ list and forwarded to the TCP-based attack detection algorithm, as they indicate potential TCP-based attacks.

The TCP session grouping process is shown in Algorithm 1.

Algorithm 1 Session grouping

Input: Capture network packets from a pcap file Output: Categorised TCP sessions (complete and incomplete three-way handshake sessions) 1. Begin 2. Import packets from pcap file 3. Create complete3way, incomplete3way to store categorised sessions 4. Define session tracking variables: three_way = False, four_way = False, terminate = False 5.   6. For each packet p in packets sequence do: 7.    If (p.tcp_flag = SYN) then 8.       Create empty list newsession 9.    Add packet to newsession 10.    If (p.tcp_flag = SYN-ACK) then 11.       Proceed to the next packet 12.    If (p.tcp_flag = ACK) and (three_way = False) then 13.       Set three_way = True 14.    If (p.tcp_flag = FIN-ACK) then 15.       Proceed to the next packet 16.    If (p.tcp_flag = ACK) and (four_way = False) then 17.       Set four_way = True 18.    If (p.tcp_flag = RST) or (p.tcp_flag = RST-ACK) then 19.       Set terminated = True 20.   21. For each session in newsession do 22.    If three_way = False then: 23.       Store session in incomplete3way 24.    Else: 25.       Store session in complete3way 26. End algorithm

4.3. TCP-Based DoS/DDoS Attack Detection Algorithm (Algorithm 2)

The TCP-based DoS/DDoS attack detection algorithm detects potential TCP-based attacks. These attacks attempt to overwhelm the target by transmitting excessive TCP packets without completing session establishment. Thus, this algorithm focuses on analysing incomplete three-way handshake sessions. The algorithm is designed to detect and identify five TCP-based attack types—SYN, ACK, FIN, FIN-ACK, and RST—by analysing TCP flag information.

To detect TCP-based attacks, we adopted batch processing and threshold-based techniques to detect and identify the attacks. Batch processing is used to manage and analyse packets by maintaining a set of recently received packets while filtering out outdated ones to ensure accurate detection. The threshold-based technique is used to define the acceptance condition for analysis and flag potential attacks when thresholds are exceeded. The threshold value can vary depending on the baseline behaviour of device communication and the specific network deployment scenarios.

Our TCP-based DoS/DDoS attack detection algorithm consists of three stages:

• Initialisation: Variable generation and incomplete three-way handshake sessions importing from Algorithm 1 (Section 4.2).
• Batch processing: Analyses a set of recently received packets while filtering outdated packets. This ensures that outdated packets will not affect the attack detection process.
• TCP attack classification: Detects and classifies TCP-based attacks based on TCP flag values.

At the beginning of the algorithm, five variables—syn_attack, fin_attack, finack_attack, ack_attack, and rst_attack—are created to store detected attack packets based on their TCP flag values. Additionally, an ‘uncategorised’ group is created to store packets that cannot be classified as attack packets. Two key thresholds have also been created in this stage for TCP-based attack detection, which can be described as follows.

max_time: The maximum time threshold for considering packets as part of the same batch for further analysis on attack detection. Packets exceeding this threshold are considered outdated and removed.

max_packet: The maximum number of packets within a batch for attack detection and identification.

The system imports incomplete three-way handshake sessions (incomplete3way) from Algorithm 1 (Section 4.2) and extracts timestamps from each packet for time-based analysis in the batch processing stage.

Then, the system proceeds to the batch processing stage for filtering outdated packets. This step is necessary because some packets may still remain and could affect the attack detection process if not filtered out. In this stage, each packet in the current batch (current_batch) is checked against the newly incoming packet. The time difference between packets is calculated and compared against ‘max_time’. If the packet time difference exceeds this threshold, it is removed from the batch. Otherwise, it remains for further analysis. After filtering, the batch is updated with the new packets to ensure only recent traffic is considered while filtering outdated packets.

Next, the system proceeds to the TCP attack classification stage. The algorithm determines the total number of packets in the batch and compares it against the ‘max_packet’. If the threshold is exceeded, the attack type is determined based on TCP flag values to classify and store each attack packet into corresponding lists. Finally, the algorithm returns separated lists of detected attack packets categorised by type.

The TCP attack detection process is shown in Algorithm 2.

Algorithm 2 TCP-based DoS/DDoS attack detection

Input: Incomplete three-way handshake sessions (incomplete3way) from Algorithm 1 Output: Lists of detected TCP attack packets 1. Begin 2. Import incomplete3way from Algorithm 1 3. Define current_batch, syn_attack, fin_attack, finack_attack, ack_attack, rst_attack, uncategorised 4. Define max_time (time threshold), max_packet (packet threshold) 5.   6. For each packet p in incomplete3way do 7.    Create new_batch to store recent packets 8.   9.    //Retain only packets within the time threshold 10.    For each packet pb in current_batch do 11.       If (p.time-pb.time) ≤ max_time then 12.          Include pb in new_batch 13.    Update current_batch with new_batch 14.    Add p to current_batch 15.   16.    //Detect Flood Attacks 17.    If size of current_batch ≥ max_packet then 18.       Categorize p based on TCP flag 19.          - SYN → Store p to syn_attack 20.          - FIN → Store p to fin_attack 21.          - FIN-ACK → Store p to finack_attack 22.          - ACK → Store p to ack_attack 23.          - RST → Store p to rst_attack 24.          - Otherwise → Store p to uncategorised 25. Return syn_attack, fin_attack, finack_attack, ack_attack, rst_attack, uncategorised 26. End Algorithm

4.4. MQTT CONNECT DoS/DDoS Attack Detection Algorithm (Algorithm 3)

MQTT CONNECT attacks occur during the MQTT broker connection phase, following a completion of a TCP three-way handshake. These attacks generate multiple illegitimate sessions by transmitting excessive CONNECT packets without completing the previous session, leading to service disruption on the MQTT broker and repeated session creation.

Due to the pattern of CONNECT attacks, we designed a detection algorithm to perform two stages of attack detection, as follows:

• Single-session attack detection: Examines each session individually to identify unusually high volumes of connect packets.
• Multiple-session attack detection: Analyses multiple sessions from the same client to detect excessive reconnection attempts.

At the beginning of the algorithm, seven variables are initialised, as follows:

• connect: Store the extracted CONNECT packets from the imported sessions.
• client_timestamp: Store the timestamp of packets for inter-arrival time calculation.
• connect_attack: Store packets identified as CONNECT attack packets.
• uncategorised: Store packets that cannot be definitively classified as attacks.
• client_count: Record the number of CONNECT packets across sessions per client.

To detect CONNECT attacks, the threshold-based technique is applied for both single-session and multiple-session analysis. Three detection thresholds are defined at the start of the algorithm:

• max_single: The maximum number of connect packets threshold in a single session. it is used to detect potential CONNECT attacks on individual sessions.
• max_multiple: The maximum number of connect packets threshold across multiple sessions. This threshold is for detecting CONNECT attacks on multiple sessions transmitted from the same client.
• min_time: The minimum inter-arrival time threshold for detecting malicious activity. This threshold is used to compare against the packet inter-arrival time to detect potential attacks.

The process begins by importing complete three-way sessions (complete3way) from Algorithm 1 (Section 4.2) and extracting CONNECT packets. The timestamp of each CONNECT packet is extracted and stored per client in ‘client_timestamps[packet.src_ip]’. Then, the system calculated the inter-arrival time (deltatime) of CONNECT packets for the attack detection stage.

Next, the system performs per-session and cross-session attack detection stages. This is achieved by determining the number of detected CONNECT packets in single and multiple sessions per client and comparing them against ‘max_single’ and ‘max_multiple’. If the threshold is exceeded, the system then compares the calculated inter-arrival time against the min_time threshold. If the time is below than threshold, the attack packet is determined based on the MQTT message type to classify and store correspondly (connect_attack, uncategorised).

The algorithm for MQTT CONNECT attack detection is shown in Algorithm 3.

Algorithm 3 MQTT CONNECT DoS/DDoS attack detection

Input: complete3way (set of sessions with completed three-way handshakes) Output: connect_attack, uncategorised packets 1. Begin 2. //Initialization 3. Import complete3way from Algorithm 1 4. Create storage for: connect, client_timestamps, connect_attack, uncategorised, client_connect_count 5. Define max_single, max_multi, min_time 6.   7. //Extract MQTT packets & track timestamps 8. For each session in complete3way do 9.    For each packet p in session do 10.      If p is CONNECT then 11.         Store p in connect, client_timestamps[p.src_ip] 12.         client_count[p.src_ip] = client_count[p.src_ip] + 1 13.   14. //Single-session and multiple-session attack detection 15. For each client do 16.    Compute deltatime (inter-arrival time) for CONNECT packets 17.   18.    If size of connect > max_single OR client_count[client] > max_multiple then: 19.      If any deltatime in CONNECT < min_time then 20.         Append CONNECT packets to connect_attack 21.      Else 22.         Append CONNECT packets to uncategorised 23.   24. Return connect_attack, uncategorised 25. End Algorithm

4.5. MQTT PUBLISH DoS/DDoS Attack Detection Algorithm (Algorithm 4)

Unlike CONNECT attacks, MQTT PUBLISH attacks occur within a single active session after establishing a connection to the MQTT broker. Therefore, attack detection is conducted only through per-session analysis.

The algorithm creates the following five variables to store information, as follows:

publish: Stores extracted PUBLISH packets.

publish_attack: Stores detected PUBLISH attack packets.

uncategorised: Stores packets that cannot be considered as an attack.

The algorithm applies the following two thresholds for attack detection:

• max_packet: The maximum number of packets threshold. It is used to detect unusually high volumes of PUBLISH packets.
• min_time: The minimum packet inter-arrival time threshold. It is introduced to detect suspicious activity that indicates potential PUBLISH attacks.

These thresholds are defined at the beginning of the algorithm.

The attack detection begins by detecting PUBLISH packets in complete three-way sessions (complete3way) from Algorithm 1 (Section 4.2), storing them in the ‘PUBLISH’ list. Then, the system determines the size of detected ‘PUBISH’ packets and compares them against ‘max_packet’. If the number of packets exceeds a threshold, the system then calculates the packet inter-arrival times (deltatime) and compares them against ‘min_time’. If the calculated time is below the ‘min_time’ threshold, these packets are then classified as attack packets and stored in the ‘publish_attack’ lists. Any inconclusive packets will be marked as ‘uncategorised’ for further continuous monitoring.

MQTT PUBLISH DoS/DDoS attack detection is shown in Algorithm 4.

Algorithm 4 MQTT PUBLISH DoS/DDoS attack detection

Input: complete3way (sessions with completed three-way handshakes) Output: publish_attack, uncategorised 1. Begin 2. Import complete3way from Algorithm 1 3. Create storage for publish, publish_attack, uncategorised 4. Define max_packet (packet thresholds) and min_time (inter-arrival threshold) 5. //Extract MQTT PUBLISH packets 6. For each session in complete3way do: 7.    For each packet p in session do: 8.       If p is PUBLISH then Store p in publish 9. 10. //Compute inter-arrival times  11. Compute deltatime (inter-arrival time) for PUBLISH packets 12.   13. //Detect PUBLISH and SUBSCRIBE attacks 14. For each client do: 15.    If size of publish > max_packet OR any deltatime < min_time then: 16.       Add publish to publish_attack 17.    Else: 18.       Add publish to uncategorised 19. Return publish_attack, uncategorised 20. End Algorithm

5. Evaluation

5.1. Testbed Setup

We set up a network testbed to collect network packets of both TCP and MQTT protocols. The testbed consists of two simulated sensors and an MQTT broker. The sensors establish a connection with an MQTT broker to transmit measure values. The specifications of the two sensors are shown in

Table 2. Temperature sensor specifications.

Sensor 1		Sensor 2
IP address	192.168.10.5	IP address	192.168.10.6
Protocol usage	TCP and MQTT	Protocol usage	TCP and MQTT
MQTT port	1883	MQTT port	1883
Measuring range	$15 \mathrm{to} 50 °\mathrm{C}$	Measuring range	$15 \mathrm{to} 50 °\mathrm{C}$
Publish topic	test/tempvalue_1	Publish topic	test/tempvalue_2
Publish Interval	Every 60 s	Publish Interval	Every 70 s

.

Each sensor runs as a virtual machine (VM) component to simulate real IoT device communication. They establish a connection with the central MQTT broker via an internal NAT network. On the broker side, we implemented Mosquitto as an MQTT broker for managing MQTT communication. The broker also acts as a network packet collector using tcpdump to capture communication traffic and export it to a pcap file for attack detection analysis. The testbed network connection diagram is shown in Figure 5.

5.2. Detection Accuracy Evaluation

To determine the accuracy of attack detection, we first calculated the false-negative rate by comparing the detection output produced by our proposed algorithm to the collected data in a pcap file. The equation for calculating the false-negative rate is as follows:

$$FN=\left({\displaystyle \frac{P_m}{P_t}}\right)\times 100$$

(1)

where:

• $FN$: False-negative rate of the proposed algorithm (percentage).
• $P_m$: Number of attack packets that were not detected by the proposed algorithm.
• $P_t$: Total injected attack packets in the original pcap file.

The calculated false-negative rate is then used to determine the detection accuracy to evaluate the performance of our proposed algorithm, as follows:

$$A=100-FN$$

(2)

where:

• $A$: Detection accuracy rate of the proposed algorithm (percentage).

5.3. TCP Attack Detection and Identification Performance Analysis

To evaluate the TCP-based attack detection of our proposed algorithms, we collected samples of simulated DoS attack network traffic during two hours of data collection. We injected SYN, FIN, FIN-ACK, ACK, and RST attacks using the hping3 tool running from two simulated sensor devices to an MQTT broker. The collected network packets were then exported to a pcap file containing both benign and attack traffic for attack detection analysis.

To prepare the evaluation of our proposed TCP-based attack detection algorithm, we configured the threshold in Algorithm 2 (Section 4.3) as follows:

• max_time: 1 s.
• max_packet: 5 packets.

TCP-based attack detection was conducted by inputting the pcap file that contained both benign and TCP attack traffic into the algorithm. It then analysed the incomplete three-way handshake sessions to detect and flag the attack type based on the extracted TCP flag. The detection results compared to the original pcap file are shown in

Table 3. TCP-based attack detection results.

Sensor 1 (192.168.10.5)			Sensor 2 (192.168.10.6)
Attack Type	Total Packets (pcap)	Total Detected Packets	Attack Type	Total Packets (pcap)	Total Detected Packets
SYN	9416	9416	SYN	5306	5306
FIN	9124	9124	FIN	9217	9217
FIN-ACK	14,379	14,379	FIN-ACK	10,917	10,917
ACK	27,203	27,203	ACK	3770	3770
RST	8611	8611	RST	11,495	11,495

.

The detection results from our TCP-based attack detection algorithms indicate a false-negative rate of 0 and achieve 100% detection accuracy in identifying all five types of TCP-based attacks.

This high accuracy is attributed to the distinct pattern of TCP-based DoS/DDoS attacks, which transmit excessive amounts of packets without completing the three-way handshake process. Since our detection algorithm is specifically designed to analyse incomplete three-way sessions, it can accurately detect and classify injected attack packets, unlike an MQTT-based attack, which poses more complex attack patterns.

5.4. MQTT Attack Detection and Identification Performance Analysis

For MQTT-based attack detection, we collected samples of simulated DoS/DDoS separately from TCP-based attack traffic. We set up a testbed and collected packets for two hours. However, unlike TCP-based attack detection, we cannot use hping3 to inject the MQTT attack, as it does not support it. Instead, we developed Python scripts to perform CONNECT and PUBLISH attacks.

To simulate MQTT DoS/DDoS network traffic, we ran each MQTT attack script to launch attacks from sensor devices to an MQTT broker at different times during the data collection process. For an MQTT PUBLISH attack, we launched attack packets under the same MQTT topics for measured value transmission: ‘test/tempvalue_1’ for sensor 1 and ‘test/tempvalue_2’ for sensor 2. This evaluated whether our NBA system can detect attacks when the attackers use legitimate MQTT topics to mimic normal traffic.

We set the thresholds for the CONNECT attack detection (Algorithm 3 (Section 4.3)) as follows:

• max_single: 3 packets.
• max_multiple: 10 packets.
• min_time: 1 s.

We have also set the thresholds for PUBLISH attack detection (Algorithm 4 (Section 4.4)) as follows:

• max_packet: 10 packets
• min_time: 1 s

The detection results compared with the original pcap file are shown in

Table 4. MQTT-based attack detection results.

Sensor 1 (192.168.10.5)			Sensor 2 (192.168.10.6)
Attack Type	Total Packets (pcap)	Total Detected Packets	Attack Type	Total Packets (pcap)	Total Detected Packets
CONNECT	1000	999	CONNECT	1000	999
PUBLISH	66,274	66,274	PUBLISH	33,628	33,628

.

To calculate the detection accuracy, we adopted Equations (1) and (2) to calculate the detection accuracy of each attack detection, which can be expressed as follows:

CONNECT attack detection accuracy: 99.9%.

PUBLISH attack detection accuracy: 100%.

The detection results in Table 4 show that our proposed MQTT-based attack detection algorithm is able to detect PUBLISH attacks at 100% accuracy, while the CONNECT attack detection missed one packet for both sensors, resulting in 99.9% detection accuracy.

A chart of attack detection accuracy of our proposed algorithms against our own data generated from the testbed is shown in Figure 6.

The difference in attack detection accuracy between CONNECT and PUBLISH is due to their attack patterns. The CONNECT attack exhibits more complex behaviour, as it generates multiple illegitimate sessions without completing previous sessions to perform an attack on an MQTT broker. This complexity increases the likelihood of missed detection on the last illegitimate session if it does not meet the detection criteria or remains uncertain. In contrast, the PUBLISH attack occurs within a single active session after establishing a connection with an MQTT broker. Due to its simpler attack pattern compared to the CONNECT attack, it is able to be detected with high accuracy.

5.5. Evaluation with Publicly Published Datasets

We evaluated the performance of our proposed TCP- and MQTT-based attack detection algorithms using publicly available datasets. BoT-IoT [30] was selected to evaluate TCP-based attack detection. For MQTT-based attack detection, we used the CICIoMT2024 [51] and MQTTset [52] datasets to evaluate MQTT CONNECT and PUBLISH attacks, respectively.

The original BoT-IoT dataset includes injected SYN flood traffic, which we used to assess the detection accuracy of our TCP-based attack detection algorithm. For this evaluation, we analysed 5000 SYN flood packets from the dataset. The threshold values for Algorithm 2 (Section 4.3) were configured as follows:

• max_time: 1 s.
• max_packet: 5 packets.

The detection results are presented in

Table 5. Attack detection results against BoT-IoT dataset.

BoT-IoT (TCP-Based)
Attack Type	Total Packets (pcap)	Total Detected Packets
SYN	4960	4953

.

Our proposed TCP-based attack detection algorithm achieved a detection accuracy of 99.86% against the BoT-IoT dataset.

For MQTT attack detection, we conducted an evaluation using CICIoMT2024 and MQTTset. The CICIoMT2024 dataset includes MQTT CONNECT attack packets, while MQTTset contains MQTT PUBLISH attack packets. To assess the performance of our proposed MQTT-based attack detection algorithms, we applied the following threshold settings for MQTT CONNECT attack detection, as defined in Algorithm 3 (Section 4.4):

• max_single: 2 packets.
• max_multiple: 10 packets.
• min_time: 1 s.

For the MQTT PUBLISH attack detection evaluation, the thresholds were configured as follows:

• max_packet: 10 packets.
• min_time: 1 s.

The detection results are shown in

Table 6. Attack detection results against MQTT-based attack datasets.

CICIoMT2024			MQTTSet
Attack Type	Total Packets (pcap)	Total Detected Packets	Attack Type	Total Packets (pcap)	Total Detected Packets
CONNECT	144,390	143,041	PUBLISH	54,431	54,431

.

The evaluation of our MQTT-based attack detection algorithm against the CICIoMT2024 dataset for MQTT CONNECT attack detection achieved an accuracy of 99%, with 1349 packets missed. For the MQTT PUBLISH attack detection evaluation using the MQTTset dataset, our proposed algorithm achieved 100% detection accuracy. A chart of the attack detection accuracy of our proposed algorithms against publicly published datasets is shown in Figure 7.

The evaluation results against publicly available datasets may differ from our dataset collected from our testbed due to the variations in network testbed setup and communication behaviour. As shown in the evaluation, TCP-based attack detection using the BoT-IoT dataset achieved 0.14% lower accuracy compared to our dataset from our testbed, likely due to differences in the underlying communication.

For MQTT attack detection, the detection accuracy achieved from evaluating against both CICIoMT2024 and MQTTset datasets was similar to the accuracy achieved using our generated dataset in our own testbed. Specifically, the CONNECT attack detection achieved 99% accuracy, while PUBLISH flood attack detection reached 100%. However, for MQTT CONNECT attack detection, we had to adjust the ‘max_single’ threshold value to 2 to achieve maximum accuracy. When the threshold was set to 3, the detection accuracy dropped to 91%. This highlights that the threshold definition can significantly influence the detection accuracy of our proposed algorithms. As discussed in Section 4.3, the threshold may vary depending on the network deployment scenarios and the way devices communicate within the networks.

To achieve optimal detection performance, it is essential to first establish a baseline by collecting normal network traffic and analysing communication patterns. Based on this baseline, thresholds should then be fine-tuned according to the specific characteristics and requirements of the deployment environment.

6. Conclusions

This study proposes the implementation of an NBA system in MUD-based IoT networks for DoS/DDoS attack detection. The intention is to enhance the effectiveness of attack detection and reduce false-negative filtering of MUD architecture. We achieved this by designing a set of algorithms to group related traffic into network sessions to detect TCP- and MQTT-based DoS/DDoS attacks on MUD networks. To demonstrate its effectiveness, we collected benign and attack behaviour data from our testbed network and evaluated the detection accuracy of our designed algorithms on TCP- and MQTT-based attack detection. Evaluation results show that our algorithms achieve high detection accuracy on TCP- and MQTT-based DoS/DDoS attack detection. Currently, our focus is on TCP and MQTT, but this can be extended to support other network protocols.

The NBA system was developed with a focus on small-scale MUD network deployment. Large-scale deployment was beyond the scope of this study. Additionally, we have yet to evaluate detection accuracy on real-time traffic, as we evaluated the collected traffic during the data collection process. In this study, we also limited our focus to TCP- and MQTT-based attack detection.

In future work, we will evaluate our proposed algorithms in real-time traffic to determine the effectiveness of attack detection. We will also extend our NBA system to support a wider range of protocols, such as UDP and constrained application protocol (CoAP). Currently, the NBA system detects only DoS/DDoS attacks. Our future work will consider other types of attacks. Additionally, to improve the versatility of our algorithms, we will explore the implementation of an automatic threshold adjustment mechanism. Furthermore, we will work on creating MUD extensions to enhance the effectiveness of attack mitigation in IoT networks.