Machine Learning-Assisted Cryptographic Security: A Novel ECC-ANN Framework for MQTT-Based IoT Device Communication

Abstract

The Internet of Things (IoT) has surfaced as a revolutionary technology, enabling ubiquitous connectivity between devices and revolutionizing traditional lifestyles through smart automation. As IoT systems proliferate, securing device-to-device communication and server–client data exchange has become crucial. This paper presents a novel security framework that integrates elliptic curve cryptography (ECC) with artificial neural networks (ANNs) to enhance the Message Queuing Telemetry Transport (MQTT) protocol. Our study evaluated multiple machine learning algorithms, with ANN demonstrating superior performance in anomaly detection and classification. The hybrid approach not only encrypts communications but also employs the optimized ANN model to detect and classify anomalous traffic patterns. The proposed model demonstrates robust security features, successfully identifying and categorizing various attack types with 90.38% accuracy while maintaining message confidentiality through ECC encryption. Notably, this framework retains the lightweight characteristics essential for IoT devices, making it especially relevant for environments where resources are constrained. To our knowledge, this represents the first implementation of an integrated ECC-ANN approach for securing MQTT-based IoT communications, offering a promising solution for next-generation IoT security requirements.

1. Introduction

The Internet of Things (IoT) continues to be a dominant force and a significantly trending topic in networking, and extensive research has been conducted in this domain [1]. With the rapid growth in the global IoT user population, security challenges and privacy issues have intensified in communication networks. Because of the heterogeneous characteristics of IoT devices, ensuring privacy protection is a considerable challenge for any IoT application [2]. IoT primarily uses devices with restricted resources, characterized by limited processing capabilities and memory compared to traditional internet infrastructure. As a result, implementing security functionalities on these lightweight IoT devices poses significant challenges. Secure data transmission, along with maintaining privacy and trust, is a critical motivation for all IoT applications [3].

The rise in IoT has been driven by the advent of smart gadgets that can gather, process, and share data autonomously or interact with humans seamlessly. However, despite its numerous promises and potential, IoT deployment presents various security challenges due to its unattended nature and limited resources. Device-to-device (D2D) communications in IoT environments rely on several protocols, such as Data Distribution Service (DDS), Constrained Access Protocol (CoAP), and MQTT, to facilitate communication [4]. IoT protocols are integral to IoT technology as they enable devices to exchange data systematically and effectively. These protocols act as the backbone for extracting useful information from transmitted data and ensure a profitable deployment, particularly in the context of IoT device business management. Among these protocols, MQTT stands out due to its specific design for resource-constrained devices, minimal overhead, and quality of service (QoS) capabilities. However, MQTT’s default security characteristics are limited, as it primarily relies on plaintext authentication via username and password. This lack of encryption and vulnerability to attacks necessitates the implementation of additional security mechanisms [5].

Motivated by the pressing demand for robust IoT security, this research focuses on the development of a secure MQTT system to ensure reliable and protected communication in IoT environments. The IoT Security Foundation’s 2024 report [6] highlights a significant yet gradual improvement in IoT security practices, revealing that 35.59% of consumer IoT companies have now implemented a Vulnerability Disclosure Policy (VDP). This represents notable progress from 27.1% in 2023 and a substantial increase from just 9.7% in 2018. Despite this positive trend, nearly two-thirds of IoT companies still operate without formal vulnerability disclosure protocols, creating potential security risks in their device ecosystems. The findings underscore the critical need for wider VDP adoption to strengthen IoT security frameworks and build consumer confidence in connected devices, while a Palo Alto network report [7] states that 98% of data exchanges among IoT devices occur without encryption, revealing confidential data to potential attacks. Given these findings, the development of a secure MQTT protocol becomes crucial. This study addresses these challenges by integrating ECC and ANN into the MQTT protocol to improve security measures and enhance the overall efficiency of IoT systems.

This work introduces groundbreaking advancements in securing the MQTT protocol for IoT environments, addressing critical vulnerabilities with innovative solutions. At its core, the study implements a lightweight security framework leveraging ECC, ensuring encrypted message transmission that is accessible only to authorized subscribers. Beyond encryption, the research integrates machine learning into the broker node, enabling real-time detection and prediction of various attack types targeting the MQTT protocol. By training and evaluating multiple machine learning algorithms on a dataset encompassing five common attack types, the study reveals that ANN outperforms others, achieving a remarkable accuracy of 90.38%. This demonstrates the ANN’s superior capability in identifying and mitigating threats. Together, these contributions provide a comprehensive and robust security solution, overcoming the limitations of existing MQTT mechanisms and establishing the path for safer and trustworthy IoT communication.

When discussing IoT, the critical aspect is communication between devices. Cooperation between devices and user applications is the defining feature that makes IoT functional. Protocols are pivotal in enabling devices to communicate and interact, serving as the language through which devices exchange data. Although numerous customized protocols exist for implementing IoT environments, only a few have been standardized for all devices operating within IoT infrastructures [8].

Several studies have proposed solutions using various IoT protocols and approaches to enhance IoT security. For example, Ammar and colleagues [9] surveyed the security of IoT frameworks, highlighting the importance of lightweight key management for securing resource-constrained IoT devices. Hussain and colleagues [10] introduced a lightweight authentication protocol based on ECC, specifically designed to strengthen security in Internet of Drones (IoD) environments. It effectively addresses the challenges associated with resource-constrained devices by offering a secure solution that minimizes computational and communication overhead. Recognizing the growing prevalence of security threats in IoT, Khan and Salah [11] presented a comprehensive survey on IoT security challenges and mitigation strategies. Al-Garadi and colleagues [12] conducted an extensive survey on the application of machine learning and deep learning techniques to enhance IoT security. Their work highlights the effectiveness of these methods in addressing critical challenges such as anomaly detection, intrusion prevention, and malware classification. The authors emphasize the adaptability of machine learning in handling the dynamic and heterogeneous nature of IoT ecosystems. This study underscores the growing importance of intelligent, data-driven approaches in securing IoT networks against evolving cyber threats.

Hybrid cryptographic frameworks are increasingly utilized in resource-constrained environments like the Internet of Things (IoT) to balance robust security with minimal overhead. These systems often employ a two-pronged approach, using Elliptic Curve Cryptography (ECC) for key exchange due to its smaller key sizes and lower computational demands compared to RSA, and the Advanced Encryption Standard (AES) for efficient bulk data encryption. The effectiveness of this model is commonly evaluated based on performance metrics such as encryption/decryption latency, memory footprint, and energy consumption, all crucial for embedded platforms [13]. Simultaneously, to enhance system security beyond mere data encryption, some frameworks integrate the Message Queuing Telemetry Transport (MQTT) protocol with advanced access control mechanisms, like User-Managed Access (UMA). These models prioritize policy-based access control and secure session management over the optimization of cryptographic overhead, ensuring that only authenticated and authorized entities can interact with the system’s topics and data streams [14].

Other notable contributions in IoT security have proposed innovative approaches for securing IoT frameworks and devices. For instance, Aouedi and colleagues [15] provided a comprehensive survey on intelligent IoT, exploring its applications, security concerns, privacy issues, and future research directions, offering valuable insights into the evolving landscape of IoT technologies. While Szymoniak and Kesar [16] provide a comprehensive survey on key agreement and authentication protocols in IoT, highlighting the critical role of secure communication in IoT environments, and Seoane and colleagues [17] evaluated the effectiveness of MQTT and CoAP protocols in the IoT ecosystem with security features, focusing on their efficiency under constrained conditions. Ani and colleagues [18] evaluated the security of the MQTT protocol in IoT systems, highlighting key vulnerabilities and proposing measures to enhance its resilience in resource-constrained environments. These advancements highlight the necessity of adaptive and innovative approaches to protect IoT ecosystems from evolving threats. Collectively, these findings highlight the importance of resilient and scalable security solutions in IoT networks.

Despite all the above advancements, significant gaps remain in addressing the security of pub-sub systems, particularly MQTT. Most previous studies focused on binary classification, identifying normal and abnormal conditions without specifying attack types. Moreover, traditional protocols like Rivest Shamir Adleman (RSA) are unsuitable for IoT systems due to their computational overhead, making ECC a more viable alternative. The purpose of this work is to close these disparities by addressing a multi-class classification problem, distinguishing between normal records and multiple attack types in the MQTT protocol. As far as we are aware, this is the first work to put together ECC with ANN for securing MQTT-based IoT communication, offering a robust and efficient solution that addresses both data encryption and real-time threat prediction. This approach significantly enhances the security of IoT ecosystems, providing better network management while adhering to the resource constraints of IoT devices.

2. Design and Methodology

After having decided on the MQTT protocol as the IoT protocol for the solution of our research problem, the two key aspects of this work came into place, i.e., security and lightweight communication; both have to be considered for the Protocol to run smoothly over IoT devices. Research was performed in this area to analyze and evaluate the related work which has been performed on the MQTT protocol regarding security. As stated before, MQTT is a very lightweight protocol, and one of the aims of this work is to keep it as lightweight as possible by providing, at the same time, secure communication between the IoT devices using the MQTT protocol without adding much more overhead, which would make this Protocol not lightweight.

As highlighted earlier, IoT devices encounter a variety of issues and constraints that underscore the need for secure communication techniques. The inherent resource limitations of these devices make it challenging to deploy comprehensive security solutions. For instance, although the TLS protocol has been adopted to ensure secure communication, its significant overhead can be impractical for devices with limited processing power and memory.

This work focuses on developing the MQTT protocol through the paho-mqtt library available in Python 3.12.7. The Python-based script comprises various functions essential for the implementation, testing, and deployment of the MQTT-based ECC protocol as a solution to establish the encryption of data exchanges between MQTT clients and the broker, as well as an ANN model deployed in the broker node to identify and predict the status of every connected device. The publish/subscribe architecture of MQTT requires asynchronous client-to-broker communication. Except in the event of denial of service or flooding attacks, which seek to deceive or compromise the system by depleting the broker, the design guarantees end-to-end encryption, ensuring that sensitive information sent by the broker is encrypted and that data cannot be compromised by the attacks.

2.1. Overall System Description

Each device connected to the system sends the request link packet to the broker as a network packet; from the connect packet, the broker will analyze the network traffic using the ANN model to identify whether the client is an attacker. If yes, the system administrator will be required to perform a security action before harming the system. Otherwise, no security action is needed, and the client will perform the desired action; accordingly, if it is the publisher, it will send the encrypted data using the public key. Moreover, the subscriber will decrypt the data with the support of its private key (see Figure 1).

This design intends to reduce the amount of MQTT messages the device needs to send and receive, the processing memory, and the bandwidth overhead associated with encryption on devices with limited resources. The developed system employs asymmetric-key data encryption to guarantee secrecy and integrity to the message flow between IoT devices, with an ANN model assisting in threat detection and identification. Figure 1 depicts the MQTT architecture used in our framework. In this model, IoT devices act as publishers or subscribers, exchanging messages through a central broker. The broker is responsible for receiving messages from publishers, filtering them based on topics, and forwarding them to the appropriate subscribers. A topic is a hierarchical string that categorizes messages and serves as the logical channel for communication. In practical deployments, the broker can be implemented either as an on-premises server or as a cloud-based service; however, for simplicity, it is represented here as a central entity without specifying its physical location.

For enhanced security, the system incorporates authentication, access control, and key management through a dedicated Key Management Service (KMS) (see Figure 2).

A high-level design of the system is portrayed by the architecture in Figure 1, which identifies the key elements and their interdependencies. It illustrates a traditional MQTT architecture in which the central broker facilitates asynchronous communication between publishing and subscribing devices. Furthermore, KMS is a publisher and subscriber that offers a management service to the system’s devices. Given MQTT’s centralized structure, every device relies on the broker for communication.

The ECC-ANN framework indirectly reduces MQTT traffic by optimizing the cryptographic processing and decision-making at the edge device level. By performing lightweight pre-filtering and decision-making locally, the device only publishes necessary messages, thereby reducing the number of MQTT publish events. Additionally, the ECC-based encryption is integrated efficiently, minimizing the payload size while maintaining strong security, which further contributes to lowering overall network traffic.

The Key Management System (KMS) in our framework is a trusted entity responsible for generating, storing, and securely distributing symmetric keys among authorized parties. The key exchange process occurs over secure channels to ensure confidentiality and integrity during transmission. By centralizing key management, the KMS simplifies the distribution process and reduces the risk of key compromise, enabling efficient and secure communication between IoT devices and other components in the system.

Since asymmetric-key encryption is applied, the data is encrypted and decrypted using a pair of keys. In addition to offering other necessary features that eliminate the requirement for local device performance, this service offers such functionality in a safe and manageable manner. It complies with the asynchronous nature of the system that works perfectly with the MQTT paradigm by communicating and running concurrently with the broker. Furthermore, the KMS continuously monitors and analyses all traffic within the system, enhancing overall security and operational oversight. Figure 2 shows how logically our Protocol encrypts and decrypts messages using key exchange, then in the following section, we are going to show how ECC executes this process.

2.2. Elliptic Curve and Key Exchange

The square root of a cubic equation over a prime finite field (F_p) usually defines a two-space graph known as an elliptic curve (E). In mathematics, an elliptic curve is extracted from the difference between the square function and the cubic function, equalized to a linear function C, as follows:

$$y^2- x^3 = c where c=ax+b$$

$$y^2-x^3=ax+b$$

(1)

$$y^2=x^3+ax+b$$

Finally, an elliptic curve is described by a collection of points defined by the solution to the following equation:

$$E\left(F_p\right)=\{\left(x,y\right)|y^2=x^3+ax+b\}$$

(2)

where a, b, and x are part of F_p in addition to a special point known as the point of infinity. Identity operations for points on the curve are offered by the point of infinity (O).

$$a,b,xandy\in F_p$$

(3)

Additionally, it is essential to satisfy a condition that guarantees that equation E has no repeated roots, ensuring the curve remains non-singular. The following equation, which is a standard criterion that must be fulfilled, describes this condition:

$$4a^3 + 27b^2 \ne 0$$

(4)

To formulate the discrete logarithm problem (DLP) using elliptic curves, a sufficiently large cyclic group of elements is necessary. This begins by identifying the group elements as the set of points that satisfy the elliptic curve equation. Next, the group operations are defined on these points. As we will describe in the following part, point addition and point doubling constitute group operations on an elliptic curve.

Point doubling is the process of adding the same point on the curve to itself, whereas point addition is the addition of two distinct points on the curve.

Plotting the elliptic curve equation gives us a symmetric graph to the x-axis. If we draw a line on the graph, it will touch a maximum of 3 points.

Two distinct points give the point addition of Elliptic Curve E over the prime field F_p:

$$P\left(x_1,y_1\right)\in E\left(F_p\right)andQ\left(x_2,y_2\right)\in E\left(F_p\right)$$

(5)

Such that Q ≠ P, and the point addition are given by $Q+P= R\left(x_3,y_3\right)$ Here, the coordinates of the third point R on the curve are calculated by adding two points, P and Q:

$$\left(x_3,y_3\right)=\left(x_2,y_2\right)+\left(x_1,y_1\right)$$

$$x_3=m^2-x_2-x_1\left(modp\right),$$

(6)

$$y_3=m\left(x_1-x_3\right)-y_1\left(modp\right)$$

$$m=\left(y_2-y_1\right)/\left(x_2-x_1\right)\left(modp\right)$$

The other group operation on an elliptic curve is point doubling. Point P was added to itself throughout this operation. A tangent line was traced through the curve using this technique. The second point is where the curve and the sketched tangent line cross.

Point doubling $P+P=2P$

For the point doubling, we only have one point $P \left(x_1, y_1\right) \in E\left(F_p\right)$

Point P is then replicated to obtain the outcome, which appears as $2P = P + P:$

$$\left(x_3,y_3\right)=\left(x_1,y_1\right)+\left(x_1,y_1\right)$$

$$x_3=m^2-2x_1\left(modp\right)$$

(7)

$$y_3=m\left(x_1-x_3\right)-y_1\left(modp\right)$$

$$m=\left(3x_2^1+a\right)/2y_1\left(modp\right)$$

Here, m refers to the tangent line’s slope across point P.

For encryption, we have to choose one random positive integer K. Based on that value of K, we will calculate the cipher point C_m that we will send to the receiver (User B).

$$C_m=\left\{KG,P_M+K{P}_B\right\}$$

(8)

C_m is the cipher point for the plain text P_M, where G represents the base point generator of the elliptic curve, K is the point on the curve which is randomly chosen uniformly from the range of 1 ≤ K ≤ n − 1 where n is the order of the base point G on the elliptic curve, and P_B is the public key of user B (receiver), as we saw in the previous step. After calculating the cipher point, it is delivered to the receiver. In the next section, we will show how the receiver performs the decryption process.

For decryption, at the receiver side, we have the cipher point C_m sent by user A:

$$C_m=\left\{KG,P_M+K{P}_B\right\}$$

where KG is the ephemeral key component and P_M + KP_B is the masked message point.

We multiply the x-coordinate of the cipher point C_m by the receiver’s private key N_B:

$$KG \times N_B$$

Then subtract KG × N_B from the y-coordinate of the cipher point C_m:

$$P_M+K{P}_B-\left(KG\times N_B\right)$$

(9)

Previously, we have seen that:

$$P_B= N_B\times G$$

(10)

Let us replace $N_B\times G \mathrm{with} P_B$ in the previous equation to obtain the following:

$$P_M+\left(K{P}_B-K{P}_B\right)=P_M$$

P_M = P_M

The original message point P_M is obtained by subtracting KP_B from the second component of the cipher. This ensures that only the intended recipient, who knows the private key N_B, can retrieve the original message P_M. The equation effectively demonstrates the correctness of the ECC decryption process.

Finally, the receiver or user B obtains the same P_M point as the sender, which is the primary purpose of the ECC algorithm.

ECC operates on the irreversibility principle, just like RSA. In essence, it is simple to calculate in one direction but extremely difficult to reverse it and return to the starting place. A number in ECC, which is multiplied by another number to produce another point, represents a point on the curve. You have to locate the new point on the curve to solve this puzzle. Although you know the initial point, it is nearly impossible to discover a new one due to the mathematical structure of ECC, making it highly secure.

The above process clearly shows how ECC works to prevent unauthorized users from accessing the data on the MQTT protocol. However, while the data is protected, we must also protect the system from unknown attackers, known as anomaly detection. Machine-learning algorithms will help us identify them.

2.3. Machine Learning Algorithm

In machine learning, an algorithm consists of a collection of steps that are applied to data in order to generate a model, which means that the algorithm turns a dataset into a model. Technically, given the input variable (X), the machine learning algorithm predicts the output variable (Y) by estimating the target function (f). The three main categories of machine learning algorithms are reinforcement learning, unsupervised learning, and supervised learning. Its interdisciplinary nature enables the development of powerful algorithms capable of addressing complex problems in various domains based on prediction and pattern discovery [19].

In supervised learning, the intended output of the model is known, although the underlying relationships within the data remain unclear. Training involves using one dataset to teach the model and another to evaluate its performance. In unsupervised learning, neither the model nor its intended output is predefined; instead, the model seeks to uncover patterns and similarities within the data, grouping them accordingly. Reinforcement learning aims to improve the model through a trial-and-error process, where it receives rewards based on its output. These rewards help the model adapt and improve by determining a value that signifies how well it performed in making decisions. Numerous machine learning techniques are widely employed in cybersecurity to detect network discrepancies and they have been shown to deliver high prediction rates [20]. The ML model is built using these techniques on the training dataset, and it is then examined for any malicious activity against the provided dataset. We refer to this kind of learning as supervised learning [21]. Popular supervised machine learning techniques, such as Decision Tree [22], Random Forest [23], Gradient Boost [24], Naive Bayes [25], Artificial Neural Networks [26], and Multilayer Perceptron [27], are typically used in machine learning mitigation in the MQTT protocol.

We have implemented and analyzed some algorithms individually to develop a machine-learning model, which can identify and predict the type of attacker. Finally, we have made a comparison between them. Before implementation, we observed the kind of data, and data preprocessing was conducted on the dataset. Cleaning, visualization, and feature engineering are the procedures that make up data preprocessing. These procedures transformed the data into feature vectors, which were subsequently divided into a 70% training set and a 30% testing set. The training set was utilized in the learning algorithm to develop a final model using various classifiers. Finally, the developed model was evaluated on the testing set using multiple evaluation metrics. (see Figure 3).

This work has proven that ANNs have the highest capacity for modeling and learning complex and non-linear relationships. Since many of the relationships between inputs and outputs in real life are complex and non-linear, this is crucial. Additionally, ANN does not place any limitations on the input variables (such as how they should be distributed), in contrast to many other prediction methods. The neural network architecture is shown in Figure 4.

Neural networks are an effective learning model that draws their core principles from the analogy of biological neurons. In some of the documents, ANN is also called “artificial neural systems,” “parallel distributed processing systems,” or “connectionist systems.” ANN consists of a large number of interconnected units arranged in patterns that facilitate communication among them. These units, known as neurons, are basic processors that function in parallel. Artificial neurons are conventionally structured into layers: the input layer, the hidden layers, and the output layer (see Figure 4). The input layer consists of the features of the dataset that you aim to train your neural network on, where the number of features corresponds to the number of input layers, and the output layer consists of the number of target variables, notably classes. Hidden layers rely on input and output layers, determined using the Trials and Errors concept. The most critical component of the ML model is the data. So, let us look at the dataset we used in this work.

2.4. Dataset

The efficiency of any ML model is largely determined by the quality of the dataset. The dataset used for training the algorithm must be fed into an ML algorithm to build the model, and then the accuracy of this model is ensured by the test dataset. The more data fed to the ML algorithm, the more precisely the model can learn and improve. Hence, providing the dataset’s quality is essential in our work to effectively and efficiently identify and predict the type of attack facing our system. Implementation of Supervised Machine Learning algorithms has been used in research for several years, mainly to improve the security of IoT. A common challenge highlighted in previous research is that many authors had to build their own datasets in order to train and test algorithms. To address this, Vaccari et al. [28] introduced and released MQTTset, a publicly available dataset designed for research purposes. This initiative eliminates the need for researchers to develop their own datasets, making it easier to study machine learning approaches for predicting and mitigating attacks on MQTT systems while enabling more consistent and comparable results.

The MQTTset dataset was created to incorporate both malicious and legitimate traffic. The authors used eight distinct sensors: a door lock, humidity, fan speed, temperature, smoke detector, CO-gas, motion sensor, and light intensity all are communicated via the MQTT protocol to create the MQTTset dataset. Two rooms were used to accommodate those eight sensors; then, the authors recorded the traffic in the MQTT protocol. As expected, the MQTT protocol was subjected to attacks, leading to the generation of both legitimate and malicious traffic records. Brute Force Authentication Attacks, SlowITe, and DoS attack variants were among the malicious traffic. Sensors and the broker communicate in a restricted access area, both physically and digitally, where the sensor network is deployed. The broker itself is the source of the network traffic. Rather, in order to carry out the cyberattacks, the rogue node is directly connected to the broker during the attack phases. Because of the nature of the chosen assaults, the attacker node’s location inside the network is unnecessary because its goal is to target the MQTT broker.

Based on the type of sensor, each one is set up to initiate communication at a particular moment. For example, at predetermined intervals, such as once an hour, a temperature sensor may provide data on the ambient temperature. However, a motion sensor only uses the network to communicate when it detects mobility. Every sensor is configured with a topic that the MQTT broker uses and a data profile. The information used by the sensors, such as the ranges that humidity or temperature sensors use or the directives that door lock sensors follow, makes up the data profile. Rather, the topic is the name of the medium that distributes or receives data. Furthermore, some of the network-connected sensors have subscriber features that allow them to obtain data that has been shared on the network in addition to transmitting data. A packet capture (PCAP) file, which was recorded during the creation of MQTTset data, represents the created MQTT traffic. The MQTTset comprises real attacks executed against the targeted MQTT broker, along with additional PCAP files that can be used to validate the multi-class classification model within the dataset. Figure 5 illustrates the correlation matrix of the selected features, highlighting the relationships and dependencies among them. This visualization provides a clear understanding of how each feature interacts with the others, assisting in the identification of patterns within the dataset. Data correlation is the visualization of the dataset according to how data are distributed among different classes.

2.5. Data Preprocessing

The dataset underwent a rigorous preprocessing and feature engineering pipeline to ensure robustness and compatibility with machine learning algorithms. Hexadecimal values present in network protocol fields (e.g., tcp.flags, mqtt.hdrflags) were converted to their integer equivalents through base-16 decoding, while categorical attributes were numerically encoded using label encoding techniques. Missing values were systematically imputed with zeros to preserve dataset integrity, and the target variable for multi-class attack classification was similarly encoded. Subsequently, all features were standardized via Z-score normalization to eliminate scale disparities and ensure equitable contribution during model training. To enhance model generalizability and reduce computational complexity and overhead, feature selection was employed to retain the most informative predictors, thereby mitigating redundancy. Finally, the processed dataset was partitioned into training and testing subsets in a 70:30 ratio to facilitate robust performance evaluation. This comprehensive data preprocessing pipeline ensured that the data was optimally prepared for analysis and model development while preserving critical patterns in network traffic behavior.

3. Implementation and Results

Through implementation, we briefly describe the organizational environment in which we implemented our system. We have read various documents detailing the security and challenges that IoT devices face, as mentioned in the related literature section. This section consists of the implementation and results discussion of our MQTT encryption and decryption of the data, alongside machine learning algorithms to classify all incoming IoT devices. We explain the software and hardware used to implement our system and the implementation details. Firstly, the implementation follows the diagram represented in Figure 6. It shows that only the publisher and subscriber can encrypt and decrypt the data, respectively, and the broker hosts and manages all the protocol’s activities.

3.1. Implementation Details

We have used the VirtualBox tool to simulate our implementation by creating four virtual machines: Publisher, subscriber, broker, and attacker. After that, we used Python 3.12.7 to develop and test our MQTT protocol and found it working correctly, as shown in Figure 7 and Figure 8. However, the attacker can access the data being transmitted by our Protocol. That is why we integrated ECC into our Protocol to encrypt the data. The last part was carried out by developing a machine-learning model for the broker to identify and predict the type of device connected to the broker. Furthermore, we have represented the system hardware, software, and tools that we have used to implement our Protocol.

Hardware specifications:

• OMEN by HP Laptop
• 1 TB of Hard Disk and 512 GB of solid-state drive (SSD)
• Processor: Intel(R) Core (TM) i7-7700HQ CPU @ 2.80 GHz (8 CPUs)
• RAM: 16,384 MB

Software specifications: We used an updated version of Windows, a VirtualBox, to create and manage virtual machines, and Python 3.12.7 to develop our Protocol and machine learning model. After setting up our working environment and implementing the proposed approaches, we will discuss the outcome results in the next part.

3.2. Results Discussion

In this part, we explain several screenshots of the outcomes gained during the implementation phase of our Protocol and discuss them. We executed the proposed schemes and mathematical model in the Python 3.12.7 programming language. We proposed several scenarios to verify and analyze the proposed schemes. We started by developing the MQTT protocol without security. Now, publishers and subscribers can exchange data insecurely.

Messages sent by the publisher: After designing our model and algorithm, we jumped into Python 3.12.7 to write scripts that could implement our idea. Figure 7 showcases a screenshot of sample messages we used to evaluate the communication of the publisher device in real time using our custom MQTT protocol developed in Python 3.12.7 programming language with the support of PyCharm (2021.1.3) as an integrated development environment (IDE) and the paho-mqtt library.

Messages received by subscriber: The subscriber client, subscribed to the same topic as the publisher via the broker, successfully receives all messages published by the publisher device. Figure 8 shows a screenshot of the exact messages delivered through the broker, confirming real-time communication.

At this level, our Protocol is vulnerable because no security is implemented, and an attacker can easily sniff the data passing on it by using different sniffing tools such as Wireshark. In the next part, we are going to use an attacker machine with the help of Wireshark as a sniffing tool to read all the messages passing on this Protocol.

As mentioned earlier, MQTT is not able to establish security by itself. That is why we designed a cryptographic framework to secure our Protocol. The attacker uses the Wireshark tool to sniff the communication of unsecured systems and can access the messages passing between the publisher and the subscriber through the broker. The sniffed message by the attacker is “IOT IS AWESOME”. With the help of ECC, our data will be encrypted, and the attacker will not be able to read the data in our system because the data is encrypted, and only authorized clients are able to decrypt the data.

We can ensure that our Protocol is well protected from attackers to access the data in our system. Thus, our data is fully encrypted by the ECC algorithm. Even if the data is secured, we also need to secure our system against those attackers who block our system or slow down our network performance. After determining that ECC is lightweight, we also need to evaluate our ANN model and compare it with other well-known machine learning models.

4. Ann Model Evaluation

In this part, we address the findings obtained during the assessment of our ANN model and provide insight into its accuracy, F1 score, and confusion matrices. A confusion matrix is a table commonly used to evaluate how well a classification model performs on a test dataset with known true values. Accuracy is the ratio of correct predictions to the total number of inputs. Moreover, the F1 score is a weighted average of precision and recall. Given that the MQTTset dataset contains various types of MQTT traffic, the system addresses a multi-class classification problem. It has successfully managed to not only detect the presence of an attack but also identify and predict the exact type of attack.

Table 1. Hyper-parameter configuration.

Number	Hyperparameters	Settings
0	Input layer	33
1	Hidden layer	3
2	Hidden neurons	100 (50, 30, 20)
3	Output layer	6
4	Batch size	256
5	Epochs	50
6	Lost function	Sparse_categorical_crossentropy
7	Activation function in the hidden layer	ReLu
8	Activation function in the output layer	Softmax
9	Learning rate	0.001
10	Optimizer	Adam

shows the final coefficients that give the highest results after trying multiple coefficients while tuning different hyperparameters by using the trial and error method:

In this study, ANN hyperparameters were initially selected using a trial-and-error approach to establish a proof-of-concept. We acknowledge that this method is not fully systematic. Future work will incorporate automated hyperparameter optimization techniques, such as grid search and Bayesian optimization, to improve model performance and provide a more rigorous and reproducible evaluation.

4.1. Performance Evaluation

The confusion matrix represented in

Table 2. Performance evaluation matrix.

	Legitimate	Dos	Bruteforce	Malformed	SlowITE	Flood
legitimate	3807	473	0	19	52	0
Dos	282	35,084	0	3650	60	1
bruteforce	2	4	89	89	0	0
malformed	49	2818	0	46,756	14	2
SlowITE	1490	246	0	297	1245	0
flood	0	1	0	0	0	2760

indicates the strong performance of our model, highlighting reliable and accurate results achieved through hyperparameter tuning. This reflects the efficiency of the model and its capacity to deliver precise predictions.

Table 3. Classification report.

	Precision	Recall	F1-Score	Support
brute force	0.68	0.87	0.76	4351
DoS	0.91	0.90	0.90	39,077
Flood	1.00	0.48	0.65	184
Legitimate	0.92	0.94	0.93	49,639
Malformed	0.91	0.38	0.54	3278
SlowITe	1.00	1.00	1.00	2761
accuracy			0.90	99,290
Macro avg	0.90	0.76	0.80	99,290
Weighted avg	0.91	0.90	0.90	99,290

represents a comprehensive breakdown of key performance indicators for our ANN model. It includes precision, recall, F1-score and accuracy, offering a comprehensive evaluation of the capacity of the model to make accurate predictions.

While the overall performance of the ANN model is strong, we acknowledge that recall for certain attack types, particularly “malformed” (0.38) and “flood” (0.48), is notably lower than for other classes. This limitation is largely due to class imbalance in the MQTTset dataset, where these categories are underrepresented compared to “legitimate” and “DoS” traffic. As a result, the model tends to prioritize majority classes during training, leading to reduced sensitivity to minority attack types. Addressing this challenge will require strategies such as class balancing, oversampling, data augmentation, or cost-sensitive learning to improve detection performance on these rare but critical threats. We highlight this as an important direction for future work to strengthen the robustness of the ECC–ANN framework across all attack categories.

Figure 9 indicates that loss is constantly decreasing over time, which proves that our ANN is working perfectly. A number of experiments were carried out to evaluate our model’s performance. With an accuracy, F1 score, and Macro F1 score ratio of 90.38%, 90.09%, and 84.52%, respectively. These experiments confirmed that the model’s loss had stabilized well before reaching the maximum epoch count, and the reported accuracy (~90.38%) remained consistent across multiple runs, demonstrating that the original 8 epochs were sufficient for early convergence.

The assessment results of the classifier adopted in the MQTT attack prediction framework are displayed.

4.2. Results Comparison

ANN and other algorithms are compared to justify the proposed algorithm’s effectiveness. To do so, we developed four more models based on four different algorithms, notably “Decision Tree (DT), Gradient Boost (GB), Naïve Bayes (NV), as well as Random Forest (RF)”. Numerous experiments were carried out to measure and compare the performance of our classifier and some of the well-known classifiers used in the identification and prediction frameworks. The accuracy and F1 score evaluation results of the classifiers utilized in the MQTT attack prediction framework are displayed. The results of the models in all investigations are reported in

Table 4. Accuracy and F1 comparison.

Algorithms	Accuracy	F1 Score	Macro F1 Score
Neural Network	0.9038271729277872	0.9009158054232087	0.8452
Decision Tree	0.9031322388961628	0.9009088402539802	0.7813
Random forest	0.9029308087420687	0.9009131314572056	0.8104
Gradient Boost	0.7931513747608017	0.8268049294767623	0.8066
Naïve Bayes	0.670863128210293	0.7581608917548786	0.6784

, along with the accuracy and F1 score of the particular ML algorithms.

The outcomes achieved differ in terms of F1 score, Macro F1 score, and accuracy. Our model achieved the highest accuracy ratio, Macro F1 score, and F1 score, with values of 90.38%, 84.52% and 90.09%, respectively. Naïve Bayes represents the weakest outcomes, with accuracy, Macro F1 score, and F1 scores of 67.08%, 67.84% and 75.81%, respectively (see Figure 10 and Figure 11).

While overall accuracy and F1 score are commonly reported in classification tasks, they can be misleading in imbalanced datasets. In our case, the “legitimate” and “DoS” classes dominate the dataset, contributing disproportionately to the accuracy metric due to their large support. This dominance risks obscuring the model’s performance on minority classes such as “bruteforce,” “malformed,” “SlowITE,” and “flood”.

To address this, we emphasize the use of the macro F1 score, which treats all classes equally regardless of their frequency. Unlike weighted averages, which scale performance by class support, macro F1 provides a more balanced view of the classifier’s effectiveness across all categories. This is particularly important in security-related applications, where detecting rare but critical attack types is essential.

For instance, our best-performing model (Neural Network) achieved a macro F1 score of 0.8452, indicating strong performance across both dominant and minority classes. A comparative analysis of macro F1 scores across all evaluated algorithms is presented in Figure 12, highlighting the relative strengths of each approach beyond accuracy alone.

To ensure the robustness of our findings, we repeated the training and evaluation process for both the Artificial Neural Network (ANN) and Decision Tree (DT) classifiers across 10 independent runs using different random seeds.

Table 5. Comparative performance of ANN and DT with statistical significance testing.

Model	Mean Accuracy (%)	Standard Deviation	95%CI (±)	p-Value (Paired t-Test)
ANN	90.42	0.05	±0.03	0.012
DT	90.29	0.06	±0.04

summarizes the mean accuracy, standard deviation, and the results of a paired t-test comparing the two models.

ANN consistently outperformed the Decision Tree across all runs, with a mean accuracy improvement of 0.13 percentage points. Although the absolute difference appears small, the paired t-test confirms that this improvement is statistically significant (p < 0.05). Therefore, we conclude that the ANN provides a modest but statistically reliable advantage over the Decision Tree for this multi-class attack classification task.

4.3. Computational and Memory Overhead Analysis

To validate the lightweight nature of ECC-ANN under resource-constrained conditions, we measured its computational and memory overhead on the same experimental setup described in Section 3, where four virtual machines (VMs) simulated IoT devices. Each VM was configured with 1 vCPU (2.0 GHz), 1 GB vRAM, and Ubuntu 20.04 LTS, reflecting the resource limitations typical of edge IoT nodes. The MQTT broker was hosted on a separate VM with 2 vCPUs and 2 GB vRAM. Metrics included model size, peak RAM usage, CPU utilization, and average inference latency, which were recorded using psutil and Linux system monitors during inference. Each experiment was repeated five times, and mean values are reported. The summarized results are presented in Figure 13.

On simulated IoT devices, the performance of the proposed ECC-ANN model was evaluated against a standard ANN and a Decision Tree. The results demonstrated that ECC-ANN significantly reduced resource consumption compared to the baseline ANN, exhibiting a 31% decrease in memory footprint (57 KB vs. 82 KB), a 40% reduction in RAM usage (35 MB vs. 60 MB), and a 27% lower CPU utilization (19% vs. 26%). Furthermore, ECC-ANN achieved a 28% improvement in latency (11.7 ms vs. 16.2 ms). While the Decision Tree model remained the most resource-efficient with metrics of 82 KB, 57 MB, 21% CPU, and 8.6 ms, the findings confirm that ECC-ANN provides a favorable balance between computational efficiency and predictive performance. This establishes its viability for deployment in resource-constrained IoT environments.

4.4. Length Key Comparison Between ECC and RSA

Table 6. ECC and RSA key sizes comparison.

Security Level (bits)	ECC Key Length (bits)	RSA Key Length (bits)
256	512	15,360
192	384	7680
128	256	3072
112	224	2048
80	160	1024

compares ECC and RSA based on key length and corresponding security levels. ECC achieves equivalent cryptographic strength with significantly smaller keys than RSA, enhancing efficiency and reducing computational overhead. For instance, a 256-bit ECC key provides security comparable to a 3072-bit RSA key. This efficiency makes ECC particularly suitable for resource-constrained environments like IoT. The comparison highlights ECC’s advantage in delivering strong security with minimal resource consumption [29].

One of the standout advantages of ECC is its ability to generate significantly shorter keys compared to RSA. This compact key size brings a host of benefits, including streamlined data management, reduced hardware demands (such as smaller buffers, less memory, and minimal data storage), lower bandwidth consumption when transmitting keys across networks, and extended battery life in devices where efficiency is crucial, like mobile phones and various IoT devices.

4.5. Novelty and Comparative Analysis

Prior studies on IoT security have primarily focused on hybrid cryptographic schemes that combine symmetric and asymmetric algorithms, such as AES with ECC, to achieve confidentiality and integrity with minimal resource overhead on constrained devices. These solutions demonstrate efficiency in key exchange and encryption but do not incorporate intelligent threat detection mechanisms. Similarly, MQTT security enhancements, including UMA-based frameworks, emphasize access control and policy enforcement rather than integrated anomaly detection. In contrast, the proposed ECC-ANN framework introduces a dual-layer defense by coupling lightweight ECC-based key management with an optimized ANN for real-time attack classification. This integration ensures both secure MQTT communication and proactive intrusion detection while maintaining low computational and memory overhead, thereby addressing gaps in existing hybrid approaches.

5. Conclusions

Our experimental findings demonstrate a robust, dual-layer security framework that effectively shields MQTT-based IoT communications from multiple attack vectors. While the integration of ECC provides a formidable first line of defense against eavesdropping by encrypting data streams, we discovered that encryption alone was insufficient, as attackers could still execute denial-of-service attacks by flooding the broker with connection requests. This vulnerability stems from the broker’s inherent inability to distinguish between legitimate and malicious connection attempts, prompting us to develop an innovative artificial neural network (ANN) model for real-time threat detection. Our multi-classification model serves as an intelligent gatekeeper, analyzing connection patterns to identify and categorize potential threats. When tested against the five most prevalent attack types in our comprehensive dataset, the ANN model achieved remarkable performance metrics with accuracy of 90.3827%, Macro F1 score of 84.5207% and F1 score of 0.9009%, validating our approach of combining cryptographic security with machine learning-based threat detection to create a more resilient IoT communication system. This approach represents a significant step forward in safeguarding IoT networks against evolving cyber threats.

Looking ahead, we envision several strategic enhancements to fortify our security framework. Our primary focus is on expanding the neural network model’s capabilities by enriching our dataset with a broader spectrum of attack signatures. However, the practical implementation of our approach faces several noteworthy challenges in real-world scenarios. A significant hurdle lies in the current infrastructure landscape, where many server platforms and control panels have yet to incorporate support for ECC SSL/TLS certificates. Furthermore, our current attack identification model faces inherent limitations in its detection scope. To address this, we propose developing a comprehensive, continuously evolving dataset that not only encompasses a vast array of existing attack patterns but also maintains adaptability to emerging threats. This dynamic approach ensures our system remains resilient against the ever-evolving landscape of cybersecurity challenges.

Although this study focused on key length and theoretical security strength, it focused on evaluating the proposed ECC–ANN framework under simulated network conditions and benchmark datasets. Practical performance aspects such as encryption/decryption time, throughput, and energy consumption of IoT hardware were not evaluated. These factors are crucial for real-world deployments, and future work will include empirical testing on representative IoT platforms to provide a more comprehensive assessment of algorithm suitability.

Moreover, while the proposed ECC-ANN framework enhances the cryptographic and predictive security of MQTT-based IoT communication, the integration of machine learning also introduces its own security considerations. In particular, adversarial attacks targeting the ANN model, such as evasion attacks, where maliciously crafted inputs are designed to bypass detection or classification, represent a potential limitation. Future work should address more robust training techniques (e.g., adversarial training, defensive distillation, and ensemble learning) and explore resistance against advanced adversarial machine learning attacks (e.g., evasion attacks).

It is also worth noting that while this dataset provides realistic MQTT-based traffic patterns, we agree that testing on additional datasets such as UNSW-NB15 or Bot-IoT would further validate the generalizability of our approach. Due to scope and resource constraints, this was not included in the current study but is highly recommended for future work.