1. Introduction
Smart devices have become a part of our daily life. Some examples include intelligent light bulbs, motion sensors, cameras, and vacuum cleaners in smart home devices. This is because, through them, we can manage and control the various aspects of our homes in real time from a remote location. Smart devices can also improve efficiency and quality of life in smart cities (for example, intelligent street lighting, traffic, or waste management systems). In industry, intelligent devices help monitor and optimise production processes and automate production lines. Health care (such as heart rhythm control), agriculture (such as soil moisture monitoring), the transport sector, sports, and security can also utilise these devices. Smart devices can function independently or receive remote control and monitoring. These devices can also perform specific actions without human intervention [
1,
2,
3,
4].
The Internet of Things (IoT) denotes a worldwide network of interconnecting devices capable of collecting data from the environment, analysing the aggregated information, and reacting accordingly based on autonomous or human-directed commands. Through vast arrays of sensors and actuators amalgamated into one coordinated system, IoT technologies will continue to drive innovation in how we interact with appliances, vehicles, infrastructure, and more. IoT integrates the physical world with the Internet, creating more effective, intelligent, and innovative solutions. This connection to the network enables everyday objects, industrial machines, and the entire urban infrastructure to become more intelligent and unlock additional possibilities. IoT creates a coherent system that enhances the capabilities of participating entities and introduces network intelligence to facilitate decision-making and easy data exchange [
5,
6].
The use of smart devices has its advantages and disadvantages. First, these devices can be designed to perceive their working environment, gather information, and carry out particular tasks depending on particular circumstances [
7,
8]. For instance, in the context of intelligent vehicles, we can enhance the protection of road users (pedestrians, cyclists, drivers) through the monitoring of their well-being as well as the vehicle itself [
9,
10].
Conversely, using smart devices may engender issues related to technology or security. Establishing networks that connect such devices frequently necessitates the development of an infrastructure that integrates several technologies. Issues emerge with device interoperability, restricted processing and storage capacities, or Internet connectivity [
7]. These challenges may pertain to communication security, users, and data. Communication between IoT devices can be facilitated by employing a wireless network that serves users operating within a specific building, utilising a local wireless network with Internet connectivity, or establishing Internet connections that enable each device to access the Internet. Network administrators must guarantee communication security between devices and users and the integrity of sent and processed data in each specified solution [
11].
Threats to internet access include hacking of users, devices, data, and complete computer systems and networks. Many protective countermeasures are also possible in IoT to prevent cyber-attacks [
12,
13]. We can categorise several forms of attacks. For example, in a sensor node/IoT device-based capture attack, the attacker gains control over the device to control the network. He then separately withdraws the node from the network and replaces it as a malicious node [
14,
15,
16]. In an impersonation attack, the attacker abuses the identities of different network entities, including a user, a server, and an IoT device [
17,
18]. In a guessing assault, the assailant attempts to deduce the user’s password or other authentication credentials [
19]. In a significant compromise impersonation attack [
20], the adversary leverages the client’s certificate on its device to become the client.
We can approach defense and security mechanisms in IoT environments from two perspectives. The first perspective is connected with the primary goal of these mechanisms, which is to protect against network attacks (defense mechanisms). Some more significant tools incorporate these methods, such as the Intrusion Detection System (IDS) or Intrusion Prevention System (IPS). IDS and IPS are high-end security measures to prevent different kinds of threats. An IDS is a device that analyses the network traffic and searches for any abnormal behaviour that could indicate an attack. The IDS system notifies administrators about the incident when it detects a potential threat. However, it does not take any automatic action to stop the attack. IPS, on the other hand, works similarly to IDS. Still, in addition to detecting threats, it can also take steps to block them, such as blocking network packets, resetting connections, or deactivating infected user accounts. IPS offers preventive measures and reactions at the moment. This is very important in preventing threats from becoming bigger [
21,
22,
23].
From the second perspective point of view, we can also look at different techniques that can be used to safeguard users, devices, data, or even networks, referred to as security mechanisms. In this category of security mechanisms, communication security protocols implement various cryptographic methods such as encryption, anonymisation, or hashing for the messages to be exchanged. Thus, they are considered the primary security mechanism in IoT systems [
24,
25,
26].
1.1. Motivations and Contributions
Technology is a part of our life. We utilise a range of sophisticated technologies that primarily enhance our daily lives while facilitating the transfer of substantial volumes of data. Frequently, the transmitted communications include confidential information about users’ devices. Thus, data protection is an essential part of intelligent systems’ functioning. Users require the endorsement of safety while using technical facilities, thereby ensuring the protection of human life in the real and virtual world.
Malicious users in IoT environments expose several stages of the communication process to security vulnerabilities. For instance, these users attempt to intercept data and subsequently exploit it. As IoT technology moves forward, so do the methods used to attack it. We must keep reviewing and updating our defense and security measures regularly. Appropriately chosen methods will undoubtedly enhance the security level of connected devices. Thus, this review provides an overview of existing defense and security mechanisms in IoT environments, including Intrusion Detection Systems and Intrusion Prevention Systems and security protocols ensuring data security.
How defense and security systems work in IoT can give readers insight into the latest theory and real-world application developments. In this section, we will shed light on the security problems and weak points IoT devices face when they are part of these networks. We will also discuss the security levels of IDSs, IPSs, and security protocols used in IoTs. We will also focus on the challenges and needs of the newly developed defense mechanisms.
The main points of this article are as follows:
This paper provides a detailed analysis of the current state of defense and security of the IoT, focusing on protocols and systems such as IDSs, IPSs, and secure communication protocols.
This paper also outlines the vulnerabilities of the IoT communication mechanisms and the risks of data interception and misuse by adversaries. It highlights the stages where such vulnerabilities are most likely to occur.
This paper also seeks to explain how cyber-attacks have advanced with the advancement in technology in the area of IoT. Therefore, there is a need to develop and look at the security measures in light of the current technology.
This paper analyses the efficiency of IDS, IPS, and security protocols to understand the efforts made towards enhancing data protection and the IoT system.
This article aims at discussing the challenges that would be encountered when implementing security in IoT. The article addresses the practical challenges and requirements for implementing modern defense systems, considering the limitations and prerequisites of recently developed IoT security solutions.
The article provides a balanced perspective for academic researchers and practitioners in the field by bridging the gap between theoretical insights and practical applications of IoT security.
The article emphasises the need for regular updates on defense mechanisms and points out future research opportunities, encouraging ongoing advancements in IoT security.
This research paper presents a deep and thorough survey of IoT security and defensive mechanisms by providing an integrated taxonomy of how security protocols integrate with anomaly detection systems. In contrast to all earlier reviews, this library-focused work gives pertinent detail about the functionality of such mechanisms in real-world IoT scenarios, including strengths and weaknesses.
This study innovatively contributes a detailed analysis of interdependencies among various attacks and defense approaches. This topic has not been previously covered in reviews. Another aspect of this paper is the emerging concerns in IoT security, like scalability, energy efficiency, and privacy, with possible actionable solutions, such as lightweight protocols and AI-based anomaly detection.
1.2. Methodology
The requirements of systematic literature reviews motivated us to establish a systematic review of the security mechanisms in IoT. First, we identified relevant keywords and phrases, such as “IoT security”, “authentication protocols”, “anomaly detection”, and “cyber-attacks”, to search across various academic databases (mainly Google Scholar, DBLP, IEEE Xplore). Inclusion criteria target articles published in the last decade, emphasising peer-reviewed papers and high-impact journals.
To this end, we adopted a snowball sampling technique, studying the citations and references in the selected articles to complement our dataset. We thoroughly read all the publications to understand the proposed mechanisms, their application in IoT scenarios, and their performance under various conditions. This approach helped us collect the best contributions to IoT security and defense while reducing duplication.
The review was carried out at four levels: (1) identification of relevant literature, (2) a preliminary screening of title and abstract texts, (3) a detailed evaluation of entire texts, and (4) synthesis of findings into thematic categories. By building a solid scientific framework piece by piece, readers from various fields can comprehend recent trends and challenges on the Internet of Things, just as they would in their research areas.
1.3. Organization
Apart from
Section 1, this article consists of six sections. Some examples of authoring practices over the past few years will be provided in
Section 2, where we will critically evaluate the retrieved literature’s strengths and weaknesses.
Section 3 provides a somewhat detailed and comprehensive analysis of the essential ways and methods concerning the security of IoT systems, such as the basic theory of this technology protection, the principles of cryptography, the most common cyber-attacks on IoT-based networks, and the counterparties employed for event processing and detection of anomaly data.
Section 4 overviews a suite of security mechanisms for IoT systems, focusing on authentication and key agreement protocols. We also provide an in-depth analysis of the defense schemes, which are tailor-designed for the IoT, highlighting their strengths and weaknesses.
Section 5 overviews existing defense mechanisms dedicated to IoT environment IDSs, anomaly and cyber-attack detection systems, and malware detection systems). In
Section 6, the synthesis of our findings presents a comprehensive examination and framework for future study in IoT security. Lastly,
Section 7 summarises this manuscript, including conclusions from the analyses carried out and plans for the future.
6. Discussion
This manuscript focuses on two security aspects of IoT environments: security protocols and anomaly and cyber-attack detection tools. Regarding security protocols, we concentrated on realising authentication and key agreement goals. The protocols may aim to achieve one or both goals throughout their functioning. The reviewed protocols use cryptographic algorithms to accomplish their objectives and communicate securely. The susceptibility of these protocols to attacks and the provision of crucial security features have been confirmed using a variety of instruments and techniques.
We provide a summary of the updated protocols’ goals in
Table 4. According to the analyses performed, we have identified three categories of protocols. This illustrates the necessity of developing protocols primarily for user authentication. Since the reconciliation and agreement of session keys are crucial components of communication, creating and implementing this protocol is likewise crucial to communication security.
Table 5 summarises the protocols discussed regarding their uses and interoperability. We highlighted IoT solutions such as medicine and healthcare, edge or cloud computing, crowdsourcing, and smart homes. We assigned other protocols as cross-domain because they may be used in different resolutions.
In
Table 6, we indicate what attacks the discussed security protocols are resistant to. The authors of the indicated papers provided formal and informal security proofs and highlighted the attacks to which their protocol is resistant. The indication + signifies that the authors have demonstrated the resilience of their proposed protocol against attacks. The indication - means that the protocol has not been verified to be vulnerable to attacks. We highlighted the fourteen most common attacks—the less common attacks we included in the Otherscolumn. By SKTI, we assigned a Session Key Temporary Information attack, and by KSK, we assigned a Known Session Key attack.
We observed that replay, MITM, and impersonation attacks are IoT environments’ most frequently verified threats. They were tested in almost all papers. In IoT environments, the effects of these three attacks can be hazardous because IoT devices often control critical systems and have fewer security resources. These attacks can lead to physical failures, endangering user safety, data theft, and compromising entire IoT systems, seriously affecting critical applications (for example, healthcare and industry).
Replay attacks can lead to repeating old, already executed commands (for example, commands to open doors, turn on/off a device, change temperature), which can cause undesirable and dangerous actions. In monitoring systems, the attacker can send old sensor data, misleading operators or decision algorithms. In the case of MITM attacks, confidential information sent between IoT devices, such as health monitoring data (for example, medical devices), vehicle location data, or data from monitoring cameras, can be intercepted. The attacker can also modify data sent to a device, for example, controlling critical operations (such as door locks, alarm systems, and water pumps), leading to dangerous failures. In addition, the attacker can manipulate data, gaining administrative access to IoT devices, which allows for controlling the operation of the devices. In impersonation assaults, the assailant can seize control of the system. By impersonating a trusted device or user, the attacker can penetrate the entire IoT network and subsequently gain control of systems such as heating, building automation, or smart locks. The attacker can also introduce their device into the IoT network, impersonating a legitimate one to track network traffic or send false data.
Table 7 gives a rundown of the security features in the protocols we looked at. We have compiled a list of features and marked whether each protocol has that feature (shown as +). The - sign means the authors did not submit any information about that feature. The analysis showed that anonymity and secrecy are the most desirable security properties for security protocols in IoT environments.
Both features in the IoT context are crucial in protecting privacy, data security, and user trust. Anonymity protects personal data and prevents the tracking of a user’s identity. This is especially important for devices monitoring users’ daily activities (for example, smartwatches and smart speakers). This helps maintain user anonymity and complicates the efforts of companies and third parties to build detailed behavioural profiles based on data collected from IoT devices. Furthermore, anonymity makes it more difficult for cybercriminals to target specific individuals, as linking device activity to a particular user becomes challenging. Confidentiality ensures that the data collected by IoT devices is safeguarded against unauthorised access, thereby minimising the risk of leaking sensitive information. Additionally, data transmission between IoT devices is secured against interception, which lowers the chances of MITM attacks. Moreover, ensuring data confidentiality increases users’ trust in IoT devices and service providers, which is crucial for adopting the technology and increasing its popularity.
All authors of the examined studies conducted performance evaluations of their regimens. They compared their proposals with similar communication, processing, and energy consumption options. According to their findings, the suggested techniques outperform similar solutions in every study.
The authentication process is a vital part of communication in IoT environments. It involves verifying the identities of the parties engaged in the communication. During this process, we can use one or more factors; the more factors we incorporate, the more secure the authentication becomes. Relying solely on passwords can be a weak and vulnerable security measure, as attackers can intercept, guess, or crack them. This is why biometrics is often considered a better option, as it helps prevent spoofing or impersonation attacks.
The session keys (in securing communication) are essential for IoT security. Timestamped and protected with one-time session keys, messages guard against replay and man-in-the-middle attacks. This strategy enables the system to determine whether a legitimate network node produced the processed message and assesses if an attacker has intercepted and resent it. However, this improves the audience’s understanding of this vital security mechanism because it highlights the complexity of maintaining secure communications. Although the technical aspects may seem daunting, grasping these concepts is necessary for effective security measures in the IoT landscape.
Besides security considerations, the scalability of IoT protocols is a crucial factor. IoT devices have limited processing power. Therefore, the protocol’s energy must not be depleted by computations on other devices while it is operating. To ensure this, lightweight cryptographic algorithms are highly recommended to create authentication or key agreement protocols. These algorithms provide a suitable level of data security without straining system resources, thereby reassuring the audience about these protocols’ efficiency and resource management.
The future challenges of the security mechanisms dedicated to the IoT environments include enhancing scalability and optimising energy consumption, privacy enhancements, resilience against various attacks, and validation (via formal and informal methods). Scalability depends on the cryptographic techniques and hardware parameters used. So, future research should focus on cloud environments, their optimisation, and methods for lightweight data processing in such infrastructures, including blockchain consensus algorithms. Low-power communication protocols (supported by low-computational-overhead encryption techniques) can enhance the energy efficiency of communication in IoT environments. Consequently, sophisticated cryptographic techniques can improve privacy. The examined defense mechanisms guarantee robustness against diverse attackers.
Considering the need for experimental verification of the analysed protocols, we conducted security and simulation performance tests of these protocols. We used the ProVerif tool for security tests, which did not find any attack on the discussed protocols. In the case of performance tests, we simulated the execution of the entire protocol. Each proposed protocol has been designed distinctively, featuring varying phases and stages, diverse cryptographic algorithms, and different sent objects. We simulated these protocols utilising a methodology that will incorporate the capability of calculating execution times for alternative cryptographic techniques, including elliptic curve cryptography and blockchain technology. We considered the simulation environment with the following technical parameters of IoT devices: RAM: 16 GB, CPU Cores: 8.
Figure 2 presents obtained performance results. Protocols such as [
61,
121] with short execution times are optimal for applications requiring low latency, such as real-time IoT. Solutions with longer execution times, such as [
131,
137], may be better suited for environments where higher security is a priority at the expense of performance.
In the case of the reviewed anomaly and cyber-attack detection tools, in
Table 8 we prepared a summary that contains the realised defense method, used AI techniques, dedicated environments, and used datasets and key results or findings. These considerations allow us to emphasise employing various AI methods for anomaly and cyber-attack detection. IDSs and IPSs play complementary roles in security systems. IDSs focus on monitoring network traffic and identifying suspicious events or anomalies, which allows analysts to respond quickly. On the other hand, IPSs actively block potential threats to minimise the risk of attacks in real time. It is worth emphasising that these systems should work together to provide comprehensive protection. IDSs and IPSs increasingly use advanced anomaly detection techniques like machine learning and behavioural analysis. Although these approaches increase the effectiveness of detecting new or unknown threats, they can generate false alarms, one of the main challenges in their practical implementation. IDSs offer a more reactive approach, enabling analysis and incident response after detection. IPSs, on the other hand, introduce an element of proactive protection, but their operation may introduce an additional risk of blocking legitimate traffic (so-called false positives).
Also, in
Figure 3, we summarise the accuracy and precision of the proposed defense methods confirmed by the authors. Here, we assigned only solutions where such values are indicated. The authors of other papers did not highlight received accuracy. It is worth mentioning that the proposed methods achieve high accuracy in anomaly and cyber-attack detection.
Due to the limited amount of information available regarding the effectiveness (accuracy) of the methods considered for detecting attacks and anomalies in IoT environments, we decided to conduct experimental studies. These studies aimed to verify the effectiveness of selected defense mechanisms based on various datasets commonly used in the literature, such as CIC-DDoS2019 or IoTID20. The experiments focused on methods based on algorithms widely used in attack detection in IoT environments. The choice of algorithms such as Random Forest (RF), Long Short-Term Memory (LSTM), Support Vector Machines (SVMs), or Graph Neural Networks (GNNs) was derived from how frequently they appear in the scientific literature, the effectiveness reported in existing studies, and the potential for their application in various IoT contexts.
In particular, RF and LSTM were included based on their ability to work with big datasets and efficiently perform classification tasks. GNNs have been selected because they can work on graph-structured data and are promising in detecting more complex attack patterns. As a classical model, SVM is a reference point and allows for comparing results with more modern approaches. It is key for assessing essential parameters such as precision, sensitivity, F1-score, and false alarm rate for their use in large-scale analysis.
The findings from the conducted studies are shown in
Table 9, which contrasts the obtained results with those reported in the reviewed literature. The analysis of these results indicates that the selected methods, such as using Random Forest and LSTM algorithms in IDSs, show higher effectiveness than classical methods, such as SVM. For example, RF-based models obtained the highest F1-score value on the IoTID20 set, which indicates their high ability to balance precision and sensitivity. In turn, the methods using GNN algorithms achieved lower values for some metrics, which suggests the need for further optimisation in the context of IoT. These studies emphasise that selecting the appropriate model should depend on the data’s specificity and the system’s requirements, which creates room for further experiments and improvements.
Furthermore, detecting anomalies in large systems, such as IoT, requires scalable algorithms to process enormous volumes of real-time data. With increasing devices and data, systems must maintain high performance and low latency. Cybercriminals are constantly developing new attack techniques, leading to new patterns and types of anomalies. Detection algorithms must be flexible enough to recognise new attacks (zero-day) and adapt to changing threats. The main challenge is to find a balance between minimising the number of false alarms and high-threat detection. False positives may result in reduced operational efficiency and increased incident management costs. Such systems also have to deal with the issue of available resources. Resource-constrained systems, like IoT devices, often have limited computing capabilities. Therefore, detection methods must be energy-efficient, lightweight, and resource-optimised models. Internal attacks also hold significant importance. Typically, traditional IDSs and IPSs concentrate on detecting external attacks. Detecting anomalies related to internal threats (for example, employee actions) is problematic because this activity can resemble typical behaviour. IoT networks and distributed systems experience various anomalies, from attacks to unexpected system failures. Detecting all anomalies with a single technique is challenging because each type requires specific characteristics and data processing techniques.
Detecting anomalies in large systems like IoT systems requires scalable algorithms for real-time data. As devices and data grow, systems must maintain high performance and low latency. Detection algorithms must be flexible to recognise new attacks and adapt to changing threats. Balancing false alarms with high-threat detection is crucial, as false positives can reduce operational efficiency and incident management costs. Detecting internal threats is also challenging due to specific characteristics and data processing techniques.
We also decided to present a detailed comparison of different security solutions for IoT environments, focusing on their applications, advantages, and limitations. This comparison is presented in
Table 10. Each solution offers unique benefits in different IoT contexts but has limitations. Introducing such solutions requires understanding the trade-offs between computational requirements, implementation costs, and attack resistance. The presented comparison highlights the need for further research into solutions that can be more universal, scalable, and adapted to the constraints of IoT environments.
Research Limitations and Future Directions
While thoroughly observing defense and security in networks of connected devices, this research has several limitations. For starters, the rapid advancement of IoT technology and its security models makes it practically impossible to keep up with the unfolding events, a task that may not be achievable. Despite our efforts to include the most relevant and up-to-date research, this review may not cover emerging mechanisms. In this work, we reviewed the relevant methods from a theoretical perspective. Most surveyed methods do not consider practical implementation challenges regarding IoT devices’ scalability, computational overheads, and constrained energy resources. Therefore, this indicates a gap in the studies conducted, and further investigations should analyse the practical deployment of these mechanisms in realistic IoT scenarios.
Another significant limitation is the lack of thorough benchmarking regarding the IDS and anomaly detection frameworks considered. Typically, there is a lack of standard datasets or uniform metrics for comparing various approaches and drawing definitive conclusions about the relative merit of the competing methods. This work finally concentrates on the defense mechanisms, not extensively considering interdisciplinary factors, such as user behaviour, organisation policy, or economic factors that may influence IoT security.
Also, we can highlight here some directions for future work:
Future research should emphasise real-world testing of IDSs and anomaly detection frameworks, addressing false positives, adaptability, and long-term performance in dynamic IoT environments.
The development of consistent evaluation standards and publicly available datasets for the mechanisms in IoT security will help ensure fairness in innovation comparisons.
When designing low-power smart devices, creating power-efficient and computationally easy-to-perform security schemes is essential.
7. Conclusions
This manuscript comprehensively surveys security and defense mechanisms proposed for IoT environments. We focused on key agreement and authentication protocols (security mechanisms) and anomaly and cyber-attack detection methods (defense mechanisms). Our examination of the theoretical dimensions of IoT ecosystems, security and defense strategies, and potential cyber-attacks that may affect the security of these environments provides valuable insights and enhances the understanding of IoT security.
This review is distinctive by approaching the security mechanisms of IoT from a two-legged stance—the security protocols and security systems. It systematically compares authentication and key agreement protocols, highlighting their crucial role in securing communication in IoT, unlike similar works. Furthermore, the paper demonstrates the complementary nature of intrusion detection and prevention systems while stressing the need to integrate newer AI techniques for better anomaly detection.
This review is not like other reviews because apart from compiling previous works, it has also found some gaps in research that needed further investigation, such as energy-efficient designs and adaptable algorithms for combating persistent threats. Such leads are the pillars of possible future work, such as lightweight cryptographic protocols or scalable methods powered by AI against specific, localised attack scenarios.
Regarding the security mechanisms represented by security protocols, we highlighted the crucial role of authentication and key agreement processes during the communication between smart devices. The first process ensures the parties’ identities, and the second process improves the messages’ encryption. In reviewing anomaly and cyber-attack detection methods, we highlighted that we can use various AI techniques. The systems, like IDSs and IPSs, play complementary roles in security. They focus on monitoring network traffic and identifying suspicious events.
Furthermore, we emphasised the obstacles that security and defense systems must confront. Future security problems for IoT environments encompass enhancing scalability, optimising energy efficiency, augmenting privacy, and assuring resistance against cyber-attacks. We may address these concerns by optimising cloud infrastructure, blockchain consensus mechanisms, lightweight data processing algorithms, and sophisticated cryptographic protocols.
Detecting anomalies in large systems like IoT requires scalable algorithms for real-time data processing. As devices and data grow, systems must maintain high performance and low latency. Detection algorithms must be flexible to recognise new attacks and adapt to changing threats. Balancing false alarms with high-threat detection is crucial, as false positives can reduce operational efficiency and incident management costs. Energy-efficient and resource-optimised detection methods are essential in resource-constrained systems. Detecting internal threats is also challenging due to specific characteristics and data processing techniques.
Based on remarks from this review, our future research will focus on creating new security methods, such as lightweight, energy-efficient, and scalable security protocols. We will remember that IoT devices have limited resources, which makes implementing cryptographic algorithms too demanding. These limitations are connected to limited computing power, memory, or battery. Also, we will focus on testing AI methods in the case of less-known or niche attacks, which can have hazardous consequences for users and their data or devices.