Mitigation of Risks Associated with Distrustful Routers in OSPF Networks—An Enhanced Method

Abstract

Packet routing in computer networks provides complex challenges in environments with distrustful routers due to security vulnerabilities or potential malicious behaviors. The literature offers solutions to the problem designed for different types of networks. This paper introduces a novel method to mitigate risks associated with distrustful routers by constructing secure and efficient routing paths in Open Shortest Path First (OSPF) networks. Networks in which routing is carried out based on OSPF protocols are currently the most widespread, hence ensuring the security of data transmission in such networks is urgently needed. In turn, distrustful routers can degrade the overall security and performance of the network, creating vulnerabilities that can be used for malicious purposes. The proposed method is based on the Dijkstra algorithm which is enhanced to identify and mitigate the risk connected with potential distrustful network nodes. Analysis of the proposed method shows its ability to build efficient routes exclusively through trusted routers if such paths exist. As a criterion for effectiveness, a metric such as the channel weight is used. The proposed method is validated using applications across networks of varying topologies and sizes, including large-scale networks. For networks containing post-distrustful routers to which there is no path without distrustful nodes, the proposed method is able to build the shortest paths that are marked as not secure but have a minimum number of distrustful nodes on their path. In scenarios with multiple compromised routers with different locations in the network, the proposed method significantly increases network resilience.

1. Introduction

Packet routing is a fundamental aspect of computer networks. It establishes a reliable way for the end-to-end transmission of packets between routers within a complex network infrastructure. This process ensures that data are successfully sent from source node and received by designated target router, achieving network-wide connectivity.

Numerous studies have focused on optimizing and securing the packet routing process. These efforts have brought attention to a variety of challenges, including issues related to overall network efficiency and routing security. While these problems are extensively documented, the increasing complexity of contemporary network architectures and technologies continues to present challenges. As a result, there is a need for innovative routing optimization methods designed to address specific usage scenarios.

A computer network is composed of interconnected routers that communicate through various types of communication channels. Depending on the network’s complexity and purpose, different routing protocols may be employed to optimize data transmission and ensure reliable communication.

Within a network, the path of a packet is determined by the routing protocol. Open Shortest Path First (OSPF) protocol is a widely used link-state protocol, ideal for larger enterprise networks and autonomous systems. It calculates the shortest path using the Dijkstra algorithm and quickly adapts to changes, making it more efficient and scalable than RIP (Routing Information Protocol) [1].

The Dijkstra algorithm is an efficient method for solving the single-source shortest path problem and is commonly used. It works by iteratively updating the shortest known distance to each node as it incorporates new nodes from the network into the shortest path calculation [2].

Network security plays a vital role in protecting data in interconnected systems. Security issues in packet routing are of particular concern, as attackers can exploit routing protocols to compromise the network. One potential risk is the compromise of a network node, resulting in the loss of its trustworthiness. When a node is compromised, it may act maliciously and sabotage the network by stealing traffic or degrading network performance [3]. A distrustful router is capable of correctly receiving and transmitting packets. However, there is a possibility of its use for malicious purposes due to its administrative subordination. In distrustful routers, Deep Packet Inspection can be used not for its intended purpose, but for intercepting data, redirecting it, and collecting prohibited statistics. There are various reasons why a router can be labeled as distrustful, for example, due to its location in a temporarily occupied territory or due to a suspicious change in behavior policy. Such decisions can be made by the operational and technical management system of telecommunications networks. In large corporations, to prevent data leakage, certain nodes can also be administratively transferred to the distrustful class.

Trust-based routing employs new criteria for calculating packet routes. Trust represents the information about the network node security risks or vulnerabilities. If a node is distrustful, transmitting packets to or through it can introduce security risks. Therefore, the primary goal of trust-based routing is to minimize these risks by reducing the transmission of packets through distrustful nodes. Since the reason for the router trust loss discussed in this article is not the router’s technical state or current state of the network but the threat of the router’s use for malicious purposes, a router status change is a rare occurrence.

In Wireless Sensor Networks (WSNs), existing solutions for secure routing emphasize protecting against cyberattacks that could destabilize energy distribution and create hazards [4,5]. Trust-based routing solutions establish trust levels for nodes based on their behavior, enhancing resilience by isolating potentially compromised nodes. However, due to the inherent constraints of WSNs—such as limited energy, memory, and computational resources—achieving comprehensive security remains complex.

Due to the limitations of WSN, achieving Quality of Service (QoS) is challenging. However, trust-based cryptographic routing algorithms offer a viable solution by mitigating risks and ensuring secure routing while maintaining QoS [6,7]. Routing in vehicular ad hoc networks (VANETs) is vital for facilitating efficient communication between vehicles and infrastructure components. Combined with frequent link disconnections, VANETs necessitate secure and reliable routing protocols. Trust-based geographic routing protocol for VANETs (TGRV) can be used to address challenges of routing in highly mobile and dynamic vehicular networks, particularly in the presence of malicious vehicles [8,9].

Mobile ad hoc networks (MANETs) operate in highly dynamic and rapidly changing environments. With the growing demand for interconnected mobile devices, these networks are becoming increasingly dense and complex, which introduces significant challenges to routing security [10]. Trust-based routing mechanisms have emerged as a promising solution to address these issues. One approach introduces FLSTMT-LAR (Federated Learning Long Short-Term Memory Trust-Aware Location-Aided Routing), a novel routing framework for MANETs that integrates federated learning with LSTM-based trust prediction to enhance both security and efficiency [11]. By dynamically assessing trust levels and optimizing multiple objectives, such as trustworthiness, energy efficiency, and latency, FLSTMT-LAR demonstrates superior performance compared to traditional routing protocols. Additionally, another study [12] proposes an innovative method for calculating node trust levels based on their reputation while employing cryptographic algorithms for secure group key exchange after identifying malicious nodes.

In IoT networks, security routing problems are particularly critical due to the constrained resources of devices and the dynamic nature of the network topology. The work [13] overviews efficient secure routing mechanisms for the most used low-powered IoT networks. Proposed solutions for addressing security routing problems in IoT networks often focus on trust-based routing, where nodes in the network evaluate the reliability and behavior of their peers before establishing communication paths [14,15]. The paper [16] is devoted to secure routing in IoT with an orientation to software-defined networking and machine learning techniques. The work [17] is focused on IoT-6G technologies and proposes adaptive and energy-efficient routing protocol, having the prime motivation of prioritizing critical data with on-time delivery.

Additionally, a routing method in the network must consider GDPR requirements when selecting the optimal path for a packet. Violating these regulations and transmitting packets, containing sensitive information through unsecured nodes or communication channels poses a direct threat to data security within the network [18,19]. This is particularly crucial in highly secure environments such as medical or banking systems [20]. GDPR compliance routing can be considered a specific case of routing with untrusted nodes. Therefore, the same methods can be used to achieve secure GDPR compliant routing with distrustful routers.

Trust-based routing can be integrated into the security subsystem of info-communication systems to enhance resilience against typed cyber-physical attacks, leveraging trust metrics as dynamic parameters in modeling the system’s end states and optimizing the probability of attack neutralization over time [21].

The problem of routing with distrustful nodes lies in constructing the most efficient and secure path for a packet within a network that may contain distrustful nodes. Unlike standard routing problems, this task involves additional conditions related to the trustworthiness of nodes through which the packets pass. This ensures control over the confidentiality of information transmitted through the network. Channels in the network have metric values assigned to them, so the route with the smallest sum of channel metrics will be the most effective for sending a packet. The metric value can correspond to the channel’s physical characteristics, for example, the throughput channel, can be equal to 1 when the metric is the number of hops to the destination [22] or be an integral criterion [23]. Distrustful nodes pose a security threat due to their potential to intercept or alter traffic passing through them. There can be a different number of distrustful nodes in the network and they can connect with both trusted and other distrustful nodes, further complicating the routing task. In the article [24], an approach to bypassing an unreliable router is proposed, which can be used in complex network topologies by modifying it and adapting to different application scenarios.

Overall, the topic of routing in networks with untrusted nodes remains highly relevant, finds application in various network types, and requires further investigation to address its inherent challenges effectively. A common approach to ensuring reliable routing is focused on removing unreliable routers from the network that cannot guarantee the correct performance of relay functions. In this regard, such routers are removed from consideration for building routing tables. As a result, end nodes that are connected to such routers will not receive data addressed to them. In this work, the loss of the router’s ability to ensure data confidentiality, rather than its technical malfunction, is used as the main criterion for improving routing security. Due to this, we have the opportunity to ensure data transmission to all network clients and at the same time increase data transmission security by building routing tables that provide bypassing of confidentially distrustful routers.

The main contributions of this paper are as follows:

• We proposed an approach to obtain the shortest path tree for construction routing tables in a network when using a weighting criterion adapted to minimize traffic through distrustful nodes, thereby enhancing network security;
• Our proposed algorithm finds the shortest path tree when using a weighting criterion without intermediate distrustful routers if such a path exists and with the minimum possible number of distrustful nodes for cases when there is no shortest path without using distrustful nodes;
• We proposed an approach to improving routing security by introducing an additional channel to bypass a distrustful router when there is no secure path in the existing network.

2. Materials and Methods

In this article, we consider a network that uses OSPF routing protocol and, therefore, employs the Dijkstra algorithm. The classical Dijkstra algorithm accepts a list of nodes and the weights of the paths connecting them. The Dijkstra algorithm constructs a table of the shortest paths from a starting node to all other nodes in the network. When a transmission is required, it utilizes this precomputed path to route the packet.

However, the classical algorithm does not account for security concerns when determining the shortest path, which may lead to a route passing through a distrustful node. Routers can be classified as either trustful or distrustful, but the trust metric does not influence connections between nodes or their weights, so the network graph remains unchanged.

Specialized internet services may classify certain nodes as distrustful based on predefined criteria. They take into consideration node behavior, Internet provider and security information, and router geolocation. At any given point in time, multiple untrustworthy nodes may be active within the network. This information is available to all routers and can be used during the routing process. Providing trust in the routing process increases its complexity but allows for security risks to be mitigated at the routing level.

To prevent potential security issues, the shortest path should not include distrustful routers, unless the target node is distrustful or there are no other paths to the target node except the path with the distrustful router. In case the distrustful router is included in shortest path, such a path should be marked as distrustful. Sensitive information should not be transmitted through distrustful paths, as it may be vulnerable to interception by third parties.

To mitigate the risk, a new method is proposed that modifies the Dijkstra algorithm to minimize packet traffic through distrustful nodes. A detailed description of the proposed method is provided below.

Consider a network represented as a graph that contains routers denoted as nodes on the graph. Edges represent full-duplex channels, and their weights correspond to channel metric values. To account for the presence of multiple distrustful routers, the method finds distrustful nodes and mitigates their impact during the computation of the shortest paths. We will consider node A as a router for which the shortest path tree is to be built. The shortest path is defined as the route with the lowest cumulative metric value while avoiding routes that include distrustful nodes.

Distrustful nodes are penalized by assigning artificially high distance scores to them during path calculations when finding detours. Specifically, when a node is identified as distrustful, its distance score is reset to infinity. This ensures that paths passing through distrustful nodes are deprioritized in the shortest path computation.

In this context,

• AS(v): The cumulative metric value of the path from the root node A to the current node v.
• c(w, v): The weight of the channel between nodes w and v.

The step-by-step implementation of the proposed method:

• Start with a set of nodes N that initially contains only the source node and set U that is initially empty. Set N = {A}, U = {}.
• Assign distance scores for all neighboring nodes of A based on their direct connection cost: AS(v) = c(A, v), where c(A, v) is the cost of reaching node v from A.
• Identify a node w that is not in the set N and has the smallest distance score AS(w).
• If node w is found to be distrustful, add w to set U. Reset the distance score of node w to infinity by setting AS(w) = infinity to prevent using paths through w.
• Otherwise, add w to the set N.
• Update the distance scores for all nodes not in N by recalculating them as follows: AS(v) = min(AS(v), AS(w) + c(w, v)), where c(w, v) is the cost of the connection from w to v.
• Repeat steps 3 through 6 until all nodes have been processed or there are no more reachable nodes outside N.
• Repeat steps 3 through 6 for all nodes in U until there are no more nodes left.

This modified approach reduces the influence of distrustful routers by penalizing their inclusion in path computations and prioritizing secure, trustworthy paths.

3. Results and Discussion

To analyze the proposed method in detail and its effectiveness, we will apply it to various network topology examples of differing complexities. These topologies include variations in the number of active routers, the number of links connected to them, and differing numbers of distrustful routers, representing possible real-world scenarios. We assume that connection weights are accurately assigned for each topology before executing the routing process.

3.1. Small Network

3.1.1. Small Network Without Distrustful Routers

For better clarity, we will first show how the proposed method works in a small network with a limited number of routers. Consider the network in Figure 1; it has seven routers, and all of them are trusted.

In this network example, node A is designated as a router, for which a shortest path tree must be found to build a routing table. It is visually highlighted with a green border. This scenario does not include distrustful nodes, and the proposed method works according to the classical Dijkstra algorithm. The resulting shortest path tree is shown in Figure 2. The connections used for the shortest path tree are highlighted in green.

3.1.2. Small Network with One Distrustful Router

Let us show the method’s functionality in the presence of a distrustful router, and present a basic scenario by altering the trust level of one router within the network. Specifically, node F in Figure 3 will be designated as distrustful and shown with a red border on the topology map, while all other network elements remain unchanged.

The proposed method is re-run with the new configuration. As a result, the shortest path tree for the modified topology was built (Figure 4). Green lines represent trusted channels, red lines—distrustful part of shortest path tree. This scenario provides a foundational example of how the method identifies and manages distrustful nodes, ensuring that paths bypass or account for potentially unreliable nodes.

Shortest path trees allow the router to make a decision about choosing a Gateway; for our example, they are given in

Table 1. The comparison of routing decisions about choosing a Gateway made by Router A.

Destination	Classical Method	The Proposed Method
D	B	C
E	B	C
G	B	B

.

In the provided network, node G is accessible only through the distrustful router, node F. This introduces a security risk, as any data packets routed to node G must pass through node F, where they may be intercepted or otherwise compromised. Despite this vulnerability, it remains possible to transmit certain information to node G, with an awareness of the potential risks involved. Consequently, node G should still be incorporated into the shortest path tree, though its connection via node F is highlighted in red color.

To better show the difference between the classical method based on Dijkstra’s algorithm and the proposed method, we show fragments of routing tables for node A using both methods (Figure 5). Additionally, for better illustration, the shortest paths from the found trees have been added to the table.

The comparison of routing decisions made by Router A using the classical and the proposed methods is presented in Table 1. The difference lies in the route changes for destination nodes D and E; we can see Gateway B is replaced by Gateway C. As a result, the path to destination node D is (A→C→D) instead of the shortest (for used metric) path (A→B→F→D), and to destination E (A→C→D→E) instead of the shortest path (A→B→F→D→E). Note that the paths found by the proposed method do not contain a distrustful node F while the classical method contains a distrustful node in both cases. The paths of the classical method are shorter in terms of weight (six for destination D and seven for E) since for the proposed method it is seven for D and eight for E. The increase in weight is a payment for ensuring secure data transportation.

This simple example is designed to demonstrate another possibility of the proposed method. As shown above, the paths found by the proposed method have a higher weight concerning the selected criterion than for the classical approach. However, they win by the criterion of minimizing the number of intermediate hops, i.e., the path that bypasses the distrustful node at the same time has fewer hops. This can be especially useful for time-critical traffic, such as speech.

3.2. Larger Network with Clusters

The proposed method is adaptable to different networks and is able to work in large and complex topologies, which are commonly encountered in real-world scenarios. It can also be applied to routing between autonomous systems (AS). Each AS connects to the broader network through external gateways, which can represent their cluster of routers in a network diagram. The proposed method can then be employed to calculate the optimal route for packets that need to be sent between AS edge points.

Let us consider a network composed of multiple clusters. Theoretically, every router in the network can be or become a distrustful node. A specific case involves distrustful nodes being the entry point for a cluster, acting as the only entry point for transmitting packets to certain clusters.

In real-world scenarios, networks often contain multiple points of vulnerability, with several nodes being potentially distrustful due to various security risks. The proposed method is designed to handle such complexities and produce a shortest paths tree that mitigates security concerns while maintaining efficient routing.

To analyze the performance of the proposed method, let us examine a simplified network with clusters, depicted in Figure 6. This network consists of 19 routers, connected into three interconnected clusters: Cluster 1, Cluster 2, and Cluster 3. The clusters include the following nodes: A-B-C-D-E-G-F (Cluster 1), P-Q-R-S (Cluster 2), and H-M-N-I-L-J-O-K (Cluster 3). Routers P and M are distrustful. In this example, Cluster 2 is connected to the external network only through distrustful router P. There are no alternative channels to access the cluster, making node P the sole router on the path to the specified cluster. For Cluster 3, the distrustful router M is inside it and is the only way to access router N. The proposed method identifies potentially compromised routes and adapts routing decisions to maintain efficiency in isolated clusters while mitigating security vulnerabilities.

The shortest path tree of this network with root A obtained using the classical method is shown in Figure 7. Green lines represent trusted channels, red lines—distrustful part of shortest path tree. Eight routers (Q, R, S, N, L, J, K, and O) will relay data through distrustful routers.

The shortest path tree for this network obtained by using the proposed method is shown in Figure 8.

As we can see, the number of routers with distrustful routers on the path has been reduced to four (Q, R, S, and N). This is half the number when using the classic approach. In this configuration, all traffic directed toward Cluster 2 must pass through the distrustful router, node P. Also, all traffic to router N must pass through the distrustful router M.

Let us consider the fragment of the routing table for node H, which connects Cluster 3 with other clusters (Figure 9). The path column is added for better example visualization.

Packets to Cluster 1 can be freely sent through router E, but packets to Cluster 2 still must be sent through distrustful router P. To better compare the proposed method with the classical method, let us examine the nodes within Cluster 3. For this purpose, we will construct a comparison of the routing decisions made by Router H (

Table 2. The comparison of routing decisions made by Router H in larger network with distrustful routers in clusters.

Destination	Gateway (Dijkstra’s Algorithm)	Gateway (Proposed Method)
N	M	M
L	I	M
O	I	M
K	I	M
J	I	M

). The proposed method selects a longer path but safer path compared to the classical one for packets within Cluster 3.

This approach demonstrates the method’s effectiveness across the considered network topologies, as it consistently constructs a shortest path tree that reduces exposure to distrustful nodes.

A closer look at the above example confirms that the proposed method successfully established secure connections bypassing the distrustful router in Cluster 3.

However, the problem persists in Cluster 2, where the distrustful router P serves as the only entry point for the cluster. As a result, all traffic to the cluster must pass through the distrustful router P. This is because there are no physical paths to bypass this unreliable router. To enable the use of the proposed method for networks that have clusters with only one entry point into them, it is recommended to include an additional channel that will connect another cluster to the next router node after the untrusted one. Among software solutions, only the use of end-to-end encryption may be recommended.

3.3. Network Topology Optimization

The case where a certain cluster can be connected to the external network by a single edge router that has become untrusted may not be isolated. Let us consider a similar case for the network shown in Figure 6. According to the recommendations made above, we will add the HQ channel to this network that will be used as a path to bypass distrustful node P. The proposed method can be applied and thus redirect traffic through node Q, bypassing the untrusted node P.

The updated shortest path tree for the network with optimized topology is shown in Figure 10 and Figure 11, built using the classical and the proposed methods, respectively. Green lines represent trusted channels, red lines—distrustful part of shortest path tree.

A comparison of the shortest path trees in Figure 10 and Figure 11 shows that the inclusion of an additional HQ channel when using the classical method did not yield any positive results in terms of improving security. Instead, using the proposed method, the number of routers that will receive data through untrusted routers decreased from eight (for the classical method) to one.

3.4. A Complex Configuration Large Network with Different Options for Distrustful Node Locations

To visualize the operation of the proposed method, a software solution was developed that presents the shortest path trees in networks of various dimensions and topological complexity. Real-world telecommunication network data are located in the databases of national or private Network Operation Centers. To ensure network security, the content of these databases is information with restricted access in order. Therefore, for verification, we will use a hypothetical network topology, noting that it, like a real network, is described by a similar mathematical model such as a weighted graph. The difference will be only in the number of nodes of the graph, the connections between them, and the weights of the edges. To provide a clear visualization, we limited the network dimension in our example to 31 routers, 7 of which were selected as distrustful. Untrustworthy nodes are located both at the edges of the network and inside, singly or grouped. Figure 12 illustrates this example and presents the shortest path tree for router A constructed using the proposed method. Green lines represent trusted channels, red lines—distrustful part of shortest path tree. For simplicity, only source node A and neighboring nodes B, C, and D are labeled with letters in the figure. This is because the IP addresses of nodes B, C, and D are used when building the router A Routing Table to fill the Gateway field.

The analysis showed that if in a network with distrustful nodes, there are paths to all nodes without using distrustful nodes, then the developed software will find them and present them as the shortest paths tree. In this case, the delivery of packets to distrustful nodes will also be ensured. If in a network with distrustful nodes, there is no path to some nodes without using unreliable nodes, the result of the program will be a tree of shortest paths that will use the minimum possible number of unreliable nodes, and ensure the delivery of packets to distrustful nodes. The proposed method was verified on different types of networks.

The proposed approach improves the security of data transmission in networks with distrustful nodes. It guarantees the avoidance of the use of distrustful routers for data transit, provided that such paths exist. The price for this is a decrease in network efficiency because the weights of the proposed paths are greater than those calculated by the classical algorithm. The decrease in efficiency could be reduced by developing an adaptive application of the proposed method depending on the type of traffic and its security requirements.

4. Conclusions

The method developed in this paper has been proven to increase security in networks containing routers under administrative control, which causes untrust in their management policy. Increased security is achieved by minimizing transit traffic through distrustful routers and by enabling the delivery of packets addressed to end devices directly connected to distrustful routers. In this paper, we propose an approach to overcome the challenge by developing an algorithm and software, and by analyzing their effectiveness.

The main task of the method is to obtain a shortest path tree in which distrustful routers are located only at the endpoints of the tree branches. Such a tree guarantees that transit flows through distrustful routers will not be transmitted at all, and therefore minimizes transit traffic through them.

A program solution was developed to build various network topologies and validate the effectiveness of the proposed solution, including scenarios involving multiple distrustful routers with different locations, both single or groups. The input data are a weighted network graph indicating the location of distrustful nodes, and the output data are a tree of the shortest paths. If there are paths to all destination nodes without using distrustful nodes, the tree contains these nodes at the ends of the branches. If such paths to deliver packets to certain destinations without using distrustful routers are absent, their number on the path is minimized. Based on the obtained trees, routing tables are then constructed.

The proposed method can be integrated into existing network management systems by replacing the Dijkstra algorithm used to find the shortest path in the OSPF protocol with its modified version. At the same time, it is necessary to organize a procedure for assigning routers to the distrust status and removing this status in the databases of network management centers. Incorrectly marking a router as untrusted in general will degrade performance in some areas of the network, but will not lead to a breakdown in its performance. Incorrectly returning a router to the trusted status will also not disrupt the network, but will restore the threat of insecure data transmission.

For cases where a secure route does not exist, but the task of reliable routing exists an approach to improving routing security by introducing an add channel to bypass the distrustful router is proposed and analyzed.

The analysis conducted in this paper is focused narrowly on routing security. However, since it is based on Dijkstra’s algorithm for IP-oriented networks, by default, when searching for the shortest path tree, one of the possible metrics is used as a criterion. It is represented by the channel weight, which can reflect various indicators such as throughput, latency, packet loss, or computational overhead. For the modified network, we strongly recommend keeping the same metric used in the network whose security needs to be improved, unless otherwise specified. However, further research should focus on developing an approach that considers different metrics for different traffic classes when obtaining bypass paths, without changing the criterion for the main path. The proposed mechanism can be further improved by incorporating adaptive trust metrics based on real-time network behavior, such as packet loss, latency, or anomalous actions. Machine learning techniques could play a pivotal role here by analyzing historical and real-time network data to dynamically predict and adapt trust metrics for routers. Additionally, instead of assigning a fixed weight used to penalize distrustful routers, it is worth considering the possibility of using a gradient penalty system that assumes different levels of distrust. Such a system, guided by ML algorithms, could optimize routing decisions by balancing security and performance dynamically.