InMesh: A Zero-Configuration Agentless Endpoint Detection and Response System

Abstract

Endpoint Detection and Response (EDR) systems play a crucial role in continuously monitoring endpoint activities to detect, analyze, and respond to cybersecurity threats in real time. Traditional agent-based EDR systems rely on software agents installed on endpoints for data collection, which can be impractical due to the large number of devices, their mobility, and privacy concerns. In contrast, agentless EDR systems aim to overcome these limitations by remotely collecting network and host data, but they face challenges in precise data attribution because of the transient nature of network addresses. Achieving a fully zero-configuration agentless EDR system remains a significant challenge. This paper introduces InMesh, an innovative system that can identify and monitor endpoints without relying on network addressing or software agents. The effectiveness of the approach is demonstrated using real-world data.

1. Introduction

Endpoint Detection and Response (EDR) systems continuously monitor endpoint activities to identify suspicious behavior, potential malware infections, and security breaches in real time. These systems enable immediate threat detection while facilitating essential security functions such as threat hunting and forensic analysis. EDR platforms collect detailed telemetry data on endpoint processes, file modifications, and network communications, which they then integrate with Security Information and Event Management (SIEM) systems. SIEMs correlate this endpoint data with logs from other security infrastructure including firewalls, intrusion detection/prevention systems (IDS/IPS), and authentication services. This integration creates a centralized platform for comprehensive security analysis and advanced threat identification.

When combined with Security Orchestration, Automation, and Response (SOAR) capabilities, these systems can automate critical defensive actions—such as isolating compromised endpoints and blocking malicious IP addresses. This unified security ecosystem substantially enhances threat detection capabilities and accelerates incident response times. By providing both enhanced visibility and analytics-ready data, EDR solutions significantly reduce the manual workload of security teams. The growing recognition of these benefits is reflected in market projections, with EDR systems expected to experience a substantial growth rate of nearly 25% from 2024 to 2032 [1], underscoring their increasing importance in modern cybersecurity architectures.

EDR systems rely on two fundamental types of data collection: network-level monitoring and host-level telemetry. Network packets can be collected either using specialized software agents installed on the endpoints or by an external system of strategically placed sensors, which are usually on a network tap or a switched port analyzer (SPAN) port. Host telemetry can be collected internally using agents or externally using remote login protocols, API integrations, syslog monitoring, and event log analysis.

Agent-based EDR systems may be impractical in certain operational environments. Incompatibility with the operating system may preclude deployment. Resource constraints on devices with limited computing capabilities, such as the Internet of Things (IoT), can limit the frequency and amount of data collected. Certain specialized devices, such as industrial control systems, do not allow the installation of EDR agents. Privacy concerns related to the collection and transmission of sensitive system data can pose challenges in implementing data collection policies. The number of unmanaged devices in enterprise networks has grown exponentially in recent years. By 2021, up to 90% of all devices were predicted to be unmanaged and unsecured [2], and the yearly number of such devices had increased by almost 31% [3].

Agentless EDR systems offer several distinct advantages over traditional Endpoint Detection and Response solutions. By eliminating the need for software installation and update mechanisms, they enable faster, simpler deployment and maintenance. Data collection occurs on separate, security-hardened devices rather than on the endpoint, enhancing security by avoiding installed software with file system or process-level access that could be compromised. This separation also ensures that the system cannot be easily disabled during an endpoint compromise. Additionally, agentless systems streamline deployment, minimize operational overhead, and readily support diverse network and device architectures. From a privacy perspective, these systems use non-invasive methods and suspend data collection when devices disconnect from the monitored network, preserving user privacy unlike traditional EDR agents, which continue collecting data regardless of device movement. This architectural approach provides both operational efficiency and enhanced privacy protection while maintaining robust security coverage.

Agentless EDR systems possess inherent security limitations that merit careful consideration. Their capabilities in detailed process and file monitoring may be restricted compared to agent-based alternatives, which may create detection gaps that sophisticated threat actors can exploit. Additionally, reliance on external security-hardened collection devices introduces a potential vulnerability pivot, allowing attackers to shift their focus toward compromising centralized infrastructure instead of individual endpoints. Moreover, while the non-invasive monitoring approach enhances privacy preservation, it also constrains real-time detection capabilities when devices disconnect from monitored networks and limits response options, as there is no agent available to execute immediate endpoint remediation. This asymmetry in monitoring further complicates matters, as it hinders the detection of malicious software installations during periods of network disconnection. Contextual limitations at the network level may also lead to an increase in false positives, necessitating additional correlation and analysis that adds to operational complexity. Consequently, security architects implementing agentless EDR solutions must devise complementary strategies to mitigate these inherent limitations.

Despite the variety of agent-based and agentless EDR solutions available in both the literature and the market, significant gaps persist in current technologies. Existing commercial EDR products often depend on invasive agents that introduce considerable operational overhead or on network-centric methods that struggle with accurate endpoint identification in dynamic environments. Open-source alternatives frequently lack the advanced fingerprinting capabilities necessary to effectively track endpoints as network conditions change. Furthermore, many current solutions remain susceptible to address spoofing and network-based impersonation attacks, resulting in critical blind spots in security monitoring. These shortcomings underscore the necessity for a more robust approach that integrates the operational benefits of agentless deployment with enhanced endpoint identification capabilities that remain effective despite changes in network configurations.

This paper presents InMesh, which is an innovative zero-configuration agentless EDR system designed to identify and monitor endpoints without depending on network addressing or software agents. InMesh employs a combination of passive network monitoring and active endpoint scanning to collect metadata independently of network addresses, utilizing endpoint-specific statistical models to accurately identify devices, even when their addresses have been spoofed, impersonated, or otherwise modified.

The remainder of this paper is structured as follows. Section 2 examines spoofing and common attack vectors that lead to endpoint compromise while also providing an overview of state-of-the-art EDR research and commercially available products. Section 3 elaborates on the InMesh system architecture, emphasizing data collection and automated endpoint detection. Section 4 offers a scenario-based evaluation that showcases InMesh’s capability to detect attacks and furnish analysts with critical information. Section 5 provides closing remarks.

2. Background

2.1. Endpoint Detection

Most agentless EDR systems rely on traffic captured by backbone data collection systems to model endpoints, as there is a lack of frameworks capable of capturing network activity at the subnet level and endpoint information without agents. These systems use network traffic addresses for attribution, which can be unreliable. An endpoint’s IP address used for Layer 3 routing can be manually assigned or altered within the subnet’s usable range or automatically through a dynamic host configuration protocol (DHCP) server. DHCP IP release and reuse allows transient devices to connect despite the limited IP address pool in the LAN’s Classless Inter-Domain Routing (CIDR). Furthermore, the IP address can change during network address translation (NAT) when crossing the subnet perimeter. The Media Access Control (MAC) address, which is used for Layer 2 switching, was designed to be immutable while being unique to the network adapter of the endpoint. However, it can be changed by users with administrative system privileges. The MAC address of a data frame is also replaced automatically when packets are routed from one subnet to another. That is, the source MAC is replaced with that of the router forwarding interface and the destination MAC with that of the next hop or final destination interface. This ensures proper Layer 2 delivery within each subnet. For these reasons, addresses from the TCP/IP stack can be an unreliable means of identifying unique devices. When statistical models are developed that rely on these addresses, agentless EDR systems may not attribute data to the correct endpoint.

Bad actors can spoof MAC addresses to impersonate existing devices or evade detection. When a MAC address overlap occurs, the Content Addressable Memory (CAM) table of the switch is updated with the interface the attacker is on as the new forwarding port. This results in a Layer 2 to Layer 1 mapping. When the attacker sends packets with the victim’s source MAC address, the switch learns that it should forward future packets intended for the victim to the attacker’s interface, since its CAM table now shows that the victim’s MAC address is at the attacker’s interface. The attacker may periodically send spoofed data frames to prevent the victim from reclaiming the interface. Successful impersonation will route all packets intended for the victim through the attacker. This Layer 2 Man-in-the-Middle (MITM) attack exploits the lack of verification on LAN switches. The attack can also be performed at Layer 3 using Address Resolution Protocol (ARP) poisoning with spoofed gratuitous ARP messages exploiting the lack of built-in verification of the ARP protocol. In both cases, the attacker can monitor, record, or alter all unencrypted network communications from the endpoint. Perimeter protection systems are designed to detect attacks at the network boundary, but they may lack sufficient visibility into internal reconnaissance, endpoint compromise, lateral movements, and data exfiltration.

IP address spoofing is predominantly associated with malicious activities, although legitimate legacy uses have existed. For example, certain Internet Service Providers (ISPs) in mobile IP networks historically relied on spoofing, assigning the IP address of the native ISP to roaming devices for billing purposes to provide Internet connectivity. However, reverse tunneling as defined in RFC 2344 [4] and proper VPN routing per RFC 4364 [5] have largely obviated these practices. Beyond these legacy implementations, address randomization has gained prominence as a tool for safeguarding user privacy by mitigating tracking and fingerprinting as well as for supporting security research and the controlled testing of network defenses. Moreover, spoofing techniques can be exploited to bypass geo-restrictions and censorship by simulating traffic originating from approved regions.

Malicious IP spoofing applications can be more sophisticated. Attackers employ spoofing to avoid detection and evade attribution in cyber crime investigations, mask the identities of botnet devices, circumvent block lists and firewalls, and prevent compromised devices from generating alerts. Typically, IP address spoofing acts as a precursor to more complex attacks, such as denial of service (DoS), distributed denial of service (DDoS), spam, and phishing—rather than constituting an attack on its own. In particular, reflection and amplification attacks in the context of DoS/DDoS exploit spoofed source IPs to induce auxiliary servers to send overwhelming volumes of traffic to the target. Similarly, MAC address spoofing is used to bypass weak authentication mechanisms like MAC address allow-listing and to facilitate ARP poisoning for MITM attacks, wherein bidirectional communication is disrupted as responses are misdirected. A notable exception is double spoofing [6], which involves manipulating TCP sequence numbers by exploiting vulnerabilities in legacy sequence number generators, thereby enabling attackers to glean information about a target without exposing their actual IP address.

To achieve effective endpoint detection when MAC and IP addresses have been manipulated, EDR systems can use system attributes such as OS information, active services, and host keys as well as behavioral indicators such as average inbound and outbound promiscuity, traffic producer–consumer ratios (PCRs), and top talkers. Although these traffic characteristics may fluctuate significantly in the short term, they provide valuable data to develop a statistical model unique to each endpoint. Over time, these features stabilize, enabling precise endpoint behavior modeling. Effective and accurate endpoint modeling requires data collection at Layer 2, as MAC addresses change when traffic is routed between local area networks, making it essential to include inter-LAN communication and capture MAC address information from the traffic. To capture such data, a distributed sensor network is required. Figure 1 shows data collectors operating on Layer 2 routers or switches installed on each LAN or VLAN and a coordinator we call the ’core’ operating on Layer 7 for coordination and storage. The green arrow shows the flow of sensor data and the red arrows show the network connections. The placement and data flow are depicted using a simple two-subnet network but can be extrapolated into more complex topologies where sensor nodes will remain in each LAN while the core node will move up to the backbone.

2.2. Endpoint Security

Initially, attackers perform a reconnaissance of the network topology and its endpoints. This phase offers vital insights into the network layout and identifies intermediate devices that might need to be breached before the attacker can access the target endpoint. Reconnaissance and compromise are an iterative and incremental process. In Figure 1, the attacker’s endpoint performs host discovery and vulnerability scanning on the same LAN using the A1 connection and on other LANs using the A2 connection. In this example, the collection of network data from the router or the Layer 3 switch does not reveal the A1 connection. More complex networks may have a deeper hierarchy in which the A2 connection may also not be detected.

Vulnerable endpoints are exploited using payloads for known vulnerabilities, zero-day vectors, and brute-forced login credentials. Endpoints can also be compromised through insider threats, where disgruntled employees or individuals with authorized access inadvertently or deliberately facilitate unauthorized access, which is often through social engineering or physical access methods. Once an endpoint is compromised, an attacker can escalate privileges to the root or administrator level by exploiting various vulnerabilities. This allows the attacker to disable preventive software such as antivirus, anti-malware, and EDR agents. Furthermore, adversaries can deploy malicious tools as part of an advanced persistent threat (APT). A command-and-control (C2) channel is typically established, facilitating communication with the attacker. This is usually accomplished via a temporary virtual private server (VPS) hosted on the Internet, utilizing DNS tunneling with a predetermined domain name. In Figure 1, the attackers use the A3 connection to establish the control channel with C2. Although perimeter protection systems are designed primarily to detect attacks at the network boundary, they may not have sufficient visibility into lateral movement or stealthy internal reconnaissance, potentially allowing an attacker to advance far before detection. More advanced threats can be fully autonomous, staying silent until the target endpoint has been reached and they are ready to exfiltrate the data. Data are usually exfiltrated to an attacker-owned VPS via DNS tunneling, email, Bittorrent protocol, file transfer protocol (FTP), or secure shell (SSH), uploading to a cloud service or paste site such as Pastebin or Github, or on physical media [7]. Additionally, sophisticated attackers may leverage Tor for anonymized data transfer, use messaging apps like Telegram for discreet communication, store data in cloud storage hidden under legitimate content, or even utilize blockchain technology for obfuscation and decentralized data hosting. By the time perimeter security devices detect the attack, a substantial number of devices within the network may already be compromised. In the worst-case scenario, these attacks could be completely overlooked because current systems primarily seek known indicators and signatures [8].

2.3. State-of-the-Art

Figure 2 illustrates a variety of network security solutions, which are categorized by deployment location and network stack layer. Solutions such as threat intelligence gateways, firewalls, and IDS/IPS focus on securing the network infrastructure itself. In contrast, antivirus, anti-malware, DLP, WAFs, and email security solutions focus on protecting specific services or endpoints. Network segmentation, often using a demilitarized zone (DMZ), helps isolate Internet-exposed resources, which are more susceptible to threats. Within the DMZ, resources are managed to integrate agent-based EDR systems. Additional security measures include analyzing network flows [9,10], monitoring system resources such as CPU and RAM [11,12,13], tracking file system changes [14,15,16], and examining system logs [17]. Solutions tailored for mobile devices have been identified [18,19]. Agentless EDR systems offer enhanced protection for unmanaged devices, such as mobile and transient bring-your-own devices (BYOD), utilizing network traffic data and endpoint telemetry, although they are not yet extensively documented in the literature.

Table 1. Commercially available EDR systems and their capabilities (adapted from [20]).

Name	Mobile	On-Prem	Cloud-Based	Agentless
Microsoft Defender				Yes
Infocyte				Yes
Cybereason				Yes
Bitdefender		Yes	Yes	No
BlackBerry Cylance	Yes	Yes	No
Broadcom (Symantec)		Yes	No
Check Point	Yes	Yes	No
Cisco			Yes	No
CrowdStrike			Yes	No
ESET			Yes	No
FireEye			Yes	No
Fortinet			Yes	No
F-Secure			Yes	No
Kaspersky			Yes	No
McAfee		Yes	Yes	No
Panda Security			Yes	No
SentinelOne			Yes	No
Sophos			Yes	No
Trend Micro			Yes	No
VMWare Carbon Black			Yes	No
Red Hat Openshift			Yes	No

summarizes commercially available EDR systems and their capabilities. Only Microsoft, Infocyte, and Cybereason vendors have agentless EDR products. We also compare them with respect to the coverage of mobile devices and whether the appliance is hosted on the client’s premises or on the vendor’s cloud. On-prem installations can provide more flexibility, and cloud-based systems can provide more up-to-date threat analysis.

Microsoft Defender for Endpoint (MDE) [21] primarily uses a persistent agent-based architecture with limited agentless capabilities for network device discovery and vulnerability assessment, which requires sensor deployment for full endpoint telemetry. In contrast, Infocyte’s [22] hybrid architecture utilizes Forensic State Analysis (FSA) for both agentless assessments and ongoing monitoring. MDE streams the real-time telemetry of process creation, memory activities, and network communications with throttling to optimize bandwidth, while Infocyte allows adjustable collection frequencies for continuous monitoring and periodic forensic captures. Latency varies across each system, with MDE offering near-real-time detection with variable latency based on processing queues and ML model execution. Infocyte’s latency depends on the deployment mode with continuous monitoring being fast and scheduled captures introducing periodic delays. Privacy implications differ in that MDE collects extensive behavioral data with a 180-day retention in Azure, requiring exclusion policies to address privacy concerns, while Infocyte’s FSA can reduce monitoring overhead but needs extensive allow-listing and raises issues with policy separation between servers and workstations.

InMesh offers a fundamentally distinct approach compared to open source solutions such as Zeek [23] and Suricata [24]. Although these tools possess notable strengths, they face challenges in subnet-level attribution. Zeek excels in detailed protocol analysis and network traffic behavior, whereas Suricata serves as a high-performance IDS/IPS that utilizes signatures and anomaly detection for network flows. However, both tools typically operate at a higher network layer, which can hinder their ability to accurately attribute traffic to specific endpoints within a subnet, especially in dynamic environments where IP addresses frequently change. In contrast, InMesh improves detection capabilities by collecting data at each LAN before routing subnet traffic, enabling more precise endpoint identification. This LAN-centric approach mitigates the challenges faced by traditional tools, ensuring that attacks often overlooked by systems reliant on passive analysis or signature-based detection are effectively captured and analyzed.

The number of datasets available for EDR system development that include both network traces and endpoint telemetry is very limited. One such dataset is CIC-IDS2017 developed by the Canadian Institute for Cybersecurity at the University of New Brunswick [25]. CIC-IDS2017 contains five days of simulated traffic and endpoint data. Protocols considered include email, HTTP, HTTPS, FTP, and SSH. Implemented attacks and vulnerabilities include Brute Force FTP, Brute Force SSH, DoS, Heartbleed, Web Attack, Infiltration, Botnet, and DDoS. CIC-IDS2017 comprises more than 80 network flow features as well as endpoint data ranging from memory dumps to system calls from hosts running Microsoft Windows, Apple MacOS, and Ubuntu Linux. The main drawback is that the data are synthetic.

3. InMesh EDR

In this section, we describe the InMesh system architecture with a focus on data collection and automated endpoint detection. Accurate endpoint detection is vital for forensic investigations carried out by security analysts. InMesh provides the precise address-to-timestamp mapping needed to facilitate the extraction of flow data and SIEM logs linked to a specific host for a given time frame. InMesh also enables the continuous monitoring of behavioral changes over time, allowing baseline behavior to be established as well as anomalies or malicious activities to be detected.

3.1. Data Collection

The discovery of endpoints is a prerequisite for their subsequent detection. This process is facilitated by sensor nodes that use both passive detection and active probing. Passive detection is achieved by having the sensor nodes observe the initial packet transmitted by a newly connected endpoint. This typically involves the endpoint broadcasting a gratuitous ARP message, which serves to announce or update its IP and MAC address mapping across the subnet. Unlike conventional ARP, gratuitous ARP is an unsolicited broadcast initiated by the endpoint itself. Active probing involves periodic querying of the subnet for endpoints that have yet to communicate. This is achieved using ARP packets for IPv4 networks and the Neighbor Discovery Protocol (NDP) for IPv6 networks. The probing frequency is influenced by the size of the CIDR IP address range. The minimum time required to discover an endpoint is approximately 250 ms, but actual discovery times depend on network conditions, device power configurations, and, in the case of wireless connections, Wi-Fi signal strength. It is important to note that devices that connect to the network without transmitting packets or responding to probes are considered non-threatening and are not monitored.

Once an endpoint is discovered, the sensor node actively probes to extract system characteristics, which is followed by passive monitoring of its network traffic for a predetermined duration known as $t_S$. An endpoint snapshot, which includes the sensor node ID and the collected information, is created, timestamped, and securely transmitted to the core node via a WireGuard VPN configured in a hub-and-spoke topology. This VPN ensures authentication and encryption for communication between the sensor nodes and the core node as well as between federated sites. To optimize communication efficiency, feature reduction is employed, minimizing overhead between federated sites. As illustrated in Figure 3, federation enables the secure sharing of endpoint models across geographically dispersed sites. This capability enhances visibility beyond the network perimeter, improves behavioral insights during roaming, and reduces the time required to establish new endpoint models when a device connects to a federated site for the first time.

A network flow is primarily identified by the five-tuple: Layer 2 MAC address, Layer 3 source and destination IP addresses, and Layer 4 protocol. Network features are extracted from packet headers, including TCP sequence numbers, TCP window size, type of service (TOS) value, time to live (TTL) and timestamps. Additional statistics are generated after binning packets into flows, such as packet count, byte count, flow duration, packet loss, retransmissions, packet rate, inter-packet arrival time, and runtime.

When the destination endpoint identifies missing packets using TCP sequence numbers, it requests their resending. The source retransmits these packets, and we capture the number of retransmissions per flow. The jitter is calculated when the packets arrive out of order, also using TCP sequence numbers. High jitter can negatively impact VoIP, video conferencing, and streaming.

Network features are categorized into source and destination groups, encompassing metrics such as the number of packets, bytes, application-level bytes, retransmissions, packet rate, inter-packet arrival time, jitter, TCP window advertisement, and statistics such as mean, maximum, and minimum packet sizes. We also compute percentage metrics for specific features, including packet loss and retransmissions. These features are gathered from packet headers by monitoring traffic from the mirror port. InMesh does not require deep packet inspection to collect payload data, eliminating the need for TLS decryption, which can be resource intensive.

Endpoint features are derived from special packet responses that require at least one open port and one closed port for effective OS detection. An open port indicates an active service responding to incoming messages. Sometimes, ports might be filtered, whereby communication is blocked by a firewall or network obstacle, preventing any response to probes. In contrast, closed ports lack listening services. The port status can change when services are deployed, responding to requests. For endpoints with public IP addresses, we collect reverse DNS data. We also gather the device type, common platform enumeration (CPE), OS information such as kernel version, hop distance from sensors, and traceroute details. For each open port, we record the port number, service name and version as well as service data including host keys, HTTP headers, and banner information. Ingress and egress promiscuity is calculated for the endpoint as a whole and per open port.

In total, 128 network features and 13 + N endpoint features are collected, where N is the number of open ports. For a full description of these features, we refer to [26].

3.2. Feature Reduction

The goal of endpoint detection is to accurately attribute activities performed by each endpoint even when their MAC or IP addresses have changed or devices have physically been moved to different subnets. As an agentless EDR system, InMesh performs endpoint detection at the core node using endpoint snapshots. System attributes, such as OS and port information, provide useful information on the system characteristics of the endpoint. Traffic activity statistics help characterize the behavior of the endpoint on the network. However, not all the collected features are necessary or suitable for endpoint detection.

Table 2. Features used by InMesh for endpoint detection.

Network Features (Numerical)	Endpoint Features (Categorical)
Average flow duration	Operating system
Average TCP window size	Device type
Average packets per flow	Hop distance
Average packet rate	Number of open ports, N
Average PCR	Open port 0 (4-tuple)
Average time to live	Open port 1 (4-tuple)
Ingress promiscuity	⋯
Egress promiscuity	Open port N-1 (4-tuple)

lists the subset of endpoint and network features that are ultimately used. Next, we describe how these features were selected. Note that network features are numerical, while endpoint features are categorical.

The analysis of numerical features often involves some type of data transformation. Common approaches involve standardization, which involves subtracting the mean and dividing by the standard deviation, and normalization, which involves scaling using the minimum and maximum values. However, the recalculation needed each time a new endpoint snapshot is added is not scalable. Furthermore, endpoint behaviors tend to drift over time, which could be problematic for transformations based on historical values. Instead, we simply apply a logarithmic transformation to minimize the effect of large data differences.

Some network features are derived using others, such as total bytes, which is the sum of source and destination bytes, and total packets, which is the sum of source and destination packets. The network features that characterize a single flow, such as the protocol and TCP options, are not useful outside of the context of that flow. We do not consider these features for endpoint detection. We also exclude MAC and IP addresses from the network features. Furthermore, we omit features that are not common to all networks, such as the autonomous system (AS) number, which is used for routing packets between Internet Service Providers (ISPs), and the multiprotocol label switching (MPLS) identifier, which is considered the Layer 2.5 that provides faster switching in wide area networks (WANs). This reduces the endpoint description to 48 network features.

Feature reduction techniques that map to a lower-dimensional space, such as principal component analysis (PCA), are detrimental to security analysts who need to understand the physical meaning of features when investigating an incident [27]. Instead, we use information entropy to eliminate features in the original high-dimensional space [28]. Here, we compare AIC and CAIC which are defined as follows:

$$\begin{array}{ccc}\hfill \mathrm{AIC}\left(k\right)& =& -2logL\left({\theta }_k\right)+2k\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill \mathrm{CAIC}\left(k\right)& =& -2logL\left({\theta }_k\right)+(1+logN)k\hfill \end{array}$$

(2)

Variable k represents the number of features considered, while ${\theta }_k$ denotes the corresponding feature vector. The term $L\left({\theta }_k\right)$ indicates the value of the maximum likelihood estimate and N stands for the total number of samples. CAIC penalizes the use of many features more strongly than AIC. Although not discussed here, recursive feature elimination could be an alternative method to consider. Figure 4 demonstrates the application of the AIC and CAIC criteria to the 48 network features. Although AIC shows relatively poor differentiation, CAIC indicates that good performance can be achieved with nine features. The post-analysis reveals that these nine features cover 99% of the endpoint-relevant information present in the flow data.

The small subset of 4 + N endpoint features considered contain those that are universally available for all endpoints and are not correlated with other features, namely, OS type, device type, hop distance and number of open ports. Reverse DNS is excluded, as the information may suffer from incomplete local domain names. OS details such as kernel version are excluded, since they are difficult to source. Traceroute is excluded, since hop distance provides similar information.

3.3. WRCCDC 2020

We used Western Regional Collegiate Cyber Defense Competition (WRCCDC) 2020 [29] packet captures to test InMesh’s endpoint detection capabilities in an adversarial network environment. The packet captures were obtained from a live capture-the-flag (CTF) competition where red team adversaries were given access to isolated network pods containing enterprise-grade Cisco routers, Palo Alto firewalls, and various virtual machines through jump boxes, and the blue team was tasked with defending the network. The red team employed various attack services and vectors including system access exploitation through buffer overflow attacks, credential dumping scripts, exploitation frameworks like Metasploit, and default credential attacks. For data theft, they utilized SQL injection queries, file inclusion techniques, automated scripts, and MITM attacks via ARP spoofing. Service disruption was achieved using SYN-flood tools, resource exhaustion scripts, deliberate corruption of configuration files, and tools to modify service permissions. Penetration techniques included MAC address spoofing, packet manipulation, SSH tunneling, and automated vulnerability scanning. Lastly, for advanced persistence, they installed backdoor shells, created scheduled tasks, modified startup scripts, embedded web shells, and deployed reverse shells.

3.4. Network Feature Modeling

We used the InDepth cyber range [30] to create a simple statistical model of the squared Euclidean distance ($d_{SE}$) between the network features of successive endpoint snapshots to allow InMesh to determine whether an endpoint snapshot belongs to a known endpoint. Four subnets with different firewall rules found in a typical network are modeled. The work reported here mainly used the Clearnet LAN, which contains user devices. The endpoints were made to match the WRCCDC 2020 dataset and thus consisted of hosts running Microsoft Windows, Apple MacOS, Ubuntu Desktop, and Kali Linux. Packet capture files (PCAPs) were used to replay network traffic with the help of the tcpreplay tool. Data were averaged for $t_S=60$ s. We have found this time interval between two consecutive endpoint snapshots to produce good results. Longer time intervals provide more accurate endpoint detection results at the cost of a slower response. Conversely, shorter time intervals lead to a faster response but produce more volatile endpoint snapshots, since the data captured may not contain enough behavioral information. In addition, we noticed that active probing can in practice take up to 60 s, especially for battery-powered mobile devices. If the time between consecutive endpoint snapshots is less than $t_S$, the decision is made with high confidence because the two samples have a high temporal resolution. Devices can become unreachable for a number of reasons, e.g., physical disconnection, poor Wi-Fi signal, network congestion, or power saving starting to occur. If the time between consecutive endpoint snapshots exceeds a DHCP lease period, which is 24 h in most cases, DHCP lease renewal may have taken place. If the lease period expires and the DHCP client has not yet renewed its IP configuration data, then the DHCP client can lose the IP configuration data and begins the DHCP lease generation process again, which may result in a different IP address, although modern DHCP servers attempt to release the same IP if available.

Figure 5a provides a histogram of the distance between the averaged endpoint snapshots produced by the same endpoint. The data follow a one-sided Gaussian distribution with near zero mean and a small standard deviation of 0.9. Close to 94% of the data fall below a threshold of 2.6, corresponding to three standard deviations. Figure 5b provides a histogram of the distance between the averaged endpoint snapshots produced by two different endpoints. The data follow a slightly left-skewed Gaussian distribution with a large mean greater than 100.0 and a standard deviation of 130.3. We conclude that the threshold of 2.6 can serve to discriminate between the same and different endpoints.

3.5. Endpoint Detection

The InMesh endpoint detection algorithm is executed by the core node. Behavioral and spatial information is used to determine whether the endpoint associated with a new endpoint snapshot is known in the network, has been impersonated, has been spoofed, has physically moved to another subnet, or is new altogether. Although ’impersonation’ is achieved by spoofing, we use the term impersonation in the context of an attacker taking over an address that belongs to another device in the network. We use the term ‘spoofing’ to refer to changing an address to avoid detection by using an address that is not in active use. We describe the details next. See Algorithm 1 for pseudo-code.

First, MAC and IP addresses are compared against endpoint snapshots stored for known endpoints to see if the new endpoint snapshot might belong to one of them. A match leads to the calculation of the Hamming distance ($d_H$) for the endpoint features. To handle services that go offline, only open ports are considered, and the port information tuple is aligned using the port number. A value of $d_H$ of 0 or 1 indicates that the endpoint is known, in which case the network features acquired for the considered time window are averaged and compared against the stored average. If the squared Euclidean distance $d_{SE}$ between the two falls below 2.6, the endpoint is considered to be known; otherwise, impersonation is said to have occurred and an alert is raised. A value of $d_H$ greater than 1 is also taken to indicate impersonation. In either case, the new endpoint snapshot is labeled accordingly and is appended to the list of endpoint snapshots for the endpoint for which the MAC and IP addresses matched.

Algorithm 1: InMesh Endpoint Detection

When the search for matching MAC and IP addresses is unsuccessful, the new endpoint snapshot is compared against the most recent endpoint snapshot appended to each known endpoint. When a $d_H$ value of 0 or 1 is observed together with a $d_{SE}$ value less than 2.6, the new endpoint snapshot is considered to represent spoofing if from the same LAN as the sensor node and physical movement if from a different LAN than the sensor node. The new endpoint snapshot is labeled accordingly and is appended to the endpoint for which the features match. If large $d_H$ or $d_{SE}$ values are observed, the endpoint snapshot is viewed to represent a new device which is added to the database that stores all information.

4. Use-Case Scenarios

In this section, we demonstrate InMesh’s ability to accurately detect and assess endpoint behaviors for five different attack scenarios. We used the InDepth cyber-range combined with WRCCDC 2020 packet captures [29] replayed using the tcpreplay tool to replicate real-world threats. Ettercap was used for ARP poisoning, Nmap for reconnaissance, and dnscat2 for DNS tunneling [27].

Scenarios 1, 2, and 3 show that endpoints can be identified using statistical models specific to each endpoint. These scenarios address MAC and IP address spoofing, endpoint impersonation by attackers, and the physical relocation of endpoints across different LAN segments. Scenarios 4 and 5 cover the initiation and conclusion of a typical attack chain. The reconnaissance phase involves scanning for vulnerabilities, while the end stage focuses on data exfiltration through covert DNS channels. These scenarios simulate the workflow of security analysts tasked with identifying endpoints and attributing network traffic when investigating occurrences of data reconnaissance and exfiltration.

4.1. Endpoint Detection When Mac and IP Addresses Are Spoofed

In this scenario, we studied the identification of an endpoint when its MAC and IP addresses have been spoofed. We made a Kali Linux endpoint located within the Clearnet LAN do the spoofing and note that a spoofed IP address must remain within the correct CIDR range. Otherwise, traffic will not be properly routed to and from the endpoint, resulting in a communication failure. InMesh was able to correctly identify the endpoint by comparing endpoint snapshots before and after spoofing. Endpoint snapshots revealed changes in the endpoint addresses, while the behavior of the endpoint remained consistent. The results would have been similar if the spoofing had occurred within another LAN, as the event was contained within the original LAN’s network architecture.

4.2. Endpoint Detection During Impersonation Attacks

In this scenario, the attacker successfully impersonated another endpoint’s identity within the network. Using a Kali Linux endpoint, the Microsoft Windows and Ubuntu Desktop endpoints were able to impersonate each other by spoofing MAC and IP addresses. This type of IP address impersonation is typically associated with MITM attacks and identity masking. To take over a victim’s IP address, an attacker sends an ARP packet containing their MAC address along with the victim’s IP address, which is a process known as ARP poisoning. This can target either the entire LAN or specific devices. For instance, during our tests, InMesh detected the impersonation by capturing replayed packet traffic leaving the victim’s Clearnet LAN. We also simulated an MITM attack in which a Kali Linux endpoint poisoned the ARP table entries related to Microsoft Windows and Ubuntu Desktop hosts to intercept bidirectional traffic between them. In addition to enabling traffic snooping, MITM attacks can disrupt critical systems by redirecting traffic to a non-existent IP address or executing malicious activities while impersonating another device. InMesh effectively detected the MITM attack by identifying the use of identical addresses by two different endpoints on the same LAN.

4.3. Identifying Endpoints When They Move Among LAN Segments

In this scenario, select endpoints were disconnected from one LAN and connected to another LAN, simulating physical movement. Mobile devices, which are rapidly increasing in number, are more difficult to track compared to stationary devices since they connect, disconnect, and move between different subnets. Relying on the MAC addresses obtained when devices are authenticated would be insufficient, since they can be spoofed. The behavioral and spatial information used by InMesh’s endpoint detection algorithm, on the other hand, enabled the physical movement to be identified based on the endpoint snapshots.

4.4. Detecting Reconnaissance Attacks

In this scenario, we investigated reconnaissance attacks with a focus on identifying operating systems and service versions. The primary tool used was Nmap, chosen for its ability to gather comprehensive information about the host, which is crucial in the reconnaissance phase. Potential vulnerabilities were detected by using Nmap to scan well-known ports (0–1023) on the network endpoints. By identifying services with outdated or unpatched versions, an attacker might use a tool like Metasploit to launch targeted attacks based on publicly disclosed security flaws and vulnerabilities (CVEs). The choice to postpone updates to the latest patches or service versions often stems from compatibility concerns, as updates can disrupt existing configurations. This is especially true for systems that are dependent on legacy software or hardware, such as industrial control systems, which makes them highly vulnerable to attacks. Despite these practical challenges, it should be noted that we strongly recommend maintaining all systems with the latest patches and operating system versions. InMesh identified changes in the attacker’s network traffic promiscuity as payloads were distributed to various IPs and ports. In particular, we used default Nmap settings without stealth options to maintain transparency in detection and study the potential network packet signatures of such attacks. It is important to note that the use of stealth options could allow attackers to evade detection, but analyzing this aspect was beyond the scope of the study.

4.5. Detecting Data Exfiltration Attacks

Data exfiltration is often the final step in espionage or sabotage attacks, leveraging unsuspecting protocols such as DNS, HTTP, HTTPS, FTP, SSH, RDP, or ICMP for covert data channels. In this scenario, we simulated data exfiltration using the dnscat2 tool, transferring local data to an external C2 server. This method segments, encrypts, and embeds data into DNS queries, taking advantage of the fact that DNS traffic is often under-monitored compared to more commonly scrutinized protocols. Traditional intrusion detection systems, especially those limited to perimeter monitoring, often fail to identify these attacks. However, the subnet-level data collected by InMesh provided critical information that enabled more precise forensic investigations, identifying subtle shifts in endpoint behavior, and revealing the source of attacks. Thus, the continuous monitoring of network traffic, including metrics like producer–consumer ratio, is necessary to highlight anomalies and interpret the full implications of data exfiltration.

4.6. Discussion

The five scenarios presented illustrate InMesh’s capability to accurately detect and attribute endpoint behaviors in the context of various cybersecurity threats. They encompass the detection of endpoints despite MAC and IP address spoofing, the identification of impersonation attacks through ARP poisoning, the tracking of endpoints physically transitioning across LAN segments, the recognition of reconnaissance activities utilizing Nmap scans, and the uncovering of data exfiltration via covert DNS channels. Collectively, they demonstrate InMesh’s effectiveness in leveraging statistical modeling and behavioral analytics to enhance Endpoint Detection and Response even within complex and adversarial network environments.

Sophisticated adversaries may employ several techniques to evade agentless EDR systems like InMesh by exploiting fundamental limitations in its detection methodology. By maintaining minimal network footprints through extremely slow data transmission rates that fall below statistical significance thresholds, attackers can avoid triggering changes to the endpoint statistical model. Furthermore, adversaries may implement patience-based evasion strategies in which malware remains dormant while connected to monitored networks, only establishing command and control (C2) communications after the compromised device physically disconnects from the supervised infrastructure, effectively exploiting the monitoring blind spot acknowledged in the system architecture. Additionally, advanced port obfuscation technologies can be deployed to mask malicious services during active probing sessions, including dynamic port knocking mechanisms, temporary service suspension during detection windows, and service signature manipulation that presents benign-appearing responses to fingerprint attempts. Such countermeasures can deliberately preserve the endpoint’s statistical profile within expected parameters ($d_{SE}<2.6$) while concealing malicious capabilities, thereby maintaining the appearance of legitimate endpoint behavior despite compromise.

The data collection architecture and unique device fingerprinting capabilities of InMesh may be used to develop machine learning based IDS/IPS systems. However, caution should be exercised, since adversarial attacks that deliberately inject malicious input may impact classification. Such attacks can include gradient-based attacks that exploit mathematical properties to create minimal but effective perturbations; optimization-based approaches that systematically search for inputs maximizing classification error; feature space manipulation targeting influential decision-making features; and decision boundary attacks that methodically identify the minimal changes needed to cross classification boundaries. Mitigating such attacks is beyond the scope of this paper, and we defer development thereof to future work.

5. Conclusions

In this paper, we have introduced InMesh, which is a zero-configuration agentless EDR system. By generating statistical models for each endpoint based on traffic and host data, this novel system is effective in identifying endpoints, even when network identifiers have been compromised and across different LAN segments. Using data from the WRCCDC 2020 cybersecurity competition, we evaluated the efficacy of InMesh by studying five real-world attack scenarios that cover spoofing, impersonation, lateral movement, reconnaissance, and data exfiltration. The experimental work confirmed the ability of InMesh to robustly perform endpoint detection without relying on network addressing and software agents. Future work will focus on improving performance, aggregating authentication logs from external systems such as databases, VPNs, and enterprise applications, and addressing the unique challenges that exist for supporting machine learning based endpoint detection.