Elevating Security in Migration: An Enhanced Trusted Execution Environment-Based Generic Virtual Remote Attestation Scheme

Abstract

Cloud computing, as the most widely applied and prominent domain of distributed systems, has brought numerous advantages to users, including high resource sharing efficiency, strong availability, and excellent scalability. However, the complexity of cloud computing environments also introduces various risks and challenges. In the current landscape with numerous cloud service providers and diverse hardware configurations in cloud environments, addressing challenges such as establishing trust chains, achieving general-purpose virtual remote attestation, and ensuring secure virtual machine migration becomes a crucial issue that traditional remote attestation architectures cannot adequately handle. Confronted with these issues in a heterogeneous multi-cloud environment, we present a targeted solution—a secure migration-enabled generic virtual remote attestation architecture based on improved TEE. We introduce a hardware trusted module to establish and bind with a Virtual Root of Trust (VRoT), addressing the challenge of trust chain establishment. Simultaneously, our architecture utilizes the VRoT within TEE to realize a general-purpose virtual remote attestation solution across heterogeneous hardware configurations. Furthermore, we design a controller deployed in the trusted domain to verify migration conditions, facilitate key exchange, and manage the migration process, ensuring the security and integrity of virtual machine migration. Lastly, we conduct rigorous experiments to measure the overhead and performance of our proposed remote attestation scheme and virtual machine secure migration process. The results unequivocally demonstrate that our architecture provides better generality and migration security with only marginal overhead compared to other traditional remote attestation solutions.

1. Introduction

Over the last two decades, distributed technologies have undergone rapid evolution, finding applications in diverse scenarios such as cloud computing infrastructure, big data processing, blockchain, and edge computing. Among these, cloud computing stands out as one of the most pivotal and extensively utilized application domains within distributed systems. It has emerged as the preeminent choice for computing and storage solutions for myriad organizations.The advent of cloud computing offers a multitude of advantages, including resource elasticity, scalability, efficient resource sharing and utilization, and cost reduction in operation and maintenance. These benefits enable users to direct their focus exclusively towards their core business activities, without the need for concerns related to infrastructure maintenance and management. This, in turn, significantly enhances operational efficiency, attracting an increasing number of enterprises, corporations, and governmental agencies to adopt cloud-based solutions for their core business operations.

Nonetheless, while cloud computing offers substantial advantages, it simultaneously ushers in a host of novel security risks and challenges. Diverging from traditional, localized user–business operations, the decision to embrace cloud-based solutions signifies a departure from complete trust and control over the cloud environment in which the business operates. This departure arises due to the shared hardware resources among multiple tenants within the cloud, preventing the user’s business from running within a dedicated, fully trusted domain. Additionally, it remains uncertain whether cloud service providers can be unequivocally deemed trustworthy. The assumption that cloud operators will perpetually act with integrity cannot be taken for granted. In cases where cloud service providers harbor malicious intent or employ individuals with malicious motives internally, security breaches originating within the cloud can lead to more severe consequences. As a result, the imperative concerns encompass how to assure the security and privacy of user-uploaded business data within the cloud and how to instill trust within the cloud environment. These issues have surged to the forefront as pressing challenges demanding resolution.

In conventional hardware environments, remote attestation technologies [1,2,3,4,5] play a pivotal role in verifying confidentiality, integrity, and the trustworthiness of an entity operating within a secure execution environment, thus ensuring security and privacy while establishing an unbroken chain of trust. Remote attestation typically relies on a trusted root as the foundational point for trust. However, in virtualized contexts, such as cloud computing, the absence of a firmly anchored trusted root introduces challenges in constructing a trust chain within the cloud environment as seamlessly and directly as in physical settings. Consequently, preserving the confidentiality, integrity, and security of assets stored in the cloud proves intricate. In response to these challenges, virtualization security technologies, including vTPM [6], SGX [7], SEV-SNP [8], and TDX [9], have been devised to forge trusted roots or establish secure execution domains within virtualized environments, ameliorating the complexities associated with remote attestation in virtual scenarios. Among these technologies, vTPM virtualizes the Trusted Platform Module (TPM), emulating the functionalities of a physical TPM within a virtual machine. This furnishes a secure storage area for the virtual machine to safeguard keys, certificates, and other sensitive data, offering encryption and signature capabilities to ensure the security of the virtual machine’s data and communication. In contrast, SGX and TDX are predominantly employed to cultivate secure and trustworthy execution environments within virtual machines, segregating code and data within a Trusted Execution Environment (TEE) to shield them against potential attacks. SEV-SNP introduces encrypted virtual machines, incorporating features such as encrypted memory, memory integrity protection, and hardware nested paging to accomplish virtual machine isolation and augment data security in virtualized environments.

By leveraging the aforementioned virtualization security technologies, researchers have introduced a multitude of virtual remote attestation (VRA) schemes founded on these technologies [10,11,12,13,14] to enable remote attestation of entities operating within a distributed virtualization cloud environment. Regrettably, while most of these schemes meticulously address the design and architecture of the VRA process itself, they often overlook the critical aspect of facilitating secure virtual machine migration. Furthermore, even though there are solutions [10] advocating for stateless and transient vTPM to mitigate data leakage during migration, this approach still falls short in addressing the challenge of preserving vital information post-migration, and it struggles to meet the demanding security and integrity requisites of migration procedures. Simultaneously, the majority of existing VRA schemes rely heavily on one or two specific virtualization security technologies, thereby exhibiting a pronounced reliance on precise hardware prerequisites and technical implementations. As a result, they lack the versatility necessary to adapt to diverse hardware configurations and technical environments. Given the diverse array of infrastructure offered by various cloud service providers in a multi-cloud setting, characterized by varying hardware components, a pressing need emerges for a universally applicable VRA solution that can transcend these hardware disparities.

In this paper, we propose a novel approach named VRASM, a generic virtual remote attestation scheme which is based on enhanced TEE. VRASM is designed to address the critical need for ensuring the secure migration of virtual machines (VMs) and eliminating the reliance of VRA on specific hardware configurations. Unlike many VRA schemes that impose specific requirements for the implementation of TEE, VRASM disregards the particular physical instantiation of TEE. Whether it involves an Intel security processor, AMD security processor, or other security chips, as long as the hardware can demonstrate its capability to provide TEE functionality, it can be employed within the VRASM scheme. Moreover, VRASM enhances TEE functionality by introducing a Virtual Root of Trust associated with a physical hardware trusted module (HTM) in TEE and facilitates secure cross-cloud and even cross-hardware migration of virtual machines through controller assistance. Consequently, VRASM represents a generic virtual remote attestation scheme that is not reliant on specific hardware.

Contributions. In short, this paper makes the following contributions:

• We propose a generic virtual remote attestation scheme which can be adapted to multiple hardware environments.
• We present a novel virtual remote attestation scheme which supports the secure migration of virtual machines across clouds without loss or leakage of confidential data and sensitive information.
• We propose a VRASM scheme which bestows VM with a Virtual Root of Trust (VRoT). This VRoT not only functions as an identity credential for VMs but also serves as a pivotal trust anchor, enabling VMs to place their trust in a virtualized environment.
• We evaluate the performance overhead introduced by the proposed schema and the overhead required for VM migration through the presented scheme.

Roadmap. The rest of this paper is organized as follows. Section 2 presents the background. Section 3 gives the overview of VRASM. Section 4 describes the specific details and process of the VRASM scheme. Section 5 introduces the overall security migration process. Section 6 shows the evaluations of the scheme and migration. Finally, Section 7 gives the conclusions of this paper.

2. Background

In this section, we will provide a concise introduction to Trusted Execution Environments and remote attestation, as these techniques constitute the foundational elements of the VRASM scheme.

2.1. Trusted Execution Environments

Trusted Execution Environments (TEEs) are secure computing environments that amalgamate both hardware and software elements, dedicated to enforcing isolation and safeguarding sensitive data and mission-critical code against threats like malware, physical attacks, and side-channel vulnerabilities. The fundamental constituents of a TEE typically encompass a secure processor, a TEE-specific operating system, application containers, and security libraries. These components collaborate to establish a secure execution environment, commonly referred to as a Trusted Execution Environment, where applications and data can operate securely, impervious to interference from the host operating system or other applications. TEEs leverage hardware security modules (e.g., ARM TrustZone [15]) or specialized security processors (e.g., Intel SGX [7], Intel TDX [9], AMD SEV [8]). With TEE technology, even in scenarios where the host operating system has been compromised or manipulated by malicious attackers, the code and data executing within the TEE remain highly protected from potential attackers. Hence, it is feasible to securely store sensitive data and execute programs that require protection within the TEE, such as running a remote attestation program to guarantee the authenticity and integrity of the remote attestation process.

2.2. Remote Attestation

Remote attestation is a security technique employed to validate the integrity, authenticity, and trustworthiness of a remote system or device via a network. This approach enables one party to securely assess the state and configuration of another party’s device or software, even in the absence of direct physical access or trust in the remote entity. Remote attestation finds widespread application in scenarios like securing cloud computing, Internet of Things (IoT) devices, and remote servers.

Remote attestation relies on a challenge–response mechanism [16], typically functioning as follows:

• Attestation Request: The party initiating the attestation, usually referred to as the verifier, dispatches a challenge or request to the remote device or system under evaluation.
• Remote System Response: The remote system, acting as the prover, generates an attestation report, which typically encompasses details regarding its hardware, software configuration, and possibly other pertinent data.
• Remote Attestation Evidence: Subsequently, the remote system signs this report using its cryptographic keys and transmits it back to the verifier.
• Verification: The verifier employs cryptographic techniques to authenticate the attestation evidence, ensuring its integrity and its origin from a trusted party. This step may also include verifying the authenticity of the signer’s identity.
• Trust Assessment: Based on the verification results, the verifier can make an evaluation of the trustworthiness of the remote system. If the verification is successful, the verifier can trust that the remote system is in a known, secure state. In case of discrepancies or failed verification, appropriate security measures can be initiated, such as denying access to the remote system.

For a multitude of diverse requirements and applications, a spectrum of remote attestation schemes have been proposed. These schemes can be broadly classified into two categories based on the specific application scenarios: physical remote attestation and virtual remote attestation. Traditional remote attestation schemes predominantly fall under the category of physical remote attestation. They can be further categorized into software-based RA [1,11], hardware-based RA [17], and software/hardware hybrid RA [4,18]. These schemes are primarily designed for applications in the realm of Internet of Things, Wireless Sensor Network, and similar scenarios, where the verification of the actual device’s integrity is paramount. On the other hand, virtual remote attestation [10,11,12,13,14] primarily finds utility in virtualization environments, where the focus lies on validating the confidentiality, integrity, and trustworthiness of virtual machines within cloud computing environments.

Remote attestation is crucial for security and trust in distributed and cloud environments, especially when the verifier cannot have complete trust in the remote systems it interacts with. It helps prevent various types of attacks and provides a level of assurance about the integrity and configuration of remote devices or software. Technologies like TPM, Intel’s Software Guard Extensions (SGX) (Intel, Santa Clara, CA, USA), Physical Unclonable Functions (PUFs) [19], and various cryptographic protocols are often used in implementing remote attestation.

3. Overview of VRASM

VRASM endeavors to tackle the challenge of remote attestation the trustworthiness of virtual machines and ensure secure migration. To this end, we propose an innovative scheme. In this section, we introduce VRASM’s threat model, design goals and key approach.

3.1. Threat Model

Virtual remote attestation schemes are typically deployed within distributed cloud computing environments. In the virtualized cloud environment, security concerns extend beyond the conventional threats such as viruses, Trojans, and hackers, common in traditional attack scenarios. Cloud environments introduce specific threats, including malicious cloud tenants, unscrupulous cloud service providers, and even potentially nefarious internal staff. In an effort to offer comprehensive protection against a wide array of potential attackers, this paper assumes a highly capable adversary with access to the underlying hardware, cloud infrastructure, hypervisor, virtual machines, operating systems, and the ability to compromise them. Consequently, all these layers are considered untrustworthy. As shown in Figure 1a, within the VRASM architecture, our trust is confined to the secure processor and the hardware trusted module. As long as these components remain uncompromised, the TEE maintains internal isolation and protection, safeguarding sensitive data stored within it and any critical operations performed therein.

While the VRASM scheme certainly provides extensive defense against a wide range of attacks, it is essential to note that our threat model does not encompass side-channel attacks or transient execution attacks. A common attack pattern involves exploiting transient vulnerabilities to gain access to data within the TEE and subsequently exfiltrate the data using side-channel attacks. While our scheme does not provide direct defense against side-channel and transient execution attacks, we can draw from existing research for mitigation strategies. For instance, in the case of transient vulnerabilities like Zombieload [20], which pertain to microarchitecture data sampling (MDS), we can employ mitigation measures by deactivating the use of microarchitecture buffers for internal TEE operations. Similarly, for side-channel vulnerabilities associated with the Performance Monitoring Unit (PMU), such as PMU-Spill [21], we can recommend that chip vendors implement mitigations at the hardware level within the TEE by avoiding concurrent use of both mechanisms. Overall, it is important to emphasize that countering transient execution attacks and side-channel attacks primarily falls within the purview of hardware vendors. Consequently, our scheme does not delve extensively into this aspect.

3.2. Design Goals

In light of existing shortcomings, threat models, practical requisites, and other pertinent factors, we have formulated the VRASM schema with the explicit intent of realizing the following three primary design objectives:

• Hardware-Agnostic: The VRASM schema is designed to be hardware-agnostic. It accommodates deployment across a range of certified hardware environments, without being tied to any particular vendor’s hardware or technology. This commitment to hardware neutrality underscores the openness and universality of VRASM schema, thereby affording users a wider array of choices.
• Serves as a trust anchor for virtual machines: The VRASM schema establishes a Virtual Root of Trust for each virtual machine, serving as the foundational point of trust in virtualization environments. This trust anchor paves the way for virtual remote attestation, secure migration, and various other trust-centric operations.
• Enabling Secure VM Migration: The VRASM schema is meticulously crafted to facilitate secure, uninterrupted migration of virtual machines (VMs) to designated secure migration destinations. This process is executed without compromising vital information, safeguarding sensitive data, or succumbing to potential tampering by malicious attackers. This comprehensive approach to VM migration mitigates the associated risks and effectively curtails migration costs.

3.3. Key Approach

With the three primary goals mentioned above, we present VRASM, a novel generic virtual remote attestation scheme. Figure 1a illustrates its components. In contrast to the vTPM architecture depicted in Figure 1b, VRASM features a more streamlined Trusted Computing Base (TCB). The hardware components of VRASM primarily comprise two essential elements: a secure processor endowed with TEE capabilities and a hardware trusted module (HTM).

Presently, all leading processor vendors offer secure processors with TEE capabilities. For instance, certain CPUs in Intel’s Xeon E3 series support SGX functionality, some CPUs within AMD’s Cortex-A series provide TrustZone capabilities, and specific CPUs in the EPYC series offer SEV-SNP functionality. It is imperative to underscore the openness of the VRASM solution—even in cases where a secure processor is not available, users can leverage other hardware components that offer TEE functionality to implement the VRASM solution.

Another integral component essential for the VRASM solution is the HTM. The HTM serves as the hardware trusted root of the solution and the initial point in the chain of trust. Its primary functions encompass the generation and binding of the Virtual Root of Trust within the virtual TEE, as well as ensuring secure and seamless VM migration. It is pertinent to note that the HTM component does not impose specific hardware prerequisites and can be realized through various combinations of functionalities provided by various hardware elements. The HTM component does not entail specified hardware requirements and can be implemented via hardware components like TPM/TCM, HSM, PUF, and similar hardware that offer the relevant functional combinations.

Our VRASM schema facilitates the establishment of a secure TEE within the VM by harnessing the capabilities of both the secure processor and the HTM components. These components play a pivotal role in safeguarding the Virtual Root of Trust (VRoT) and executing the intricate VRA process, which will be expanded upon in Section 4. Additionally, the VRASM schema leverages the HTM component to ensure the secure migration of trusted VMs, with comprehensive insights provided in Section 5.

4. Virtual Remote Attestation

This section provides a comprehensive overview of the VRASM scheme, which is delineated into two primary phases: the initialization phase and the virtual remote attestation phase. The initialization phase encompasses the establishment of the TEE and the creation of the VRoT within it. This phase encompasses the necessary setup procedures required by the VRASM scheme, facilitating subsequent processes such as virtual remote attestation and secure VM migration. In the virtual remote attestation phase, the verifier engages in integrity verification and trustworthiness assessment. This is achieved by meticulously examining the specified evidence of the virtual machine, which has been authenticated by the VRoT.

4.1. Initialization

Figure 2 illustrates the general flow of the initialization phase of VRASM as follows:

(1) TEE Establishment. When a VM is initially created, a secure processor with TEE creation capabilities is responsible for establishing a TEE within the VM. The precise details of TEE creation may vary depending on the specific hardware platform employed. For instance, if an AMD secure processor is used, it employs the SEV-SNP technique to encrypt the entire VM, effectively transforming it into a TEE. Although the exact implementation may differ across hardware platforms, the outcome remains consistent within the context of our VRASM scheme.

(2) TEE Verification. Steps ②, ③, and ④ in Figure 2 show that when the TEE creation is complete, the TEE is required to furnish evidence to the TEE verification server to initiate TEE attestation. Subsequently, the TEE verification server receives this evidence and undertakes a verification process to ascertain the authenticity and security of the TEE. This verification primarily centers around validating the security processor’s factory-injected endorsement key, which serves as a litmus test for the genuine reliability of the security processor. This, in turn, substantiates that the TEE is running in a real and effective environment, as opposed to a spurious environment that could potentially be orchestrated by an attacker. The core objective is to thwart attacks targeting counterfeit TEEs. Presently, most of the prominent processor manufacturers offer TEE verification mechanisms. For example, Intel provides the Intel SGX Intel Attestation Server (IAS) service. Hence, it suffices to validate the TEE using the relevant scheme.

(3) VRoT Creation. Upon successful validation of the TEE, it attains a state of trustworthiness. Subsequently, the HTM undertakes the generation of the essential security key for the VRoT, along with the corresponding signature certificate. These cryptographic artifacts are then dispatched to and securely stored within the TEE, thereby designating the VRoT as the exclusive and paramount trust anchor for the VM. This establishment is effectuated through the invocation of the TEE’s security interface, which in turn initiates a secure communication channel with the VRoT. The VRoT serves a dual role, functioning not only as the VM’s identity credential but also as the authoritative signatory entity for the production of virtual remote attestation evidence. This multifaceted capacity ensures the formal integrity and validity of the evidence, while concurrently bestowing enhanced security for the VM migration process.

After completing the above process, the initialization phase ends, having laid the foundational groundwork for subsequent operational phases. Before advancing to the subsequent phase, it is imperative to provide a more granular details of the initialization phase.

Hardware Trusted Module. The hardware trusted module (HTM) is a critical hardware component of the VRASM scheme. As its name suggests, the HTM plays two pivotal roles within the scheme: firstly, it serves as a Hardware Root of Trust (HRoT) to facilitate the generation and binding of VRoT, thereby initiating the trust chain. Secondly, it assumes responsibility for ensuring the secure migration of VMs.

An HTM must possess a minimum set of functionalities, including key generation, key management, encryption and decryption capabilities, random number generation, secure interaction interfaces, and a secure clock. Additionally, the HTM necessitates elements such as the manufacturer’s signing certificate and the endorsement key, which are intrinsic to conventional Hardware Root of Trust components. Only when these prerequisites are met can the HTM perform the two aforementioned functions. Furthermore, for the purpose of augmenting security and facilitating the realization of additional functionalities, an extension of the HTM may be contemplated. The VRASM scheme exhibits flexibility in HTM implementation, as it solely mandates the inclusion of specific functions, without imposing constraints on the hardware format. Therefore, the HTM can be realized not only as a standalone chip but also as an assembly of constituent components. In the pursuit of HTM realization, designers have the option to create a dedicated HTM chip or consider leveraging a combination of pre-existing physical hardware components, such as TPM/TCM, HSM, and PUF, contingent upon the specific context and requirements.

Furthermore, it is important to note that, due to the possibility of implementing HTM using extended hardware such as TPM, HTM also has the capability to support boot-time attestation. Therefore, if there are security requirements for attestation during the boot process, it is feasible to incorporate HTM into the initialization process described above, ensuring the comprehensive and accurate startup of the virtual machine through HTM during the VM boot phase. In our proposed solution, we have intentionally omitted the inclusion of this process by default, as we have streamlined the integrity verification of virtual machines in the subsequent remote attestation process. However, in consideration of comprehensive security enhancements, we leave the extensibility of the boot-time attestation capability for future specific security requirements.

Virtual Root of Trust. In the VRASM scheme, the VRoT is a minimalist implementation initiated by the HRoT generation. The HRoT undertakes the generation of a secure key pair for the VRoT and employs its private key to sign the VRoT’s public key, along with pertinent information, thereby generating a certificate that securely binds it to the VRoT. This key is then transmitted to the TEE through a secure channel established by invoking the TEE interface. The HRoT retains solely the certificate information, deleting the original copy of the key.

The VRoT serves multiple purposes within the scheme:

• The VRoT enables VMs to authenticate their identity by verifying the VRoT’s certificate, thus thwarting identity forgery and tampering.
• The VRoT supports virtual remote attestation by signing and encrypting the evidence, ensuring the integrity and confidentiality of the VRA process.
• The VRoT guarantees the secure and unaltered migration of VMs.
• The VRoT facilitates encryption, decryption, and signature operations within the TEE, thereby reducing the HRoT’s performance burden, thus preventing performance bottlenecks akin to those experienced by the TPM in scenarios involving multiple vTPMs.

The implementation of VRoT enhances the security performance of the TEE and plays a pivotal role in the VRASM scheme.

4.2. Virtual Remote Attestation

While concluding the initialization phase, the remote attestation of the TEE is accomplished simultaneously. Consequently, we can seamlessly and securely execute subsequent procedures, including virtual remote attestation, within the TEE. The comprehensive VRA process for VRASM is illustrated in Figure 3.

As shown in the steps depicted in the Figure 3, the concrete process of virtual remote attestation in our proposed scheme is outlined as follows. The attestation server initiates the attestation process by sending an attestation request to the virtual machine, instructing it to perform virtual remote attestation. Upon receiving the attestation command, the virtual machine forwards it to the HTM. Subsequently, the HTM, upon receiving the request, sends the certificate of the VRoT corresponding to the virtual machine to the attestation server. After receiving the certificate containing the VRoT public key, the attestation server determines the files and other confidential information that the virtual machine needs to attest, encrypts them using the VRoT public key, and sends the encrypted data back to the virtual machine. Upon receiving the attestation list, the virtual machine securely transports it into the TEE through the TEE interface. The attestation program running in the TEE decrypts the attestation list using the VRoT private key, collects the required evidence item by item according to the list, and hashes them. Finally, the collected attestation evidence is sent to the attestation server, which, in turn, verifies the trustworthiness of the virtual machine based on the provided attestation evidence. It is important to note that communication between the attestation server and the virtual machine is conducted through secure channels such as TLS to ensure security.

In this manner, upon completing the overall process of virtual remote attestation, the attestation server has ample grounds to determine the trustworthiness of the virtual machine. In comparison to other virtual remote attestation schemes, our approach boasts several advantages. Firstly, our virtual remote attestation scheme exhibits universality. Unlike schemes relying on TPM or SGX, our approach only necessitates the presence of TEE and HTM. Consequently, our hardware-agnostic open and universal VRA scheme effectively addresses the challenge of significant variations in environments provided by different cloud service providers in today’s multi-cloud settings, making it applicable across diverse scenarios and preventing vendor lock-in. Secondly, our VRA scheme affords a high degree of flexibility. By freely adjusting the list of evidence items to be attested on the attestation list, our approach dynamically tailors the granularity of remote attestation to balance diverse security and efficiency requirements.

5. Secure Migration

In this section, we will comprehensively delineate the model details within our proposed VRASM scheme, elucidate the secure migration process, and expound specifically on why the migration procedure we propose ensures safety.

5.1. Details of the Model

Before delving into the process of secure migration for virtual machines, it is imperative to first introduce certain parts within the model that have not been previously expounded upon.

• Controller. The controller plays a pivotal role throughout the entire migration process. Its primary responsibilities encompass overseeing the migration of virtual machines, determining the feasibility of the migration, and facilitating secure key exchanges between different HTMs. Simultaneously, the controller assumes a partial role as a attestation server during the migration process, verifying whether the cloud environment meets migration prerequisites and ensuring the security integrity of the virtual machines. Consequently, the security of the controller is paramount in determining the safety of the virtual machine migration process. Hence, it is imperative to deploy the controller in a secure and trustworthy environment.
• Trust Domain. To ensure the security of the controller, it is imperative to deploy it within a trust domain. We recommend placing the trust domain within a secure realm that users can control, allowing them to guarantee the security of virtual machine migration. However, for the sake of flexibility, if users place trust in the cloud service provider, the controller can also be deployed in a secure and trustworthy cloud environment.
• Cloud. A cloud environment that supports secure migration must possess HTM and the capability to construct TEE. In the absence of remote attestation by the controller to verify the hardware capabilities of either migration party’s cloud environment, the controller will deem the migration unfeasible. This precaution ensures the security of both cloud environments involved in the migration and mitigates the risk of malicious attacks on the migrated virtual machines.

5.2. The Secure Migration Process of Virtual Machines

Figure 4 illustrates the overall process of secure virtual machine migration. The numbers and specific steps in the figure correspond one-to-one, and we will provide a detailed explanation of each step in numerical order. Additionally, it is essential to note that communication between HTM and the controller in the process should preferably occur through secure channels such as TLS to ensure security.

• The source virtual machine located in Cloud 1 sends a migration request to the controller situated in the trust domain controlled by the user.
• Upon receiving the migration request, the controller simultaneously conducts remote attestation for both the source virtual machine in Cloud 1 and the destination Cloud 2. This process ensures the authenticity and effectiveness of the HTM and the capability to provide TEE in both clouds. Once both clouds have successfully passed the remote attestation process, the controller can ensure that they meet the security requirements for migration, thereby preventing potential malicious attackers from migrating virtual machines from a secure cloud environment to an insecure one for malicious purposes.
• After completing the remote attestation for the hardware security capabilities of both cloud environments, it is essential to ensure the security and trustworthiness of the virtual machine itself. This step is crucial because if the virtual machine is insecure or under the control of an attacker, it could pose a threat to the security of the destination cloud, leading to lateral movement attacks across cloud environments. Therefore, before migration, it is imperative to conduct virtual remote attestation for the virtual machine, as outlined in the previously described process, to guarantee the security and trustworthiness of the virtual machine.
• Upon the completion of the aforementioned remote attestation and virtual remote attestation processes, the controller will notify Cloud 2 to prepare for the reception of the virtual machine migration.
• After receiving the migration notification, Cloud 2 sends the public key of HTM2 to the controller for later use in the encryption of the virtual machine during subsequent steps.
• The controller sends the public key of HTM2, along with its own public key, to HTM1. The public key of HTM2 will be used to encrypt the virtual machine in step 8, while the controller’s public key is utilized in the TEE to encrypt the hash value of the private key of the VRoT. This encrypted hash is then sent back to the controller, enabling it to verify that the VRoT has not been tampered with before and after migration, ensuring the security and integrity of the VRoT.
• HTM1 invokes the interface of the TEE to send the received public key of the controller into the TEE. Within the TEE, the VRoT hashes its private key and encrypts it with the controller’s public key. Subsequently, the encrypted hash is transmitted to HTM1 via an interface. Finally, HTM1 forwards the encrypted hash back to the controller. The primary objective of this step is to ensure that, upon completion of the migration process and reconstruction of the trust chain, the VRoT remains unaltered, consistent, and secure before and after migration. This safeguards the integrity of the trust anchor for the virtual machine.
• Within the TEE, the content and certain confidential information in the TEE are first signed using the private key of HTM1. This signing process, in step with HTM1’s private key, aims to ensure that the protected content remains intact and consistent with its state before migration upon completion of the migration process. After the signing by HTM1, the confidential information that requires protection is encrypted using the public key of HTM2, received in step 6. The purpose of encryption, distinct from signing, is primarily to safeguard confidential information from unauthorized disclosure. Once both the signing and encryption steps are completed, the virtual machine is prepared for migration.
• The virtual machine undergoes migration. It is noteworthy to mention that the proposed solution can accommodate both cold migration and live migration scenarios.
• After the completion of the virtual machine migration, HTM1 sends its public key to the controller. This step is crucial for subsequent processes to verify the ongoing validity of HTM1’s signature. This verification serves as evidence that the virtual machine migration process was not compromised or tampered with by attackers, thereby confirming the security of the migration process.
• The controller forwards the received public key of HTM1 to HTM2 for subsequent signature verification in the upcoming steps.
• At this stage, Cloud 2 has successfully received the migrated virtual machine. It is imperative to initiate the reconstruction of the TEE within the virtual machine, which entails completing the VRASM initialization process. This step is essential for reconstructing the trust chain from HTM to the VRoT. Once the TEE is established, subsequent operations must be performed within the TEE to ensure the security of the data. Within the TEE, HTM2 decrypts the confidential information using its private key and subsequently verifies the virtual machine’s signature to ensure it originates from HTM1 and is valid, using the public key received in the previous step. With the completion of decryption and signature verification operations, the secure migration process of the virtual machine is essentially concluded.
• To ensure the security of the virtual machine, it is imperative to conduct an additional round of virtual remote attestation after the completion of the secure virtual machine migration process. This step is essential to verify the post-migration security and integrity of the virtual machine.

5.3. Security Analysis

After providing a detailed exposition of our proposed secure virtual machine migration scheme, it is imperative to conduct a comprehensive security assessment of the overall virtual machine migration process.

• Data Leakage Protection. In our proposed solution, before virtual machine migration, the VRoT and critical confidential data within the virtual machine are encrypted using HTM. Consequently, during the migration process, attackers are thwarted from accessing the virtual machine’s sensitive data, thus preventing data leakage. The migrated virtual machine is encrypted with the public key of the destination HTM, ensuring that only the destination can decrypt the confidential data within the virtual machine. Importantly, all encryption and decryption operations take place within the TEE. After the virtual machine arrives at the destination, decryption of sensitive data and the VRoT can only occur within the TEE after its reconstruction, preventing potential data leaks. Additionally, since communication between HTM and the controller is conducted through secure channels such as TLS, the information exchanged during this process remains secure. Therefore, our proposed virtual machine migration scheme provides robust privacy and data security protection.
• Defense Against Tampering. Addressing the integrity protection concern before and after virtual machine migration is a cornerstone of our proposed solution, achieved through a robust signing mechanism. Firstly, before migration, the confidential data within the virtual machine are signed by the HTM at the migration source. After migration, a simple verification using the public key of the source HTM reveals whether the virtual machine data have been tampered with. Secondly, before migration, the hash value of the VRoT of the virtual machine is encrypted using the controller’s key and stored at the controller. After migration, a virtual remote attestation can be employed to verify if the hash value of the VRoT remains consistent with the pre-migration state. Through this dual-signature mechanism, our proposed virtual machine migration scheme effectively ensures the prevention of malicious tampering with critical virtual machine data, providing robust resistance against attacks such as man-in-the-middle attacks.
• Adaptation to Heterogeneous Environments. Our proposed migration scheme exhibits strong versatility, making it well suited for diverse and heterogeneous cloud environments. The requirements for our solution are minimal, necessitating only HTM and hardware with TEE functionality. This results in a small TCB. As a result, regardless of the complexity of the underlying cloud environment, as long as it can provide this minimal TCB, our solution is applicable and ensures the security of the migration process.

6. Implementation and Evaluation

6.1. Implementation Details

The test scenario was based on the premise of implementing the virtual remote attestation and VM Secure Migration on different platforms. The measurement results were utilized to verify VRASM’s adaption to heterogeneous hardware and to assess performance overhead. The two aforementioned processes were conducted on the two following platform:

• SlimTEE: Inspired by OP-TEE, we created a simple Trusted Execution Environment named SlimTEE based on C++ and OpenSSL library. Its security is higher than bare metal operation systems but inferior to complicated TEEs like AMD SEV or Intel SGX. It provides core functions like isolating data from directly reading/writing, calculating hash code for strings, and encrypting/decrypting information by RSA keys.
• Intel SGX: Intel Software Guard Extensions (SGX) (Intel, Santa Clara, CA, USA) is an innovative security technology developed by Intel that enables the creation of a secure area within the processor called an enclave. This enclave is isolated from the normal operation of the computer and is designed to keep the code and data it processes safe from unauthorized access or tampering, even if the operating system or other software is compromised. The official Intel SGX SDK for Linux provides developers with the ability to compile projects in SGX simulation mode, without the need for physical processors to be equipped with SGX, since only certain models of Intel processors are equipped with SGX. Our experiments are conducted based on the SGX simulation mode.

For the process of virtual remote attestation, each platform repeated it 100 times, while for VM Secure Migration from one hardware to another, each platform repeated it 20 times.

In the processes of virtual remote attestation and VM Secure Migration, the HTMs which can be seen as clients are coded with C++, while the attestation/verification server and controller are coded with Python. In C++, the core steps like creating secure communication channels and encryption/decryption are dependent on OpenSSL library, while this work are supported by cryptography in Python. The processes all run on virtual machines with the Ubuntu 20.04 operating system. Further details of the physical hardware platform where the implementations conducted, as well as the respective lines of code for each module, are in Figure 5.

6.2. Evaluation

(1) Deployment and scalability.

The essential functions on the VRASM client side, such as encryption, decryption, and hash code calculation, are based on the OpenSSL library. This foundation provides VRASM with a highly scalable property that allows it to operate effectively across diverse and heterogeneous cloud environments and on various hardware platforms, provided that OpenSSL is installed in the software environment. Figure 6 demonstrates the external interface we designed for VRoT. Figure 7 illustrates how we implement encryption, decryption, and hash calculation in VRASM using different Trusted Execution Environments (TEEs) with minor modifications.

(2) Execution overhead.

Memory. VRASM requires no more than 4096 bytes of permanent storage to store the RSA key pair of VRoT, and no more than 2048 bytes of permanent storage to store the confidential message of VM which is used for attestation. During execution, the usage of maximum temporary storage (RAM) in a single client may come up to 30 KB, for allocating and initializing memory to receive from or send to the server/controller.

Runtime.Figure 8 shows the runtime overhead of virtual remote attestation, VM Secure Migration between SlimTEE and Intel SGX, and on the two platforms, respectively. The results obtained in an environment without TEE are used to demonstrate the impact of network transmission latency and the time cost of encryption and decryption operations within TEE on overall performance.

Discussion. Compared with the experimental results from the environment without TEE, we can see that the additional time cost of VRASM deployed in SlimTEE and SGX is mainly due to the reading of key files, encryption and decryption, and hash value calculation. In comparison, the network transmission latency is almost negligible. At the same time, due to the more complex and reliable design of Intel SGX compared to SlimTEE, the experimental results based on Intel SGX show a significant increase in time cost for both virtual remote attestation and secure virtual machine migration compared to SlimTEE. As demonstrated by the the last two experiments, secure virtual machine migration from SlimTEE to Intel SGX and, reversely, the hardware-independent feature of VRASM have been confirmed. We have verified VRASM’s scalability across heterogeneous environments, and have determined that its overhead in terms of time and memory is within acceptable limits.

6.3. Security

Man-in-the-Middle Attacks. In VRASM, security in communication is provided by the SSL/TLS protocol, which is based on the OpenSSL library. Even if the attacker could steal the encrypted messages from the channel, they still cannot decrypt them since they do not possess the key used by the server and VM. Meanwhile, the majority of message hash codes exchanged are digitally signed with the sender’s private key and subsequently verified by the receiver. This process is designed to affirm the authenticity of the sender and ensure the integrity of the communication. Thus, VRASM can be considered secure against MITM attacks.

DoS Attacks. In the process of virtual remote attestation and VM Secure Migration, the attestation server and controller play essential roles. Even if an attacker drops a message during during the attestation or migration process, the server or controller can still initiate a new run of the process, quickly releasing the port and resource which were maliciously occupied.

Replay Attacks. An attacker might attempt to undermine the VRASM process by delaying or replaying previously transmitted packets. To counteract such tactics, the server and controller would check every message they received if it is needed in the current step. As each step necessitates a unique type of message, any message from a replay attack is promptly identified and discarded. Such design effectively thwarts replay attacks.

7. Conclusions

This paper proposed a novel approach named VRASM, a generic virtual remote attestation scheme which is based on enhanced TEE. VRASM is designed to address the critical need for ensuring the secure migration of virtual machines (VMs) and eliminating the reliance of VRA on specific hardware configurations which are commonly seen in traditional VRA solutions. VRASM introduced the concept of VRoT, which serves as a vital trust anchor for VMs, enabling secure operations within a virtualized environment and providing robust identity credentials. Extensive evaluations demonstrate that VRASM incurs acceptable overhead in terms of time and memory while providing strong security guarantees against common attacks such as replay, DoS, and man-in-the-middle (MITM) attacks. Future work will focus on further optimizing the scheme and exploring its integration with emerging technologies such as blockchain for enhanced trust and transparency in distributed systems.