Architecture for Self-Evolution of 6G Core Network Based on Intelligent Decision Making

Abstract

The rapid progress of 6G mobile communication technologies has sparked a great deal research interests. The 6G core network architecture faces formidable challenges due to the escalating complexity of network service demands and diverse application scenarios. In response, our research endeavors to tackle these challenges by proposing a self-evolving architecture based on intelligent decision making. Inspired by the principles of biological morphological evolution, our architecture empowers the core network to dynamically adapt and reshape itself in order to effectively address the evolving communication environments. To facilitate this self-evolutionary process, we introduce a comprehensive framework encompassing mechanisms, architecture, agents, and algorithms that enable the network to autonomously generate and optimize its own structure, thereby ensuring adaptability to a wide range of application scenarios. By conducting concept proof simulation experiments, we have demonstrated the effectiveness of our self-evolution algorithm, which enables the 6G core network to make rational evolving decisions and exhibit remarkable adaptability to various application scenarios.

1. Introduction

With the large-scale deployment of 5G commercialization, the concept and technologies of 6G (Sixth Generation) have attracted significant attention [1,2]. The 6G mobile communication system is envisioned to be an intelligent infrastructure for universal connection of exponential growth heterogeneous devices of massive application scenarios, such as smart cities, multi-sensory extended reality, and tactile internet, etc. The goal of 6G is to establish a seamless global coverage network across space, air, oceans, and land with high transmission rate, low end-to-end latency, and customized service provisioning [1].

The 6G system primarily distinguishes itself from 5G by its endogenous intelligence capability supported by AI (Artificial Intelligence) to meet the constantly changing requirements of users and applications [3]. AI algorithms are ubiquitous from the cloud to the edge and are applied in many aspects of 6G, such as resource management, service orchestration, network security, and semantic communications [4]. However, it is still unclear how the AI can benefit the 6G core network architecture. To address this question, we review the long history of mobile communication systems and explore potential evolution strategies for the core network architecture. We further classify these evolution patterns into four basic forms: the emergence of new components [5], the elimination of obsolete functions [6], the separation or merger of different network components [7], and changes in component interactions [8]. We refer to these four forms of macroscopic network evolution as Network Morphology Evolution.

The above morphing of network components is designed and evolved by the pure manual effort of expert knowledge and industrial consensus, typically recorded in a series of 3GPP protocols, which is expensive and time-consuming. The previous generations of core networks were generally designed by domain experts under the assumption that the user’s requirements were predefined and predictable [9]. Meanwhile, the design process was often characterized by long research cycles and significant investments in manpower. The resulted fixed-defined architecture of core networks struggled to effectively accommodate the dynamic nature of evolving user demands [10]. Therefore, this design philosophy of core network architecture makes it hard to guarantee the diverse and dynamic customized 6G services anywhere and anytime.

Therefore, a 6G core network capable of automatically morphing its architecture according to the changing scenarios is promising. A 6G core network will explore new architecture without being restricted by traditional design paradigms. Intuitively, AI technology is the natural choice for automatically evolving the architecture of the 6G core network. However, to the best of our knowledge, little thought has been given to how AI can empower the 6G core network architecture with self-evolution capabilities.

Fortunately, research on simulating biological morphology evolution has emerged in the field of machine learning, providing inspiration for our idea of empowering the core network with self-evolution enabled by AI. Reinforcement Learning (RL) is a subfield of machine learning that trains agents to make decisions in an environment by maximizing a cumulative reward signal through trial-and-error interactions, enabling them to adapt and improve performance over time. Gupta Agrim et al. [11] introduced Deep Evolutionary Reinforcement Learning (DERL) to evolve diverse agent morphologies to learn challenging locomotion and manipulation tasks in complex environments. Wang Tingwu et al. [12] proposed the Neural Graph Evolution (NGE) algorithm, using Graph Neural Networks (GNNs) to describe embodied agents and simple mutation primitives to represent continuous evolution in the environment. Yuan Ye et al. [13] proposed the Transformation and Control policy (Transform2ACT) algorithm, which incorporated the design procedure of an agent into its decision-making process. Moreover, this approach enables joint optimization of agent design and control as well as experience sharing across different designs.

We consider that the evolution paradigm of the 6G core network closely resembles that of biological evolution, where the core network architecture resembles the form of a biological organism. In biological evolution, organisms evolve in response to varying environments by changing components in a three-level hierarchical structure, with cells forming tissues, and tissues forming bodies. Under the context of the 6G core network, following the idea of the SBA (Service-based Architecture) framework of 5G [14], there is thought to be a similar three-level hierarchical structure, where Microservices compose Network Functions, and the Network Functions compose the Core Network. To adapt to the changing scenarios, varying microservices can compose different network functions and then core networks. That is, the automatic decomposition and recomposition of elements at each level, triggered by the dynamic user requirements of the 6G system, provide the essential fundamental drive of self-evolution.

Similar to biological evolution, here we use the term self-evolution to refer to the intelligent capability of the 6G core network to autonomously adjust and optimize its structure in response to environmental changes during the operation process. Self-evolution enables networks to better adapt to complex and dynamic communication environments and respond quickly and accurately to different network requirements and application scenarios. The self-evolution paradigm allow the 6G core network to break the static architectures standardized by predefined protocols and to explore arbitrary novel structures beyond the known forms of human expert knowledge. In comparison to architectural research endeavors in 5G, such as Saha et al.’s comprehensive security analysis framework based on machine learning [15] for the 5G core network to enhance its security, the integration of AI in 6G is inherent rather than an externally attached module.

While self-evolution can be achieved under the principles of multiple frameworks, here we specifically adopt the idea of self-generation, which entails the network’s inherent capacity to generate novel structures to effectively adapt to emerging communication environments. As aforementioned, we consider 6G core networks as organic entities, with their network structure resembling the form of a biological organism. As shown in Figure 1, similar to how living organisms interact with the external environment, observe the environment, and choose the best body form and behavior for survival and competition, we propose an intelligent entity named Self-Evolving Agent following the basic idea of reinforcement learning for the 6G core network, aiming to actively morph the core network architecture to try to fit the changing environment by obtaining the feedback reward as much as possible. This process constitutes a closed loop for the agent of perception, reasoning, decision-making, and execution, enabling networks to autonomously learn, continuously iterate, and optimize their existing forms to meet new network requirements and challenges.

To further clarify the novelty of our self-evolution strategy, we review the literature on AI integrated mobile network systems. Currently, numerous research works mainly focus on the use of AI for performance optimization [16,17] in topics including wireless network resource allocation, edge computing scheduling, channel allocation, network slicing, and service chain composition. However, they generally assume that all the optimizations are conducted under the assumption that the network structure is not allowed to change. Thus, their optimization results are restricted by the given network architecture. As this type of optimization can be typically represented as a set of learnable and solvable model parameters of AI algorithms [10,18], we refer to this pattern as Network Parameter Evolution, to parallel with the aforementioned pattern of Network Morphology Evolution. While the Network Parameter Evolution tends to evolve the network at the micro-level with partial optimization in a short time span, such as seconds or minutes, the Network Morphology Evolution focuses on evolving the network at the macro-level, with global changes in a long time span, such as days, months, or even years.

In summary, we aim to empower the 6G core network architecture with self-evolution capabilities to enhance its environmental adaptability. This involves enabling the network architecture and communication protocols to support self-learning, self-generation, and self-operation and maintenance, ultimately leading to a new architecture of the 6G core network. Our primary contributions include:

(1)

We propose an architecture for the 6G core network that enables self-evolution based on intelligent decision making. We elaborate the core ideas and mechanisms of network self-evolution and formulate the self-evolution task for the 6G core network.

(2)

We propose a reinforcement learning-based self-evolving agent, which can be applied in cloud and Edge Core Networks to morph the network architecture in response to the changes of the communication environment.

(3)

Finally, We validate our self-evolution algorithm through experimental simulations. The results demonstrate that the self-evolution agent can make rational evolutionary decisions and enable the core network to exhibit promising environmental adaptability.

The rest of this paper is organized as follows. In Section 2, we introduce the evolution of core network architecture and discuss relevant works on Network Intelligence. In Section 3, we present the Problem Formulation for Network Self-Evolution. In Section 4, we propose the Self-Evolving Pipeline and a deployment architecture for intelligent agents in the 6G core network. In Section 5, we present the details of our algorithm. Section 6 is dedicated to the validation of our algorithm, where we conduct experiments to assess its performance. Finally, Section 7 is used to conclude our research.

2. Related Works

2.1. The Evolution of Core Networks Preceding 5G

The mobile communication system has undergone five generations of evolution, resulting in significant changes in core network architectures. For example, the 4G transitioned from the Universal Mobile Telecommunications System (UMTS) [19] to the Evolved Packet Core (EPC) [20], a pivotal change that separated the user plan and control plan, enhancing the network’s flexibility and reliability. Subsequently, the 5G core network adopted a Service-Based System Architecture (SBA) and embraced Network Function Virtualization (NFV) [21], enabling the core network to evolve and innovate while providing flexible and scalable network services and operations.

Compared to 4G, 5G has not only emerged new functional components, such as the Network Slice Selection Function (NSSF) [5], but also eliminated obsolete components, such as the circuit-switched (CS) domain, which was discontinued in the 4G EPC core network with the introduction of the all-IP network [6]. Additionally, some network components in 4G have undergone separation or have merged, exemplified by the transformation of the Mobility Management Entity (MME) in 4G. Its functions were dispersed into various network functions, such as AMF (Access and Mobility Management Function), SMF (Session Management Function), and AUSF (Authentication Server Function) in 5G [7]. Furthermore, there has been a change in the interactions between network components in 4G, as seen in the adoption of a service bus framework between multiple Network Functions (NFs) in the 5G core network control plane [8], replacing the point-to-point mechanisms utilized in the 4G EPC.These evolutionary processes reflect the continuous efforts to improve network performance, flexibility, and adaptability in response to the ever-growing demands of modern communication.While significant progress has been made in the exploration of core network evolution, there remains few valuable works that focus on self-evolution communication, similar to our research.

2.2. 6G Core Network

In comparison to the 5G core network, the 6G core network architecture should be redesigned to achieve a powerful, flexible, and intelligent network [22,23,24,25,26]. Most existing works have analyzed the challenges of the 6G communication system and proposed new architectural solutions for the 6G network [27,28,29]. Yuanzhe Li et al. [29] proposed a cognitive service architecture for the 6G core network, inspired by the nervous system of the octopus, to enhance the core network and meet the increasing quality of service requirements and complex scenarios. Xinran Fang et al. [30] introduced basic models for integrated satellite terrestrial networks. To achieve seamless global wireless signal coverage, Chao Wang et al. [31] proposed a 6G-supported space-air-ground-sea integrated network (SAGSIN) architecture. Through reconsidering the 5G Service Based Architecture (SBA) functional split, Marius Corici et al. [32] proposed a new “Organic 6G Network” concept and a new service architecture for 6G core networks based on advancements in the software services adopted. Zhang et al. [33] introduced a novel concept called “intellicise”, which describes the integration of next-generation networking technologies and AI in wireless networks. The “intellicise” network actively takes systematic entropy reduction as the global optimization objective, adaptively reshapes information systems, and ultimately endows itself with inherent intelligence and simplicity. Maier et al. [34] introduced an ESPN architecture that leverages artificial-intelligence-enhanced computing to explore the flourishing development of multisensory Extended Reality (XR) experiences within multiverse cross-reality environments in the context of 6G. Notably, Cai Lin et al. [35] proposed a Self-Evolution and Transformation (SET) architecture, where a protocol control agent is deployed in each network entity to handle flow/packet level control. This agent can assemble, configure, and exchange protocol functions, thereby enabling the protocol to change and self-evolve. However, here we specifically focus on the self-evolution within the 6G core network architecture.

Because there is still no standardized consensus on how the 6G core network would look, we adopt the hierarchical architecture proposed in [29] as the backbone of the 6G core network and follow the Service Based Architecture (SBA) as the structure organization principle of Network Function (NF). Specifically, as shown in Figure 2, the 6G core network adopts a two-layer topology with Edge Core networks and Cloud Core Networks. The Edge Core Networks sink to the edge of the network and act like a peripheral control plane for special domains or usage scenarios. The Cloud Core Network plays the role of the central brain to coordinate multiple Edge Core Networks and no longer directly participates in communication. Each layer of the core network is composed of a set of Network Functions (NF). For each NF, there is a two-level service structure. That is, the microservices serve as the fine-grained units and are then composed to form coarse-grained Network Functions (NFs), catering to diverse communication requirements. With the composition capability of different service granularity, the 6G core network would provide connectivity and support to users under various application scenarios. The integrated terrestrial, aerial, and maritime networks in 6G, built upon the foundation of terrestrial cellular mobile networks and deeply integrated with broadband satellite communications, offer extensive coverage, flexible deployment, and efficient broadcasting capabilities. To enable information exchange and sharing between different networks and to provide customized communication services to a variety of users, these networks also require seamless integration with other heterogeneous networks [36].

The two-level structures of each NF in the Edge Core Network are elaborated in the right part of Figure 2. For instance, the AMF (Access and Management Function) [14] is responsible for mobility management and access control, which are composed of microservices for User Authentication, Session Management, Security, and Policy Control. For example, the Mobility Microservice is responsible for tracking the location of user equipment (UE) and managing handovers between different network cells. Similarly, the SMF (Session Management Function) [14] is responsible for efficient session management. It incorporates microservices such as Statistics and Reporting, Session Management, Traffic Control, and User Data Management. These microservices collectively contribute to ensuring smooth session establishment and maintenance. Furthermore, the AUSF (Authentication Server Function) [14] plays a crucial role in user authentication and security operations. It consists of several microservices, including UE Authentication, UE Authorization, Security Policy Management, and User Data Management. These microservices work together to authenticate users, manage security policies, and handle user-related data. Note that for each individual microservice, such as UE authentication microservices, it would be contained by different NFs. As for the NFs in the Cloud Core Network, there is yet no detailed technique solution consensus on them, thus we do not plot the NFs and microservices in the Cloud Core Network as we do for the Edge Core Network. We believe that, following the strategy of self-evolution, there is potential for the Cloud Core Network to find its optimal NF functionalities and the corresponding microservice compositions, which would be left to our future work.

2.3. Network Intelligence

With the exponential proliferation of mobile devices and data, the seamless integration and rapid development of AI and 6G have gained widespread recognition. The key characteristic of 6G is the emergence of network endogenous intelligence, often referred to as AI-enabled networks, which can perceive, analyze, and make optimal decisions autonomously [37]. Research has shown that AI-enabled 6G networks will gradually be applied to major network issues, including advanced radio interfaces, intelligent traffic control, security protection, management, and coordination [38]. Khattak et al. [39] believe that the AI-enabled 6G networks will also have significant impacts on all other related vertical fields, such as the mobile health applications supported by 6G, which will change human life. Cai et al. [40] proposed a 6G mailbox theory to enable distributed algorithm embedding for network intelligence.

The combination of AI and the network emerged earlier than the 6G era. At that time, network intelligence was primarily manifested through automatic network management and orchestration, as well as network optimization leveraging machine learning and big data analysis. As shown in Figure 3, the 3GPP working group has been promoting the standardization process of 5G Network Intelligence. R8-R10 proposed an intelligent strategy known as the Self-Organizing Network (SON) [41]. R15 introduced the 5G Network Data Analytics Function (NWDAF) and R16 defined a centralized architecture for 5G big data analysis services [42]. R17 and R18 designed a layered intelligent network architecture to further advance the application and standardization of Network Intelligence [43]. However, although these efforts have promoted the application and standardization of the industry, there is still a considerable gap between them and the true network intelligence, with limitations in deployment flexibility and scalability.

It is obvious that the approach of utilizing AI to merely “patch” network operations and management in 5G networks is no longer viable. As mentioned in Section 1, what we need is a network agent that can learn and evolve itself according to changes in the environment. Nasralla et al. [44] and Azari et al. [45] introduced the technological evolution from 5G to 6G from different vertical fields of network applications, which laid the foundation for our thinking on the evolution of generations. Lv, Z. et al. [46] studied the evolution and prospects of network architecture and proposed a conceptual architecture for 6G networks that encompasses holistic network virtualization and network intelligence (AI).

Based on the above works, a 6G network will manifest as an intelligent agent with continuous evolutionary capabilities. Current research on the evolution architecture of the 6G core network is still in its infancy stage. There are primarily two obstacles: (1) Designing network architectures for diverse application scenarios requires extensive search space and significant computing power. (2) Evaluating the endogenous AI generated by the 6G network poses great challenges. Given the aforementioned analysis, our research focus primarily lies in establishing an intelligent network self-evolving architecture for the 6G core network.

3. Problem Formulation

To precisely and concisely describe the mechanisms of network self-evolution, in this section we formulate the task of network self-evolution as an intelligent decision-making problem based on Reinforcement Learning (RL) [47]. We first define the service hierarchy for the 6G core network. Then, we model the network self-evolution problem as a Markov Decision Process (MDP) under the RL framework. We define the fundamental concepts of the Environment and the Agent, which is targeted to learn an optimal policy to evolve the 6G core network to maximize the reward from the Environment.

3.1. 6G Core Network Hierarchy

To formulate the self-evolving architecture and mechanism, we need to define the 6G core network composition hierarchy to illustrate the three-level service structure for the Network Function (NF).

Definition 1 (6G Core Network, CN).

At time t, the 6G core network, $C{N}_t$, based on distributed core network technology, is composed of k core network subnets, $SC{N}_t$, where $C{N}_t=\{SC{N}_t^1,SC{N}_t^2,\dots ,SC{N}_t^k\}$. The term subnet here refers to the Edge Core Network and the Cloud Core Network shown in Figure 2, where we would use them interchangeably. Each subnet is independently deployed and operated, and signaling communication is allowed among the subnets through the Cloud Core Network to coordinate their behaviours. Now, we consider 3 types of evolvable elements for each subnet: the Network Function (F), the Network Protocol (P), and the subnet Topology (T). The Network Function (F) is discussed in the following definition. The Protocol (P) presents the signaling message procedures among Network Functions. The Topology (T) indicates the physical connection structure between Network Functions (F). Then, a subnet, $SCN$, at time t can be represented as a set of its evolvable elements: $SC{N}_t=\{F_t,P_t,T_t\}$. The subnet $SC{N}_t^i$ is regarded differently to the subnet $SC{N}_t^j$ when either of their 3 elements are different at time t. With the evolving process forward, the subnet $SC{N}_t^i$ is allowed to change to $SC{N}_{t+T}^i$. When either subnet $SC{N}_t^i$ is changed, the core network $C{N}_t$ is considered to be evolved.

Definition 2 (6G core network Functions, F).

The network functionalities of the 6G core network are assumed to be divided into a set of highly reusable microservices, denoted as $MS=\{m{s}_1,m{s}_2,\dots ,m{s}_k\}$; k is total number of microservices. Each microservice, ms, implements a basic and decoupled network service, such as user authentication, session establishment, etc. Multiple microservices can be recombined together to constitute the new core network functions: $F=\{M{S}_1,M{S}_2,\dots ,M{S}_n\}$. When deploying the 6G core network, each network function element (NF) implements a subset of the core network functions: $NF\subset F$. For example, Figure 4 shows the constituent microservices of AMF, as well as the corresponding functions and data.

3.2. Self-Evolution Task Formulation

Given the definitions of the 6G core network, we formulate the self-evolution task of the 6G core network as a model of Reinforcement Learning (RL) [47]. Reinforcement Learning enables an agent to learn optimal decision-making strategies through interactions with its environment. Specifically, RL is usually modeled as Markov Decision Processes (MDPs), denoted as $M=(S,A,\gamma ,P,R)$, where the elements in the tuple M represent the state space, action space, discount factor, transition probability function, and reward function, respectively. Starting from an initial state, $s_0$, the agent iteratively samples an action, $a_t$, according to its policy, ${\pi }_E$, while the environment generates the next state, $s_{t+1}$, based on the transition function, $P\left(s_{t+1}|a_t{,s}_t\right)$, and assigns a reward, $r_t$, to the agent. The goal of the agent is to learn a policy, ${\pi }_E$, that maximizes the expected total discounted reward: $J\left(\pi \right)=E_{{\pi }_E}\left[{\sum }_{t=0}^H{\gamma }^t{r}_t\right]$, where H is the time horizon experienced by the agent. Following the framework of RL, we define the basic elements of the self-evolution task for 6G Core Network.

Definition 3 (Environment, ENV).

The landscape of 6G core network application scenarios exhibits significant diversity. Specifically, the i-th network application scenario at time t, such as satellite or emergency scenarios, can be denoted as $E_{i,t}$. Here the introduction of time t indicates the constantly changing nature of the application scenarios. The overall application scenarios, $EN{V}_t$, of the 6G core network at time t can be denoted as a collection of K individual application scenarios, i.e., $EN{V}_t=\{E_{1,t},E_{2,t},\dots ,E_{K,t}\}$. The role of the Environment, $EN{V}_t$, is to provide observations and rewards to the Agent to learn the evolving policy.

Definition 4 (Agent).

An Agent is an intelligent model learning the optimal evolving policy. For Cloud Core Networks and Edge Core Networks, they will deploy their own Agents to coordinately maximize the overall rewards by changing their function, F, protocol, P, and topology, T.

Definition 5 (State).

Theoretically, all the information that might drive the changes of the core network would be viewed as the State, including but not limited to the user demands, the current QoS level, microservices contained within the NFs, traffic on physical servers containing the NFs, and CPU and memory utilization of those servers. In practice, the data of State could be collected and provided to the Agent by the network management system. When considering the long-term self-evolution nature of the core network, the State prefers the statistic results over a large time span rather than the direct raw data.

Definition 6 (Action).

The Action is defined as all the legal modification selections applied to the function, F, the protocol, P, and the topology, T, of the core network. The recomposition of existing microservices, the decomposition of services into new microservices, the emergence of a new kind of NF, and the re-deployment of the NFs among physical servers are all possible instances of Action. When the Agent executes the selected actions, the Environment ENV would feedback corresponding rewards to guide the Agent to learn the optimal evolving policy.

Definition 7 (Reward).

The reward can be divided into two categories: functionality reward and performance reward. Because the evolved core network is automatically learned by the Agent instead of being produced by following the pre-standardized solution, it is not theoretically guaranteed to work as well as the human-defined one. Therefore, the functionality reward is used to measure whether the evolved core network would implement the targeted functions of the control plane. The performance reward is usually defined according to the QoS (Quality of Service), QoE (Quality of Experience), and/or operation KPI (Key Performance Indicator) benchmarks to assess the efficiency of the evolved policy. The two types of reward would be appropriately mixed to formulate the evolving goal of the Agent.

We summarize the defined symbols and list their corresponding meanings in

Table 1. Symbol Denotations.

Symbols	Meanings	Symbols	Meanings
$E_{i,t}$	The i-th network application scenario at time t	$C{N}_t$	6G core network at time t
$SC{N}_t^i$	The i-th subnet at time t	$EC{N}_t$	The Edge Core Network at time t
$CC{N}_t$	The Cloud Core Network at time t	F	6G core network function
P	Core Network protocol	T	Core Network topology
$m{s}_i$	The i-th microservice	$MS$	The set of microservices
g	Network evolution gene	${\pi }_E$	Network self-evolution policy
$GLib$	Network evolution gene library	$EN{V}_t$	The application scenarios at time t
$NF$	Subset of the core network functions

for convenience. Some of the symbols will be further explained in the following section.

4. Self-Evolving Pipeline

In this section, we propose a cyclical pipeline for the evolving agent to complete the endless evolution of the 6G core network. The design considerations behind the pipeline stem from insights of the biological evolution process. We posit that each subnet SCN experiences a unique environment and evolves through the intelligent modification of its genes and physical structure. In addition to the pipeline for self-evolution, we also propose a deployment architecture for the intelligent agents in the 6G core network.

4.1. Pipeline Cycle of Evolution

Figure 5 illustrates the self-evolving pipeline for the 6G core network. The pipeline is analogous to the biological evolution process, consisting of three distinct stages: (1) gene generation; (2) gene composition to construct the body; and (3) natural selection to identify the optimal composition. In this context, the gene refers to the microservice, the body is the subnet SCNs, the optimal composition is the evolved 6G core network adaptive to the given application environment, and the natural selection is the self-evolving algorithm that will be later detailed in the next section. It is important to note that due to the constantly changing environments, the three-stage pipeline will be run iteratively to maintain the adaptability of the resulting 6G core network. The period of each pipeline cycle is primarily determined by the rate of changes in the environment. For instance, when application scenarios are slowly changing over a long period of time, the pipeline might evolve once every few months or even years. On the other hand, when a new application scenario suddenly emerges, the pipeline tends to quickly modify the core network structure within days or minutes.

The pipeline begins with the generation of network evolution genes, the most fundamental components in a self-evolving 6G core network. These genes, also known as network microservices, serve as the building blocks for the NFs of the core network. Each gene possesses specific attributes, including its function and the data it processes. For instance, a registration microservice handles user registration requests by collecting user identification information.

There are typically two methods for generating genes: manual definition and deployment by human domain experts, or automatic extraction using machine learning algorithms from the current core network. Considering the increasing complexity of the 6G core network, it is usually beyond human ability to manually generate the microservices, thus we prefer to exploit the latter method. At each time step, t, the evolving agent automatically extracts a set of genes according to the current architecture of the 6G core network to compose the gene library, $GLi{b}_t=\{g_1,g_2,\dots ,g_k\}$. This set is then updated to the next version, $GLi{b}_{t+1}$, by adding new genes or removing old ones. The in-depth details of the machine learning algorithm used for gene extraction will be discussed in the following section.

In the second stage, with the gene library, $GLi{b}_t$, the objective of the pipeline is to reconstruct the subnet SCN by appropriately combining genes into various functions to form a set of NFs. Specifically, given the current structure of $SC{N}_t$, the evolving agent aims to learn the composing strategy, ${\pi }_E$, that decides how to select the genes and compose them into special NFs, which in turn affects the overall structure of the next time-step SCNs. That is, $SC{N}_{t+1}={\pi }_E(SC{N}_t,GLi{b}_t,E_t)$, where $E_t$ is the application environment for the subnet $SC{N}_t$. Different $E_t$ would lead to different variants of SCN, each with its own network function, F, communication protocol, P, and topology, T. For a given $SC{N}^i$, the corresponding strategy, ${\pi }_E^i$, would produce a group of individual microservice composition plans, analogous to the concept of population in the process of natural selection.

In the third stage, the pipeline evaluates the fitness of each individual SCN plan and selects the most suitable one for survival. The evolving agent assesses the composed SCN by assigning corresponding rewards based on metrics such as evolutionary success rate, evolutionary speed, service performance, and resource costs. The evolutionary success rate measures the completeness of the evolved core network’s required functionality. The evolutionary speed evaluates the response time from changing the environment to the convergence time of the evolutionary strategy. Service performance and resource cost are usually used together to measure the trade-off decided by the evolution policy.

With these rewards, the pipeline repeatedly optimizes the gene library, $Glib$, the gene composition plan for SCNs, and the agent evolution policy. Once convergence is achieved, the newly produced architecture of the 6G core network is deployed to replace the old one, completing a cycle of evolution.

4.2. Deployment Architecture of Pipeline

After discussing the working stages of the pipeline, we propose a distributed architecture to deploy the evolving agent. As depicted in Figure 6, the 6G core network hierarchy comprises a Cloud Core Network (CCN) and multiple Edge Core Networks (ECNs), each catering to customized communication scenarios. The CCN is responsible for managing and coordinating ECNs and does not directly involve the processing of user requirements.

At the top-layer of the architecture, the central CCN deploys AI models as its cloud evolving agent. This cloud agent is responsible for gathering information about itself and all ECNs to learn the evolving policy for the CCN. It creates training data and updates its evolving policy to maximize rewards, ensuring that the CCN evolves in line with its agent’s decisions. To optimize resource sharing and reuse the microservices among all ECNs, it is feasible to deploy only a single global gene library, $Glib$, in the CCN. The CCN’s agent is responsible for creating, maintaining, and updating the $Glib$ to ensure that all the ECNs can efficiently access and utilize the shared microservices. By doing so, the self-evolving architecture can achieve greater scalability and consistence while minimizing resource consumption. To enhance the intelligence of the cloud agent and ensure automatic maintenance of the $Glib$, a knowledge graph is often deployed in the CCN, which incorporates domain expert prior knowledge and the evolving preferences of 6G service operators. This allows the cloud agent to learn from the experts’ insights and make informed decisions about how to evolve the network. By incorporating this knowledge into its evolving policy, the cloud agent can improve its performance and help the CCN evolve more efficiently. For instance, we can construct a knowledge graph with the entities and the relationships containing knowledge of 3GPP protocols, Cloud Evolution Agent, and the statistics of the CCN to reduce the decision space of the CCN evolution.

At the bottom-layer of the architecture, there are multiple ECNs designed for various application scenarios. Each ECN is equipped with its own evolving agent, which receives edge data, such as the signaling throughput, the distribution of various signaling types, and quality of service (QoS) of UEs and is responsible for making decisions about how to evolve its host ECN. Specifically, the edge agent learns how to use the genes from the global gene library, $GLib$, residing on the CCN, to compose desirable network functions. The significant changes in the architecture of numerous ECNs will result in changes to the environment of the CCN, such as signaling throughput of the ECN and signaling distribution of the ECN. Changes in environment will result in alterations to the composition of microservices, and thereby evolute the CCN on NFs, and then the Edge Core Network structure. As the application scenario of the ECN changes, as shown in Figure 6, where the applications expand from single autonomous vehicles to a mixture of services including autonomous vehicles, video surveillance, and Internet of Things (IoT), the agent decides to reshape the structure of the ECN from two NFs into three NFs, while also altering the internal microservice compositions of each NF. In the long-term, the evolving policies of different agents will be highly distinct due to their ECNs serving different application scenarios. As a result, the architecture of the ECNs will also vary accordingly. The significant changes in the architecture of numerous ECNs will result in changes to the environment of the CCN, such as signaling throughput of the ECN and signaling distribution of the ECN. Changes in environment will result in alterations to the composition of microservices, and thereby evolute the CCN.

In the hierarchical self-evolution architecture of the 6G core network, the agents in the ECNs and CCNs form a distributed multi-agent system, which means that the evolving policies learned by each agent are not generally independent of others. Due to the complexity of joint decision making among all agents, although there have been valuable reinforcement learning algorithms for multi-agent systems published in the literature, we leave this work for further research. To streamline the deployment process for the evolving agent, the MLops (Machine Learning Operations) techniques would be employed to support agent development, deployment, and operation. In the context of this paper, the agents would prefer to utilize the RL (Reinforcement Learning) framework and associated algorithms to make decisions for the 6G core network evolution. Because the agent is natively integrated into the core network, it offers a distinct advantage over external AI solutions for the current 5G core network in terms of comprehensive state perception, real-time decision-making, economic interaction, and efficient resource utilization.

5. Self-Evolving Algorithm

As shown in Section 4.1, the self-evolving pipeline is composed of three stages from the genetic and evolutionary point of view. In this section, we will provide a clear definition of the three stages from the perspective of algorithm modeling and detail the specific methods for each stage: (1) GLib Generation Stage; (2) Composition Stage; (3) Evaluation Stage.

5.1. Formulation of Self-Evolving Pipeline

As mentioned in Section 3.2, we formulate the Self-Evolving Process of the 6G core network as a framework of Reinforcement Learning, which is usually modeled as a Markov Decision Process (MDPs): $M=\left(S_t,A_t,\gamma ,P,R\right)$. In particular, each stage, ${\phi }_t$, of the pipeline is modeled as an individual MDP. On the one hand, they have different action spaces and different strategies. On the other hand, each stage is closely related to one another, together forming a self-evolving cycle, $SC{N}_{t+1}={\pi }_E(SC{N}_t,GLi{b}_t,E_t)$.

(1) GLib Generation Stage: At stage ${\phi }_t=0$, the agent applies actions, $A_t^G$, to manually generate microservices by splitting the core network functions into reusable building blocks, and then updates the $GLi{b}_t$ with the newly extracted microservices. The strategy, ${\pi }_{E0}$, in this stage makes action selection based on the current environmental information, $E_t$, and gene pool, $GLi{b}_t$.

(2) Composition Stage: At stage ${\phi }_t=1$, the agent applies action, $A_t^C$, to select existing microservices from the $GLi{b}_t$, which is updated in stage ${\phi }_t=0$, and combines them into network elements based on the strategy, ${\pi }_{E1}$. The output of stage ${\phi }_t$ = 1 for each SCN is a set of candidate microservices compositions, constructed by adding or reducing microservices to existing ${NF}_t$, also with connections between ${NF}_t$ to generate static network topology. It is important to note that, at this stage, the agent does not need to directly interact with the application environment.

(3) Evaluation Stage: At stage ${\phi }_t=2$, the agent applies actions, $A_t^E$, to organize the obtained ${NF}_t$ and network topology deduced from stage ${\phi }_t=1$ into a service function chain for its host SCN, and then evaluate arrangement performance by interacting with the environment, $E_t$. The strategy for achieving the above goals is called ${\pi }_{E2}$.

It is important to note that the time scale, t, in our algorithm is different from the commonly used training scale, which may represent a long time range. So the update of network policy is not real-time. At each time step, t, $E_t$ represents all environmental information of the current application scenarios, overall network latency, and CPU and memory usage of ${NF}_t$. The policy of the Self-Evolving agent is denoted as below.

$${\pi }_E=\left\{\begin{array}{c}{\pi }_{E0}\left(A_t^G|GLi{b}_t,{\phi }_t\right){\phi }_t=0\\ {\pi }_{E1}\left(A_t^C|GLi{b}_t,E_t,{NF}_t,{\phi }_t\right){\phi }_t=1\hfill \\ {\pi }_{E2}\left(A_t^E|E_t,{NF}_t,{\phi }_t\right){\phi }_t=2\end{array}\right.$$

(1)

In order to prevent human intervention in network evolution, we do not set reward functions in stages ${\phi }_t=0$ and ${\phi }_t=1$, and only interact with the application environment in the stage ${\phi }_t=2$, which will be helpful to fully evaluate the completion of specified tasks by self-evolving networks. The detailed explanation of the reward function will be provided in Section 5.3.

After we formulate the self-evolving task of the core network, the learning of the optimal evolving policy, ${\pi }_E$, for the agent would be accomplished by popular RL algorithms. Here, we employ Proximal Policy Optimization (PPO) [48], a popular RL algorithm based on Policy Gradient (PG). PPO is particularly suitable for our approach because it has a Kullback–Leibler (KL) divergence between current and old policies, which can prevent large changes, thus avoiding catastrophic failure.

5.2. GLib Generation Stage

The goal of $GLib$ generation is to decompose the Core Network Functions into multiple microservices. This approach follows the insight of decomposing monolithic software into smaller, more manageable components known as microservices [49,50]. By regarding the SCN of current time step t as a monolithic software system, we can decompose it into various microservices with distinct functionalities and corresponding data, which would be used as raw materials to compose new network elements in the second stage, ${\phi }_t=1$. The optimal decomposition of the current SCN is learned through the reward principle of higher cohesion within the microservices and lower dependency among them, which offers advantages of easier maintenance, improved scalability, and reduced coupling risks.

In particular, we employed Go-Callviz [51] to analyze the code structure of the current core network functions. Go-Callviz is an open-source project for visualizing the call graph of the Go program, which shows the relationships between functions and methods, including which functions are called by which others and how many times each function is called. By leveraging this information, the Go-Callviz generates an intuitive graphical representation of the call graph, which serves as the input for our gene decomposition algorithm.

We denote the obtained call graph as a directed graph, $G(V,E)$, where V is a set of entities and edge, $e_{i,j}\in E$, represents the connection between entities i and j. We define three distinct types of nodes in graph G: data node ($V_d$), function node ($V_f$), and module node($V_m$). These nodes correspond to the primary attributes of a microservice, with the first two representing the data and functionality, respectively. The module node, on the other hand, refers to the package under analysis, and it responds to the functional correlation in the graph clustering algorithm, indicating a stronger correlation between functions within the same package. Given the three types of nodes, we define five types of directed weighted edges:

• module–function ($e_{m,f}$): if a function, f, belongs to package m, then the weight of the edge $w_{m,f}=1$, otherwise $w_{m,f}=0$.
• function–function ($e_{f_1,f_2}$): if there are calling relationships between function $f_1$ and $f_2$, then they are connected by edge with weight, as below:

  $$w_{f_1,f_2}=\left\{\begin{array}{cc}{\psi }_{f_1,f_2}^{n-1},\hfill & \mathrm{Invocation}\mathrm{relationship}\mathrm{between}{f}_1\mathrm{and}{f}_2\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  (2)

  where $\psi \le 1$ and n represents the depth of function calls. For example, if there is a function call chain $f_1\to f_2\to f_3$, then $n=1$ in $w_{f_1,f_2}$, while $n=2$ in $w_{f_1,f_3}$.
• function–data ($e_{f,d}$): when function $v_f$ has a read/write operation on data $v_d$, the edge weight between them is defined as follows:

  $$w_{f,d}=\frac{T\left(f,d\right)}{max\left(T\left(d\right)\right)},$$

  (3)

  where $max\left(T\left(d\right)\right)$ represents the maximum number of times all functions, $v_f\in V$, perform read/write operations on data node $v_d$. $T\left(f,d\right)$ represents the number of times function $v_f$ performs operations on data node $v_d$.
• data–data ($e_{d_1,d_2}$): The weight of edge $w_{d_1,d_2}=1$ when data $d_1$ is the member of data $d_2$, otherwise $w_{d_1,d_2}=0$.
• module–function ($e_{m,f}$): The weight of edge $w_{m,d}=1$ when f is in package m, otherwise $w_{m,f}=0$.

To further clarify call graph G, in

Table 2. Examples of nodes and edges in the calling graph of the Free5GC [52] project, where source and target refer to the nodes and relationships are the types of edges. Function nodes are represented by the function path in the source code.

Source Node	Relationship	Target Node
github.com/free5gc/amf/communication .HTTPAMFStatusChangeSubscribeModify	function–data	SubscriptionDatarequestBody
github.com/free5gc/amf/communication .HTTPCreateUEContext	function–data	CreateUeContextRequest
AccessAndMobilityData	data–data	UserLocation
AccessAndMobilityData	data–data	AccessType
github.com/free5gc/udm/consumer .SendRegisterNFInstance	function–function	(github.comfree5gc/openapi/ Nnrf_NFManagement .NFInstanceIDDocumentApiService) .RegisterNFInstance
github.com/free5gc/amf/gmm .HandleRegistrationRequest	function–function	github.com/free5gc/amf/gmm/message .SendRegistrationReject
github.com/free5gc/amf/communication	module–function	github.com/free5gc/amf/communication .HTTPAMFStatusChangeSubscribeModify

we present several examples of utilizing Go-Callviz to analyze the source code of Free5GC [52], an open-source project for 5th generation (5G) mobile core networks. Because the number of nodes obtained from Free5GC exceeds a few hundred, Table 2 only shows a few of the source and target nodes along with their respective edge connections. Note that because there are currently no popular projects for 6G core networks, we only use Free5GC for the concept proof of the self-evolution in this gene generation algorithm and later experiments.

The nodes in calling graph G provide the fine-grained materials to generate microservices. Generally, the extraction of microservices follows the idea of “high cohesion, low coupling”. High cohesion implies that each microservice is focused on a specific and well-defined functionality, ensuring its internal components work together effectively. Low coupling indicates that the microservices are loosely interconnected, minimizing dependencies and enabling independent development, deployment, and scalability.

To meet this principle, we propose to use the graph clustering Louvain algorithm [53] to cluster the nodes into a set of coarse-grained microservices, each of which has independent function and data. In particular, the input of the clustering algorithm is calling graph G, and each cluster output by the algorithm is treated as a microservice, that is, a gene $g\in GLib$. The clustering algorithm ensures that the resulting microservices exhibit higher coherence within them and lower coupling to one another, which aligns with the desired characteristics of being easily manageable and reusable.

In Figure 7, we present an experimental example of six clustered microservices, along with the corresponding nodes and edges in the user traffic control microservice. It is evident that the extracted microservices, such as AMF Session Management, Authentication, and SMF Session Management, are highly coherent with the function requirements of current 5G core network NFs, including AMF, SMF, and AUFS. This suggests that the clustering strategy on the calling graph of the core network is desirable for achieving the goal of gene generation.

To further quantify the quality of the obtained microservices, we use four evaluation metrics to assess them [54]: (1) Centripetal coupling (Ca): the number of other microservices that depend on the classes within a given microservice. A higher value indicates greater stability of the microservices; (2) Centrifugal coupling (Ce): the number of other microservices that the modules within a given microservices depend on. A higher value indicates a higher dependency and lower stability of the microservices; (3) Instability (I): the level of instability of microservices by calculating $\frac{CE}{CE+CA}$. $I\in [0,1]$, where 0 indicates the most stable microservice that does not depend on any other microservices and $I=1$ represents extreme instability; (4) Relational Cohesion (RC): the ratio between the number of internal relationships (function–function, data–data, function–data) and the number of functions and data within a microservice. A higher RC indicates higher cohesion within the microservice. The evaluation results of Free5GC clustering will be later shown in Section 6.3. Finally, it is noteworthy that the proposed graph clustering algorithm does not constitute a typical candidate for solving MDP problems. In this context, we consider the call graph as the state, $S_t$, of the MDP, with the clustering selection of a particular node as action, $A_t^G$, and the four evaluation metrics as local rewards, R. With this viewpoint, the policy, ${\pi }_{E0}$, is learned when the clustering algorithm converges.

5.3. Composition and Evaluation Stage

In stage ${\phi }_t=0$, the $GLi{b}_t$ is obtained through decomposing the Core Network Functions into multiple microservices, which are input for the next two stages. The goal of composition stage ${\phi }_t=1$ is to automatically learn how to compose microservices generated in the first stage, ${\phi }_t=0$, into $N{F}_t$ and set the connections between $N{F}_t$ to form a static network topology. In the next stage, ${\phi }_t=2$, the evolving agent organizes the obtained ${NF}_t$ and network topology deduced from stage ${\phi }_t=1$ into a dynamic service function chain and then interact with the environment, $E_t$, to receive feedback on a reward, $r_t$. Although these two stages are modeled as relatively independent MDPs and use different methods, the ${NF}_t$ and network topology generated in stage ${\phi }_t=1$ will not be evaluated separately, but as a component of the service function chain in the third stage, ${\phi }_t=2$. It can only be regarded as the completion of a training cycle after finishing both the second and the third stages. The optimal service chain can be obtained through multiple loops of Reinforcement Learning. Therefore, we will merge the two stages together for a more comprehensive discussion.

As mentioned in Section 5.1, we model the composition task as a typical MDP. The state, $S_t$, includes the gene library, $GLi{b}_t$, the application scenarios, $E_t$, and the current set of network elements, $N{F}_t$. The agent makes decisions at each time step as to whether or not to add or remove microservices from $N{F}_t$ and modify the connection topology between them, so the action set $A_t^C$ consists of three options: ${\mathbf{A}}_{\mathbf{1}}$: Add-Microservice, which adds a microservice onto a selected network element; ${\mathbf{A}}_{\mathbf{2}}$: Del-MicroService, which deletes a microservice from a selected network element; and ${\mathbf{A}}_{\mathbf{3}}$: Set-Connection, which represents whether to set connections between $N{F}_t$ as 0 or 1. Note that there is no direct reward function, R, at the composition stage ${\phi }_t=1$, and we later use the reward of stage ${\phi }_t=2$ to supervise the learning of composition. As a result, policy ${\pi }_{E1}=\left(A_t^C|GLi{b}_t,E_t,{NF}_t,{\phi }_t=1\right)$ enables rapid growth and adaptation in the network topology by generating diverse $N{F}_t$ and connections between them. The microservices being composed into the same NF are integrated into more compact modules, eliminating internal interfaces and sharing data between them. This approach effectively reduces resource consumption within the $NF$.

After the composition stage, we need to evaluate whether or not the generated $N{F}_t$ and static network topology can effectively complete specific tasks. Here we select the method of generating dynamic service function chains to evaluate performance. The state includes the application scenarios, $E_t$, and the current set of network elements, $N{F}_t$, in stage ${\phi }_t=2$. At each time step, t, the action $A_t^E$ traverses through existing connection relationships from the current network element and selects one network element for the next time step, $t+1$. Policy ${\pi }_{E2}=\left(A_t^E|E_t,{NF}_t,{\phi }_t=2\right)$ encourages exploration of multiple service chain generation paths.

In dynamic environments, service composition needs to support the recovery service-based applications from unexpected violations of not only function but also QoS, and the key factor in deciding which microservices are appropriate to be combined is their ability to provide a more modular and scalable service function with high performance and efficient resource consumption. Therefore, the reward, $\mathbf{R}$, for evaluation takes into account two factors: (1) whether or not the composed network elements, $N{F}_t$, can form a complete SCN that meets the functional requirements of the core network; (2) whether or not the tasks are completed with high Quality of Service (QoS). In other words, $R=R_{requirement}+R_{Qos}$, where $R_{requirement}$ mainly includes a reward for completing a signaling process and each step taken in the network evaluation stage. $R_{Qos}$ is determined by network performance, including CPU occupancy, time latency, and specific rewards for different tasks.

The pseudocode of the Self-Evolving Algorithm 1 is presented bellow. The algorithm takes the initial strategy, ${\pi }_E$, network elements, ${NF}_{t0}$, gene pool, $GLi{b}_{t0}$, and the initial state of the environment, $E_{t0}$, as inputs, and then outputs an update strategy, ${\pi }_E={\pi }_{E0},{\pi }_{E1},{\pi }_{E2}$, after completing three stages of training. In the first stage, ${\phi }_t=0$, the self-evolving agent samples action $A_t^G$ to split microservices and update the gene pool, $GLi{b}_t$ through policy ${\pi }_{E0}$. In the second stage, ${\phi }_t=1$, the agent samples action $A_t^C$ to reorganize microservices to form ${NF}_t$ and topologies through policy ${\pi }_{E1}$. In the third stage, ${\phi }_t=2$, the agent samples action $A_t^E$ to generate dynamic service chains and interact with the environment, $E_t$, to return reward function $r_t$ through policy ${\pi }_{E2}$. All parameters are stored in memory pool M, and we use Proximal Policy Optimization to update the strategy.

Algorithm 1 Self-Evolving Algorithm

Initialize Self-Evolving policy ${\pi }_E,{NF}_{t0},GLi{b}_{t0},E_{t0}$ while not reaching max iterations do    $memorypoolM\leftarrow ⌀$    initialize ${\phi }_t=0,r_t=0$    sample action $A_t^G\sim {\pi }_{E0}$    $GLi{b}_{t+1}\leftarrow$ apply $A_t^G$ to modify $GLi{b}_t$    store $(r_t,A_t^G,GLi{b}_t,{\phi }_t,E_t,{\pi }_{E0})$ into M    while M not reaching batch size do      for ${\phi }_t=1$ do         sample action $A_t^C\sim {\pi }_{E1}$         $N{F}_{t+1}\leftarrow$ apply $A_t^C$ to modify $N{F}_t$      end for      for ${\phi }_t=2$ do         sample action $A_t^E\sim {\pi }_{E2}$         $ServiceChain\leftarrow$ apply $A_t^E$      end for      $r_t\leftarrow R$      store $(r_t,A_t^E,{\phi }_t,E_t,N{F}_t,{\pi }_{E1},{\pi }_{E2})$ into M    end while    update ${\pi }_E={\pi }_{E0},{\pi }_{E1},{\pi }_{E2}$ with $PPO$ using samples in M end while

6. Self-Evolution Experiment

In this section, we conducted a simplified toy experiment as proof of concept for the proposed self-evolving mechanism of a 6G core network. Because it was challenging to clearly demonstrate the evolution results when directly applying the self-evolving algorithm to the fully functional core network due to its inherent complexities, we simulated a smaller core network, based on the Free5GC [52] project, with limited functionalities and application scenarios, enabling us to conduct controllable experiments and assess the efficiency of our proposed approach. Furthermore, due to the current lack of consensus on NF structures and signaling procedures for the 6G core network, we opted to utilize the existing 5G core network as our experimental platform. In addition, for the sake of simplicity, we only simulated an Edge Core Network. The joint evolution of the Cloud Core Network and multiple Edge Core Networks was left for future work.

6.1. Application Scenarios

For the single simulated Edge Core Network, we designed two communication scenarios that served as the external environment to drive its evolution: Emergency communication and Satellite mobile communication. In the Emergency scenario, low latency of service was essential to ensure the safety of people and equipment [55]. On the other hand, in the Satellite scenario, communication had to be completed under certain resource constraints [56]. The varying demands of these two scenarios led to different outcomes for the core network’s evolution.

For the two application scenarios, as shown in Figure 8, we assigned the Edge Core Network with three tasks: user registration, PDU session establishment, and PDU session modification. Correspondingly, the involved NFs included AUSF (Authentication Server Function), UDM (Unified Data Management), PCF (Policy Control Function), AMF (Access and Mobility Management Function), and SMF (Session Management Function). Note that we also simplified the signaling message procedures of the 5G core network standardization for these three tasks to focus on understanding the evolution.

We further differentiated the two scenarios by setting different task parameters according to their respective characteristics. As shown in

Table 3. Simulation parameter settings for the application scenarios. $\lambda$ is the total arrive rate of 3 task requests, proportion denotes the mix ratio of each task, and the latency/cpu occupation performance requirement is represented by Inequation to emphasize the special communication characters of the Emergency and Satellite scenarios. The General scenario here is set as a baseline with only basic communication requirements that can be met without any evolution.

Scenario	$\mathit{\lambda }$ (times/s)	Type and Proportion of Services	Performance Requirement
Emergency	0.8	Registration:PDU Establishment:PDU Modification = 1:10:5	$t_{emerg}<$$t_{general}$
Satellite	0.8	Registration:PDU Establishment:PDU Modification = 1:10:5	$cp{u}_{sate}<$$cp{u}_{general}$
General	0.3	Registration:PDU Establishment:PDU Modification = 1:10:5	―

, we set the arrive rate, $\lambda$, of task requests, the mixture proportion of tasks, and the performance requirements for the Emergency and Satellite scenarios. We also designed a General scenario as a baseline in order to set the special performance requirements for the Emergency and Satellite scenarios. We primarily focused on the communication flows of UE registration and session establishment, assuming a ratio of 1:10:5 for UE registration, session establishment, and session modification (with session establishment being more frequent than UE registration). We initialized the simulated core network with the settings in

Table 4. Simulation initialization settings. In each Network Function, the related microservice and its functions and data are listed.

Network Function	Microservice Name	Microservice Function	Microservice Data
AMF	Registration	Process device registration and Deregistration requests	UE identity, UE context information, Security information
SMF	Session	Manage and modify sessions	UE identity, session information
SMF	Connection	Establish N4 sessions	UE identity, session information
AUSF	Authentication	Authenticate UE	UE identity, security information
AUSF	Authorization	Authorize UE	UE identity, security information

, and the evolution pipeline was started by generating a gene library, $GLib$, with these settings.

6.2. Experiment Settings

We used an NVIDIA TITAN 1080 graphics card and two Intel(R) Xeon(R) CPU E5-2620 [email protected] GHz processors (8 cores each) for our experiments. We conducted self-evolution experiments in the Gym [57] simulation environment, which is an open-source platform providing a set of tools and algorithms for creating, evaluating, and optimizing Reinforcement Learning (RL) environments. To overcome Gym’s lack of a dedicated self-evolution environment setting, we developed a custom simulation environment with User Equipment (UE), microservices, and Network Functions (NF) for self-evolution experiments. The UE serves as a mere signaling transceiver within the system. In addition, the design of our microservices and NFs draws inspiration from Free5GC’s architecture. However, in contrast to Free5GC, we have omitted the intermediate processing steps in our microservices and network functions. Instead, we focus solely on calculating performance during user interactions.

Because the first stage, ${\phi }_t=0$, of the evolution pipeline has already been discussed in Section 5.2, in this section, we only simulated the composition and evaluation stages using the gene microservices derived from stage ${\phi }_t=0$. The simulation environment provides state information for agent training, including message experienced latency and CPU occupation. The action spaces of stages ${\phi }_t=1$ and ${\phi }_t=2$ use the definitions in Section 5.3. The rewards, $R_{requirement}$, are specified as follows:

$$R_{requirement}=\sum _{i=1}^{n_{\mathrm{tasks}}}{r}_c+\sum _{i=1}^{n_{\mathrm{step}}}{p}_s$$

(4)

where the agent is rewarded with $r_c$ when it successfully completes each of the three tasks and a penalty, $p_s$, is imposed on the length of the message step of the task to encourage the agent to optimize the NF compositions in order to shorten the service chain for efficiency purposes. The reward, $R_{Qos}$, is determined according to the latency and CPU occupation preferences of the three tasks. For the latency, we focus on the registration latency, $t_r$, PDU session establishment latency, $t_s$, and PDU session modification latency, $t_m$. The total latency, $t_{tot}=t_r+t_s+t_m$, is set as part of $R_{Qos}$. The registration latency is estimated as $t_r={\sum }_{i=1}^{n_{r\text{\_}ms}}{t}_p+{\delta }_s·n_{\mathrm{link}}$, where $n_{r\_ms}$ represents the number of microservices related to registration, $t_p$ is the processing latency, $n_{link}$ is the number of connection links, and ${\delta }_s$ is the transmission speed set to the link. The PDU session establishment latency, $t_s$, and PDU session modification latency, $t_m$, are calculated in a similar way to $t_r$. As for the CPU occupancy, we considered the total CPU occupancy, $cp{u}_{tot}$, as a reward by adding up the respective CPU usages for each task: registration $cp{u}_r$, PDU session establishment $cp{u}_s$, and PDU session modification $cp{u}_m$.

Furthermore, we took into account the hop number of the service chain required to finish a task as the third part of the reward, R, which encourages the agent to quickly find the shortest signaling message path in the evolved network. Therefore, the service chain lengths of the above three tasks were added together as the reward, $ho{p}_{tot}$. Thus, the total reward, R, we used in the experiment was set as:

$$R=R_{requirement}-{\omega }_1{t}_{tot}-{\omega }_{2}cp{u}_{tot}-{\omega }_{3}ho{p}_{tot}+C_R$$

(5)

where $C_R$ is a constant to ensure that the reward remains positive, and ${\omega }_1,{\omega }_2,{\omega }_3$ are hyperparameters to weight the three QoS factors. The agent obtains a high reward when it can complete the three tasks with lower latency, small CPU utilization, and shorter signaling service chains. The setting of reward hyperparameters are listed in

Table 5. Hyperparameters of reward settings. Different scenarios have different QoS reward weights corresponding to their evolution preferences. For the task completion reward, $r_c$, the agent receives a positive reward of 200 when it successfully completes a task, and a negative penalty of $-1000$ when it fails to do so.

Hyperparameter	Value	Emergency	Value	Satellite	Value
task completion, $r_c$	200/−1000	${\omega }_1$	1000	${\omega }_1$	0
composition penalty, $p_s$	−50	${\omega }_2$	0	${\omega }_2$	11
constant, $C_R$	2000	${\omega }_3$	5	${\omega }_3$	5

.

We employed the Proximal Policy Optimization (PPO) algorithm [48] to train the agent. The PPO network consisted of a Multi-Layer Perceptron (MLP), and the output layer utilized the softmax function to determine the action with the highest probability. The hyperparameters of the PPO model are outlined in

Table 6. PPO hyperparameter settings.

Hyperparameter	Value	Hyperparameter	Value
Train epoch	50	batch_size	2048
PolicyNet Learning Rate	$5\times {10}^{-4}$	ValueNet Learning Rate	$3\times {10}^{-4}$
clip $ϵ$	0.2	discount $\gamma$	0.995

.

6.3. Experiment Results

First, we verify the quality of the generated microservices in $GLib$ using the metrics defined in Section 5.2. We extract in total, 57 microservices from AMF, SMF, and AUSF of Free5GC by using the graph clustering algorithm, some examples of which are shown in

Table 7. Evaluation results of part of the 57 extracted microservices. Ca refers to the Centripetal coupling, Ce is the Centrifugal coupling, I is the Instability, and RC is the Relational Cohesion.

Microservice	Coupling			Cohesion
	$Ca$	$Ce$	I	$RC$
Mobility	10	4	0.285	1.1518
Authentication	2	3	0.600	1.2439
AMF Session Management	5	3	0.375	1.1609
User traffic control	3	5	0.625	1.32
SMF Session Management	5	10	0.666	1.1181
Average	5	5	0.500	1.1989

. It can be observed that the Mobility microservice has great stability with high $Ca=10$. The microservices of User Traffic Control and SMF Session Management are more dependent to each other, with high $Ce=5/10$, because they are extracted from the same SMF. The relative low Instability, I, indicates that the extracted microservices have high stability, and the relatively high Relational Cohesion, $RC$, suggests a well-structured and efficient microservice architecture. On average, the microservices obtained are desirable due to their relatively low coupling and high coherence.

We present the results of the self-evolution algorithm for the Emergency and Satellite scenarios in Figure 9. In particular, we demonstrate the reward convergence while also displaying the changes in CPU occupancy and latency for each scenario. We can see that the algorithm can converge quickly within only 10 epochs. After that, the newly evolved core network can significantly reduce the CPU occupancy and latency. This highlights the ability of the evolving agent to automatically adapt to various scenarios by reconstructing itself. It has been observed that the reward for both scenarios exhibits some fluctuations, which may be attributed to the agent’s random exploration. For instance, during the 33rd epoch of the Emergency reward shown in Figure 9a, the agent generated an authentication NF but failed to add it as one hop into the service chain during the registration process, resulting in the inability to complete the registration task and, consequently, a considerable performance degradation on the reward, and the latency, as shown in Figure 9c,e.

To further illustrate the evolving process of the core network during training, we conducted a case study by plotting the core network structure from the perspective of its derived NFs and their constituent microservices in Figure 10. At each inference step, the trained agent decides to enrich the existing NF by adding a new microservice or generating a new NF. For the Emergency scenario, the agent decides to generate two NFs and connect them directly. The function of NF1 is equivalent to the traditional AUSF, while NF2 combines the functions of traditional AMF and SMF into one NF, which significantly reduces the signaling process latency and meets the primary requirements of the Emergency scenario. For the Satellite scenario, the agent constructs an evolved core network with three NFs. NF1 performs the same functions as traditional AUSF. The function of the traditional SMF is split by the agent into two parts, one is put into the NF2 as a Connection microservice and the other is put into the NF3 as a Session microservice, which integrates with the Registration microservice to form a new NF3. In other words, the NF3 combines part functions of the traditional SMF and AMF. This structure results in a reduction in CPU occupancy for task completion. From Figure 10, it can be seen that the agent has the ability to evolve the core network according to different scenarios by modifying the composition and topology of NFs.

Finally, we compare the performance of the core networks generated by our self-evolution algorithm in the Emergency and Satellite scenarios to the General core network defined in Table 3 that was not subjected to evolution. As shown in Figure 11, in the Emergency scenario, the evolved network reduced the delays on the three tasks at the cost of increasing resource consumption. Specifically, the core network consolidated the session establishment and modification functions from SMF into AMF, resulting in lower delays for session establishment and modification. However, this led to an increase in resource consumption for AMF during registration. In the Satellite scenario, the evolved network reduced both CPU consumption and latency. The reason for this is that the evolving agent splits the SMF into two parts, separating session modification from session establishment, thereby reducing the CPU occupancy and delays for session modification.

7. Conclusions

In this paper, we propose a self-evolution architecture for the 6G core network, which is designed to accommodate diverse and constantly changing scenarios through intelligent decision making. Our contributions encompass elucidating the core ideas and mechanisms of 6G core network self-evolution, designing a pipeline, and introducing a reinforcement learning-based self-evolving agent along with a relevant algorithm. Through experimental simulations, we have validated the effectiveness of our self-evolution algorithm, demonstrating its ability to empower the agent with decision-making capabilities and ensure favorable environmental adaptability. For future work, we consider three points. First, we have discussed the evolving approaches for the Function, F, and Topology, T, of a $SCN=\{F,P,T\}$; however, we have not explicitly addressed the Protocol, P. When the network functions’ NFs are reconstructed, the corresponding core network protocols, which can be regarded as a conversation between NFs, would also be automatically regenerated. The AIGC (AI Generated Content) provides potential mechanisms for this aspect. Second, considering the physical or virtual environment of the Edge Core Network, the physical deployment of evolved network function instances should also be studied. Finally, the multi-agent evolution algorithm for the joint-optimization for the Cloud Core Networks and Edge Core Networks would also be studied.