*4.5. Experiment Configurations*

The main configuration parameters of the airport simulation are the number of areas, cameras, gates, and counters. As mentioned in the previous section, the whole airport is divided into areas, each with a fog device and a set of cameras, gates, and counters. KAIA main building area is around 1.2 km2, therefore we assumed that each simulated area is around 100 m<sup>2</sup> and ranged our areas parameter from 100 to 1000 to cover the 1 km2, having 10 configurations in total. Each area has two cameras (ranging from 200 to 2000 cameras), and in total 40 gates and 120 counters distributed across different areas. These configurations aim to show the architecture performance when the whole environment scales up from 360 to 2160 devices.

Table 6 lists the configurations of the Smart Airport sensors including the type of workload they generate, and the distribution of inter-arrival time. Camera has deterministic distribution as it generates workloads regularly every 5 ms while gate and counters have uniform distribution (5 to 20 ms) as they depend on the passenger arrival. Table 7 lists the various workloads alternating between modules for the Smart Gate Control and Smart Counter applications, while the Smart Surveillance workloads is listed in Table 3.


### **Table 6.** Smart Airport: Sensors Configuration.


### **Table 7.** Smart airport: workloads configuration.

### *4.6. Results and Analysis*

This section will present our results of IoE-6G and IoE-Fog-6G scenarios for the 10 configurations in terms of network usage, application loop end-to-end delay, and energy consumption. Figure 5a shows the network usage in GB/s. Deploying modules on fog devices in the IoE-Fog-6G scenario decrease the volume of data sent to the cloud by 36% for 2160 devices. The difference between the two scenarios network usage increased from 13% to 36% with the increase in the number of devices which approves that IoE-Fog-6G architecture will have a greater impact when the network scale-up and will alleviate traffic jams around the datacenter.

**Figure 5.** Smart airport case study results: (**a**) Total network usage, (**b**) Smart surveillance application average loop end-to-end delay on a log scale, (**c**) Smart gate application average loop end-to-end Delay on a log scale, (**d**) Smart counter application average loop end-to-end delay, (**e**) Network energy consumption, and (**f**) Estimated energy cost.

Figure 5b shows the average application loop end-to-end delay of the two scenarios for the Smart Surveillance control loop. The main control loop for Smart Surveillance is the tuple of modules (Camera, Motion Detector, Obj Detector, Obj Tracker, Camera Ctrl). There is a huge difference in the delay of the applications between the two scenarios. The IoE-Fog-6G provided a faster response than the IoE-6G as most of the processing is done on the edge and fog devices. In addition, because the data here is a stream of video, a huge amount of time is reduced when the unnecessary transformation is avoided. In Figure 5c, the average application loop end-to-end delay for Smart Gate is shown. The tuple (Barcode Reader, Boarding Processor, Auth., Gate Ctrl) is the main control loop for the Smart Gate. Similar to Surveillance, the delay is significantly less in the IoE-Fog-6G scenario. In addition, here we can see the benefit of proactive caching that the Smart Counter performs when it processes a passenger as the passenger authentication information is transformed into the gate fog. This guarantees the availability of authentication information before the passenger arrival which improved the loop latency. Finally, the average application loop end-to-end delay for a Smart Counter is shown Figure 5d. The tuple (Counter Device, Check Info, Passenger Processing, Auth. Info Provider, Passenger Processing, Counter Ctrl) is the main control loop for the Smart Counter application. The result here di ffers from the results of Surveillance and Gate applications because of the way modules are placed in this application. As shown in Figure 4, the Auth. Info Provider module is always located in the cloud in both scenarios, so when the number of devices increases the pressure on the datacenter increase which means higher delay.

Figure 5e shows the average energy consumption for network data transfer. The energy consumption is reduced in the IoE-Fog-6G scenario by around 3 MWh with 2160 devices, and this di fference is also expected to be larger with more devices joining. Similarly, Figure 5f shows the cost of energy for 2160 devices reduced by \$3500 per day with fog deployment which is around \$1,260,000 per year saving.

### **5. Case Study 2: Smart District**

In this section, the second case study is presented which evaluates the e ffectiveness of deploying edge/fog layer to the IoE/6G network in a smart district system. This case study di ffers from the smart airport case study as it represents an outdoor area and uses the 6G base stations as fog devices. In the following subsections, we will discuss the use of IoE in the smart district and its applications. Then, our experiment design, configuration, and results will be presented.

### *5.1. IoE in Smart District*

The smart district is the building block for smart cities and various applications can be involved to provide the smartness that will improve the quality of life. Challenges such as energy and utility provision, healthcare, education, transport, waste management, environment, and many others must be solved e fficiently and e ffectively in a smart ecosystem [91]. Smart sensors and IoE devices provide real-time monitoring of the district and act inelegantly. Applications such as parking guidance systems, parking lots monitoring, parking lot reservation, parking entrance, and security management, use sensors such as infrared sensors, ultrasonic sensors, inductive loop detectors, cameras, RFID, magnetometer, and/or microwave radar [92]. For energy provisioning and managemen<sup>t</sup> systems, smart grids might be utilized to provide a real-time monitoring and control using sensors that can read parameters such as voltage, current, power flow, and temperature [93].

### *5.2. Smart District: Architectural Overview*

King Abdullah Economic City (KAEC) is one of the new cities in Saudi Arabia that aims to provide a new way of living, working, and playing. KAEC is around 173 km2, so we choose one of its districts that is the Bayla Sun district which is around 4 km2. Bayla Sun is one of the active areas at KAEC and has di fferent facilities including, a residential area, college, parks, hotels, and a resort, a fire station, restaurants, and many others. Figure 6 shows a screenshot of Bayla Sun district from google map. The active area is divided into small areas of a size around 100 m2. Each is supported by one fog device (6G station). Three applications of the smart district are simulated Smart Surveillance, Smart Meters, and Smart Bins. Three types of edge devices: Smart Camera, Smart Meter, and Smart Bin are considered, and each of them is connected to its specific sensor or actuator depending on the system. The IoE-6G or IoE-Fog-6G scenarios have the same physical infrastructures but di fferent modules placemen. Figure 7 shows the detailed architectural design of both scenarios and their application modules placement. In the following subsections, we will discuss the Smart Meter and Smart Bin applications. Smart Surveillance application were already explained in the previous sections.

**Figure 6.** Simulated King Abdullah Economic City's (KAEC's) Bayla Sun District Layout.

### *5.3. Application: Smart Meter*

The Smart Meter application is the energy managemen<sup>t</sup> and analysis system in the district where voltage and current sensors monitor electricity usage and detect power cut in a real-time manner. This application consists of six modules: Meter, Meter Monitor, Electricity Controller (Elect Controller), Outage Notifier, User Interface, and Meter Control (Meter Ctrl). The Meter Monitor module is always located in the smart meter device. It receives reading (*meter\_ reading*) from the Meter sensors and it sends the reading to the Elect Controller. It also detects an outage and sends an outage signal (*outage\_ status*) to the Outage Notifier. The Electricity Controller module is the main processing module and it is located in the cloud on the IoE-6G scenario and fog node on the IoE-Fog-6G scenario. It receives status (*meter\_ status*) from Meter Monitor to be evaluated and will send electricity analysis (*elect\_analysis*) to the User Interface and controls (*ctrl\_params*) to Meter Ctrl. The Outage Notifier module is also located in the cloud on the IoE-6G scenario and in meter (edge) on the IoE-Fog-6G scenario and it receives an outage signal (*outage\_ status*) from the Meter Monitor and sends an outage signal to the local operator. The User Interface is always located in the cloud and it receives electricity analysis (*elect\_analysis*) and presented to the user.

### *5.4. Application: Smart Bin*

The Smart Bin application is the smart waste managemen<sup>t</sup> system that optimize and monitor waste collection and recycling in a real-time manner. Smart Bins have sensors that use ultrasonic beams to sense fill-levels and type of waste, such as mixed waste, paper, glass, or metal. This application consists of six modules: Bin, Bin Monitor, Bins Coordinator (Bins Coord), Full Notifier, User Interface, and Bin Control (Bin Ctrl). The Bin Monitor module is always located in the smart bin device. It reads the bin fill-levels from the sensors and it sends the reading (*bin\_ reading*) to the Bins Coord. It also detects a full bins state and sends a full signal (*full\_status*) to the Full Notifier for real-time responses. The Bins Coord module is the main waste managemen<sup>t</sup> module and it is located in the cloud on the IoE-6G scenario and fog node on the IoE-Fog-6G scenario. It receives bin status (*bin\_ status*) from Bin Monitor to be analyzed and will send waste conditions (*waste\_cond*) to the User Interface and controls (*ctrl\_params*) to Bin Ctrl. The Full Notifier module is located in the cloud on the IoE-6G scenario and bin on the IoE-Fog-6G scenario. It receives a full signal (*full\_status*) from the Bin Monitor and sends a full signal to the local operator for collection. The User interface module similar to other applications is always located in the cloud. It receives waste conditions (*waste\_cond*) from the Bin Monitor and presents it to the user.

 **Figure 7.** Smart District: (**a**) IoE-6G Scenario and (**b**) IoE-Fog-6G Scenario.

### *5.5. Experiment Configurations*

The main configuration parameters of the district simulation are the number of areas, cameras, meters, and bins. The number of areas is fixed to 100 to represent the active areas of KAEC's Bayla Sun district as shown in Figure 6. In each area, we specify the number of cameras, meters, and bins on it. In this study, 10 configurations were also simulated. The aim here is to show the architecture performance when the granularity of IoE devices increases. Therefore, the number of cameras, meters, and bins is increased from 1 to 10 per area (fog device) which means that the total number of end devices ranges from 300 to 2100 devices. The sensors in this case study are configured to periodically generate workloads following a deterministic distribution of 5 ms. Table 8 list the properties of all workloads

alternating between application modules for the two applications, while the Smart Surveillance is listed in Table 3.


**Table 8.** Smart District: Workloads Configuration.

### *5.6. Results and Analysis*

This section discusses the Smart District results of both IoE-6G and IoE-Fog-6G scenarios for the 10 configurations simulated in terms of network usage, application loop end-to-end delay, and energy consumption. Figure 8a shows that the difference in network usage between the two scenarios remains study with the growth in the granularity of IoE devices, at an average of 38% reduction in network usage. This clearly shows the role of edge/fog deployment in reducing the pressure on the 6G network, even when the number of IoE devices grows, by performing computation near the users and avoid transferring data to data centers as much as possible.

**Figure 8.** Smart district case study results: (**a**) Total network usage, (**b**) Smart surveillance application average loop end-to-end delay on a log scale, (**c**) Network energy consumption, and (**d**) Estimated energy cost.

We evaluated the end-to-end delay of the three applications, Smart Surveillance, Smart Meter, and Smart Bin Figure 8b shows the Smart Surveillance application result, while the other applications

results are not presented here as they have a similar pattern which is due to the similarity in the module placement. All applications showed a significant reduction in the end-to-end delay of the application's main control loop due to the local analysis on the fog. Smart Surveillance delay ranged from 8 to 10 ms for IoE-Fog-6G and from 6 to 7 s for IoE-6G scenario. Smart Meter and Smart Bin delay ranged from 10 to 12 ms for IoE-Fog-6G and from 4 to 5 s for IoE-6G scenario. The slight increase in the delay is due to the growth in the granularity of IoE devices which increases the pressure on fog devices with more workloads arriving at them.

Figure 8c shows the average energy consumption of network data transfer decreased by about 3 MWh for 3000 devices when the fog is deployed in the network at the IoE-Fog-6G scenario. The cost of energy, as shown in Figure 8d, also decreased at the same rate with a total saving of around \$1,280,000 per year for 3000 devices.

### **6. Distributed Artificial Intelligence-as-a-Service (DAIaaS)**

AI is critical in embedding smartness into smart cities and societies. Due to the exponential increase in the number IoE devices, a pressing need to reduce latencies for real-time sensing and control, privacy constraints, and other challenges, the existing cloud-based AIaaS model, even with fog and edge computing support, is not sustainable. Distributed Artificial Intelligence-as-a-Service (DAIaaS) will facilitate standardization of distributed AI provisioning in smart environments, which in turn will allow developers of applications, networks, systems, etc., to focus on the domain-specific details without worrying about distributed training and inference. Eventually, it will help systemize the mass-production of technologies for smarter environments. We describe in this section DAIaaS and investigate it using four di fferent scenarios.

### *6.1. DAIaaS: Architectural Overview*

Figure 9 shows four DAIaaS provisioning scenarios (Scenarios A, B, C, and D) for distributed training and inference to investigate various design choices and their performance. DAIaaS comprises several modules that represent typical core operations in AI workflow, including Data Collection, Data Aggregation (Data Agg.), Data Fusion, Data Pre-Processing (Data Prep.), Model Building, and Analytics. In an AI application, data (*data*) generated from various sensors connected to the edge devices are sent to the Data Collection module. The collected data (*datac*) is then sent by Data Collection to the Data Aggregation module that will combine them in a unified form and structure. Aggregated data (*dataag*) then will be passed to the Data Fusion module where the data from various sources will be fused to reduce uncertainty and produce enhanced forms of the data. Then, fused data (*dataf*) will be pre-processed by the Data Pre-Processing module where any missing data, noises, and drift are treated, and data are reduced and transformed as necessary. Finally, Preprocessed data (*datap*) are passed to the Model Building to train or retrain the model. When the model (*model*) is ready, the Analytics module will represent the inference step where the model is used to produce a decision or prediction as a result (*results*). We discuss next the four scenarios in detail.

### *6.2. Scenario A: Training*/*Retraining and Inference at Cloud*

In the first DAIaaS scenario (Figure 9a), all the data are sent to the datacenter (clouds) to be processed. Therefore, all the AI computation and processing modules are located in the datacenter including Data Agg., Data Fusion, Data Prep., Model Building, and Analytics, except the Data Collection module that will be located on the edges to receive data from sensors. Figure 9a shows these modules, their arrangements in the two network layers, and the workloads between them as a directed graph. Table 9 lists the di fferent workloads passing between the modules and the required resources in terms of the network and CPU computations. We will see in Section 6.5 that this scenario will facilitate a high-level of computing and storage resources, allowing the applications to run higher accuracy models on large volumes of data at the expense of higher delays.

 **Figure 9.** DAIaaS: (**a**) Scenarios A and B and (**b**) Scenarios C and D.

### *6.3. Scenario B: Training*/*Retraining at Cloud & Inference at Edge*

The model is built, trained, and retrained on the cloud in the second DAIaaS scenario (see depiction in Figure 9a, and the workload configuration in Table 9), but a smaller version of the model is built and sent to the edge devices. Edges in this scenario are responsible for inference, and passing the inference results to the actuators. The cloud layer contains the modules: Data Agg., Data Fusion, Data Prep., Model Building, and Create Distributed Model (Create Dist. ML). The Create Dist. ML module is an extra module that generates a smaller version of the model called Dist. model (*modeld*) that is sent it to the edges. Techniques such as distillation, pruning, and quantization can be used to reduce the model size with minimum effect on model accuracy. The Data Collection, Local Analytics, and Receive Distributed Model (Receive Dist. Model) modules are placed at the edge. The Local Analytics module will use the model received from the cloud to generate results in a real-time manner. The Receive Dist. Model module is required to receive the distributed model (*rec\_modeld*) from the cloud and passed to the Local Analytics.

The Data Collection module will use the Local Analytics module 90% of the time but because the edge-local model will be outdated after a while, it will offload the data to the cloud 10% of the time to retrain the model and receive a new model. We will see in Section 6.5 that this scenario will facilitate a high-level of computing and storage resources, however, the model accuracy will be affected due to smaller, somewhat outdated models running locally on edges with the advantage of faster response times due to analytics at the edge.


### **Table 9.** DAIaaS: Workloads configuration.

### *6.4. Scenarios C and D: Training*/*Retraining at Cloud and Fog and Inference at Edge*

The scenarios C and D contain the extra (fog) layer (see Figure 9b and Table 9). The model building and retraining happens in both cloud and fog layers but the model retraining at the cloud is done less often to reduce latency. The main AI modules (Data Agg., Data Fusion, Data Prep., Model Building, and Create Dist. ML) are located in both the datacenter and the fog. There is no direct communication between the edge and the cloud. The Create Dist. ML module is required in both the cloud and fog layers to create smaller models namely Cloud dist. model (*modelC\_d*) and Fog dist. model (*modelF\_d*). These smaller models are able to fit within the available resources in their parent layers. The Receive Dist. Model modules are needed in the fog to receive the distributed model from the cloud (*rec\_modelC\_d*). Similarly, it is needed on the edge to receive the distributed model from the fog (*rec\_modelF\_d*). The edges will have the Receive Dist. Model, Data Collection, and Local Analytics modules, which will work similar to Scenario B to generate results in a real-time manner.

Figure 9b shows that both scenarios C and D have the same modules and arrangements, however, they differ in the time they are retrained on the cloud. In Scenarios C and D, the Data Collection module will use the Local Analytics module 90% of the time, as was the case in Scenario B, however, in this case, it will offload the data to the fog layer 10% of the time to retrain the model and receive a new model. In Scenarios C, the fog will retrain the model using the new data received from its edge

devices 90% of the time locally and 10% of the time on cloud. In Scenario D, this split is 99% at fog and 1% at cloud. These scenarios provide relatively lower accuracy than Scenarios A and B but with the benefits of lower latencies.

### *6.5. Results and Analysis*

We now discuss the results for the four scenarios. All four scenarios are investigated using the same number of edge devices (varying between 50, 100, up to 500). Scenarios A and B have no fog devices, while in Scenarios C and D, the number of fogs is fixed at 50. Figure 10a shows the network usage of the DAIaaS model for the four scenarios. Note that the network usage of scenario A is exponentially increasing compared to the other scenarios with the number of edges increases. This clearly shows that offering AI as services on different levels (edge and fog) will reduce the pressure on the 6G network compared to using merely Cloud AIaaS (as in scenario A). In the case of 500 edges, the usage is reduced from 6 GB/s in scenario A to 2 GB/s in scenarios B, C, and D which is a three-fold improvement.

**Figure 10.** DAIaaS results: (**a**) Total network usage and (**b**) Average loop end-to-end delay for all requests on a log scale.

Figure 10b shows the average end-to-end delay for all AI applications in the four scenarios. The number of loops differs in each scenario, and also the frequency of each loop depends on the configuration. Scenario A has one loop, which is the tuple (Data Collection, Data Agg., Data Fusion, Data Prep., Model Building, Analytics, Actuator). Scenario B has two loops one to the cloud for model creation (Data Collection, Data Agg., Data Fusion, Data Prep., Model Building, Create Dist. ML, Receive Dist. Model) and the second in the edge to ge<sup>t</sup> the results (Data Collection, Local Analytics, Actuator). Scenario C and D have three loops, one to the cloud for model creation (Fog Data Collection, Data Agg., Data Fusion, Data Prep., Model Building, Create Dist. ML, Receive Dist. Model), the second to the fog, also, for model creation (Data Collection, Fog Data Collection, Fog Data Agg., Fog Data Fusion, Fog Data Prep., Fog Model Building, Fog Create Dist. ML, Receive Fog Dist. Model) and the third in the edge to ge<sup>t</sup> the results (Data Collection, Local Analytics, Actuator). Note in Figure 10b that the delay has been reduced significantly from Scenario A (all AI at the cloud) to other scenarios because in the other scenarios data are processed less often in the cloud. Scenario D has the lowest delay of around 8 ms (for 500 edges) as only 1% of the time data travels to the cloud while in scenario A the highest delay of around 5 s (for 500 edges) is because 100% of the time data travels to the cloud.

Figure 11a shows the average energy consumption of the network data transfer for the four scenarios. The energy consumption decreases by more than 7 MWh (for 500 edges) from scenario A at cloud (12 MWh) to scenarios B, C, and D (4 MWh). The cost of energy also decreases at the same rate, as shown in Figure 11b, with a total saving of around \$3.1 million per year for 500 edges.

**Figure 11.** DAIaaS results: (**a**) Network energy consumption and (**b**) Network energy cost.

### **7. Conclusions and Future Work**

In this paper, we proposed a framework for DAIaaS provisioning for IoE and 6G environments and evaluated it using three case studies comprising eight scenarios, nine applications and delivery models, and 50 distinct sensor and software modules. These case studies have allowed us to investigate various design choices for DAIaaS and helped to identify the performance bottlenecks. The first two case studies modelled real-life physical environments. We were able to see the benefits of various computation placement policies, allowing us to reduce the end-to-end delay, network usage, energy consumption, and annual energy cost by 99.8%, 33%, 3 MW, and 36%, on average, respectively. The third case study investigated various AI delivery models, without regard to the underlying applications. Again, we were able to identify various design choices that allowed us to reduce the end-to-end delay, network usage, energy consumption, and annual energy cost by 99%, 66%, 8 MW, and 66%, on average, respectively. We also showed that certain design choices may lead to lower performance (e.g., higher latencies) at the cost of higher AI accuracies and vice versa.

To the best of our knowledge, this is the first work where distributed AI as a service has been proposed, modeled, and investigated. This work will have a far-reaching impact on developing next-generation digital infrastructure for smarter societies by facilitating standardization of distributed AI provisioning, allowing developers to focus on the domain-specific details without worrying about distributed training and inference, and by helping systemize the mass-production of technologies for smarter environments. The future work will focus on improving the depth and breadth of the DAIaaS framework in terms of the case studies, applications, sensors, and software modules, and AI delivery models, and thereby, on developing new strategies, models, and paradigms for the provision of distributed AI services.

**Author Contributions:** Conceptualization, N.J. and R.M.; methodology, N.J. and R.M.; software, N.J.; validation, N.J. and R.M.; formal analysis, N.J. and R.M.; investigation, N.J. and R.M.; resources, R.M., I.K., and A.A.; data curation, N.J.; writing—original draft preparation, N.J. and R.M.; writing—review and editing, R.M., A.A., and I.K.; visualization, N.J.; supervision, R.M.; project administration, R.M., I.K., and A.A.; funding acquisition, R.M., A.A., and I.K. All authors have read and agreed to the published version of the manuscript.

**Funding:** This project was funded by the Deanship of Scientific Research (DSR) at King Abdulaziz University, Jeddah, under gran<sup>t</sup> number RG-10-611-38. The authors, therefore, acknowledge with thanks the DSR for their technical and financial support.

**Acknowledgments:** The experiments reported in this paper were performed on the Aziz supercomputer at KAU.

**Conflicts of Interest:** The authors declare no conflict of interest.
