**2. VPSF Design**

The VPSF stores user data in the extra storage space of sensor devices while guaranteeing the expected network lifetime of the IoT local network. Accordingly, the VPSF averages the remaining energy of the sensor devices across the IoT local network by building virtual private storage consisting of a specific group of sensor devices, and reduces communication overhead by using the modified BlockS of the CoAP. In this section, the VPSF design is described in detail.

#### *2.1. VPSF Architecture*

Figure 1 illustrates the VPSF architecture consisting of the user device, VPSF gateway, and sensor devices. It is assumed that VPSF gateway and sensor devices are static and all sensor devices are within the transmission range of the VPSF gateway. The user device is responsible for the request for data storing and retrieval. In the figure, the user device requests to store the user data. If the user device has the data that need to be stored, it transmits both the storing request message and the data to the VPSF gateway. In contrast, it transmits only the retrieval request message to the VPSF gateway when data retrieval is needed. The storing and retrieval request messages are identified by the VPSF gateway through a message type field in the header of each message.

**Figure 1.** Virtual private storage framework (VPSF) architecture.

The VPSF gateway builds virtual storage by logically integrating the extra data volumes of the multiple sensor devices and manages it by maintaining a virtual storage table (VST) containing the total amount of extra data volumes of all sensor devices in the network and the information of each sensor device (device ID, extra data volume size, and residual energy). For storing data, the VPSF gateway divides the user data into several blocks and selects the sensor devices in which the blocks will be stored based on their residual energy. Then, the VPSF gateway forwards the blocks to the selected sensor devices and maintains a block information table (BIT) containing the information of the stored blocks (original user data identifier, block sequence number, block size, and device ID of the sensor device where the block is stored). The selected sensor device stores only the information contained in the payload of the blocks in binary format. For data retrieval, upon receiving the retrieval request message, the VPSF gateway checks the identifier of the requested data and then searches the BIT to find the device ID of sensor devices where the blocks for the requested data are stored. It then requests the blocks of the corresponding sensor devices in order of sequence number. After the VPSF gateway obtains all blocks, it integrates them into a single data set and transmits the integrated data to the user device.

Each sensor device serves as storage for the blocks. The storage space of a sensor device is divided into data and system volumes by the VPSF gateway—each sensor device separately maintains two volumes. The data volume is a part of the sensor device's storage space that is used to build virtual storage, and the system volume is a storage space required to perform the sensor device's unique operations (e.g., sensor and actuator control). Therefore, upon receiving the blocks from the VPSF gateway, the sensor device stores them in the data volume within its storage space. For data retrieval, it transmits the requested blocks to the VPSF gateway.

The VPSF can have a variety of use cases, such as healthcare, financial, and multimedia data storage, in which it is necessary to consider the physical resources of sensor devices running VPSF such as battery capacity and storage space. One of the potential use cases for VPSF is the personal healthcare data storage. In this use case, the wearable or mobile user devices periodically collect users' health information and generate the healthcare data files. The generated files are transmitted to the VPSF gateway. Then, the VPSF gateway divides the files into multiple blocks and store them in the extra storage space of the sensor devices. In the use case, users maintain personal health information via their own local storage, which is logically created by merging the extra storage space of sensor devices. Therefore, VPSF eliminates the need for users to store sensitive health information in storage managed by third-party service providers, and allows users to maintain it by themselves without additional hardware installation.

#### *2.2. Operations of VPSF Gateway*

The VPSF gateway performs three operations: (1) virtual storage building, (2) sensor device selection, and (3) consecutive block retransmission. In the following subsections, the operations of the VPSF gateway are described in detail.

#### 2.2.1. Virtual Storage Building

In a virtual storage building, the VPSF gateway first partitions the storage space of a sensor device into two volumes: system and data. The storage space of the system volume should be sufficient to store the system data needed for installing and running the operating system (OS), applications, and task data generated by processing specific tasks. It is difficult to determine the storage space for the system volume because each sensor device may use a different OS and applications. Consequently, we consider two assumptions to determine the storage space for the system volume. First, the storage space required to store system data is determined by examining the filled storage space when the sensor device does not perform any task, provided as a constant value in VPSF. Second, the sensor device generates task data of the same size at periodic intervals (task data generation interval) and temporally accumulates the data in the storage space. The sensor device transmits the accumulated task data to the intended destination and removes it from the storage space at specific intervals (task data removal interval). The storage space for the data volume (extra storage space) for the *i*-th sensor device (*DVi*) is given by

$$DVV\_i = T\_i - Sys\_i - Task\_i = T\_i - Sys\_i - SizeTask\_i \frac{RunTask\_i}{GenTask\_i} \tag{1}$$

where *Ti*, *Sysi*, *Taski*, *SizeTaski*, *GenTaski*, and *RemTaski* are the total storage space, the storage space required to store system data, the storage space required to store task data, the task data size, the task data generation interval, and the task data removal interval for the *i*-th sensor device, respectively. The VPSF gateway then logically integrates the data volume of all sensor devices to build virtual private storage. The total storage space of virtual storage is equal to the sum of the storage space of the data volumes. The total storage space of the virtual private storage and the storage space of the data volume for each sensor device are listed in the VST. The VST is updated when a new sensor device is added or the existing sensor device is depleted.

#### 2.2.2. Sensor Device Selection

The purpose of sensor device selection is to guarantee the expected lifetime of the IoT local network by averaging the residual energy of sensor devices, which may differ for each device. Accordingly, when building virtual private storage, the VPSF gateway examines the residual energy of each sensor device and maintains it using the VST. The residual energy values of all sensor devices are represented by **E** = [*e*0,*e*2, ... ,*en*−1], where *n* is the number of sensor devices. Sensor device selection is initiated when the VPSF gateway receives the storing request message, which contains the size of the user data, from the user device. Algorithm 1 defines the sensor device selection procedure. In the algorithm, the VPSF gateway iteratively performs sensor device selection. Before initiating sensor device selection, the VPSF gateway initializes variables to zero: the number of iterations (*k*), the device ID of the sensor device selected in the *k*-th iteration (*SSDk*), and the sum of the data volume of the selected sensor devices (*SDS*). It then calls the list of residual energy for each sensor device from the VST and initiates sensor device selection. During sensor device selection, the VPSF gateway selects a sensor device with the highest residual energy (based on the VST) and verifies whether the data volume of the selected sensor device is large enough to store the user data; if it is not, the sensor device with the next highest residual energy is selected from the remaining list. The VPSF gateway then sums the data volumes of all selected sensor devices and compares the result with the size of the user data. This operation repeats until the sum of the data volumes of the selected sensor devices becomes larger than the size of the user data (*SizeUsr*). After terminating sensor device selection, the VPSF gateway divides the user data into multiple identically-sized blocks and sequentially transmits the blocks to the selected sensor devices.


#### 2.2.3. Consecutive Block Retransmission

Consecutive block retransmission aims to reduce communication overhead in the VPSF. Accordingly, the BlockS option of the CoAP is modified to retransmit lost blocks without any request message and then applied to the VPSF. The existing BlockS option of the CoAP uses the non-confirmable (NON) message, which does not require acknowledgment when transmitting the blocks. This allows the server to transmit multiple blocks consecutively. For the existing BlockS option,

the confirmable (CON) message that requires acknowledgment is regularly transmitted to verify the client's state. The server transmits the CON message when the number of consecutive transmissions reaches the maximum window size (SPDYWND), which is predefined based on the client's capacity limit. The number of blocks transmitted consecutively is equal to the value of SPDYWND. However, for retransmission, the existing BlockS option only uses the CON message, causing unnecessary retransmission request messages to be transmitted repeatedly.

To solve this problem, in VPSF, the existing protocol is modified to inform the server of the sequence number of the lost blocks. Upon receiving the blocks, the client identifies the lost blocks by examining the gap between their sequence numbers and maintains the sequence number of the lost blocks as a list. The client then piggybacks the list to the acknowledgment and transmits it to the server, thereby providing the server with the sequence number of the lost blocks. Consequently, the server consecutively transmits the lost blocks as NON messages without any retransmission request messages.

Figure 2 illustrates an operational example of the existing and modified BlockS option of the CoAP. MID is the message ID, T is the token, and S is the value of the BlockS option (the sequence number of the blocks, whether more blocks are following, the size of the block, and SPDYWND). In the example, the size of the block is set to 64 B, and SPDYWND is set to 4. The example demonstrates that the modified protocol reduces communication overhead by eliminating retransmission request messages.

(**a**)

**Figure 2.** *Cont*.

**Figure 2.** Comparison of two operational examples: (**a**) existing speedy block-wise transfer in CoAP; (**b**) modified speedy block-wise transfer in CoAP.

#### **3. Implementation and Performance Evaluation**

The VPSF performance was evaluated through implementation and extensive experiments. The user device was implemented using a personal computer running the Windows 10 OS, and the VPSF gateway and sensor devices were implemented using the open-source hardware Raspberry Pi 3 Model B+ running the Linux-based Raspbian OS.

Figure 3 depicts the VPSF implementation structure. The VPSF gateway was placed between the user device and multiple sensor devices. It communicates with the user device using the hypertext transfer protocol (HTTP) and communicates with the sensor devices using the CoAP. To implement the CoAP for the VPSF, we used the open-source software *libcoap*, a C-implementation of CoAP. We developed a monitoring application running on the VPSF gateway to examine the residual energy of each sensor device and the delay required to successfully store blocks. Each device was equipped with a wireless interface supporting IEEE 802.11 b/g/n and used a 100 Mbps data rate for data transmission.

For the experiment, we implemented four sensor devices, each initially set to have a different residual energy. Table 1 lists the initial settings for the residual energy of each sensor device. In the experiment, the size of user data varied from 100 MB to 1 GB. The block size was fixed as 1000 B, so the number of blocks per user data varied from 105 to 106. Each sensor device was equipped with a 32 GB SD card, and the storage space of each data volume was set equally to 20 GB. SPDYWND was set to 100 to consecutively transmit 100 blocks. To evaluate the VPSF performance, the VPSF experiment results were compared to legacy virtual private storage (legacy VPS), in which the gateway transmits the same number of blocks to each sensor device using the existing BlockS option of the CoAP.

**Figure 3.** VPSF implementation structure.



Figure 4 depicts the changes in the lifetime of the sensor device with the lowest residual energy (Sensor Device 1) when the size of user data increases. The lifetime of the IoT local network is determined by the lifetime of the sensor device with the lowest residual energy. In this experiment, the user device was set to transmit user data continuously until the energy of Sensor Device 1 was depleted. Moreover, the sensor devices were set to overwrite user data to prevent the storage space of its data volume from filling up completely.

**Figure 4.** Lifetime of sensor device with the lowest residual energy.

The VPSF exhibited a longer lifetime than legacy VPS, owing to the VPSF's support of sensor device selection and the modified BlockS option. Specifically, Sensor Device 1 was used to store the blocks only after the residual energy of the other sensor devices reached that of Sensor Device 1. Furthermore, the lost blocks were retransmitted without any retransmission request messages in the VPSF. In contrast, in legacy VPS, the same number of blocks were stored in each sensor device whenever the user data were transmitted from the user device, and the retransmission was conducted with the retransmission request messages. In both cases, as the size of user data increased, the lifetime decreased; if the size of user data increased, the sensor device received and stored more blocks because the communication overhead caused by the transmission interval and the user data header decreased. On average, the VPSF exhibited a 65.85% longer lifetime than that of legacy VPS.

Figure 5 depicts the residual energy of each sensor device. In the experiment, the user device was configured to transmit 100 times to the VPSF gateway. Two sizes of user data (500 MB and 1 GB) were considered. In both cases, VPSF had a smaller difference in residual energy among sensor devices compared to legacy VPS because, unlike legacy VPS, the VPSF selected and used some of the sensor devices to store the blocks. Only Sensor Device 4 was used in the 500 MB case, and Sensor Devices 3 and 4 were used in the 1 GB case. The difference in residual energy among sensor devices can be mathematically represented by the fairness index *F* [35]. The fairness index is calculated as

$$F = \frac{\left(\sum\_{i=1}^{n} c\_i\right)^2}{n\sum\_{i=1}^{n} c\_i^2} \tag{2}$$

**Figure 5.** Residual energy of each sensor device.

The higher the fairness index, the smaller the difference in residual energy. Table 2 lists the fairness index of the VPSF and legacy VPS for each case. For 500 MB and 1 GB, respectively, the VPSF exhibited 7.58% and 19.77% higher fairness indexes than legacy VPS.


Figure 6 depicts the variance in retransmission delay per user data when the size of user data changes. During the experiment, the block delivery ratio was 92.1%, on average, and 7.9% of blocks were lost and retransmitted. The VPSF employed the modified BlockS option, which enabled the server (VPSF gateway) to consecutively retransmit the lost blocks. The retransmission delay of the VPSF was 25.6% shorter than that of legacy VPS, on average. During the experiment, the retransmission delay increased as the size of user data increased—the larger the size of the user data, the higher the number of lost blocks.

**Figure 6.** Retransmission delay based on size of user data.

#### **4. Conclusions**

In this paper, we presented a VPSF that uses the extra storage space of sensor devices in an IoT local network to store user data while guaranteeing the expected network lifetime. The VPSF gateway performs three operations to build and maintain virtual private storage: (1) virtual storage building, (2) sensor device selection, and (3) consecutive block retransmission. The first operation builds virtual private storage by integrating the extra storage space of sensor devices. The second operation selects a sensor device to store blocks based on the device's residual energy to guarantee the expected lifetime of the IoT local network. The third operation enables the VPSF gateway to retransmit the lost blocks without retransmission request messages to reduce communication overhead. To verify the feasibility of the VPSF, an experiment was performed using a specific implementation and the results were compared with legacy VPS. The results demonstrate that the VPSF exhibited a 65.85% longer lifetime and 25.6% shorter retransmission delay than those of legacy VPS.

**Author Contributions:** E.-J.K. conceived and designed the overall framework; H.-H.L. performed the open source-based implementation and experiment; J.-H.K. interpreted and analyzed the data; H.-H.L., J.-H.K., and E.-J.K. wrote the paper; E.-J.K. guided the research direction and supervised the entire research process. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2019R1I1A1A01059787).

**Conflicts of Interest:** The authors declare no conflict of interest.
