Multi-Server Multi-Function Distributed Computation

Abstract

The work here studies the communication cost for a multi-server multi-task distributed computation framework, as well as for a broad class of functions and data statistics. Considering the framework where a user seeks the computation of multiple complex (conceivably non-linear) tasks from a set of distributed servers, we establish the communication cost upper bounds for a variety of data statistics, function classes, and data placements across the servers. To do so, we proceed to apply, for the first time here, Körner’s characteristic graph approach—which is known to capture the structural properties of data and functions—to the promising framework of multi-server multi-task distributed computing. Going beyond the general expressions, and in order to offer clearer insight, we also consider the well-known scenario of cyclic dataset placement and linearly separable functions over the binary field, in which case, our approach exhibits considerable gains over the state of the art. Similar gains are identified for the case of multi-linear functions.

1. Introduction

Distributed computing plays an increasingly significant role in accelerating the execution of computationally challenging and complex computational tasks. This growth in influence is rooted in the innate capability of distributed computing to parallelize computational loads across multiple servers. This same parallelization renders distributed computing as an indispensable tool for addressing a wide array of complex computational challenges, spanning scientific simulations, and extracting various spatial data distributions [1], data-intensive analyses for cloud computing [2], and machine learning [3], as well as applications in various other fields such as computational fluid dynamics [4], high-quality graphics for movie and game rendering [5], and a variety of medical applications [6], to name just a few. In the center of this ever-increasing presence of parallelized computing stand modern parallel processing techniques, such as MapReduce [7,8,9] and Spark [10,11].

However, for distributed computing to achieve the desirable parallelization effect, there is an undeniable need for massive information exchange to and from the various network nodes. Reducing this communication load is essential for scalability [12,13,14,15] in various topologies [16,17,18]. Central to the effort to reduce communication costs stand coding techniques such as those found in [19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36], including gradient coding [21] and different variants of coded distributed computing that nicely yield gains in reliability, scalability, computation speed, and cost-effectiveness [24]. Similar communication-load aspects are often addressed via polynomial codes [37], which can mitigate stragglers and enhance the recovery threshold, while MatDot codes, devised in [31,38] for secure distributed matrix multiplication, can decrease the number of transmissions for distributed matrix multiplication. This same emphasis on reducing communication costs is even more prominent in works like [31,34,35,38,39,40,41,42,43,44,45,46], which again, focus on distributed matrix multiplication. For example, focusing on a cyclic dataset placement model, the work in [39] provided useful achievability results, while the authors of [35] have characterized achievability and converse bounds for secure distributed matrix multiplication. Furthermore, the work in [34] found creative methods to exploit the correlation between the entries of the matrix product in order to reduce the cost of communication.

1.1. The Multi-Server Multi-Function Distributed Computing Setting and the Need for Accounting for General Non-Linear Functions

As computing requirements become increasingly challenging, distributed computing models have also evolved to be increasingly complex. One such recent model is the multi-server multi-function distributed computing model that consists of a master node, a set of distributed servers, and a user demanding the computation of multiple functions. The master contains the set of all datasets and allocates them to the servers, which are then responsible for computing a set of specific subfunctions for the datasets. This multi-server multi-function setting was recently studied by Wan et al. in [39] for the class of linearly separable functions, which nicely captures a wide range of real-world tasks [7] such as convolution [41], the discrete Fourier transform [47], and a variety of other cases as well. This same work bounded the communication cost, employing linear encoding and linear decoding that leverage the structure of requests.

At the same time, however, there is growing need to consider more general classes of functions, including non-linear functions, such as is often the case with subfunctions that produce intermediate values in MapReduce operations [7] or that relate to quantization [48], classification [49], and optimization [50]. Intense interest can also be identified in the aforementioned problem of distributed matrix multiplication, which has been explored in a plethora of works, which include [35,42,45,51,52,53], with a diverse focus that entails secrecy [45,51,53], as well as precision and stragglers [14,35,42,52], to name a few. In addition to matrix multiplication, other important non-linear function classes include sparse polynomial multiplication [54], permutation invariant functions [55]—which often appear in multi-agent settings and have applications in learning, combinatorics, and graph neural networks—as well as nomographic functions [56,57], which can appear in the context of sensor networks and which have strong connections with interference exploitation and lattice codes, as nicely revealed in [56,57].

Our own work here is indeed motivated by this emerging need for distributed computing of non-linear functions, and our goal is to now consider general functions in the context of the multi-server multi-function distributed computing framework while also capturing dataset statistics and correlations and while exploiting the structural properties of the (possibly non-linear) functions requested by the user. For this purpose, we go beyond the linear coding approaches in [39,58,59] and devise demand-based encoding–decoding solutions. Furthermore, we adopt—in the context of the multi-server multi-function framework—the powerful tools from characteristic graphs that are specifically geared toward capturing both the statistical structure of the data as well as the properties of functions beyond the linear case. To help the reader better understand our motivation and contribution, we proceed with a brief discussion on data structure and characteristic graphs.

1.2. Data Correlation and Structure

Crucial in reducing the communication bottleneck of distributed computing is an ability to capture the structure that appears in modern datasets. Indeed, even before computing considerations come into play, capturing the general structure of the data has been crucial in reducing the communication load in various scenarios such as those in the seminal work by Slepian–Wolf [60] and Cover [61]. Similarly, when function computation is introduced, data structure can be a key component. In the context of computing, we have seen the seminal work by Körner and Marton [62], which focused on efficient compression of the modulo 2 sum of two statistically dependent sources, while Lalitha et al. [63] explored linear combinations of multiple statistically dependent sources. Furthermore, for general bivariate functions of correlated sources, when one of the sources is available as side information, the work of Yamamoto [64] generalized the pioneering work of Wyner and Ziv [65] to provide a rate-distortion characterization for the function computation setting.

It is the case, however, that when the computational model becomes more involved—as is the case in our multi-server multi-function scenario here—the data may often be treated as unstructured and independent [39,58,66,67,68]. This naturally allows for crucial analytical tractability, but it may often ignore the potential benefits of accounting for statistical skews and correlations in data when aiming to reduce communication costs in distributed computing. Furthermore, this comes at a time when more and more function computation settings—such as in medical imaging analysis [69], data fusion, and group inferences [70], as well as predictive modeling for artificial intelligence [71]—entail datasets with prominent dependencies and correlations. While various works, such as those by Körner–Marton [62], Han–Kobayashi [72], Yamamoto [64], Alon–Orlitsky [73], and Orlitsky–Roche [74], provide crucial breakthroughs in exploiting data structure, to the best of our knowledge, in the context of fully distributed function computation, the structure in functions and data has yet to be considered simultaneously.

1.3. Characteristic Graphs

To jointly account for this structure in both data and functions, we draw from the powerful literature on characteristic graphs, introduced by Körner for source coding [75] and used in data compression [62,73,74,76,77,78], cryptography [79], image processing [80], and bioinformatics [81]. For example, toward understanding the fundamental limits of distributed functional compression, the work in [75] devised the graph entropy approach in order to provide the best possible encoding rate of an information source with vanishing error probability. This same approach, while capturing both function structure and source structure, was presented for the case of one source, and it is not directly applicable to the distributed computing setting. Similarly, the zero-error side information setting in [73] and the lossy encoding setting in [64,74] use Körner’s graph entropy [75] approach to capture both function structure and source structure but were again presented for the case of one source. A similar focus can be found in the works in [73,74,76,77,79]. The same characteristic graph approach nicely used by Feizi and Médard in [82] for the distributed computing setting, for a simple distributed computing framework, and in the absence of considerations for the data structure.

Characteristic graphs, which are used in fully distributed architectures to compress information, can allow us to capture various data statistics and correlations, various data placement arrangements, and various function types. This versatility motivates us to employ characteristic graphs in our multi-server multi-function architecture for distributed computing of non-linear functions.

1.4. Contributions

In this paper, leveraging fundamental principles from source and functional compression, as well as graph theory, we study a general multi-server multi-function distributed computing framework composed of a single user requesting a set of functions, which are computed with the assistance of distributed servers that have partial access to the datasets. To achieve our goal, we consider the use of Körner’s characteristic graph framework [75] in our multi-server multi-function setting and proceed to establish upper bounds on the achievable sum-rates reflecting the setting’s communication requirements.

By extending, for the first time here, Körner’s characteristic graph framework [75] to the new multi-server multi-function setting, we are able to reflect the nature of the functions and data statistics in order to allow each server to build a codebook of encoding functions that determine the transmitted information. Each server, using its own codebook, can transmit a function (or a set of functions) of the subfunctions of the data available in its storage and to then provide the user with sufficient information for evaluating the demanded functions. The codebooks allow for a substantial reduction in the communication load.

The employed approach allows us to account for general dataset statistics, correlations, dataset placement, and function classes, thus yielding gains over the state of the art [39,60], as showcased in our examples for the case of linearly separable functions in the presence of statistically skewed data, as well as for the case of multi-linear functions where the gains are particularly prominent, again under statistically skewed data. For this last case of multi-linear functions, we provide an upper bound on the achievable sum-rate (see Section 4.2) under a cyclic placement of the data that reside in the binary field. We also provide a generalization of some elements in the existing works on linearly separable functions [39,58].

In the end, our work demonstrates the power of using characteristic-graph-based encoding for exploiting the structural properties of functions and data in distributed computing, as well as provides insights into fundamental compression limits, all for the broad scenario of multi-server multi-function distributed computation.

1.5. Paper Organization

The rest of this paper is structured as follows. Section 2 describes the system model for the multi-server multi-function architecture, and Section 3 details the main results on the communication cost or sum-rate bounds under the general dataset distributions and correlations, dataset placement models, and general function classes requested by the user over a field of characteristic $q\ge 2$, through employing the characteristic graph approach, and contrasts the sum-rate with the relevant prior works, e.g., [39,60]. Finally, we summarize our key results and outline possible future directions in Section 5. We provide a primer for the key definitions and results on characteristic graphs and their fundamental compression limits in Appendix A and give proofs of our main results in Appendix B.

Notation: We denote by $H\left(X\right)=\mathbb{E}[-log{P}_X\left(X\right)]$ the Shannon entropy of random variable X drawn from distribution or probability mass function (PMF) $P_X$. Let $P_{X_1,X_2}$ be the joint PMF of two random variables $X_1$ and $X_2$, where $X_1$ and $X_2$ are not necessarily independent and identically distributed (i.i.d.), i.e., equivalently, the joint PMF is not in product form. The notation $X\sim \mathrm{Bern}\left(ϵ\right)$ denotes that X is Bernoulli distributed with parameter $ϵ\in [0,1]$. Let $h(·)$ denote the binary entropy function and $H_B\left(B(n,ϵ)\right)$ denote the entropy of a binomial random variable of size $n\in \mathbb{N}$, with $ϵ\in [0,1]$ modeling the success probability of each Boolean-valued outcome. The notation $X_{\mathcal{S}}=\{X_i:i\in \mathcal{S}\}$ denotes a subset of servers with indices $i\in \mathcal{S}$ for $\mathcal{S}\subseteq \Omega$. The notation ${\mathcal{S}}^c=\Omega \setminus \mathcal{S}$ denotes the complement of $\mathcal{S}$. We denote the probability of an event A by $\mathbb{P}\left(A\right)$. The notation $1_{x\in A}$ denotes the indicator function, which takes the value 1 if $x\in A$ and 0 otherwise. The notation $G_{X_i}$ denotes the characteristic graph that server $i\in \Omega$ builds for computing $F\left(X_{\Omega }\right)$. The measures $H_{G_X}\left(X\right)$ and $H_{G_X}\left(X\right|Y)$ denote the entropy of characteristic graph $G_X$ and the conditional graph entropy for random variable X given Y, respectively. The notation $\mathcal{T}(N,K,K_c,M,N_r)$ shows the topology of the distributed system. We note that ${\mathcal{Z}}_i$ denotes the indices of datasets stored in $i\in \Omega$, and the notation $K_n\left(\mathcal{S}\right)=\left|{\mathcal{Z}}_{\mathcal{S}}\right|=\left|{\bigcup }_{i\in \mathcal{S}}{\mathcal{Z}}_i\right|$ represents the cardinality of the datasets in the union of the sets in $\mathcal{S}$ for a given subset $\mathcal{S}\subseteq \Omega$ of servers. We also note that $\left[N\right]=\{1,2,\dots ,N\}$, $N\in {\mathbb{Z}}^{+}$, and $[a:b]=\{a,a+1,\dots ,b\}$ for $a,b\in {\mathbb{Z}}^{+}$ such that $a<b$. We use the convention $mod\{b,a\}=a$ if a divides b. We provide the notation in

Table 1. Notation.

Distributed-Computation-System-Related Definitions	Symbols
Number of distributed servers; set of distributed servers; capacity of a server	N; $\Omega$; M
Set of datasets; dataset catalog size	${\left\{D_k\right\}}_{k\in \left[K\right]}$; $K=\left|\mathcal{K}\right|$
Subfunction $k\in {\mathcal{Z}}_i\subseteq \left[K\right]$	$W_k=h_k\left(D_k\right)$
The number of symbols in each $W_k$; blocklength	L; n
Set of indices of datasets assigned to server $i\in \Omega$ such that $\left|{\mathcal{Z}}_i\right|\le M$	${\mathcal{Z}}_i\subseteq \left[K\right]$
Set of subfunctions corresponding to a subset of servers with indices $i\in \mathcal{S}$ for $\mathcal{S}\subseteq \Omega$	$X_{\mathcal{S}}=\{X_i:i\in \mathcal{S}\}$
Recovery threshold	$N_r$
Number of demanded functions by the user	$K_c$
Number of symbols per transmission of server $i\in \Omega$	$T_i$
Topology of the multi-server multi-function distributed computing setting	$\mathcal{T}(N,K,K_c,M,N_r)$
Graph-Theoretic Definitions	Symbols
Characteristic graph that server i builds for computing $F\left(X_{\Omega }\right)$	$G_{X_i}$, $i\in \Omega$
Union of characteristic graphs that server i builds for computing ${\left\{F_j\left(X_{\Omega }\right)\right\}}_{j\in \left[K_c\right]}$	$G_{X_i}^{\cup }$, $i\in \Omega$
Maximal independent set (MIS); set of all MISs of $i\in \Omega$	$U_1$; $S\left(G_{X_1}\right)$
A valid coloring of $G_{X_i}$	$c_{G_{X_i}}$
n-th OR power graph; a valid coloring of the n-th OR power graph	$G_{{\mathbf{X}}_i}^n$; $c_{G_{{\mathbf{X}}_i}^n}\left({\mathbf{X}}_i\right)$
Characteristic graph entropy of $X_i$	$H_{G_{X_i}}\left(X_i\right)$
Conditional characteristic graph entropy of $X_i$ such that $i\in \mathcal{S}$ given $X_{{\mathcal{S}}^c}$	$H_{G_{X_i}}\left(X_i\right|X_{{\mathcal{S}}^c})$

.

2. System Model

This section outlines our multi-server multi-function architecture and details our main technical contributions, namely, the communication cost for the problem of distributed computing of general non-linear functions and the cost for special instances of the computation problem under some simplifying assumptions on the dataset statistics, dataset correlations, placement, and the structures of functions.

In the multi-server multi-function distributed computation framework, the master has access to the set of all datasets and distributes the datasets across the servers. The total number of servers is N, and each server has a capacity of M. Communication from the master to the servers is allowed, whereas the servers are distributed and cannot collaborate. The user requests $K_c$ functions that could be non-linear. Given the dataset assignment to the servers, any subset of $N_r$ servers is sufficient to compute the functions requested. We denote by $\mathcal{T}(N,K,K_c,M,N_r)$ the topology for the described multi-server multi-function distributed computing setting, which we detail in the following.

2.1. Datasets, Subfunctions, and Placement into Distributed Servers

There are K datasets in total, each denoted by $D_k$, $k\in \left[K\right]$. Each distributed server $i\in \Omega =\left[N\right]$ with a capacity of M is assigned a subset of datasets with indices ${\mathcal{Z}}_i\subseteq \left[K\right]$ such that $\left|{\mathcal{Z}}_i\right|=M$, where the assignments possibly overlap.

Each server computes a set of subfunctions $W_k=h_k\left(D_k\right)$ for $k\in {\mathcal{Z}}_i\subseteq \left[K\right]$, $i\in \Omega$. Datasets ${\left\{D_k\right\}}_{k\in \left[K\right]}$ could be dependent (We note that by exploiting the temporal and spatial variation or dependence of data, it is possible to decrease the communication cost.) across $\mathcal{K}$, so could ${\left\{W_k\right\}}_{k\in \left[K\right]}$. We denote the number of symbols in each $W_k$ by L, which equals the blocklength n. Let $X_i={\left\{W_k\right\}}_{k\in {\mathcal{Z}}_i}=W_{{\mathcal{Z}}_i}={\left\{h_k\left(D_k\right)\right\}}_{k\in {\mathcal{Z}}_i}$ denote the set of subfunctions of the i-th server, ${\mathcal{X}}_i$ be the alphabet of $X_i$, and $X_{\Omega }=(X_1,X_2,\dots ,X_N)$ be the set of subfunctions of all servers. We denote with ${\mathbf{W}}_k=W_{k1},W_{k2},\dots ,W_{kn}$ and ${\mathbf{X}}_i=X_{i1},X_{i2},\dots ,X_{in}\in {\mathbb{F}}_q^{|{\mathcal{Z}}_i|\times n}$, the length n sequences of subfunction $W_k$, and of $W_{{\mathcal{Z}}_i}$ assigned to server $i\in \Omega$.

2.2. Cyclic Dataset Placement Model, Computation Capacity, and Recovery Threshold

We assume that the total number of datasets K is divisible by the number of servers N, i.e., $\frac{K}{N}\doteq \Delta \in {\mathbb{Z}}^{+}$. The dataset placement on N distributed servers is conducted in a circular or cyclic manner in the number of $\Delta$ circular shifts between two consecutive servers, where the shifts are to the right and the final entries are moved to the first positions, if necessary. As a result of cyclic placement, any subset of $N_r$ servers covers the set of all datasets to compute the requested functions from the user. Given $N_r\in \left[N\right]$, each server has a storage size or computation cost of $\left|{\mathcal{Z}}_i\right|=M=\Delta (N-N_r+1)$, and the amount of dataset overlap between the consecutive servers is $\Delta (N-N_r)$.

Hence, the set of indices assigned to server $i\in \Omega$ is given as follows:

$${\mathcal{Z}}_i=\bigcup _{r=0}^{\Delta -1}\left\{mod\{i,N\}+rN,mod\{i+1,N\}+rN,\dots ,mod\{i+N-N_r,N\}+rN\right\},$$

(1)

where $X_i=W_{{\mathcal{Z}}_i}$, $i\in \Omega$. As a result of (1), the cardinality of the datasets assigned to each server meets the storage capacity constraint M with equality, i.e., $\left|{\mathcal{Z}}_i\right|=M$, for all $i\in \Omega$.

2.3. User Demands and Structure of the Computation

We address the problem of distributed lossless compression of a set of general multi-variable functions $F_j\left(X_{\Omega }\right):{\mathcal{X}}_1\times {\mathcal{X}}_2\dots \times {\mathcal{X}}_N\to {\mathbb{F}}_q$, $j\in \left[K_c\right]$, requested by the user from the set of servers, where $K_c\ge 1$, and the functions are known by the servers and the user. More specifically, the user, from a subset of distributed servers aims to compute in a lossless manner the following length n sequence as n tends to infinity:

$$F_j\left({\mathbf{X}}_{\Omega }\right)={\left\{F_j(X_{1l},X_{2l},\dots ,X_{Nl})\right\}}_{l=1}^n,j\in \left[K_c\right],$$

(2)

where $F_j(X_{1l},X_{2l},\dots ,X_{Nl})$ is the function outcome for the l-th realization $l\in \left[n\right]$, given the length n sequence. We note that the representation in (2) is the most general form of a (conceivably non-linear) multi-variate function, which encompasses the special cases of separable functions and linearly separable functions, which we discuss next.

In this work, the user seeks to compute functions that are separable to each dataset. Each demanded function $f_j(·)\in \mathbb{R}$, $j\in \left[K_c\right]$ is a function of subfunctions ${\left\{W_k\right\}}_{k\in \mathcal{K}}$ such that $W_k=h_k\left(D_k\right)\in {\mathbb{F}}_q$, where $h_k$ is a general function (could be linear or non-linear) of dataset $D_k$. Hence, using the relation $X_i=W_{{\mathcal{Z}}_i}={\left\{h_k\left(D_k\right)\right\}}_{k\in {\mathcal{Z}}_i}$, each demanded function $j\in \left[K_c\right]$ can be written in the following form:

$$f_j\left(W_{\mathcal{K}}\right)=f_j(h_1\left(D_1\right),\dots ,h_K\left(D_K\right))=F_j({\left\{h_k\left(D_k\right)\right\}}_{k\in {\mathcal{Z}}_1},\dots ,{\left\{h_k\left(D_k\right)\right\}}_{k\in {\mathcal{Z}}_N})=F_j\left(X_{\Omega }\right).$$

(3)

In the special case of linearly separable functions (Special instances of the linearly separable representation of subfunctions ${\left\{W_k\right\}}_k$ given in (4) are linear functions of the datasets $\left\{D_k\right\}$ and are denoted by $F_j={\sum }_k{\gamma }_{jk}{D}_k$.) [39], the demanded functions take the form:

$${\left\{F_j\left(X_{\Omega }\right)\right\}}_{j\in \left[K_c\right]}={\left[\begin{array}{cccc}{F}_1& F_2& \dots & F_{K_c}\end{array}\right]}^{⊺}=\mathsf{\Gamma }\mathbf{W},$$

(4)

where $\mathbf{W}={\left[\begin{array}{cccc}{W}_1& W_2& \dots & W_K\end{array}\right]}^{⊺}\in {\mathbb{F}}_q^{K\times 1}$ is the subfunction vector, and the coefficient matrix $\mathsf{\Gamma }=\left\{{\gamma }_{jk}\right\}\in {\mathbb{F}}_q^{K_c\times K}$ is known to the master node, servers, and the user. In other words, ${\left\{F_j\left(X_{\Omega }\right)\right\}}_{j\in \left[K_c\right]}$ is a set of linear maps from the subfunctions ${\left\{W_k\right\}}_k$, where $F_j\left(X_{\Omega }\right)={\sum }_{k\in \left[K\right]}{\gamma }_{jk}·W_k$. We do not restrict ${\left\{F_j\left(X_{\Omega }\right)\right\}}_{j\in \left[K_c\right]}$ to linearly separable functions, i.e., it may hold that ${\left\{F_j\left(X_{\Omega }\right)\right\}}_{j\in \left[K_c\right]}\ne \mathsf{\Gamma }\mathbf{W}$.

2.4. Communication Cost for the Characteristic-Graph-Based Computing Approach

To compute ${\left\{F_j\left({\mathbf{X}}_{\Omega }\right)\right\}}_{j\in \left[K_c\right]}$, each server $i\in \Omega$ constructs a characteristic graph, denoted by $G_{X_i}$, for compressing $X_i$. More specifically, for asymptotic lossless computation of the demanded functions, the server builds the n-th OR power $G_{{\mathbf{X}}_i}^n$ of $G_{X_i}$ for compressing ${\mathbf{X}}_i$ to determine the transmitted information. The minimal possible code rate achievable to distinguish the edges of $G_{{\mathbf{X}}_i}^n$ as $n\to \infty$ is given by the Characteristic graph entropy, $H_{G_{X_i}}\left(X_i\right)$. For a primer on key graph-theoretic concepts, characteristic-graph-related definitions, and the fundamental compression limits of characteristic graphs, we refer the reader to [76,79,82]. In this work, we solely focus on the characterization of the total communication cost from all servers to the user, i.e., the achievable sum-rate, without accounting for the costs of communication between the master and the servers and of computations performed at the servers and the user.

Each $i\in \Omega$ builds a mapping from ${\mathbf{X}}_i$ to a valid coloring of $G_{{\mathbf{X}}_i}^n$, denoted by $c_{G_{{\mathbf{X}}_i}^n}\left({\mathbf{X}}_i\right)$. The coloring $c_{G_{{\mathbf{X}}_i}^n}\left({\mathbf{X}}_i\right)$ specifies the color classes of ${\mathbf{X}}_i$ that form independent sets to distinguish the demanded function outcomes. Given an encoding function $g_i$ that models the transmission of server $i\in \Omega$ for computing ${\left\{F_j\left({\mathbf{X}}_{\Omega }\right)\right\}}_{j\in \left[K_c\right]}$, we denote by ${\mathbf{Z}}_i=g_i\left({\mathbf{X}}_i\right)=e_{X_i}\left(c_{G_{{\mathbf{X}}_i}^n}\left({\mathbf{X}}_i\right)\right)$ the color encoding performed by server $i\in \Omega$ for ${\mathbf{X}}_i$. Hence, the communication rate of server $i\in \Omega$, for a sufficiently large blocklength n, where $T_i$ is the length for the color encoding performed at $i\in \Omega$, is

$$R_i=\frac{T_i}{L}=\frac{H\left(e_{X_i}\left(c_{G_{{\mathbf{X}}_i}^n}\left({\mathbf{X}}_i\right)\right)\right)}{n}\ge H_{G_{X_i}}\left(X_i\right),i\in \Omega ,$$

(5)

where the inequality follows from exploiting the achievability of $H_{G_{X_i}}\left(X_i\right)=\underset{n\to \infty }{lim}\frac{1}{n}{H}_{G_{{\mathbf{X}}_i}^n}^{\chi }\left({\mathbf{X}}_i\right)$, where $H_{G_{{\mathbf{X}}_i}^n}^{\chi }\left({\mathbf{X}}_i\right)$ is the chromatic entropy of the graph $G_{{\mathbf{X}}_i}^n$ [73,75]. We refer the reader to Appendix A.2 for a detailed description of the notions of chromatic and graph entropies (cf. (A9) and (A10), respectively).

For the multi-server multi-function distributed setup, using the characteristic-graph-based fundamental limit in (5), an achievable sum-rate for asymptotic lossless computation is

$$R_{\mathrm{ach}}=\sum _{i\in \Omega }{R}_i\le \sum _{i\in \Omega }{H}_{G_{X_i}}\left(X_i\right).$$

(6)

We next provide our main results in Section 3.

3. Main Results

In this section, we analyze the multi-server multi-function distributed computing framework exploiting the characteristic-graph-based approach in [75]. In contrast to the previous research attempts in this direction, our solution method is general, and it captures (i) general input statistics or dataset distributions or the skew in data instead of assuming uniform distributions, (ii) correlations across datasets, (iii) any dataset placement model across servers beyond the cyclic [39] or the Maddah–Ali and Niesen [83] placements, and (iv) general function classes requested by the user, instead of focusing on a particular function type (see, e.g., [39,67,84]).

Subsequently, we delve into specific function computation scenarios. First, we present our main result (Theorem 1), which is the most general form that captures (i)–(iv). We then demonstrate (in Proposition 1) that the celebrated result of Wan et al. [Theorem 2] [39] can be obtained as a special case of Theorem 1, given that (i) the datasets are i.i.d. and uniform over q-ary fields, (ii) the placement of datasets across servers is cyclic, and (iii) the demanded functions are linearly separable, given as in (4). Under a correlated and identically distributed Bernoulli dataset model with a skewness parameter $ϵ\in (0,1)$ for datasets, we next present in Proposition 2 the achievable sum rate for computing Boolean functions. Finally, in Proposition 3, we analyze our characteristic-graph-based approach for evaluating multi-linear functions, a pertinent class of non-linear functions, under the assumption of cyclic placement and i.i.d. Bernoulli-distributed datasets with parameter $ϵ$ and derive an upper bound on the sum rate needed. To gain insight into our analytical results and demonstrate the savings in the total communication cost, we provide some numerical examples.

We next present our main theorem (Theorem 1), on the achievable communication cost for the multi-server multi-function topology, which holds for all input statistics under any correlation model across datasets and for the distributed computing of all function classes requested by the user, regardless of the data assignment over the servers’ caches. The key to capturing the structure of general functions in Theorem 1 is the utilization of a characteristic-graph-based compression technique, as proposed by Körner in [75] (For a more detailed description of characteristic graphs and their entropies, see Appendix A.2.).

Theorem 1 

(Achievable sum-rate using the characteristic graph approach for general functions and distributions). In the multi-server multi-function distributed computation model, denoted by $\mathcal{T}(N,K,K_c,M,N_r)$, under general placement of datasets, and for a set of $K_c$ general functions ${\left\{f_j\left(W_{\mathcal{K}}\right)\right\}}_{j\in \left[K_c\right]}$ requested by the user, and under general jointly distributed dataset models, including non-uniform inputs and allowing correlations across datasets, the characteristic-graph-based compression yields the following upper bound on the achievable communication rate:

$$\begin{array}{c}\hfill R_{\mathrm{ach}}\le \sum _{i=1}^{N_r}\underset{Z_i=g_i\left(X_i\right):g_i\in {\mathcal{C}}_i}{min}{H}_{G_{X_i}^{\cup }}\left(X_i\right),\end{array}$$

(7)

where

• $G_{X_i}^{\cup }=\bigcup _{j\in \left[K_c\right]}{G}_{X_i,j}$ is the union characteristic graph (We refer the reader to (A12) (Appendix A.2) for the definition of a union of characteristic graphs.) that server $i\in \Omega$ builds for computing ${\left\{f_j\left(W_{\mathcal{K}}\right)\right\}}_{j\in \left[K_c\right]}$,
• ${\mathcal{C}}_i\ni g_i$ denotes a codebook of functions that server $i\in \Omega$ uses for computing ${\left\{f_j\left(W_{\mathcal{K}}\right)\right\}}_{j\in \left[K_c\right]}$,
• each subfunction $W_k$, $k\in \mathcal{K}$ is defined over a q-ary field such that the characteristic is at least 2, and
• $Z_i=g_i\left(X_i\right)$ such that $g_i\in {\mathcal{C}}_i$ denotes the transmitted information from server $i\in \Omega$.

Proof. 

See Appendix B.1. □

Theorem 1 provides a general upper bound on the sum-rate for computing functions for general dataset statistics and correlations and the placement model and allows any function type over a field of characteristic $q\ge 2$. We note that in (7), the codebook ${\mathcal{C}}_i$ determines the structure of the union characteristic graph $G_{X_i}^{\cup }$, which, in turn, determines the distribution of $Z_i$. Therefore, the tightness of the rate upper bound relies essentially on the codebook selection. We also note that it is possible to analyze the computational complexity of building a characteristic graph and computing the bound in (7) via evaluating the complexity of the transmissions $Z_i$ determined by ${\left\{f_j\left(W_{\mathcal{K}}\right)\right\}}_{j\in \left[K_c\right]}$ for a given $i\in \Omega$. However, the current manuscript focuses primarily on the cost of communication, and we leave the computational complexity analysis to future work. Because (7) is not analytically tractable, in the following, we focus on special instances of Theorem 1 to gain insights into the effects of input statistics, dataset correlations, and special function classes in determining the total communication cost.

We next demonstrate that the achievable communication cost for the special scenario of the distributed linearly separable computation framework given in [Theorem 2] [39] is embedded by the characterization provided in Theorem 1. We next showcase the achievable sum rate result for linearly separable functions.

Proposition 1 

(Achievable sum-rate using the characteristic graph approach for linearly separable functions and i.i.d. subfunctions over ${\mathbb{F}}_q$). In the multi-server multi-function distributed computation model, denoted by $\mathcal{T}(N,K,K_c,M,N_r)$, under the cyclic placement of datasets, where $\frac{K}{N}=\Delta \in {\mathbb{Z}}^{+}$, and for a set of $K_c$ linearly separable functions, given as in (4), requested by the user, and given i.i.d. uniformly distributed subfunctions over a field of characteristic $q\ge 2$, the characteristic-graph-based compression yields the following bound on the achievable communication rate:

$$\begin{array}{c}\hfill R_{\mathrm{ach}}\le \left\{\begin{array}{c}min\{K_c,\Delta \}{N}_r,1\le K_c\le \Delta Nr,\hfill \\ min\{K_c,K\},\Delta Nr<K_c.\hfill \end{array}\right.\end{array}$$

(8)

Proof. 

See Appendix B.2. □

We note that Theorem 1 results in Proposition 1 when three conditions hold: (i) the dataset placement across servers is cyclic following the rule in (1), (ii) the subfunctions $W_{\mathcal{K}}$ are i.i.d. and uniform over ${\mathbb{F}}_q$ (see (A21) in Appendix B.2), and (iii) the codebook ${\mathcal{C}}_i$ is restricted to linear combinations of subfunctions $W_{\mathcal{K}}$, which yields that the independent sets of $G_{X_i}^{\cup }$ satisfy a set of linear constraints (We detail these linear constraints in Appendix B.2, where the set of linear equations given in (A22) is used to simplify the entropy $H_{G_{X_i}^{\cup }}\left(X_i\right)$ of the union characteristic graph $G_{X_i}^{\cup }$ via the expression given in (A20) for evaluating the upper bound given in (A18) on the achievable sum rate for computing the desired functions via exploiting the entropies of the union characteristic graphs for each of the $N_r$ servers, given the recovery threshold $N_r$.) in the variables ${\left\{W_k\right\}}_{k\in {\mathcal{Z}}_i}$. Note that the linear encoding and decoding approach for computing linearly separable functions, proposed by Wan et al. in [Theorem 2] [39], is valid over a field of characteristic $q>3$. However, in Proposition 1, the characteristic of ${\mathbb{F}}_q$ is at least 2, i.e., $q\ge 2$, generalizing [Theorem 2] [39] to larger input alphabets.

Next, we aim to demonstrate the merits of the characteristic-graph-based compression in capturing dataset correlations within the multi-server multi-function distributed computation framework. More specifically, we restrict the general input statistics in Theorem 1 such that the datasets are correlated and identically distributed, where each subfunction follows a Bernoulli distribution with the same parameter $ϵ$, i.e., $W_k\sim \mathrm{Bern}\left(ϵ\right)$, with $ϵ\in (0,1)$, and the user demands $K_c$ arbitrary Boolean functions regardless of the data assignment. Similarly to Theorem 1, the following proposition (Proposition 2) holds for general function types (Boolean) regardless of the data assignment.

Proposition 2 

(Achievable sum-rate using the characteristic graph approach for general functions and identically distributed subfunctions over ${\mathbb{F}}_2$). In the multi-server multi-function distributed computing setting, denoted by $\mathcal{T}(N,K,K_c,M,N_r)$, under the general placement of datasets, and for a set of $K_c$ Boolean functions ${\left\{f_j\left(W_{\mathcal{K}}\right)\right\}}_{j\in \left[K_c\right]}$ requested by the user, and given identically distributed and correlated subfunctions with $W_k\sim \mathrm{Bern}\left(ϵ\right)$, $k\in \left[K\right]$, where $ϵ\in (0,1)$, the characteristic-graph-based compression yields the following bound on the achievable communication rate:

$$R_{\mathrm{ach}}\le \sum _{i=1}^{N_r}\underset{Z_i=g_i\left(X_i\right):g_i\in {\mathcal{C}}_i}{min}h\left(Z_i\right),$$

(9)

where

• ${\mathcal{C}}_i\ni g_i:{\{0,1\}}^M\to \{0,1\}$ denotes a codebook of Boolean functions that server $i\in \Omega$ uses,
• $Z_i=g_i\left(X_i\right)$ such that $g_i\in {\mathcal{C}}_i$ denotes the transmitted information from server $i\in \Omega$,
• $G_{X_i}^{\cup }$ has two maximal independent sets (MISs), namely, $s_0\left(G_{X_i}^{\cup }\right)$ and $s_1\left(G_{X_i}^{\cup }\right)$, yielding $Z_i=0$ and $Z_i=1$, respectively, and
• the probability that $W_{{\mathcal{Z}}_i}$ yields the function value $Z_i=1$ is given as

  $$\mathbb{P}(Z_i=1)=\mathbb{P}(W_{{\mathcal{Z}}_i}\in s_1\left(G_{X_i}^{\cup }\right)),i\in \Omega .$$

  (10)

Proof. 

See Appendix B.3. □

While, admittedly, the above approach (Proposition 2) may not directly offer sufficient insight, it does employ the new machinery to offer a generality that allows us to plug in any set of parameters to determine the achievable performance.

Contrasting Propositions 1– 2, which give the total communication costs for computing linearly separable and Boolean functions, respectively, over ${\mathbb{F}}_2$, Proposition 2, by exploiting the skew and correlations of datasets indexed by ${\mathcal{Z}}_i$, as well as the functions’ structures via the MISs $s_0\left(G_{X_i}^{\cup }\right)$ and $s_1\left(G_{X_i}^{\cup }\right)$ of server $i\in \Omega$, demonstrates that harnessing the correlation across the datasets can indeed reduce the total communication cost versus the setting in Proposition 1, devised with the assumption of i.i.d. and uniformly distributed subfunctions.

The prior works have focused on devising distributed computation frameworks and exploring their communication costs for specific function classes. For instance, in [62], Körner and Marton have restricted the computation to be the binary sum function, and in [72], Han and Kobayashi have classified functions into two categories depending on whether they can be computed at a sum rate that is lower than that of [60]. Furthermore, the computation problem has been studied for specific topologies, e.g., the side information setting in [73,74]. Despite the existing efforts, see, e.g., [62,72,73,74], to the best of our knowledge, for the given multi-server multi-function distributed computing scenario, there is still no general framework for determining the fundamental limits of the total communication cost for computing general non-linear functions. Indeed, for this setting, the most pertinent existing work that applies to general non-linear functions and provides an upper bound on the achievable sum rate is that of Slepian–Wolf [60]. On the other hand, the upper bound on the achievable computation scheme presented in Theorem 1 can provide savings in the communication cost over [60] for functions including linearly separable functions and beyond. To that end, we exploit Theorem 1 to determine an upper bound on the achievable sum-rate for distributed computing of a multi-linear function in the form of

$$f\left(W_{\mathcal{K}}\right)=\prod _{k\in \left[K\right]}{W}_k.$$

(11)

Note that (11) is used in various scenarios, including distributed machine learning, e.g., to reduce variance in noisy datasets via ensemble learning [85] or weighted averaging [86], sensor network applications to aggregate readings for improved data analysis [87], as well as distributed optimization and financial modeling, where these functions play pivotal roles in establishing global objectives and managing risk and return [88,89].

Drawing on the utility of characteristic graphs in capturing the structures of data and functions, as well as input statistics and correlations, and the general result in Theorem 1, our next result, Proposition 3, demonstrates a new upper bound on the achievable sum rate for computing multi-linear functions within the framework of multi-server and multi-function distributed computing via exploiting conditional graph entropies.

Proposition 3 

(Achievable sum-rate using the characteristic graph approach for multi-linear functions and i.i.d. subfunctions over ${\mathbb{F}}_2$). In a multi-server multi-function distributed computing setting, denoted by $\mathcal{T}(N,K,K_c,M,N_r)$, under the cyclic placement of datasets, where $\frac{K}{N}=\Delta \in {\mathbb{Z}}^{+}$, and for computing the multi-linear function ($K_c=1$), given as in (11), requested by the user, and given i.i.d. uniformly distributed subfunctions $W_k\sim \mathrm{Bern}\left(ϵ\right)$, $k\in \left[K\right]$, for some $ϵ\in (0,1)$, the characteristic-graph-based compression yields the following bound on the achievable communication rate:

$$R_{\mathrm{ach}}\le \frac{1-{\left({ϵ}_M\right)}^{N^{*}}}{1-{ϵ}_M}·h\left({ϵ}_M\right)+{\left({ϵ}_M\right)}^{N^{*}}·1_{{\Delta }_N>0}·h\left({ϵ}_{{\xi }_N}\right),$$

(12)

where

• ${ϵ}_M={ϵ}^M$ denotes the probability that the product of M subfunctions, with $W_k\sim \mathrm{Bern}\left(ϵ\right)$ being i.i.d. across $k\in \left[K\right]$, take the value one, i.e., $\mathbb{P}\left({\prod }_{k\in \mathcal{S}:\left|\mathcal{S}\right|=M}{W}_k\right)={ϵ}_M$,
• the variable $N^{*}=⌊\frac{N}{N-N_r+1}⌋$ denotes the minimum number of servers needed to compute $f\left(W_{\mathcal{K}}\right)$, given as in (11), where each of these servers computes a disjoint product of M subfunctions, and
• the variable ${\Delta }_N=N-N^{*}·(N-N_r+1)$ represents whether an additional server is needed to aid the computation, and if ${\Delta }_N\ge 1$, then ${\xi }_N$ denotes the number of subfunctions to be computed by the additional server, and similarly to the above, $\mathbb{P}\left({\prod }_{k\in \mathcal{S}:\left|\mathcal{S}\right|={\xi }_N}{W}_k\right)={ϵ}_{{\xi }_N}$.

Proof. 

See Appendix B.4. □

We next detail two numerical examples (Section 4.1 and Section 4.2) to showcase the achievable gains in the total communication cost for Proposition 2 and Proposition 3, respectively.

4. Numerical Evaluations to Demonstrate the Achievable Gains

Given $\mathcal{T}(N,K,K_c,M,N_r)$, to gain insight into our analytical results and demonstrate the savings in the total communication cost, we provide some numerical examples. To demonstrate Proposition 2, in Section 4.1, we focus on computing linearly separable functions, and in Section 4.2 (cf. Proposition 3), we focus on multi-linear functions, respectively.

To that end, to characterize the performance of our characteristic-graph-based approach for linearly separable functions, we denote by ${\eta }_{lin}$ the gain of the sum-rate for the characteristic-graph-based approach given in (9) over the sum-rate of the distributed scheme of Wan et al. in [39], given in (8), and by ${\eta }_{SW}$ the gain of the sum-rate in (9) over the sum-rate of the fully distributed approach of Slepian–Wolf [60]. To capture general statistics, i.e., dataset skewness and correlations, and make a fair comparison, we adapt the transmission model of Wan et al. in [39] via modifying the i.i.d. dataset assumption.

We next study an example scenario (Section 4.1) for computing a class of linearly separable functions (4) over ${\mathbb{F}}_2$, where each of the demanded functions takes the form $f_j\left(W_{\mathcal{K}}\right)={\sum }_{k\in \left[K\right]}{\gamma }_{jk}{W}_{k}mod2$, $j\in \left[K_c\right]$ under a specific correlation model across subfunctions. More specifically, when the subfunctions $W_k\sim \mathrm{Bern}\left(ϵ\right)$ are identically distributed and correlated across $k\in \left[K\right]$, and $\Delta \in {\mathbb{Z}}^{+}$, we model the correlation across datasets (a) exploiting the joint PMF model in [Theorem 1] [90] and (b) for a joint PMF described in

Table 2. Joint PMF $P_{W_2,W_3}$ of $W_2$ and $W_3$ with a crossover parameter p, in Section 4.1 (Scenario II).

${\mathit{P}}_{{\mathit{W}}_2,{\mathit{W}}_3}({\mathit{W}}_2,{\mathit{W}}_3)$	${\mathit{W}}_2=0$	${\mathit{W}}_2=1$
$W_3=0$	$(1-ϵ)(1-p^{\prime })$	$ϵp$
$W_3=1$	$ϵp$	$ϵ(1-p)$

. Furthermore, we assume for $K_c>1$ that $\mathsf{\Gamma }=\left\{{\gamma }_{jk}\right\}\in {\mathbb{F}}_2^{K_c\times K}$ is full rank. For the proposed setting, we next demonstrate the achievable gains ${\eta }_{lin}$ of our proposed technique versus $ϵ$ for computing (4) as a function of skew, $ϵ$, and correlation, $\rho$, of datasets, $K_c\in \left[N_r\right]<K$, and other system parameters and showcase the results via Figures 1, 3–5.

4.1. Example Case: Distributed Computing of Linearly Separable Functions over ${\mathbb{F}}_2$

We consider the computation of the linearly separable functions given in (4) for general topologies, with general N, K, M, $N_r$, $K_c$, over ${\mathbb{F}}_2$, with an identical skew parameter $ϵ\in [0,1]$ for each subfunction, where $W_k\sim$ Bern($ϵ$), $k\in \left[K\right]$, using cyclic placement as in (1) and incorporating the correlation between the subfunctions, with the correlation coefficient denoted by $\rho$. We consider three scenarios, as described next:

• Scenario I. The number of demanded functions is ${\mathbf{K}}_{\mathbf{c}}=\mathbf{1}$, where the subfunctions could be uncorrelated or correlated.

This scenario is similar to the setting in [39], although different from [39], which is valid over a field of characteristic $q>3$, we consider ${\mathbb{F}}_2$, and in the case of correlations, i.e., when $\rho >0$, we capture the correlations across the transmissions (evaluated from subfunctions of datasets) from distributed servers, as detailed earlier in Section 3. We first assume that the subfunctions are not correlated, i.e., $\rho =0$, and evaluate ${\eta }_{lin}$ for $f\left(W_{\mathcal{K}}\right)={\sum }_{k\in \left[K\right]}{W}_{k}mod2$. The parameter of $f\left(W_{\mathcal{K}}\right)$, i.e., the probability that $f\left(W_{\mathcal{K}}\right)$ takes the value 1 can be computed using the recursive relation

$$\begin{array}{cc}\hfill \mathbb{P}\left(\sum _{k\in \mathcal{S}:\left|\mathcal{S}\right|=l\le K}{W}_{k}mod2=1\right)& =\sum _{k\in \mathcal{S}:\left|\mathcal{S}\right|=l\le K,k\mathrm{odd}}\mathbb{P}\left(B(K,ϵ)\right)=k)\hfill \\ \hfill & =(1-{ϵ}_{l-1})·ϵ+{ϵ}_{l-1}·(1-ϵ)\hfill \\ \hfill & \doteq {ϵ}_l,1<l\le K,\hfill \end{array}$$

(13)

where $B(K,ϵ)$ is the binomial PMF, and ${ϵ}_l$ is the probability of the modulo 2 sum of any $1<l\le K$ subfunctions taking the value one, with $W_k\sim \mathrm{Bern}\left(ϵ\right)$ being i.i.d. across $k\in \mathcal{S}$, with the convention ${ϵ}_1=ϵ$.

Given $N_r$, we denote by $N^{*}=⌊\frac{N}{N-N_r+1}⌋$ the minimum number of servers, corresponding to the subset ${\mathcal{N}}^{*}\subseteq \Omega$, needed to compute $f\left(W_{\mathcal{K}}\right)$, where each server, with a cache size of M, computes a sum of M subfunctions, where across these $N^{*}$ servers, the sets of subfunctions are disjoint. Hence, $\mathbb{P}\left({\sum }_{k\in \mathcal{S}:\left|\mathcal{S}\right|=M}{W}_k\right)={ϵ}_M$. Furthermore, the variable ${\Delta }_N=N-N^{*}·(N-N_r+1)$ represents whether additional servers in addition to $N^{*}$ servers are needed to aid the computation, and if ${\Delta }_N\ge 1$, then $\Delta ·{\Delta }_N\doteq {\xi }_N$ denotes the number of subfunctions to be computed by the set of additional servers, namely, ${\mathcal{I}}^{*}\in \Omega$, and similarly to the above, $\mathbb{P}\left({\sum }_{k\in \mathcal{S}:\left|\mathcal{S}\right|={\xi }_N}{W}_k\right)={ϵ}_{{\xi }_N}$, which is obtained by evaluating ${ϵ}_l$ at $l={\xi }_N$.

Adapting (8) for ${\mathbb{F}}_2$, we obtain the total communication cost $R_{ach}\left(lin\right)$ for computing the linearly separable function $f\left(W_{\mathcal{K}}\right)={\sum }_{k\in \left[K\right]}{W}_{k}mod2$ as

$$\begin{array}{c}\hfill R_{ach}\left(lin\right)=\sum _{i=1}^{N_r}h\left(\sum _{k\in {\mathcal{Z}}_i}{W}_k\right)=N_r·h\left({ϵ}_M\right).\end{array}$$

(14)

Using Proposition 2 and (13), we derive the sum rate for distributed lossless computing of $f\left(W_{\mathcal{K}}\right)$ as

$$\begin{array}{c}\hfill \sum _{i\in \Omega }{R}_i\le N^{*}·h\left({ϵ}_M\right)+1_{{\Delta }_N>0}·h\left({ϵ}_{{\xi }_N}\right),\end{array}$$

(15)

where the indicator function $1_{{\Delta }_N>0}$ captures the rate contribution from the additional server, if any. Using (15), the gain ${\eta }_{lin}$ over the linearly separable solution of [39] is presented as

$$\begin{array}{c}\hfill {\eta }_{lin}=\frac{N_r·h\left({ϵ}_M\right)}{N^{*}·h\left({ϵ}_M\right)+1_{{\Delta }_N>0}·h\left({ϵ}_{{\xi }_N}\right)},\end{array}$$

(16)

where $h\left({ϵ}_{{\xi }_N}\right)$ represents the rate needed from the set of additional servers ${\mathcal{I}}^{*}\in \Omega$, aiding the computation through communicating the sum of the remaining subfunctions in the set $\mathcal{C}\subseteq {\mathcal{Z}}_{{\mathcal{I}}^{*}}$, where the summation for these remaining functions in $\mathcal{C}\subseteq {\mathcal{Z}}_{{\mathcal{I}}^{*}}$ is denoted as ${\sum }_{k\in \mathcal{C}\subseteq {\mathcal{Z}}_{{\mathcal{I}}^{*}}:k\notin {\bigcup }_{i\in {\mathcal{N}}^{*}}{\mathcal{Z}}_i,\left|\mathcal{C}\right|={\xi }_N}{W}_k$, which cannot be captured by the set ${\mathcal{N}}^{*}$.

Given $K_c=1$ for the given modulo 2 sum function, we next incorporate the correlation model in [90] for each $W_k$, identically distributed with $W_k\sim \mathrm{Bern}\left(ϵ\right)$, and correlation $\rho$ across any two subfunctions. The formulation in [90] yields the following PMF for $f\left(W_{\mathcal{K}}\right)$:

$$\begin{array}{cc}\hfill \mathbb{P}(f\left(W_K\right)=y)& =\left(\genfrac{}{}{0pt}{}{K}{y}\right){ϵ}^y{(1-ϵ)}^{K-y}(1-\rho )·1_{y\in A_1}\hfill \\ \hfill & +{ϵ}^{\frac{y}{K}}{(1-ϵ)}^{\frac{K-y}{K}}\rho ·1_{y\in A_2},y\in \{0,\dots ,K\},\hfill \end{array}$$

(17)

where $1_{y\in A_1}$ and $1_{y\in A_2}$ are indicator functions, where $A_1=\{0,1,\dots ,K\}$ and $A_2=\{0,K\}$.

We depict the behavior of our gain, ${\eta }_{lin}$, using the same topology $\mathcal{T}(N,K,K_c,M,N_r)$ as in [39], with different system parameters $(N,K,M,N_r)$, under $\rho =0$ in Figure 1-(Left). As we increase both N and K, along with the number of active servers, $N_r$, the gain, ${\eta }_{lin}$, of the characteristic graph approach increases. This stems from the characteristic graph approach to compute functions $f\left(W_{\mathcal{K}}\right)$ of $W_{\mathcal{K}}$ using $N^{*}$ servers. From Figure 1-(Right), it is evident that by capturing correlations between the subfunctions, hence, across the servers’ caches, ${\eta }_{lin}$ grows more rapidly until it reaches the maximum of (16), corresponding to ${\eta }_{lin}=\frac{N_r}{N^{*}}=10$, attributed to full correlation (see, Figure 1-(Right)).

What we can also see is that for $\rho =0$, the gain rises with the increase in $ϵ$ and linearly grows with $\frac{N_r}{N^{*}}$. As $\rho$ increases, reaching its maximum at $\rho =1$, the gain is maximized, yielding the minimum communication cost that can be achieved with our technique. Here, the gain ${\eta }_{lin}$ is dictated by the topology and is given as ${\eta }_{lin}=\frac{N_r}{N^{*}}$. This linear relation shows that this specific topology can provide a very substantial reduction in the total communication cost, as $\rho$ goes to 1, over the state of the art [39], as shown in Figure 1-(Right) via the purple (solid) curve. Furthermore, one can draw a comparison between the characteristic graph approach and the approach in [60]. Here, we represent the gain as ${\eta }_{SW}$. It is noteworthy that the sum-rate of all servers using the coding approach of Slepian–Wolf [60] is $R_{ach}\left(SW\right)=H\left(W_{\mathcal{K}}\right)$. With $\rho =0$, this expression simplifies to $R_{ach}\left(SW\right)=K·H\left(W_k\right)$, resulting again in a substantial reduction in the communication cost, as we see from $R_{ach}\left(lin\right)$ in (14) for the same topology of the purple (solid) curve as shown in Figure 1-(Right).

• Scenario II. The number of demanded functions is ${\mathbf{K}}_{\mathbf{c}}=\mathbf{2}$, where the subfunctions could be uncorrelated or correlated.

To gain insights into the behavior of ${\eta }_{lin}$, we consider an example distributed computation model with $K=N=3$, $N_r=2$, where the subfunctions $W_1,W_2,W_3$ are assigned to $X_1$, $X_2$, and $X_3$ in a cyclic manner, with $h\left(W_k\right)=ϵ$, $k\in \left[3\right]$, and $K_c=2$ with $f_1\left(W_{\mathcal{K}}\right)=W_2$, and $f_2\left(W_{\mathcal{K}}\right)=W_2+W_3$.

Given $N_r=2$, using the characteristic graph approach for individual servers, an achievable compression scheme, for a given ordering i and j of server transmissions, relies on first compression of the characteristic graph $G_{X_i}$ constructed by server $i\in \Omega$ that has no side information and then the conditional rate needed for compressing the colors of $G_{X_j}$ for any other server $j\in \Omega \setminus i$ via incorporating the side information $Z_i=g_i\left(X_i\right)$ obtained from server $i\in \Omega$. Thus, contrasting the total communication cost associated with the possible orderings, the minimum total communication cost $R_{ach}\left(G\right)$ can be determined (We can generalize (18) to $N_r>2$, where, for a given ordering of server transmissions, any consecutive server that transmits sees all previous transmissions as side information and the best ordering that has the minimum total communication cost, i.e., $R_{ach}\left(G\right)$.). The achievable sum rate here takes the form

$$\begin{array}{c}\hfill R_{ach}\left(G\right)=min\{H_{G_{X_1}}\left(X_1\right)+H_{G_{X_2}}\left(X_2\right|Z_1),H_{G_{X_2}}\left(X_2\right)+H_{G_{X_1}}\left(X_1\right|Z_2)\}.\end{array}$$

(18)

Focusing on the characteristic graph approach, we illustrate how each server builds its union characteristic graph for simultaneously computing $f_1$ and $f_2$ according to (A12) (as detailed in Appendix A.2.1), in Figure 2. In (18), the first term corresponds to $G_{X_1}=(V_{X_1},E_{X_1})$, where $V_{X_1}={\{0,1\}}^2$ is built using the support of $W_1$ and $W_2$, and the edges $E_{X_1}$ are built based on the rule that $(x_1^1,x_1^2)\in E_{X_1}$ if $F(x_1^1,x_2)\ne F(x_1^2,x_2)$ for some $x_2\in V_{X_2}$, which, as we see here, requires two colors. Similarly, server 2 constructs $G_{X_2}=(V_{X_2},E_{X_2})$ given $Z_1$, where $V_{X_2}={\{0,1\}}^2$ using the support of $W_2$ and $W_3$, and where $Z_1$ determines $f_1=W_2$, and hence, to compute $f_2=W_2+W_3$ given $f_1=W_2$, any two vertices taking values (Here, $x_2^1=(w_2^1,w_3^1)$ and $x_2^2=(w_2^2,w_3^2)$ represent two different realizations of the pair of subfunctions $W_2$ and $W_3$.) $x_2^1=(w_2^1,w_3^1)\in V_{X_2}$ and $x_2^2=(w_2^2,w_3^2)\in V_{X_2}$ are connected if $w_3^1\ne w_3^2$. Hence, we require two distinct colors for $G_{X_2}$. As a result, the first term yields a sum rate of $h\left(ϵ\right)+h\left(ϵ\right)=2h\left(ϵ\right)$. Similarly, the second term of (18) captures the impact of $G_{X_2}=(V_{X_2},E_{X_2})$, where server 2 builds $G_{X_2}$ using the support of $W_2$ and $W_3$, and $G_{X_2}$ is a complete graph to distinguish all possible binary pairs to compute $f_1$ and $f_2$, requiring 4 different colors. Given $Z_2$, both $f_1$ and $f_2$ are deterministic. Hence, given $Z_2$, $G_{X_1}$ has no edges, which means that $H_{G_{X_1}}\left(X_1\right|Z_2)=0$. As a result, the ordering of server transmission given by the second term of (18) yields the same sum rate of $2h\left(ϵ\right)+0=2h\left(ϵ\right)$. For this setting, the minimum required rate is $R_{ach}\left(G\right)=2h\left(ϵ\right)$, and the configuration captured by the second term provides a lower recovery threshold of $N_r=1$ versus $N_r=2$ for the configurations of server transmissions given by the first term (18). The different $N_r$ achieved by these two configurations is also captured by Figure 2.

Alternatively, in the linearly separable approach [39], $N_r$ servers transmit the requested function of the datasets stored in their caches. For distributed computing of $f_1$ and $f_2$, servers 1 and 2 transmit at rate $H\left(W_2\right)=h\left(ϵ\right)$, for computing $f_1$, and at rate $H(W_2+W_3)$, for function $f_2$. As a result, the achievable communication cost is given by $R_{ach}\left(lin\right)=h\left(ϵ\right)+h(W_2+W_3)$. Here, for a fair comparison, we update the model studied in [39] to capture the correlation within each server without accounting for the correlation across the servers.

Under this setting, for $\rho =0$, we see that the gain ${\eta }_{lin}$ of the characteristic graph approach over the linearly separable solution of [39] for computing $f_1$ and $f_2$ as a function of $ϵ\in [0,1]$ takes the form

$$\begin{array}{cc}\hfill {\eta }_{lin}\left(ϵ\right)& =\frac{h\left(ϵ\right)+h\left(2ϵ\right(1-ϵ\left)\right)}{2h\left(ϵ\right)}\left\{\begin{array}{cc}=1\hfill & ,ϵ=\left\{\frac{1}{2}\right\},\hfill \\ >1\hfill & ,ϵ\in [0,1]\setminus \left\{\frac{1}{2}\right\},\hfill \end{array}\right.\hfill \end{array}$$

(19)

where ${\eta }_{lin}\left(ϵ\right)>1$ for $ϵ\ne \frac{1}{2}$ follows from the concavity of $h(·)$, which yields the inequality $h\left(2ϵ\right(1-ϵ\left)\right)\ge h\left(ϵ\right)$. Furthermore, ${\eta }_{lin}$ approaches $1.5$ as $ϵ\to \{0,1\}$ (see Figure 3).

We next examine the setting where the correlation coefficient $\rho$ is nonzero, using the joint PMF $P_{W_2,W_3}$, as depicted in Table 2, of the required subfunctions ($W_2$ and $W_3$) in computing $f_1$ and $f_2$. This PMF describes the joint PMF corresponding to a binary non-symmetric channel model, where the correlation coefficient between $W_2$ and $W_3$ is $\rho =\frac{1-p}{1-ϵ}$, and where $p^{\prime }=\frac{ϵp}{1-ϵ}$. Thus, our gain here compared to the linearly separable encoding and decoding approach of [39] is given as

$$\begin{array}{c}\hfill {\eta }_{lin}=\frac{H\left(W_2\right)+H(W_2+W_3)}{H(W_2,W_3)}=\frac{h\left(ϵ\right)+h\left(2ϵp\right)}{h\left(ϵ\right)+(1-ϵ)h\left(\frac{ϵp}{1-ϵ}\right)+ϵh\left(p\right)}.\end{array}$$

(20)

We consider now the correlation model in Table 2, where coefficient $\rho$ rises in $ϵ$ for a fixed p. In Figure 4-(Left), we illustrate the behavior of ${\eta }_{lin}$, given by (20), for computing $f_1$ and $f_2$ for $N_r=2$ as a function of p and $ϵ$, where for this setting, the correlation coefficient $\rho$ is a decreasing function of p and an increasing function of $ϵ$. We observe from (20) that the gain ${\eta }_{lin}$ satisfies ${\eta }_{lin}\ge 1$ for all $ϵ\in [0,1]$, which monotonically increases in p—and hence monotonically decreases in $\rho$ due to the relation $\rho =\frac{1-p}{1-ϵ}$—as a function of the deviation of $ϵ$ from $1/2$. For $ϵ\in (0.5,1]$, ${\eta }_{lin}$ increases in $ϵ$. For example, for $p=0.1$ then ${\eta }_{lin}\left(1\right)=1.28$, as depicted by the green (solid) curve. Similarly, given $ϵ\in [0,0.5)$, decreasing $ϵ$ results in ${\eta }_{lin}$ to exhibit a rising trend, e.g., for $p=0.9$ then ${\eta }_{lin}\left(0\right)=1.36$, as shown by the red (dash-dotted) curve. As p approaches one, ${\eta }_{lin}$ goes to $1.5$ as $ϵ$ tends to zero, which can be derived from (20). We here note that the gains are generally smaller than in the previous set of comparisons, as shown in Figure 3.

More generally, given a user request consisting of $K_c=2$ linearly separable functions (i.e., satisfying (4)), and after considering (20) beyond $N_r=2$, we see that ${\eta }_{lin}$ is at most $N_r$ as $\rho$ approaches one. We next use the joint PMF model used in obtaining (17), where we observe that $f_2\sim ({(1-ϵ)}^2(1-\rho )+(1-ϵ)\rho ,2ϵ(1-ϵ)(1-\rho ),{ϵ}^2(1-\rho )+ϵ\rho )$, to see that the gain takes the form

$$\begin{array}{c}\hfill {\eta }_{lin}=\frac{h\left(ϵ\right)+H\left(f_2\right)}{h\left(ϵ\right)+(1-ϵ)h\left({\zeta }_1\right)+ϵh\left({\zeta }_2\right)},\end{array}$$

(21)

where ${\zeta }_1=(1-ϵ)(1-\rho )+\rho$, and ${\zeta }_2=(1-ϵ)(1-\rho )$. For this model, we illustrate ${\eta }_{lin}$ versus $ϵ$ in Figure 4-(Right) for different $\rho$ values. Evaluating (21), the peak achievable gain is attained when $\rho =1$ at $f_2\sim ((1-ϵ),0,ϵ)$, yielding $H(W_2+W_3)=h\left(ϵ\right)$ and $H\left(W_3\right|W_2)=(1-ϵ)h\left(\rho \right)=0$, and hence, a gain ${\eta }_{lin}=N_r=2$, as shown by the purple (solid) curve. On the other hand, for $\rho =0$, we observe that $f_2\sim ({(1-ϵ)}^2,2ϵ(1-ϵ),{ϵ}^2)$, yielding $H(W_2+W_3)=h({(1-ϵ)}^2,2ϵ(1-ϵ),{ϵ}^2)=h\left(2ϵ(1-ϵ)\right)+({(1-ϵ)}^2+{ϵ}^2)h\left(\frac{{ϵ}^2}{{ϵ}^2+{(1-ϵ)}^2}\right)$ and $H\left(W_3\right|W_2)=(1-ϵ)h\left(ϵ\right)+ϵh\left(ϵ\right)=h\left(ϵ\right)$, and hence, it can be shown that the gain is lower bounded as ${\eta }_{lin}\ge 1.25$.

• Scenario III. The number of demanded functions is ${\mathbf{K}}_{\mathbf{c}}\in \left[{\mathbf{N}}_{\mathbf{r}}\right]$, and the number of datasets is equal to the number of servers, i.e., $\mathbf{K}=\mathbf{N}$, where the subfunctions are uncorrelated.

We now provide an achievable rate comparison between the approach in [39] and our graph-based approach, as summarized by our Proposition 1, which generalizes the result in [Theorem 2] [39] to finite fields with characteristics $q\ge 2$, for the case of $\rho =0$.

Here, to capture dataset skewness and make a fair comparison, we adapt the transmission model of Wan et al. in [39] via modifying the i.i.d. dataset assumption and taking into account the skewness incurred within each server in determining the local computations ${\sum }_{k\in \mathcal{S}:\left|\mathcal{S}\right|=M}{W}_k$ at each server.

For the linearly separable model in (4), adapted to account for our setting, exploiting the summation ${\sum }_{k\in {\mathcal{Z}}_i}{W}_k$, and ${ϵ}_M$ given in (15), the communication cost for a general number of $K_c$ with $\rho =0$ is expressed as

$$\begin{array}{c}\hfill R_{ach}\left(lin\right)=N_r·h\left({ϵ}_M\right).\end{array}$$

(22)

In (22), as $ϵ$ approaches 0 or 1, then $h\left({ϵ}_M\right)\to 0$. Subsequently, the achievable communication cost for the characteristic graph model can be determined as

$$\begin{array}{c}\hfill R_{ach}\left(G\right)=K_c·N^{*}·h\left(ϵ\right).\end{array}$$

(23)

To understand the behavior of ${\eta }_{lin}=\frac{N_r}{K_c{N}^{*}}·\frac{h\left({ϵ}_M\right)}{h\left(ϵ\right)}$, knowing that $\frac{N_r}{K_c{N}^{*}}$ is a fixed parameter, we need to examine the dynamic component $\frac{h\left({ϵ}_M\right)}{h\left(ϵ\right)}$. Exploiting Schur concavity (A real-valued function $f:{\mathbb{R}}^n\to \mathbb{R}$ is Schur concave if $f(x_1,x_2,\dots ,x_n)\le f(y_1,y_2,\dots ,y_n)$ holds whenever $(x_1,x_2,\dots ,x_n)$ majorizes $(y_1,y_2,\dots ,y_n)$, i.e., ${\sum }_{i=1}^k{x}_i\ge {\sum }_{i=1}^k{y}_i$, for all $k\in \left[n\right]$ [91].) for the binary entropy function, which tells us that $h\left(\mathbb{E}\right[X\left]\right)\ge \mathbb{E}\left[h\right(X\left)\right]$, we can see that as $ϵ$ approaches 0 or 1, then

$$\begin{array}{c}\hfill \underset{ϵ\to \{0,1\}}{lim}\frac{h\left({ϵ}_M\right)}{h\left(ϵ\right)}\le M,M\in {\mathbb{Z}}^{+},\end{array}$$

(24)

where the inequality between the left- and right-hand sides becomes loose as a function of M. As a result, as $ϵ$ approaches 0 or 1, then ${\eta }_{lin}\approx M·\frac{N_r}{K_c·N^{*}}$, which follows from exploiting (22), (23) and the achievability of the upper bound in (24). We illustrate the upper bound on ${\eta }_{lin}$ in Figure 5 and demonstrate the ${\eta }_{lin}$ behavior for $K_c$ demanded functions across various topologies with circular dataset placement, namely, for various $K=N$, i.e., when the amount of circular shift between two consecutive servers is $\Delta =\frac{K}{N}=1$ and the cache size is $M=N-N_r+1$, and for $\rho =0$ and $ϵ\le 1/2$. We focus only on plotting ${\eta }_{lin}$ for $ϵ\le 1/2$, accounting for the symmetry of the entropy function. Therefore, we only plot for $ϵ\le 1/2$. The multiplicative coefficient $\frac{N_r}{K_c{N}^{*}}$ of ${\eta }_{lin}$ determines the growth, which is depicted by the curves.

Thus, we see that for a given topology $\mathcal{T}(N,K,K_c,M,N_r)$ with $K_c$ demanded functions, for $\rho =0$, using (24), we see that ${\eta }_{lin}$ exponentially grows with term $1-ϵ$ for $ϵ\in [0,1/2]$ (Here, we note that the behavior of ${\eta }_{lin}$ is symmetric around $ϵ=1/2$.), and very substantial reduction in the total communication cost is possible as $ϵ$ approaches $\{0,1\}$, as shown in Figure 5 by the blue (solid) curve. The gain over [Theorem 2] [39], ${\eta }_{lin}$, for a given topology, changes proportionally to $\frac{N_r}{K_c{N}^{*}}$. The gain over [60], ${\eta }_{SW}$, for $\rho =0$ linearly scales (Incorporating the dataset skew to Proposition 1 ([Theorem 2] [39]), $R_{ach}\left(lin\right)$ is simplified to (22), which from (24) can linearly grow in $M=N-N_r+1$ at high skew, explaining the inferior performance of Proposition 1 over [60] as a function of the skew.) with $\frac{K}{K_c{N}^{*}}$. For instance, the gain for the blue (solid) curve in Figure 5 is ${\eta }_{SW}=10$.

In general, other functions in ${\mathbb{F}}_2$, such as bitwise $AND$ and the multi-linear function (see, e.g., Proposition 3) are more skewed and have lower entropies than linearly separable functions and, hence, are easier to compute. Therefore, the cost given in (23) can serve as an upper bound for the communication costs of those more skewed functions in ${\mathbb{F}}_2$.

We have here provided insights into the achievable gains in communication cost for several scenarios. We leave the study of ${\eta }_{lin}$ for more general topologies $\mathcal{T}(N,K,K_c,M,N_r)$ and correlation models beyond (17) devised for linearly separable functions, and beyond the joint PMF model in Table 2, as future work.

Proposition 3 illustrates the power of the characteristic graph approach in decreasing the communication cost for distributed computing of multi-linear functions, given as in (11), compared to recovering the local computations ${\prod }_{k\in \mathcal{S}:\left|\mathcal{S}\right|=M}{W}_k$ using [60]. We denote by ${\eta }_{SW}$ the gain of the sum-rate for the graph entropy-based approach given in (12)—using the conditional entropy-based sum-rate expression in (A30)—over the sum-rate of the fully distributed scheme of Slepian–Wolf [60] for computing (11). For the proposed setting, we next showcase the achievable gains ${\eta }_{SW}$ of Proposition 3 via an example and showcase the results via Figure 6.

4.2. Distributed Computation of K-Multi-Linear Functions over ${\mathbb{F}}_2$

We study the behaviors of ${\eta }_{SW}$ versus the skewness parameter $ϵ$ for computing the multi-linear function given in (11) for i.i.d. uniform $W_k\sim \mathrm{Bern}\left(ϵ\right)$, $ϵ\in [0,1/2]$ across $k\in \left[K\right]$, and for a given $\mathcal{T}(N,K,K_c,M,N_r)$ with parameters N, K, $M=\Delta (N-N_r+1)$, such that $N_r=N-1$, $K_c=1$, $\rho =0$, and the number of replicates per dataset is $\frac{MN}{K}=2$. We use Proposition 3 to determine the sum-rate upper bound and illustrate the gains $10{log}_{10}\left({\eta }_{SW}\right)$ in decibels versus $ϵ$ in Figure 6.

From the numerical results in Figure 6 (Left), we observe that the sum-rate gain of the graph entropy-based approach versus the fully distributed approach of [60], ${\eta }_{SW}$, could reach up to more than 10-fold gain in compression rate for uniform and up to ${10}^6$-fold for skewed data. The results for ${\eta }_{SW}$ showcase that our proposed scheme can guarantee an exponential rate reduction over [60] as a function of decreasing $ϵ$. Furthermore, the sum-rate gains scale linearly with the cache size M, which scales with K given $N_r=N-1$. Note that ${\eta }_{SW}$ diminishes with increasing N when M and $\Delta$ are kept fixed. In Figure 6 (Right), for $M\ll K$, a fixed total cache size $MN$, and hence, fixed K, the gain ${\eta }_{SW}$ for large N and small M is higher versus small N and large M, demonstrating the power of the graph-based approach as the topology becomes more and more distributed.

5. Conclusions

In this paper, we devised a distributed computation framework for general function classes in multi-server multi-function, single-user topologies. Specifically, we analyzed the upper bounds for the communication cost for computing in such topologies, exploiting Körner’s characteristic graph entropy, by incorporating the structures in the dataset and functions, as well as the dataset correlations. To showcase the achievable gains of our framework and perceive the roles of dataset statistics, correlations, and function classes, we performed several experiments under cyclic dataset placement over a field of characteristic two. Our numerical evaluations for distributed computing of linearly separable functions, as demonstrated in Section 4.1 via three scenarios, indicate that by incorporating dataset correlations and skew, it is possible to achieve a very substantial reduction in the total communication cost over the state of the art. Similarly, for distributed computing of multi-linear functions, in Section 4.2, we demonstrate a very substantial reduction in the total communication cost versus the state of the art. Our main results (Theorem 1 and Propositions 1–3) and observations through the examples help us gain insights into reducing the communication cost of distributed computation by taking into account the structures of datasets (skew and correlations) and functions (characteristic graphs).

The potential directions include providing a tighter achievability result for Theorem 1 and devising a converse bound on the sum-rate. They involve conducting experiments under the scheme of the coded scheme of Maddah–Ali and Niesen detailed in [83] in order to capture the finer-grained granularity of placement that can help tighten the achievable rates. They also involve, beyond the special cases detailed in Propositions 1–3, exploring the achievable gains for a broader set of distributed computation scenarios, e.g., over-the-air computing, cluster computing, coded computing, distributed gradient descent, or more generally, distributed optimization and learning and goal-oriented and semantic communication frameworks, which can be reinforced by compression by capturing the skewness, correlations, and placement of datasets, the structures of functions, and topology.