Remote Work in Post-Pandemic Reality—Multi-Criteria Evaluation of Teleconferencing Software

Abstract

The pandemic period has made remote work a reality in many organizations. Despite the possible negative aspects of this form of work, many employers and employees appreciate its flexibility and effectiveness. Therefore, employers are looking for the most optimal tools to support this form of work. However, this may be difficult due to their complexity, different functionality, or different conditions of the company’s operations. Decisions on the choice of a given solution are usually made in a group of decision makers. Often their subjective assessments differ from each other, making it even more difficult to make a decision. The aim of this article is to propose a methodological solution supporting the assessment of the most popular teleconferencing systems and generating their ranking. The feature of this solutions is the combination of two important methodological aspects facilitating the selection process. The first one concerns the possibility of taking into account quantitative and qualitative criteria expressed linguistically and of an uncertain nature in the assessment (NEAT F-PROMETHEE method). The second one is related to the possibility of taking into account the assessments of many experts, including the consensus study between them (PROSA GDSS method). The use of these combined methods to assess teleconferencing platforms made it possible to create their ranking and indicate the solution that best meets the adopted criteria (based on experts’ opinions). The Microsoft Teams system turned out to be this solution, whose functionality, usability, multi-platform aspect and other elements turned out to be crucial in the context of the overall assessment. The results obtained may be a guideline for managers and decision makers facing the choice of a tool supporting remote work.

1. Introduction

Lockdown has changed the perception of working methods in many jobs that do not require the physical presence of an employee. If given such an opportunity, a significant number of companies took advantage of the opportunity to work remotely. For example, in the United States, the share of work from home increased from 14.4% to 39.6% of total employment between February and May 2020 [1]. At that time, about 70% of employees who could work from home decided to do so [2]. A year later, in June 2021, this rate dropped to 28.5%, which is about twice as high as before the pandemic [1]. The importance of remote work has also definitely increased in other countries worldwide, although this increase has not been uniform. In Germany, in April 2020, approx. 26% of employees worked entirely from home, and approx. 35% combined remote work with their presence in the workplace [3]. In France, this rate was around 25% in the same period, slightly higher in Italy [4], and, for example, in Japan, this rate was lower, reaching around 17% in June 2020 [5].

The pandemic and the lockdown have forced the acceleration to transform the employment models so that they comply with the rules of remote work. Moreover, in most countries worldwide, there was a lack of legal regulations adapted to the new situation, such as those concerning the organization and control of work (providing a workplace by the employer, occupational health and safety) or employment conditions [6,7,8]. The massive transition to remote work caused by the pandemic revealed both the advantages and limitations of such functioning [9]. A broad discussion began, covering many aspects that previously were not so important for both employees and employers. The new conditions turned out to be beneficial for a part of the society, they were positively received, but for another part they were not necessarily so. Difficulties in separating work from personal life can be a big problem. The lack of a clear separation of these two spheres of life may cause an imbalance between personal (family) life and work. An important, negative aspect affecting this balance may be (sometimes enforced by employers) the need for continuous availability via ICT (Information and Communications Technology). Work responsibilities can overshadow other activities of daily life, which in turn can negatively affect family relationships. Remote work therefore requires strong self-discipline, effective time management and the ability to focus on professional tasks at set times. Working from home can also lead to reduced interaction with other people and social isolation. The lack of direct meetings with co-workers may have a negative impact on developing mutual relations and solving professional problems. Remote communication tools (e-mails, messengers, videoconferencing) are not always as effective in conveying information as direct verbal and non-verbal communication. Another problem reported by employers is the difficulty in monitoring the performance of remote employees. Without direct supervision, in many cases it is difficult to assess the effectiveness of such work [10,11,12]. Due to many factors that may have a negative impact on remote work, this form of work may be difficult to accept for a part of society.

Other people, on the other hand, like this form of professional activity. They praise the convenience, flexibility of working hours, less stress [13,14], and in many cases they notice greater productivity [15,16,17]. Another frequently cited benefit is saving time and money for commuting to work or school [18,19] and less congestion on the roads [1,20]. In addition, as research [21,22,23,24] shows, a significant proportion of employees and employers were interested in continuing to work from home after the end of the pandemic.

An important benefit of the massive introduction of remote work during the pandemic is the reduction of greenhouse gas emissions, directly related to the so-called digital sustainability. With fewer commuters, car use and demand for public transport have decreased. This means less use of natural resources such as oil and coal, which contribute greatly to the greenhouse effect [25]. It is estimated that the mere reduction of commuting resulting from remote work could reduce CO₂ emissions from land transport by approximately 8.5% per year [24]. In addition, due to remote work, offices and other facilities were able to operate at reduced power consumption, instead of fully functioning during working hours [25]. This is due to the fact that a more extensive and efficient infrastructure is used in corporate premises (e.g., lighting systems, professional printers, photocopiers, computers with large monitors, power failure protection systems, etc.). In addition, workstations are reorganized, for example by moving employees to vacant rooms. This allows for more optimal management of housing resources, and thus reduces the energy used for lighting, ventilation, air conditioning, heating systems, etc. These and other activities related to the pandemic and the introduction of remote work resulted in a further reduction of CO₂, NO₂ and other gas emissions, and thus improved air quality [26,27]. Therefore, it can be said that remote work affects digital sustainability. Although in the case of digitization of work, environmental issues may seem negligible, the introduction of remote work is associated with reducing pollutant emissions from transport and reducing energy consumption, which is environmentally friendly and has a positive impact on sustainable development.

In the context of introducing and continuing the digital “work from home” model, the availability of software that ensures efficient communication and performance of professional responsibilities via the ICT systems is very important. Among this type of software, teleconferencing systems such as Microsoft Teams, Zoom, Google Meet (formerly Hangout) and Skype [28,29] play the most significant role. These systems differ both in the range of functions offered, as well as in the costs of use. In addition, many users may have different feelings about using these systems.

Solving the problem of selecting an ICT system for the needs and conditions of a given organization has become the main motivator of the conducted research. Proposing an appropriate methodological solution that would facilitate the assessment of existing systems supporting remote work could be a valuable contribution to improving the process of selecting these systems. The methodological approach proposed by the authors fills a research gap. It is related to the integration of several important factors regarding the procedure for making decisions about the implementation of the system. One of them is support for group decision-making and reaching consensus among decision-makers. Another concerns the possibility of expressing the assessment not only numerically, but also using uncertain qualitative data. This type of data, presented using a linguistic scale, may turn out to be more effective than quantitative data with many evaluation criteria.

Multi-Criteria Decision Analysis (MCDA) and related methods deal with the issues of evaluation and support for the selection of considered variants. There are many of these methods and they can be grouped according to various criteria. One of them divides methods into those based on the outranking relation and on the utility function. The most popular methods from the first group are: ELECTRE methods [30], PROMETHEE methods [31], NAIADE [32], MELCHIOR [33], ORESTE [34], REGIME [35] and TACTIC [36]. The second group includes such methods as: AHP [37,38,39], ANP [40,41], DEMATEL [42], MAUT, REMBRANDT, MACBETH [43], SMART [44]. There are also many variants of multi-criteria methods (e.g., fuzzy), existing methods are integrated with each other and new methodological solutions appear, e.g., PVM [45,46], D-FTOPSIS [47], TROOIL [48], q-ROFDOSM–q-ROFWZIC [49], FAHP–TOPSIS, FAHP–VIKOR, FAHP–ELECTRE, FAHP–PROMTHEE [50].

PROSA GDSS is also a method of this type [51]. It is an MCDA method designed to solve decision problems by a group of experts. PROSA GDSS at the stage of individual assessment allows to apply, e.g., the NEAT F-PROMETHEE fuzzy method [52,53], thanks to which the uncertainty of assessments can be taken into account. In addition, the GAIA procedure was developed for the PROSA GDSS method, allowing for a graphical presentation and analysis of the preferences of individual decision-makers and the relationships between them.

The aim of the article is to analyse, compare and evaluate teleconferencing systems used in remote work. These activities are to enable the identification of a system that can be considered best for business in the context of remote work and the pursuit of digital sustainability. The scientific contribution includes the combination and joint application of the PROSA GDSS and NEAT F-PROMETHEE methods, and in particular the use of the NEAT F-PROMETHEE method as part of the PROSA GDSS procedure at the stage of individual expert assessments. This is a novelty of this article, because so far the possibility of such a combination of these two MCDA methods has been signalled in the literature, but never used in practice [51]. Meanwhile, such a combination of methods makes it possible to capture both the uncertainty and imprecision of assessments, experts’ preferences and criteria weights, as well as taking into account the consensus and agreement of experts’ views in the group assessment. The NEAT F-PROMETHEE method is responsible for taking into account all uncertainties, while PROSA GDSS allows the consensus of decision-makers’ preferences to be rewarded, conflicts of preferences to be punished, and outliers to be depreciated.

Section 2 provides an overview of teleconferencing systems supporting remote work and criteria for evaluating such systems. Section 3 contains a discussion of the research approach and methods used. Section 4 presents the results of research on the assessment of individual systems supporting remote work. Section 5 discusses the obtained results. The last Section 6 contains conclusions from the conducted research.

2. Literature Review

The COVID-19 pandemic has revolutionized the way we work. Thanks to the development and adaptation to teleconferencing platforms, it has become possible to perform many daily professional duties without leaving home [54,55,56]. Teleconferencing systems adapted to remote work should be characterized by appropriate functionalities, making these systems useful for the exchange of information and communication, both synchronous and asynchronous.

The most important functionality of teleconferencing systems used in remote work is to ensure synchronous communication with other users. Such systems allow for conducting employee videoconferences and business meetings. Teleconferencing platforms usually also provide functions for screen sharing, meeting recording, creating virtual rooms, file sharing, scheduling meetings in the calendar, exchanging text messages in the chat, etc. [57,58].

In situations where it is required to exchange files between users (especially large files), tools for storing and managing data in the cloud are used (e.g., Google Drive, OneDrive, Dropbox). Such solutions are safer than e-mail (multi-factor authentication), provide a large disk space and allow to improve teamwork [59]. Another aspect of remote work is team design and content creation, both textual and graphic, as well as various types of analytical and presentation studies. Popular tools include Google Docs, Microsoft 365, Visme [60,61].

Working in teams, especially dispersed ones, may also require tools to improve the management of groups working on joint projects. Dedicated tools for this purpose make it possible to increase productivity and reduce the inefficiency of remote teams [62]. Examples of software in this category include Slack, GanttPRO, Trello, ProProf Project.

Table 1. Selected parameters of teleconference platforms supporting remote work.

Parameters	Zoom	Microsoft Teams	Google Meet	Skype
Meeting Time Limits (Max time)	F: 40 min P: 30 h	F: 60 min P: 30 h	F: 60 min P: 24 h	F: 24 h
Most advanced Paid Plan	Enterprise	Microsoft 365 Business Standard	Business Plus	Microsoft Teams
Number of Participants	F: 100, P: 500	F: 100, P: 300	F: 100, P: 500	F: 100
Record Meetings	Users can record meeting	F: Options are limited	F: Options are limited	A
Cloud File Storage	P: Unlimited storage	1 TB per organization plus 10 GB per license	P: 5 TB per user	300 MB, OneDrive
Screen sharing	A	A	A	A
Live captions	A	A	A	A
Raise hand	A	A	A	A
Virtual Background	A	A	A	A
Breakout rooms	P: A	P: A	P: A	N
Calendar integration	A	A	A	A
Video Quality	F: 720p, P: 1080p	F: 1080p, P: 1080p	F: 720p, P: 720p	1080p
Mobile application	A	A	A	A
Computer application	A	A	A	A
Join from browser	A	A	A	A
Track user engagement	A	A	A	N
Whiteboard	A	A	A (Google Jamboard)	N
Meeting attendance report	A	A	A	N
Private Chat	A	A	N	A
Co-Annotation on Screen	A	N	N	N
LMS (Learning Management System) integration	A	A	Pilot program	A
Remote control of another computer while talking	N	A	A (Chrome Remote Desktop)	N
Systems Compatibility	MacOS, Windows, iOS, Android, Linux	MacOS, Windows, iOS, Android, Linux	MacOS, Windows, iOS, Android, Linux	MacOS, Windows, iOS, Android, Linux
Safety and security	Encryption, passwords, waiting rooms	Advanced security and privacy measures	Encryption and security features	Encryption of all features
Price (per user for a month)	F: A, P: $19.99	F: A, P: $12.50	F: A, P: $18	Free

Abbreviations: F—Free Plan, P—Most advanced Paid Plan, A—Available, N—Not Available.

presents selected parameters of the most popular teleconferencing platforms supporting remote work.

Taking into account the variety of teleconferencing systems and the functionality they offer, the selection of the appropriate platform supporting remote work should be carried out individually by each organization. However, based on the analysis of the literature, it is possible to determine a certain group of features and functionalities that should be met by the systems used. Thus, on the basis of expert knowledge, an initial selection of available tools can be carried out, thus supporting the choice of the system by organizations working remotely. The assessment of the considered platforms is made taking into account the criteria, the scope and number which may vary depending on the specific decision-making situation. They are usually adopted in the amount from a few to even a dozen, often grouped into thematic categories.

Table 2. Evaluation criteria for tools supporting remote work.

Criteria	References
Number of participants allowed	[29,63]
Video feeds	[29,64,65,66,67]
Application Integrations	[29,67,68,69]
Effective communication	[64,66]
Special functions and features	[63,64,66,67,68,69]
Whiteboard	[65,67]
Chat support	[64,65,67,69,70,71,72]
Malware attacks/Face recognition attacks	[29,63]
Confidentiality of personal data	[29,65,66,67,68,69,70]
Security	[29,63,65,66,67,68,69,70,73,74]
Support System	[29,64,67,70,73,74]
Multiplatform/Mobile application	[29,67,69,73,75]
User interface	[66,67,68]
Ease of use	[29,63,64,66,67,69,70,74]
Portability/Flexibility	[64,70]
Quality of video/audio	[29,63,68,69,76]
Video recording capability	[29,63,64,65,67,68,69,73]
Performance	[29,64,73]
Compatibility	[63,65,67,73]
File, content and Screen Sharing	[63,64,65,66,67,69,71,72,73]
Hardware/Network requirements/Bandwidth consumption	[63,64,65,66,67,68,75]
Communication delay	[71,75,76]
Scalability	[67,68]
Calendaring	[67,72,75]
Cost of software/service	[29,63,66,67,68]
Cost of equipment	[29,63,68]

presents selected (based on the literature review) criteria for evaluating tools supporting remote work. These are the most frequently repeated criteria, but also those that have been recognized by the authors of the research as important due to the specificity of the analysed situation.

Based on the review of the research and the specified criteria for evaluating systems supporting remote work, it can be seen that their number is significant and they do not constitute a closed catalogue. Some of them are very detailed (e.g., face recognition attacks, calendaring), while others are formulated in a more general way (e.g., security, ease of use). However, they can be grouped into several thematic categories:

• Functionality (number of participants allowed, file, content and screen sharing, application integrations, etc.);
• Usability (ease of use, user interface, etc.);
• Effective communication (chat support, video and audio feeds, etc.);
• Multi-platform (multiplatform, mobile application, etc.);
• Security and Privacy (malware attacks, face recognition attacks, confidentiality of personal data, security, etc.);
• Technical performance (hardware requirements, network requirements, bandwidth consumption, communication delay, scalability, performance, quality of video and audio, etc.);
• Support (support system, help desk, etc.);
• Pricing (cost of software, cost of service, cost of equipment, etc.).

The presented criteria in many cases may be perceived by experts subjectively and immeasurable. For example, an expert familiar with a given teleconferencing platform may find it easier to use than the one he has not dealt with before. In addition, the ease of use cannot be measured, it can only be assessed qualitatively, of course, taking into account a certain margin of error and uncertainty. Therefore, in the assessment of teleconferencing systems supporting remote work, two aspects are important.

• Capturing multi-expert ratings, aggregating those ratings, and exploring consensus among experts.
• The ability to capture qualitative assessments, quantifiable and uncertain values.

The above requirements enforce the use of an appropriate multi-criteria assessment method. These requirements are met by combining the NEAT F-PROMETHEE and PROSA GDSS methods. The NEAT F-PROMETHEE method captures both quantitative and qualitative criteria expressed linguistically and of an uncertain nature. The PROSA GDSS method, on the other hand, makes it possible to take into account and aggregate the assessments of many experts, to study the preferences of decision-makers and the relationships between them, and to reward consensus and punish non-compliance of expert assessments and outliers. Therefore, a combination of the given MCDA methods was used in further studies.

3. Materials and Methods

3.1. NEAT F-PROMETHEE

The NEAT F-PROMETHEE method was first formulated in publication [77] and developed in subsequent studies [52,53,78,79]. It uses fuzzy arithmetic and thus is able to capture qualitative assessments expressed linguistically and uncertain values. NEAT F-PROMETHEE uses trapezoidal fuzzy numbers (TFNs) in the form of $\tilde{x}=(x_1,x_2,x_3,x_4)$. Linguistic values allow you to determine the weightings of the criteria and the performance of the alternatives. The following criteria weights can be used: Very Low—VL = (0, 0, 0.1, 0.2); Low—L = (0.1, 0.2, 0.2, 0.3); Medium Low—ML = (0.2, 0.3, 0.4, 0.5); Medium—M = (0.4, 0.5, 0.5, 0.6); Medium High—MH = (0.5, 0.6, 0.7, 0.8); High—H = (0.7, 0.8, 0.8, 0.9); Very High—VH = (0.8, 0.9, 1, 1). The following linguistic values can be used to qualitatively evaluate alternatives: Very Poor—VP = (0, 0, 1, 2); Poor—P = (1, 2, 2, 3); Medium Poor—MP = (2, 3, 4, 5); Fair—F = (4, 5, 5, 6); Medium Good—MG = (5, 6, 7, 8); Good—G = (7, 8, 8, 9); Very Good—VG = (8, 9, 10, 10). The NEAT F-PROMETHEE method considers a discrete set of $M$ fuzzy alternatives $\tilde{A}=\{\tilde{a},\tilde{b},\dots ,\tilde{m}\}$ described by criteria belonging to the set $C=\left\{c_1,c_2,\dots ,c_n\right\}$.

First, the fuzzy deviation $\tilde{d}$ between each pair of alternatives $\tilde{a},\tilde{b}$ is determined for each criterion $c_j$. Equation (1) is used for this purpose:

$$\widetilde{d_j}(\tilde{a},\tilde{b})=c_j(\tilde{a})\ominus c_j(\tilde{b})$$

(1)

Then the value $\widetilde{d_j}(\tilde{a},\tilde{b})$ is mapped using the preference function $f_j$ (2):

$$P_j(\widetilde{d_j})=(P_j(d_{j1}),P_j(d_{j2}),P_j(d_{j3}),P_j(d_{j4}))=f_j\left[\widetilde{d_j}(\tilde{a},\tilde{b})\right]$$

(2)

Six different preference functions can be used in the PROMETHEE methods, and some of them use additional thresholds: indifference ($q_j$), preference ($p_j$) and gaussian ($s_j$). In addition, the NEAT F-PROMETHEE method applies an approximation error correction when mapping the values $\widetilde{d_{\mathrm{j}}}(\tilde{a},\tilde{b})$. The correction is described by the Equation (3):

$$\left\{\begin{array}{c}{P}_j(d_{j2})=0 if \frac{u-d_{j1}}{d_{j2}-d_{j1}}>0.5 \mathrm{f}\mathrm{o}\mathrm{r} d_{j1}<u\le d_{j2}\\ P_j(d_{j3})=1 if \frac{v-d_{j4}}{d_{j3}-d_{j4}}>0.5 \mathrm{f}\mathrm{o}\mathrm{r} d_{j3}\le v<d_{j4}\end{array}\right.$$

(3)

where $u,v$ are numerical coefficients whose value depends on the preference function used. The values of $P_j(\widetilde{d_j})$ are aggregated into single criterion fuzzy net flow $\widetilde{{\varphi }_j}$ (4):

$$\widetilde{{\varphi }_j}(\tilde{a})=\frac{1}{M-1}\sum _{i=1}^M\left[P_j(\tilde{a},\widetilde{x_i})-P_j(\widetilde{x_i},\tilde{a})\right]$$

(4)

Based on the value $\widetilde{{\varphi }_j}$ fuzzy net flows are calculated expressing the overall efficiency of individual alternatives (5):

$$\widetilde{{\varphi }_{net}}(\tilde{a})=\sum _{j=1}^n\widetilde{{\varphi }_j}(\tilde{a})w_j$$

(5)

where $w_j$ is the defuzzified weight of the j-th criterion. The defuzzification of the weights is carried out based on the Equation (6):

$$w_j(\widetilde{w_j})=\frac{{(w_3)}^2+{(w_4)}^2+w_3{w}_4-{(w_1)}^2-{(w_2)}^2-w_1{w}_2}{3(w_3+w_4-w_1-w_2)}$$

(6)

After defuzzification, the weights are normalized $(\sum _{j=1}^n{w}_j=1)$. Finally, the fuzzy net flows are defuzzified (7):

$${\varphi }_{net}(\tilde{a})=\frac{{{\varphi }_{net}(\tilde{a})}_3^2+{{\varphi }_{net}(\tilde{a})}_4^2+{{\varphi }_{net}(\tilde{a})}_3{{\varphi }_{net}(\tilde{a})}_4-{{\varphi }_{net}(\tilde{a})}_1^2-{{\varphi }_{net}(\tilde{a})}_2^2-{{\varphi }_{net}(\tilde{a})}_1{{\varphi }_{net}(\tilde{a})}_2}{3({{\varphi }_{net}(\tilde{a})}_3+{{\varphi }_{net}(\tilde{a})}_4-{{\varphi }_{net}(\tilde{a})}_1-{{\varphi }_{net}(\tilde{a})}_2)}$$

(7)

3.2. PROSA GDSS

The PROSA GDSS method is an extension of the PROSA method [80,81], intended for group decision support. The PROSA GDSS method considers a set of $M$ alternatives $A=\left\{a,b,\dots ,m\right\}$ described by a set of sequences $R=\left\{{Rdm}_1,R{dm}_2,\dots ,R{dm}_K\right\}$ representing the results of the evaluation of alternatives by each of $K$ decision-makers ($dm$). Each k-th sequence is ${Rdm}_k=\left\{{\varsigma }^k(a),{\varsigma }^k(b),\dots ,{\varsigma }^k(m)\right\}$ [51,82]. If PROSA GDSS is to aggregate the results of the assessment performed using the NEAT F-PROMETHEE method, then each sequence will be of the form ${Rdm}_k=\left\{{{\varphi }_{net}}^k(\tilde{a}),{{\varphi }_{net}}^k(\tilde{b}),\dots ,{{\varphi }_{net}}^k(\tilde{m})\right\}$.

The first steps of the PROSA GDSS method are similar to the NEAT F-PROMETHEE method, with the difference that crisp numbers are considered in PROSA GDSS, and not TFNs. In addition, the calculations are performed for the k-th decision-maker, not the j-th criterion. First, the deviation between the alternatives evaluated by the k-th decision-maker (8) is calculated:

$$d_k(a,b)={\varsigma }^k(a)-{\varsigma }^k(b)$$

(8)

Then, the preference relations between the alternatives are calculated (9):

$$P_k(d_k)=\left\{\begin{array}{l}0, \hspace{1em}\mathrm{f}\mathrm{o}\mathrm{r} d_k(a,b)\le 0\\ \frac{d_k(a,b)}{2}, \mathrm{f}\mathrm{o}\mathrm{r} 0<d_k(a,b)\le 2\end{array}\right.$$

(9)

On the basis of the preference relationship, the single criterion net flows of individual decision-makers are determined, according to the Equation (10):

$${\varphi }_k(a)=\frac{1}{M-1}{\sum }_{i=1}^M\left[P_k(a,x_i)-P_k(x_i,a)\right]$$

(10)

The aggregated net flows of the alternatives are then calculated (11):

$${\varphi }_{net}(a)=\sum _{k=1}^K{\varphi }_k(a){\omega }_k$$

(11)

where ${\omega }_k$ means the importance of the k-th decision-maker (weights should add up to 1). Comparing the values of ${\varphi }_k(a)$ and ${\varphi }_{net}(a)$ allows to examine the balance (compensation) between the assessments of individual decision-makers. When ${\varphi }_k(a)\approx {\varphi }_{net}(a)$, then the evaluation of the alternative $a$ by the k-th decision-maker is balanced against the other decision-makers. When ${\varphi }_k(a)\gg \ll {\varphi }_{net}(a)$ (${\varphi }_k(a)\ll {\varphi }_{net}(a)$ or ${\varphi }_k(a)\gg {\varphi }_{net}(a)$), then the evaluation of the alternative $a$ by the k-th decision-maker is unbalanced relative to the other decision-makers. The balance analysis may affect the selection of the value of the compensation factors $s_k\in \left[\mathrm{0,1}\right]$, used in the next step of the method. A higher value of the $s_k$ coefficient will increase the influence of the balance on the obtained solution, and a lower value of this coefficient will reduce the importance of the balance. This is presented by Equations (12) and (13) defining, respectively: individual absolute deviation and individual PROSA net sustainable value:

$${AD}_k(a)=\left|{\varphi }_{net}(a)-{\varphi }_k(a)\right|s_k$$

(12)

$${PSV}_k(a)={\varphi }_k(a)-{AD}_k(a)$$

(13)

The PROSA GDSS procedure ends with the calculation of the global PROSA net sustainable value in accordance with the Equation (14):

$${PSV}_{net}(a)=\sum _{k=1}^K{PSV}_k(a){\omega }_k$$

(14)

3.3. GAIA

An important element of the PROSA GDSS method is the graphical analysis of decision-makers’ consensus using the GAIA plane. GAIA is based on the PSV performance matrix (15), where the i-th alternative is represented by the row ${\alpha }_i$, which corresponds to the point $A_i$ in the space ${\mathbb{R}}^K$. The coordinates of the ${\mathbb{R}}^K$ space are the rows ${\alpha }_i$.

$$PSV=\left(\begin{array}{cc}\begin{array}{cc}{PSV}_1(a)& {PSV}_2(a)\\ {PSV}_1(b)& {PSV}_2(b)\end{array}& \begin{array}{cc}\cdots & {PSV}_K(a)\\ \cdots & {PSV}_K(b)\end{array}\\ \begin{array}{cc}⋮& ⋮\\ {PSV}_1(m)& {PSV}_2(m)\end{array}& \begin{array}{cc}\ddots & ⋮\\ \cdots & {PSV}_K(m)\end{array}\end{array}\right)=\left(\begin{array}{c}\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\\ \begin{array}{c}⋮\\ {\alpha }_M\end{array}\end{array}\right)$$

(15)

In order to project the space ${\mathbb{R}}^K$ onto the plane ${\mathbb{R}}^2$, the variance-covariance matrix $C$ is calculated according to the Equation (16):

$$tC={PSV}^T·PSV$$

(16)

On the basis of the $C$ matrix, a set of eigenvalues $\lambda =\left\{{\lambda }_1,\dots ,{\lambda }_n\right\}$ is determined. Two largest values from the $\lambda$ set correspond to the eigenvectors $u\perp v$ forming the plane ${\mathbb{R}}^2$ and defining the coordinates of each point $A_i=(u_i,v_i)$, according to the Equation (17):

$$\left\{\begin{array}{c}{u}_i={\alpha }_i·u\\ v_i={\alpha }_i·v\end{array}\right.$$

(17)

Vectors defining the preferences of decision-makers are defined using the Equation (18):

$$\left\{\begin{array}{c}{u}_k=e_k·u\\ v_k=e_k·v\end{array}\right.$$

(18)

where $e_k$ is the k-th row of the identity matrix of the $K\times K$ size. The solution constituting a compromise between the preferences of decision-makers is calculated according to the Equation (19):

$$\left\{\begin{array}{c}{u}^{\pi }=\omega ·u\\ v^{\pi }=\omega ·v\end{array}\right.$$

(19)

By analysing the GAIA plane, one can obtain information on the preferences of decision-makers, the compatibility or lack of compliance of their preferences and the solution constituting a compromise between their views.

4. Result

The most popular teleconference systems supporting remote work were selected for the study: A1—Zoom, A2—Microsoft Teams, A3—Google Meet, A4—Skype. The versions of the systems in the most advanced payment plan, while offering the greatest functionality, were evaluated. The systems were evaluated by experts with extensive work experience in each of the evaluated systems. These were academic and administrative employees who had to familiarize themselves with the tested systems during the pandemic, and the nature of their work and working in many places forced the simultaneous use of four systems included in the study. Since it is difficult to find remote workers who have used all four of the surveyed systems, we limited the number of experts to four. However, such a number is completely sufficient for a correct group assessment. Saaty and Özdemir [83] note that the participation of one experienced and well-versed expert in a given field may be sufficient in the assessment problem, and the addition of other experts may even deteriorate the accuracy and consistency of the assessment. Therefore, the number of experts should be limited. Moreover, according to Saaty [84], in order to maintain the coherence of the assessment model, none of its components may consist of more than 7–8 elements. The decision-making model we propose is a kind of structure in which we used 4 experts, 8 criteria and 4 decision-making alternatives. Therefore, the developed decision model, including the number of experts, meets the requirements regarding the number of elements. Individual expert assessments, expressed using linguistic scales (except for the payment plan), are presented in

Table 3. System evaluations and criteria weights according to the DM1 expert.

Criteria	Weight	A1	A2	A3	A4
C1—Functionality	VH	F	G	G	MP
C2—Usability	MH	F	G	MG	F
C3—Effective communication	H	MG	MG	MG	MP
C4—Multi-platform	M	F	G	MG	F
C5—Security and privacy	MH	MP	MG	G	MG
C6—Technical performance	M	MG	MP	MG	MG
C7—Support	L	MG	MG	F	MG
C8—Pricing [$per user for a month]	MH	19.99	12.50	18	0

Abbreviations: L—Low, M—Medium, MH—Medium High, H—High, VH—Very High, MP—Medium Poor, F—Fair, MG—Medium Good, G—Good.

,

Table 4. System evaluations and criteria weights according to the DM2 expert.

Criteria	Weight	A1	A2	A3	A4
C1—Functionality	VH	G	VG	MG	VP
C2—Usability	VH	F	G	MG	F
C3—Effective communication	H	MG	G	MG	MP
C4—Multi-platform	M	MG	VG	MG	P
C5—Security and privacy	H	P	MG	MG	MP
C6—Technical performance	MH	G	F	G	P
C7—Support	L	F	MG	F	P
C8—Pricing [$per user for a month]	M	19.99	12.50	18	0

Abbreviations: L—Low, M—Medium, MH—Medium High, H—High, VH—Very High, VP—Very Poor, P—Poor, MP—Medium Poor, F—Fair, MG—Medium Good, G—Good, VG—Very Good.

,

Table 5. System evaluations and criteria weights according to the DM3 expert.

Criteria	Weight	A1	A2	A3	A4
C1—Functionality	VH	F	G	F	MG
C2—Usability	MH	F	F	F	F
C3—Effective communication	VH	F	F	F	MG
C4—Multi-platform	MH	MP	G	F	G
C5—Security and privacy	H	F	P	F	MP
C6—Technical performance	H	F	P	F	G
C7—Support	M	F	F	F	VP
C8—Pricing [$per user for a month]	VH	19.99	12.50	18	0

Abbreviations: M—Medium, MH—Medium High, H—High, VH—Very High, VP—Very Poor, P—Poor, MP—Medium Poor, F—Fair, MG—Medium Good, G—Good.

and

Table 6. System evaluations and criteria weights according to the DM4 expert.

Criteria	Weight	A1	A2	A3	A4
C1—Functionality	VH	MG	VG	G	MP
C2—Usability	MH	G	G	MG	F
C3—Effective communication	H	MG	MG	MG	F
C4—Multi-platform	M	F	G	G	F
C5—Security and privacy	H	MP	G	G	G
C6—Technical performance	M	MG	F	MG	MG
C7—Support	ML	MG	MG	MG	G
C8—Pricing [$per user for a month]	MH	19.99	12.50	18	0

Abbreviations: ML—Medium Low, M—Medium, MH—Medium High, H—High, VH—Very High, MP—Medium Poor, F—Fair, MG—Medium Good, G—Good, VG—Very Good.

.

In the first stage of the study, expert assessments were aggregated using the NEAT F-PROMETHEE method, obtaining global assessments of alternatives separately for each expert. The NEAT F-PROMETHEE method uses a preference function in the form of a V-shaped criterion with a preference threshold of $p=2$ for linguistically expressed qualitative criteria and $p=10\$$ for the ‘Pricing’ criterion. The rankings of alternatives obtained by individual experts are presented in

Table 7. Rankings of alternatives by DM1-DM4 experts.

Alternative	DM1		DM2		DM3		DM4
	${\mathit{\varphi }}_{\mathit{n}\mathit{e}\mathit{t}}^1(\tilde{\mathit{a}})$	Rank	${\mathit{\varphi }}_{\mathit{n}\mathit{e}\mathit{t}}^2(\tilde{\mathit{a}})$	Rank	${\mathit{\varphi }}_{\mathit{n}\mathit{e}\mathit{t}}^3(\tilde{\mathit{a}})$	Rank	${\mathit{\varphi }}_{\mathit{n}\mathit{e}\mathit{t}}^4(\tilde{\mathit{a}})$	Rank
A1	−0.3062	4	−0.1143	3	−0.1852	4	−0.2356	4
A2	0.2829	1	0.4928	1	−0.0167	2	0.2705	1
A3	0.2219	2	0.1463	2	−0.1140	3	0.1182	2
A4	−0.1967	3	−0.5210	4	0.3197	1	−0.1519	3

.

In the next stage of the study, individual rankings were aggregated into a group ranking, presented in

Table 8. Group expert ranking of teleconferencing systems.

	A1	A2	A3	A4
${PSV}_{net}(a)$	−0.1809	0.0797	−0.0075	−0.2444
Rank	3	1	2	4

. Aggregation was carried out using the PROSA GDSS method, assigning equal importance to experts ${\omega }_k$ and using the compensation factor $s_k=1.$ The obtained results are presented in Table 8.

The group ranking shows that according to experts, the A2 system—Microsoft Teams is rated the highest. The second place is occupied by A3—Google Meet, followed by A1—Zoom and A4—Skype, respectively. Therefore, the two highest positions are occupied by the most functionally rich teleconferencing systems, issued by two leading corporations dealing with software development and the Internet. In terms of the final group evaluation, these systems definitely outperform the other teleconferencing tools under consideration. Taking into account the consistency of the group ranking with the individual rankings, it should be pointed out that the same order of alternatives appeared in the DM2 expert ranking. Analysing all individual rankings, it is easy to see that the order of alternatives $A2\succ A3\succ A1$ is maintained in them, while the rank of the A4 alternative changes, which is the winner in the DM3 expert ranking, third in the DM1 and DM4 expert rankings, and in the DM2 decision-maker ranking and in the group ranking takes the last place. Thus, some discrepancies in the preferences of decision-makers can be noticed.

5. Discussion

5.1. Consensus among Experts

Observations related to the divergent preferences of decision-makers are confirmed by the analysis of the GAIA plane, presented in Figure 1.

The DM1–DM4 expert vectors shown in Figure 1 indicate that the preferences of DM1 and DM4 experts are mutually compatible. The analysis of the GAIA plane also confirms the earlier observation about the consistency of the DM2 expert’s ranking with the group ranking (vector π on the GAIA plane). In opposition to these rankings and preferences is the DM3 expert, whose preferences are fundamentally different from other experts. The indicated conflict of preferences of the DM3 expert with other experts is reflected in the group ranking, in which the influence of the PROSA method on the obtained results is visible. Based on the individual rankings, it may seem that in the group ranking, the alternative A4—Skype should take a higher position than the alternative A1—Zoom, because in the case of DM1, DM3 and DM4 expert rankings, A4 has a higher rank. However, during the aggregation of individual ratings in the PROSA method, a relatively large discrepancy in the ratings of the A4 alternative and the lack of expert consensus regarding the assessment of this alternative results in its penalization and lowering of the final assessment. The scale of the penalty for unbalanced alternatives is determined by the compensation coefficient $s_k$, which in this study assumed the value $s_k=1$. Therefore, in order to examine the influence of this factor on the obtained results, a sensitivity analysis was performed. During the analysis, the value of the $s_k$ coefficient was changed for the k-th expert in the range $s_k=[\mathrm{0,0.1},\dots ,2]$, leaving the values $s_k=1$ for other experts. The results of the analysis are shown in Figure 2.

Sensitivity analysis shows that the obtained solution is stable. The group ranking may change only in the case of a large decrease in the compensation factor for the DM3 expert (from $s_3=1$ to the value of approx. $s_3=0.1$). In this case, rank changes would occur between A1—Zoom and A4—Skype. In the case of other experts, the change in the value of the compensation coefficient did not change the ranks of individual alternatives. Additionally, the situation was examined when the compensation coefficient was changed linearly in the range [0, 2] for all experts at the same time. The result of this study is shown in Figure 3.

The results presented in Figure 3 are very similar to the results of individual changes in the compensation factor for the DM3 expert. These results differ in the values of the $s_k$ coefficient, at which the order of A1 and A4 alternatives changes. The difference is also noticeable in the ${PSV}_{net}$ values obtained by the different alternatives. However, the shape of the function graphs representing the results of the alternatives and the relationships between the individual graphs are almost the same as in the case of the graphs describing the results of the alternatives depending on changes in the $s_3$ coefficient. The low sensitivity of the solution to changes in the value of the compensation coefficient $s_k$ proves the high quality and correct construction of the group assessment model based on the PROSA GDSS method.

5.2. Sensitivity Analysis to Changes in Criteria Weights

Sensitivity analysis was also carried out for the weights of the criteria used by individual experts. This analysis made it possible to verify the robustness of the obtained group solution in the event of a change in the weight of a single criterion. Such a change may occur, for example, in the case of an expert’s doubts as to the applied linguistic weight. In the applied approach referred to in the literature as one-at-a-time sensitivity analysis [85], the weight of one criterion of a single expert in the range from VL to VH was changed at a time. The results of the sensitivity analysis are presented graphically in Figure 4.

The analysis of Figure 4 shows that the obtained group solution is stable. Changing the weight of any criterion in the VL-VH range does not change the order of alternatives in the group ranking. The closest to such changes is the case of criterion C3 for the DM1 expert (alternatives A1 and A4), criterion C1 for the DM3 expert (alternatives A2 and A3), and criterion C2 for the DM4 expert (alternatives A1 and A4). However, in these cases, the given pairs of alternatives are very close to each other with minimal criteria weights (VL), and additional research has shown that even with weights of 0, the order of the alternatives will not change. Since it is not possible to assign weights lower than 0, the obtained solution should also be considered stable for the indicated cases. The high robustness and stability of the obtained results with changes in the weights of the criteria proves the correctness of the construction of the evaluation model and the appropriate quality of the obtained ranking of alternatives.

5.3. Objective Validation

The solution validation in MCDA coincides with the principles of examining the credibility and stability of decision results [86]. The study of the stability and robustness of the solution presented in the previous sections was based on sensitivity analysis. However, apart from stability, the formal correctness of the obtained solution is also important, which can be verified using objective validation based on statistics. In the literature, it is proposed to use the mean and standard deviation as tools for validating a solution to a multi0criteria problem [87]. Objective validation consists in dividing the ranking of alternatives into n groups containing the same number of alternatives. Alternatives are assigned in the order resulting from the multi-criteria ranking. For each group, the mean and standard deviation are determined, and then the group ranking is generated on the basis of the mean taking into account the standard deviation [88]. This process was performed for the individual rankings presented in Table 7 and for the group ranking in Table 8. Since four alternatives were considered in the study, they were divided into two groups, assigning two alternatives to each group. In the case of individual expert rankings DM1, DM2, DM4 and group ranking, the first group consisted of alternatives A2 and A3, and the second group consisted of alternatives A1 and A4. As for the DM3 expert ranking, the first group included alternatives A4 and A2, and the second group—alternatives A3 and A1. The obtained mean and standard deviation values for each group are presented in

Table 9. Objective validation results for the DM1-DM4 expert rankings and the GDSS ranking.

Expert	Alternatives	Mean	Standard Deviation	Min	Max	Rank
DM1	G1 = {A2, A3}	0.2524	0.0431	0.2093	0.2955	1
	G2 = {A1, A4}	−0.2515	0.0774	−0.3289	−0.174	2
DM2	G1 = {A2, A3}	0.3196	0.2450	0.0745	0.5646	1
	G1 = {A1, A4}	−0.3177	0.2876	−0.6052	−0.0301	2
DM3	G1 = {A2, A4}	0.1515	0.2379	−0.0864	0.3894	1
	G1 = {A1, A3}	−0.1496	0.0503	−0.1999	−0.0993	2
DM4	G1 = {A2, A3}	0.1944	0.1077	0.0867	0.3020	1
	G2 = {A1, A4}	−0.1938	0.0592	−0.2529	−0.1346	2
GDSS	G1 = {A2, A3}	0.0361	0.0617	−0.0256	0.0978	1
	G2 = {A1, A4}	−0.2127	0.0449	−0.2576	−0.1677	2

. In addition, Figure 5 shows graphically the mean values and standard deviations.

The analysis of Table 9 and Figure 5 shows that the obtained solutions, both individual and group, are statistically reliable and formally correct. In each case, the G1 group has a higher mean value than the G2 group, also after subtracting the standard deviation (min G1) from the G1 mean and adding the standard deviation to the G2 mean (max G2). Among individual experts, from the formal side, the solutions proposed by DM1 and DM4 experts seem to be the best. In these solutions, the groups G1 and G2 are significantly distant from each other, which means that the alternatives that make up the G1 group are far from the alternatives that are part of the G2 group, so there is a strong preference between them. Unfortunately, the standard deviation for each group is very small, which means that the alternatives within each group have similar performance, so there is a weak preference between them. The solution proposed by the DM2 expert seems to be better in terms of the performance difference between the alternatives included in each group, as indicated by a large standard deviation. The problem is that although the distance between G1 and G2 is large, after taking into account the standard deviation, the maximum value of the G2 group is very close to the minimum value of the G1 group. This problem is even greater in the case of the DM3 expert solution, because after taking into account the standard deviation, the G1 and G2 groups are even closer to each other than in the case of the DM2 expert solution. The group solution (GDSS) is a kind of averaging of individual solutions, which causes its results to be “flattened”. This means that both the distances between groups G1 and G2, as well as the values of the standard deviation, are lower compared to individual solutions. However, as indicated earlier, all individual solutions as well as the group solution should be considered statistically credible and formally correct, because in no case the minimum value of the G1 group is lower than the maximum value of the G2 group.

6. Conclusions

The article deals with the problem of comparing and evaluating teleconferencing systems used in remote work. The practical goal was to create an expert ranking of teleconferencing platforms to support the selection of the optimal system by organizing remote work. The use of a combination of the NEAT F-PROMETHEE and PROSA GDSS methods in the group assessment was the methodological contribution and novelty in the article. The combination consisted in the use of the NEAT F-PROMETHEE method for individual assessment of systems by individual experts and the use of the PROSA method for aggregation of individual assessments into a group assessment. This combination allowed capturing qualitative, uncertain and imprecise expert assessments by the NEAT F-PROMETHEE method and the aggregation of many expert assessments and the examination of the consensus among experts in the PROSA GDSS method.

As a result of the evaluation carried out using the opinions of four experts, it was established that the most useful teleconferencing platform for remote work is Microsoft Teams. According to the experts participating in the study, it offers the greatest functionality, usability, multi-platform aspect, and in the opinion of most experts, the Teams software is also superior to others in terms of effective communication and support. Slightly worse assessments and the second place in the expert ranking were attributed to Google Meet software, which outperforms other systems in terms of security and privacy, and in the opinion of most experts also in terms of technical performance. The two mentioned teleconferencing systems definitely outweigh the Zoom and Skype software. In terms of technical performance, Zoom is almost equal to Google Meet software, and Skype, according to all experts, offers similar technical performance. However, in terms of other criteria, Zoom and Skype are no match for Microsoft Teams and Google Meet.

Future directions of research from the methodological side will include the development of the PROSA method towards fuzzy sets, so that the method can independently capture the uncertainty and imprecision of the input data, without the need to use methods such as NEAT F-PROMETHEE. On the other hand, further research on platforms supporting remote work will focus on a more comprehensive assessment by extending the set of assessment criteria used and the set of remote work systems examined. In the studies presented in this article, these were also the main research limitations, because the use of other criteria or even supplementing the criteria with several additional ones could affect the order of the systems in the ranking. Similarly, expanding the list of systems supporting remote work could affect the order of systems in the ranking.

This article, on the one hand, shows how to solve the problem of choosing the best platform supporting remote work (methodological aspect), and on the other hand, which solution is the best. The assessment criteria were selected based on a review of numerous world literature, and the assessment was carried out by field experts. Despite the subjective assessment, the results obtained may be a hint for many managers facing the issue of implementing solutions supporting remote work. Taking into account the analysis of the functionality of available tools and the proposed choice, you can improve the implementation process by focusing on a given solution. This, in turn, will facilitate the development of an effective usage policy (access, rules of use, privacy), technical support, data security, employee training or possible integration with other systems.