**1. Introduction**

The phenomenon of collective intelligence (CI), which is understood as an ability of a particular collective to solve problems, mainly through gathering data, generating ideas and making decisions, has been the subject of interest of many scientific disciplines in recent years. The primary characteristic of a collective showing a high CI level is its capability to solve problems in which the difficulty exceeds the capacity of an individual. CI frequently manifests itself when cooperation, competition or mutual observation gives rise to totally new solutions to the problems or leads to an increase in the ability to solve them. Contemporary studies on CI, although clearly inspired by the development of the Internet in their origins, have so far been carried out in very diverse disciplines, from biology, through social sciences and organization management, to artificial intelligence.

Several empirical studies and theoretical simulations have proven that a collective can, under certain conditions, achieve better results in problem solving than a narrow group of experts [1–5]. To date, this phenomenon has been studied both as a feature of small groups, in which ties and interactions between participants are strong and the deliberation processes lead to informed intellectual outputs [6,7], and as a statistical phenomenon resulting from the aggregation of a vast number of dispersed opinions coming from incoherent crowds [8,9]. The most promising examples of recent projects in which a high level of CI was observed have combined humans and machines, organizations, and ICT networks [3]. The current empirical studies on CI are therefore largely focused on interactions between users in online communities. In parallel, theoretical work has been

**Citation:** Olszowski, R.; Pi ˛eta, P.; Baran, S.; Chmielowski, M. Organisational Structure and Created Values. Review of Methods of Studying Collective Intelligence in Policymaking. *Entropy* **2021**, *23*, 1391. https://doi.org/10.3390/e23111391

Academic Editors: Yaniv Altshuler, Francisco Camara Pereira and Eli David

Received: 5 August 2021 Accepted: 14 October 2021 Published: 24 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

carried out to simulate collective behavior with the use of computational methods. One of the most interesting is the approach called swarm intelligence (SI), which takes its inspiration from the biological examples provided by social insects such as ants, termites, bees and flocks of birds. In this model, self-organization takes place in decentralized communities in which the logical process is multi-threaded, chaotic and parallel; in which the threads intertwine and interlock; and in which the agents exhibit adaptive behavior, while also maximizing the number of diverse future paths among the possible choices. Simulations show the possible effectiveness of such a decision model, but its application to real social processes is not easy [10–12].

The domain of policymaking (i.e., formulating public policies), which used to be strictly limited to small groups of specialists, is now increasingly opening up to the participation of wide collectives, which are not only influencing government decisions, but also enhancing citizen engagement and transparency, improving service delivery and gathering the distributed wisdom of diverse participants [13–16]. National and local governments use CI methods in the policymaking processes, such as in legislative reforms [17,18], urban strategy planning [16], analyzing large amounts of social data to detect patterns and abnormalities [19,20], using dynamic models for learning, adaptation and forecasting of policy formulation [21,22], real-time continuous policy monitoring [15,23], as well as online public debates and consultations [24,25]. Opening policymaking tasks to public participation, fuelled by the theories of participatory democracy [26,27] and the concept of deliberative democracy [28], has found its practical expression in a paradigm shift towards collaborative governance [29,30], in which policy issues are addressed by networks of governmental and non-governmental actors. However, some models of CI, especially those that are characteristic of swarm intelligence, seem to be very difficult to reconcile with the common understanding of policymaking.

Although collective intelligence has become a more common approach to policymaking, the studies on this subject have not been conducted in a systematic way. The methods of studying the theoretical models, the successful case studies, the public sphere domains in which projects can be implemented, the expected results and the factors influencing CI vary greatly depending on the scientific discipline in which they are conducted. Moreover, different research traditions often use alternative terminologies to describe the same phenomena, an example of which is the competitive use of the labels "crowdsourcing" and "collective intelligence". Furthermore, there has been no scientific literature review regarding the phenomenon of CI in the field of policymaking. Research methods and strategies used in the studies conducted so far have not been systematized either. Nevertheless, we hypothesized that the methods and strategies specific to different types of CI studies in the field of policymaking can be identified and analyzed.

In order to better understand the present state of knowledge in this field, we raised the main research question (RQ1): what methods and strategies were specific to the studies on collective intelligence in policymaking during the last 10 years? What was the trend in the number of publications by year, and what were the most common concepts that appeared in the studies concerning CI in policymaking?

To supplement the knowledge about the methods and strategies we planned to identify, additional research questions were established:

RQ2: what statistical dependencies occurred between the identified research methods? What dependencies occurred between the research methods and other features of the analyzed studies?

RQ3: in which research areas were the studies conducted? What research methods and strategies were used in the specific research areas?

RQ4: what research methods and strategies were employed in the most influential works and in the topics of special importance for the study of CI in policymaking?

To answer these questions, we conducted a systematic literature review. On this basis, using the grounded theory method, we were able to categorize the identified approaches into a list of 15 methods and strategies and subsequently performed a series of analyses, described later in this article. With the use of statistical analyses, we revealed the dependencies between different study methods, as well as between study methods and other variables. Our cross-sectional analysis has produced interesting results, which may form the foundation for future projects.

## **2. Materials and Methods**

To answer the research questions posed, we divided the work into the tasks described below. In order to answer Research Question 1, we adopted the following work plan:


The method used in the first stage of our research was a systematic literature review. This literature review followed the Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) methodology [31]. This section clearly articulates guidelines regarding the inclusion or exclusion criteria of research papers to find relevant papers in our research area. We have also clearly mentioned how and to what extent the review was performed. The PRISMA flowchart for the research process is shown in Figure 1.

**Figure 1.** Flow diagram of the article-selection process.

When selecting keywords, alternative terms of CI used in the literature were taken into account, including "collective intelligence", "crowdsourcing", "swarm intelligence", "wisdom of crowds" and "crowdlaw". These concepts, although not fully identical, have an established position, and are used by researchers to describe similar phenomena, depending on the background of individual authors (the relationships and differences between these concepts were described by Buecheler [32]). The second set of keywords included concepts related to political science, administration and governance: "policymaking" (variants: "policy-making" and "policy making"), "public policy", "political science", "public administration", "public sector" and "public governance". The Web of Science was chosen from a number of pre-selected databases (other databases considered were Scopus, Sciencedirect and EBSCO) because of its reputation for the greatest coverage and the greatest impact in terms of most cited authors and articles, as well as for the most accurate subject classification. Search engines, such as Google Scholar, were excluded, as our priority was to select peer-reviewed publications. The timeframe for the search was set for the period from 2011 to 2020. The data search was conducted on March 8, 2020. We applied the logical search to the topic (including the abstract, keywords and indexed fields), as well as the titles of the scientific articles. The inclusion criteria were focused on peer-reviewed scientific articles dealing with issues in the field of public policymaking and combining them with methods, models and concepts derived from the CI research domain. In addition, we used the language filter to focus on the publications in English.

The logical search used the following syntax: TS = (("Collective Intelligence" OR "Crowdsourcing" OR "Swarm Intelligence" OR "Wisdom of crowds" OR "Crowdlaw") AND ("Policy Making" OR "Policy-making" OR "policymaking" OR "Public Policy" OR "Public Administration" OR "Political Science" OR "Public Sector" OR "Public Governance" OR "e-participation")) OR TI = (("Collective Intelligence" OR "Crowdsourcing" OR "Swarm Intelligence" OR "Wisdom of crowds" OR "Crowdlaw") AND ("Policy Making" OR "Policy-making" OR "policymaking" OR "Public Policy" OR "Public Administration" OR "Political Science" OR "Public Sector" OR "Public Governance" OR "e-participation")).

This search led to an initial total of 169 references, and after removing the duplicates, that number reached 167. Then, in accordance with the guidelines of H. Snyder [33], the content of all articles was screened in terms of checking the inclusion criteria, according to the title-abstract-references scheme, which allowed us to identify the content that did not meet the criteria described above and remove it from the database. To focus on highquality literature, we excluded the conference proceedings, editorial materials and reviews, and excluded articles written in a language other than English. Another 10 articles were excluded during the eligibility assessment due to the fact that they obviously did not concern the topic of review (e.g., their topic was tourism, citizen science initiatives, the student learning environment, etc.). This led to the refined list of 88 results. By creating the list as described above, it was possible to check how many articles were published annually and what the trends were in the number of publications per year.

The content of the articles was evaluated by our team of 3 experts, with experience and academic backgrounds in both policymaking and information technologies (2 experts with a PhD in political science and experience working on ICT projects, and 1 expert with an MA in IT and experience in working in social projects). The preliminary analysis was made by creating lists of the most common concepts that appeared in article titles, article abstracts, original keywords, as well as KeyWords Plus. The next stage, a qualitative research step, the purpose of which was to extract the methods and strategies of studying CI in policymaking from the analyzed texts, was based on the grounded theory approach. We applied this approach for extracting the theoretical value from the selected studies, grouping and presenting the key concepts, conceptualizing and articulating the concepts and distilling the categories from them. The analysis included stages that were specific to the grounded theory method: open coding, axial coding and selective coding. The open coding stage involved an analytical process of generating high-abstraction level type categories from sets of concepts. In this stage we focused on extracting keywords specific to the analyzed texts that appeared in titles and abstracts. The analysis of keywords allowed for a preliminary division of the texts into 11 subgroups, which became the initial categories. The next stage, i.e., axial coding, aimed to identify the key processes and the main research results described in the examined articles. We adopted an iterative method of working: texts were analyzed in groups of 10, using the existing categories, and then categories were redefined, combined or divided, and their definitions were developed. The emerging categories were grounded during the progressive analysis of subsequent

texts from our sample. Then, at the stage of selective coding, the categories were finally integrated and refined [34]. Theoretical saturation was achieved when, during the analysis of the following texts, no new concepts, properties or interesting links arose [35]. Based on the review of the references included in the analyzed texts and the relevant theoretical literature, we adopted the final definitions to describe the identified methods. As a result of the analysis described above, 1 to 5 methods or strategies were identified in each reviewed text, and the general list of 15 methods of studies on CI in the field of policymaking was proposed.

After completing the work described above, we attempted to answer the additional research questions. To answer RQ 2, the following tasks were planned:


This stage of our research was a series of statistical analyses. The first two tasks were based on the simple counting of averages and the visualization of trends. Then, to analyze the dependencies between research methods, we used Pearson's Chi-squared test of independence, and Yates's correction for continuity (Yates's Chi-squared test). Next, analyzing the dependencies between research methods and other features of the analyzed studies, we had to perform a Shapiro–Wilk test of normality for all continuous variables, the Chi squared of independence test, and statistical analysis based on Pearson's Chi-squared test of independence. Finally, we used the Fisher exact test of independence.

In order to answer Research Question 3, we planned the following tasks:


Based on the WoS Research Areas, we verified in which scientific disciplines the studies were conducted, and what was their number. For the further analytical purposes, we grouped the related scientific disciplines into collections, taking into account the special position of the computer sciences and political sciences. On this basis we tracked the yearly number of studies in each research area group and the most common methods and strategies in each research area.

Finally, to answer Research Question 4, the following tasks were planned:


To analyze the most influential studies, we ranked the top 10 articles based on usage and citation criteria, obtained from the Web of Science statistics. On this basis we tracked the most common methods and strategies in each research area. Then, building the ranking of topics of special importance, to ensure data triangulation and to avoid duplicating regularities already detected, in the selection of topics we relied on a different method than the one used in the earlier stages of this work. The monographic publications concerning the issues of collective intelligence and policymaking were shortlisted. Due to the scarcity of monographic literature, only 8 publications were included in this list after the review. On this basis, an initial list of 20 concepts was compiled. Subsequently, a survey was conducted in which a group of 6 social science researchers were invited to assess the significance of the proposed issues. Thus, the final list of 7 concepts that were subject to analysis was selected, and we searched our literature database for keywords specific to each of these concepts. The identified sub-groups of studies were analyzed in terms of the research methods and strategies that were adopted.

### **3. Results**

#### *3.1. Methods and Strategies of Studying CI in Policymaking*
