3.4.2. The Analysis of Topics of Special Interest

The last stage of our analysis was to examine, within the reviewed literature, the topics of special interest for the research on CI in policymaking. To ensure data triangulation, and to avoid duplicating regularities that were already detected, in the selection of topics we relied on a different method than the one used in the earlier stages of the work. When selecting specific topics for analysis, we relied on monographs concerning issues of collective intelligence and policymaking, published after 1990. The method of selecting topics for analysis is described in Appendix D. The final list of seven topics included: Citizenship, Communities, Consensus, Deliberation, Diversity, Local governance and Urban development, and Open data.

Next, we searched our literature database for the keywords specific to each of these topics. The topic-oriented subgroups of studies were created, based on the occurrence of the related keywords. The results are presented in Table 13.


**Table 13.** Saturation of the analyzed research studies with selected topics of interest.

The four most popular topic-oriented subgroups were analyzed in terms of the methods and strategies that were adopted in the conducted research. The aim was to verify to what extent the reviewed literature relates to the examined topics, and what research methods were used in the studies focused on these topics. The results of the analysis are shown in Figure 12. The source data are presented in Table A3 in Appendix B.

**Figure 12.** Method usage within the most influential studies compared to all the reviewed studies. The assignment of particular methods and strategies to the labels numbered from 1 to 15 as described in Table 2.

#### **4. Discussion**

The analyses conducted allowed us to conclude that throughout the whole sample the approaches that were most frequently used to study collective intelligence in the domain of policymaking were analysis of the organisational structure and analysis of the created values. Moreover, the analysis of the two most important research areas in which the studies were conducted revealed that the first of these methods is primarily peculiar to political science, and the latter is more common in computer science. Apart from this general observation, we were able to investigate a number of other issues related to the analyzed topic.

We observed that at least since 2015, the topic of CI in policymaking remains a subject of increasing interest among researchers. Although 2017 was the peak of interest, the subsequent years also demonstrated the continued popularity of this issue. Content analysis allowed for the identification of concepts that constituted the most important points of reference in the studies. The dominance of the term crowdsourcing, both in article titles and in author keywords, is noticeable. Due to the fact that this term in its original meaning mainly referred to business projects, we can see that many authors remain rooted to translating patterns developed in the commercial sector into the public sphere. This observation seems to be consistent with the analysis of research methods. The frequent use of analysis of the created values approach is also a common point with commercial projects, in which the direct results of collective effort are one of the primary subjects of interest. In turn, concepts such as the public and government frequently appearing in article abstracts, embedding the research in the political sciences domain. In addition, the KeyWords Plus analysis (based on the literature cited in the analyzed works) shows that the concepts that were most frequently referred to were innovation and participation. Note that the term innovation, in its business sense—being a multi-stage process whereby organisations transform ideas into new/improved products, service or processes [A1]—is

now increasingly used in social and political sciences to describe the process of reforming public organizations by opening them to participation [A2], which was also confirmed by our analysis.

Statistical analysis proved that some significant relationships between the research methods can be observed. The negative relationship between the analysis of created values and the analysis of collaboration model is particularly noteworthy. This can be explained by the fact that projects mainly oriented at generating new values are studied in the context of the existing governance framework. The studies on new models of intersectoral collaboration between public and private entities, when the scope of the project extends beyond the structure of one specific organization, require a different approach. The remaining relationships are fairly obvious: A common combination in the reviewed studies was to analyze the behavior and motivation of the participants at the same time. Similarly, it is not surprising that state-of-art-review and categorization of implemented projects were linked. The observed positive relationship between the analysis of created values and the 180 Day Usage Count also led to interesting observations. It can be concluded that the use of the analysis of created values method translates into increased popularity among readers. On the other hand, we can see that studies based on this method result in texts with fewer pages, which makes them more accessible to readers.

The analysis of research areas in which the studies were conducted points to the conclusion that the number and diversity of the scientific disciplines covered by the review is growing year by year. References to CI and policymaking appear in more and more specialized works related to the implementation of public policies. It shows that reflections on CI in policymaking have moved from general considerations to the application of solutions in specific domains of public policy. Secondly, the analysis of the number of studies appearing yearly in research area groups confirmed that researchers tend to be less interested in technological aspects of projects (the computer science and related group), and more in the implementation of these projects in diverse areas of administration, and in the public sphere (the political sciences and related group). As we have already emphasized, the patterns of analysis borrowed from business projects (i.e., created value analysis) were the leading methods of study in computer science. At the same time, the analysis conducted from an organizational perspective was characteristic of contemporary governance studies on CI. However, the low popularity of the analysis of the impact of AI algorithms approach was surprising. It seems that CI studies are still conducted almost entirely separately from AI studies. Despite the fact that the combination of AI and CI has been recently proposed as one of the most important topics of research, for example, in the report *Identifying Citizens' Needs by Combining AI and CI* [68] or in the works of G. Mulgan [69], it looks like this demand has not yet been answered. The relatively low popularity of the analysis of the impact on policymaking is also puzzling. It can be concluded that the practical function of CI in policymaking is often reduced to fitting CI projects into the existing administrative structure, or on increasing efficiency in achieving goals formulated at the political level, whereas actual shaping of public policy agendas is still rare. Nevertheless, the observed decline in the popularity of the analysis of organisational structure approach may herald some changes.

Research into created values is not the only approach that stands out in computer science. We also notice the popularity of studies on the e-participation processes, focused on engaging wide audiences in policymaking, which is promising in the context of future research. It is also interesting that in the political sciences, apart from research on the organizational structure, there is a significant interest in collaboration models. Reflecting on the cooperation of different types of partners, achieving mutual benefits seems to be a promising model for the future shape of policymaking.

A review of the most influential articles, taking into account both their use and citations, allowed their specific features to be captured. The innovation analysis was a particularly popular research approach in this group. Our observation may be an indication for future research that including the analysis of project innovativeness in the planned works may contribute to increased interest in research results. However, as in the other analyzed subgroups, the number of studies tracking the actual impact of CI projects on shaping public policies was still unexpectedly low. Conversely, the analysis of the eparticipation process enjoys increased popularity in this group, although only among the frequently read, though not among the most cited articles. We also noted that the articles relating to user behavior were underrepresented in this group.

Finally, the analysis of the selected topics of interest showed that the most popular concept in our sample was citizenship, and studies using this term were often associated with the method of analyzing the motivations of participants. This is in line with postulated changes in the relationship between citizens and the state, as proposed by Noveck [67] and others. The government is expected to transform from an authoritative problem-solving center into an arbiter, inviting the citizens to jointly seek the best solutions. Putting the citizens at the center of interest and studying their motivations enhances their role as active participants in the online public sphere. Another very popular concept in the analyzed sample was local governance. References to this topic could be found in over 34% of the reviewed studies. The analysis showed that cities, as well as communities (both local and based on interests), have become the main field of implementation of CI projects in the public space. In the case of cities, the organizational structure of projects was the main method of study, and in the case of communities, the values they produce were more important. It was also noted that topics with a deep theoretical foundation, such as diversity or consensus, were still not very popular among the analyzed works, which may be related to their relatively low applicability to the leading topics of citizenship and local governance.

#### **5. Conclusions**

Opening policymaking tasks to public participation has become one of the major trends in public policy in recent years. Regarding the 2030 Agenda for Sustainable Development, approved by United Nations Member States in 2015, "responsive, inclusive, participatory and representative decision-making at all levels" is one of the adopted strategic goals for the future [70]. The role of governments is substantially changing, and the emergence of new and complex social problems requires looking for new ways to collaborate in making public decisions with non-governmental actors, and with self-organized communities. For this reason, there is a need to constantly review the existing research on collective intelligence in the domains of public policy and the methods of studying this topic, which may contribute to the better planning of future implementations.

In the present study we made an attempt to identify which methods and strategies have been used so far for researching CI in policymaking. To answer Research Question 1, we conducted a systematic literature review following the PRISMA methodology, supplemented by an analysis of article titles, abstracts and keywords, the yearly number of publications, as well as qualitative research based on the grounded theory method. We identified 15 methods in the analyzed sample. The analysis of the organizational structure and analysis of the created values approaches proved to be the most frequently used approaches.

Considering Research Question 2, the analysis of statistical dependencies allowed us to identify several positive and negative correlations between research methods and between research methods and other variables (especially usage count, as well as the number of pages).

Considering Research Question 3, we found that studies were conducted mainly in computer sciences and political sciences, with the latter group, though initially less numerous, becoming dominant in recent years. We also identified which research methods were more common and which were less common in particular research areas.

Finally, considering Research Question 4, it is possible to conclude that the most influential, i.e., the most cited and the most popular articles, differed from typical studies in terms of the research methods used. A similar phenomenon occurred in relation to groups of articles built around topics of special importance.

The authors hope that by publishing this article they contributed to the systematization of knowledge about studies on collective intelligence in policymaking, showing in which areas the research has been conducted and which methods have been used for this purpose. In addition to identifying the most popular methods, we have attempted to identify the underrepresented approaches, which are promising for the future development of these studies. The present study differs significantly from the studies that were conducted in the past. None of the literature reviews on CI and public policymaking have so far developed a comprehensive list of analytical methods and approaches used in this type of research. For example, Prpi´c et al. presented the status of research focusing on three selected policy crowdsourcing techniques (virtual labor markets, tournament crowdsourcing, open collaboration), to compare them to the different stages of the policy cycle [37]; Liu et al. synthesized prior research and practices mainly to provide practical lessons for designing new projects in the public sector [52] and Linders focused on classifying citizen co-production initiatives [54]. As our review shows, some types of research have so far been extremely rare. For example, only one study in the analyzed sample concerned organizational learning, and yet, according to studies conducted by Mulgan [4] and Malone [71], it is one of the most important elements involved in collective intelligence. The state of research on the impact of CI in shaping public policy agendas, and on the use of AI algorithms in implemented projects also seems insufficient. We trust that by indicating the areas in which research is still limited, we will contribute to the better quality of future studies.

**Author Contributions:** Conceptualization, R.O.; methodology, R.O., S.B. and P.P.; validation, R.O. and M.C.; formal analysis, S.B. and P.P.; investigation, R.O. and M.C.; writing—original draft preparation, R.O.; writing—review and editing, R.O. and S.B.; visualization, S.B., P.P. and R.O.; supervision, R.O.; funding acquisition, R.O. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Narodowe Centrum Nauki (National Science Centre, Republic of Poland), the research grant UMO-2018/28/C/HS5/00543 – "Collective intelligence on the Internet: Applications in the public sphere, research methods and civic participation models" ("Kolektywna inteligencja w Internecie: zastosowania w sferze publicznej, metody badania i modele partycypacji obywatelskiej"). This research was funded from the funds granted to the Cracow University of Economics, within the framework of the POTENTIAL Program, project number 26/EIM/2021/POT.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
