Submit to this Journal Review for this Journal Propose a Special Issue

Article Menu

Share Help Cite Discuss in SciProfiles

Open AccessArticle

Peer-Review Record

Open Science Knowledge Production: Addressing Epistemological Challenges and Ethical Implications

Publications 2022, 10(3), 24; https://doi.org/10.3390/publications10030024

by Bjørn Hofmann^1,2

Reviewer 1: Anonymous

Reviewer 2:

Dimity Flanagan

Reviewer 3:

Edward Luca

Publications 2022, 10(3), 24; https://doi.org/10.3390/publications10030024

Submission received: 20 May 2022 / Revised: 2 July 2022 / Accepted: 12 July 2022 / Published: 14 July 2022

Round 1

Reviewer 1 Report

General Comments

This paper explores the extent to which the “Open Science” approach is actually achieving claimed goals of increased openness, equity, diversity, and robustness in knowledge production within the academy. The premise of the paper is that open science is unevenly, and sometimes quite dramatically, failing to achieve the values-driven goals it aims to support and is also generating unanticipated negative impacts. Open science practices have been in existence long enough for this premise to be explored, especially since a significant amount of literature is now available looking at specific gains and challenges associated with open scholarly activities. Included in the discussion are important questions and complexities that have arisen by virtue of Open Science practices. Critically assessing the actual benefits of Open Science is tremendously important since too frequently those gains either fail to be realized or generate unanticipated deleterious side-effects. This paper aims to contribute to that critical assessment and the author’s ambition to identify asserted positive outcome that never come to fruition; unanticipated negative impacts; and unrecognized epistemic challenges is significant and could help the academy adjust practices in line with stated goals and values.

The author’s methodology of synthesizing and analyzing a set of significant articles focused on these issues is a perfectly acceptable approach, but the collection of papers assembled through the provided query, while a good starting point, is overly specific on its own. A second pass at finding articles, using the possible epistemic and ethical challenges or terms identified through the first set of articles, would have generated a larger body of articles to be reviewed and provided a more solid foundation for providing the aimed for overview. The author’s criteria for removing an article from the set to be reviewed are solid and those could have been applied to this larger set of papers..

The writing needs to be revised for clarity; many passages are quite convoluted and take several readings to parse out the author’s meaning.

The author relies too heavily on quoted passages, quotes that don’t really support the author’s point (see citation 6 in section 3.1 or much of section 3.3), or lists of quoted phrases without then providing the author’s own summary analysis of those concepts. This statement from section 3.2 is an example of the latter: “OS can also change the contract and interplay between science and society (47). For example, OS can alter the peer review process and publishing as such (48), as well as gift-relationship and reciprocity within the scientific society (49). ”

Notably absent is a discussion of where there are contradictions or conflicts in the analyzed literature.

Specific Issues

In section 1.1, reference #3 is not strongly supportive of the related statement.
The last paragraph of section 1.1 includes “technology transfer” as a space in which open science is discussed, but this needs to be unpacked more if it is to be included here. Technology transfer has typically been a method of helping the private sector companies benefit from the results of higher education research; discussing how that flow may or may not been impacted by open science goals is an interesting, vast topic that can’t be explored here, but the complexity could be referenced.
In section 1.2, the author establishes some broad categories of OS goals (e.g. “the productivity goal”), which is an interesting way to group the many specific OS aims under consideration. These should be more clearly called out as the author’s own organization scheme and then used as a structure throughout this section at least, if not the entire paper. For instance, as it stands, Table 1 is very confusing. It asserts that the “epistemic advantages” listed are the most frequently mentioned without providing any evidence for that, and the entries in the “Implication” column are often vague or simply reformulations of the entries in the first column. A more interesting and helpful table would be to this all of the specific goals that fall under each of the larger umbrella goals the author has defined.
The opening paragraph of section 3 needs to be revised for clarity and better alignment with the body of the section. The author isn’t presenting 3 categories of challenges, but is instead looking at three aspects of the impact of open science: 1) claimed benefits that never occur; 2) negative outcomes; and 3) significant problems specific to open science that occur in discrete phases of the research process.
In section 3.1:

Citation 10 is about human subjects and doesn’t support this broad assertion about OS’s unproven impact on research quality.
the last sentence of the first paragraph and the first sentence of the second paragraph are redundant.
The argument that infrastructures supporting open science will shape its outcomes is true, but it is also true about “gated’ science and that commonality should be indicated here, i.e. open science is not free from contextual or situational influence.
The section about the weakness of and negative impacts of shared data is full of conjectures, and includes claims that are very old and have not borne out.
Table 2 is not helpful and should be eliminated. The entries in columns one and two are imprecise and the responses in column 3 are simplistic, almost tautological, and are discussed insufficiently in the closing part of the section.
The closing section covering the responses to the epistemic challenges needs to be developed more. As it stands, the suggestions are either unclear or very general. If they are to be provided, they need to be specific and defensible as reasonable strategies to try. For instance, the author doesn’t specific what they actually mean when they propose paying authors based on metrics, which on the face of it is a truly alarming, inequitable proposal that would have tremendous negative impacts.

In section 3.2

Why would AI/ML research be harder to replicate in OS?
The author's claim that open data generates poor research is not supported.
Paragraph 5 (“Another challenge is handling…”) needs to be rewritten as it is difficult to understand as is.
Citation 42 is very old (1998), and much has changed since.
This assertion needs to be elaborated: “Without institutions verifying the quality and validity of its knowledge, OS may come to promote epistemic pluralism (44) and polarized research (45).” How should institutions engage differently than they already do and what would be the outcomes of that?
This passage is taken direction from the citation, but is not quoted, is not analyzed, and is actually a critique of the academy in general (which is fine) as opposed to open science in particular: “Moreover, it is argued that traditional single-level epistemic approaches are inadequate. Virtue theorists focus on individuals, paternalists on environments, and collectivists on groups. Accordingly, it it is argued that we need to apply interactionist approach integrating individuals, environments and groups to reduce the distortion of bias in OS (55).”
Citation 56 is misunderstood and inaccurately used here. The source describes the ethical problems of using social media platform data, which is different than open data.

In section 3.3

Many claims in this section are unsupported. For instance, in the second paragraph the author claims that open science data can’t be easily checked, but offers no evidence for this. In fact, the opposite is true.

In section 4

The ethical implications were not really explored at all, so this statement can’t be made without adding that exploration to the article: “Ethical implications of the epistemological challenges have been revealed, such as injustice, deception and manipulation (reduced autonomy), reduced benefit, and increased harm.”

Table 3 needs to be eliminated as it confusing, doesn’t represent the results of analysis, employs a visualization with shading to imply meaning that doesn’t exist, and the “Levels” in the bottom row, (“Individual, corporate”) and are never distinguished in the table.

Author Response

Dear reviewer,

I thank you for your very relevant comments and constructive suggestions. Following them have substantially improved the manuscript. Below follows a point-by-point response to the comments and suggestions. Please also see the revised manuscript (with track changes).

RESPONSE: I am most thankful for this insightful reading of the manuscript.

RESPONSE: I am most thankful for the acknowledgement of the method. I agree that more articles could have been identified and included. In the revised manuscript, I write: “The search strategy of the literature search is not very elaborate. Neither is the database very specific. More targeted searches may have included additional references. For example, including synonyms for OS such as “open research” and “open scholarship” (Search string = (("open science” OR "open research" OR "open scholarship") AND epistem* AND ethic* AND (problem* OR challeng* OR implication OR issue)) identifies 514 references (199 references more). However, the content of the additional references do not appear to identify issues that have not been covered. Moreover, as stated, the aim was not exhaustiveness and completeness of references, but rather to provide an overview of the challenges and implications, a more elaborate search may not extend the content of the results. While not an exhaustive literature review, this revew may look like an exemplary literature review as far as it “presents only key references to reacquaint the reader with representative work that relate to the research study” (73). However, it is in line with an ethics review (in Health Technology Assessment) where it aims to provide the reader with an overview of the main issues to consider (19).”

The writing needs to be revised for clarity; many passages are quite convoluted and take several readings to parse out the author’s meaning.

RESPONSE: I fully agree. Most sections have been revised and its points have been elaborated on.

RESPONSE: I am most thankful for this point. The sections have now been revised in order to increase clarity and consistency. I have also provided summary analyses of the concepts. For example, the sections of the mentioned sentences now reads as follows: “Accordingly, it is argued that resilience and best practices are needed, as well as post-normal literacy (54). While quality and validity problems with temporal knowledge production appears to be a general problem (independent of OS) it may be that it can be enhanced or directed by OS. In any case this indicates that we need to have focus on the validity and quality of knowledge production and the institutions that contribute to this in any case – and a special focus on the aspects influenced by OS. OS can also change the contract and interplay between science and society, e.g., by power structure and distortions, but also by involvement and co-evolution (55). It can imply a “constitutional recalibration” in terms of “the recalibration of old agreements concerning the autonomy of science and its relations with its societal environment, in particular politics, law, and the economy.”(56) Moreover, it is argued that OS promotes a shift in knowledge production from industrial “productionist metaphysics” to a “post-industrial mode of consumption as use, reuse, and modification.” (57) Within science itself, OS can alter the social process, such as the peer review and publishing process (58), as well as gift-relationship and reciprocity within the scientific society (59)”

Notably absent is a discussion of where there are contradictions or conflicts in the analyzed literature.

RESPONSE: Good point. This is addressed in the following: “Moreover, the identified challenges in the three main areas are not mutually exclusive. An undermined or non-occurring benefit may also have harmful (side-)effects or what causes the benefit shortfall may also cause negative effects. A more detailed analysis (for each type of OS) is needed to fully address these issues. The aim here has been to provide an overview over challenges and implications and not an exhaustive and exclusive taxonomy. Notably, there is no consistency in the literature. What has been pointed out as epistemological challenges (or ethical implications) for some types of OS may not be relevant for other types of OS. Moreover, what some consider to be an challenge for a specific type (or part) of OS, others do not. Mapping the differences and inconsistencies with respect to the various challenges and implications is beyond the scope of this article.”

Specific Issues

In section 1.1, reference #3 is not strongly supportive of the related statement.

RESPONSE: I agree. This has bee revised.

The last paragraph of section 1.1 includes “technology transfer” as a space in which open science is discussed, but this needs to be unpacked more if it is to be included here. Technology transfer has typically been a method of helping the private sector companies benefit from the results of higher education research; discussing how that flow may or may not been impacted by open science goals is an interesting, vast topic that can’t be explored here, but the complexity could be referenced.

RESPONSE: Good point. This has now been rewritten as follows: “Correspondingly, OS is discussed in a wide range of literature streams, such as Responsible Research and Innovation (RRI), Open Innovation in Science (OIS), Citizen or Crowd Science, public engagement, technology transfer (from the originator to a secondary (less privileged) user), and third mission activities (for researchers to take responsibility for the society on whose behalf they are working). More detailed analyses of how the various challenges and implications are studied in these literature streams is beyond the research questions of this study (see below).”

In section 1.2, the author establishes some broad categories of OS goals (e.g. “the productivity goal”), which is an interesting way to group the many specific OS aims under consideration. These should be more clearly called out as the author’s own organization scheme and then used as a structure throughout this section at least, if not the entire paper. For instance, as it stands, Table 1 is very confusing. It asserts that the “epistemic advantages” listed are the most frequently mentioned without providing any evidence for that, and the entries in the “Implication” column are often vague or simply reformulations of the entries in the first column. A more interesting and helpful table would be to this all of the specific goals that fall under each of the larger umbrella goals the author has defined.

RESPONSE: I agree. I have elaborated on this in the following manner: “Very broadly, OS is envisioned to increase scientific efficiency as more people can access research data, analyses, and results. Accordingly, OS contributes to reach the productivity goal of science (4). OS is thereby expected to democratize knowledge (the democratization goal) (4). Relatedly, it is predicted to promote open culture and solidarity through fostering an attitude of sharing (5), and thereby to equitize and remedy global historical injustices in scientific knowledge-production (6, 7). By this OS is envisioned to contribute to reaching the global cognitive justice goal of science. By facilitating active participation of previously excluded groups and promoting the co-production of knowledge OS aims at addressing sustainable development goals (8). In particular, OS has been envisioned to produce locally relevant knowledge, particularly for developing countries and to be a driver of innovation (9, 10) and socioeconomic growth (8, 11).” Moreover, Table 1 has been revised.

The opening paragraph of section 3 needs to be revised for clarity and better alignment with the body of the section. The author isn’t presenting 3 categories of challenges, but is instead looking at three aspects of the impact of open science: 1) claimed benefits that never occur; 2) negative outcomes; and 3) significant problems specific to open science that occur in discrete phases of the research process.

RESPONSE: Thank you for this good point. This has been rewritten and now reads: “The identified challenges can be discussed under three aspects of the impact of OS: 1) envisioned benefits that do not occur, 2) negative effects on knowledge production, and 3) significant problems specific to open science that occur in discrete phases of the research process:: data procurement (collecting, producing, clarifying, rinsing data), analyzing data, applying algorithms or models, model parameters management (estimating, deliberating on, selecting), producing raw results, interpreting results, synthesizing results, presenting results, envisioning implications, and documenting (articles, reports, synopsis, reviews).”

In section 3.1: Citation 10 is about human subjects and doesn’t support this broad assertion about OS’s unproven impact on research quality.

RESPONSE: I could not find reference 10 in 3.1, however, I have revised the references in this section.

the last sentence of the first paragraph and the first sentence of the second paragraph are redundant.

RESPONSE: This has been revised.

The argument that infrastructures supporting open science will shape its outcomes is true, but it is also true about “gated’ science and that commonality should be indicated here, i.e. open science is not free from contextual or situational influence.

RESPONSE: Excellent point. This has been added to this section: “While all infrastructures (OS or not) may influence its content, OS may have discrete framing-effects that need special attention.”

The section about the weakness of and negative impacts of shared data is full of conjectures, and includes claims that are very old and have not borne out.

RESPONSE: I fully agree. This section has been rewritten.

Table 2 is not helpful and should be eliminated. The entries in columns one and two are imprecise and the responses in column 3 are simplistic, almost tautological, and are discussed insufficiently in the closing part of the section.

RESPONSE: The table has been supported by the other reviewers, but I have revised it in order to improve clarity and relevance.

The closing section covering the responses to the epistemic challenges needs to be developed more. As it stands, the suggestions are either unclear or very general. If they are to be provided, they need to be specific and defensible as reasonable strategies to try. For instance, the author doesn’t specific what they actually mean when they propose paying authors based on metrics, which on the face of it is a truly alarming, inequitable proposal that would have tremendous negative impacts.

RESPONSE: I agree. This section now reads as follows: “What then can we do to address these epistemological challenges and their ethical implications? As suggested in Table 2, we may need to change the epistemic governance structure, as the traditional structures related to access and use of data are bypassed. Moreover, we need actively to adapt the incentive systems and funding requirements, to avoid biased knowledge production and assure equity and justice, also in a global perspective. Accordingly, we need to make adaptive adjustments to the impact metrics and peer review system, as opennes does not assure quality. Connecting money to the metrics is but one way of acknowledging the importance of recognition and reward, but it may reinforce socioeconomic structures. Improving infrastructure facilitating the contributions and access of under-priviliged is crucial. So is countering framing-effects and compensation mechanisms to address skewed infrastructure effects. How to address the epistemological challenges and ethical implications in practice of course needs to be tailored to the specific OS context. This is well beyond the objective and scope of this article and is the task of many subsequent and specific studies. The point here is to provide an overview which hopefully can be helpful for such studies. Nonetheless, the specific solutions may do well in following such general suggestions (1).”

In section 3.2: Why would AI/ML research be harder to replicate in OS?

RESPONSE: Good point. This now reads: “Moreover, the replicability problem is even more pronounced in OS artificial intelligence (AI) and machine learning (ML) (45) due to lack of transparency and “black box problems.””

The author's claim that open data generates poor research is not supported.

RESPONSE: This is an important point. This section now reads: “Another identified challenge is that easy access to data may incentivise poor quality research. It is documented that researchers who are incentivised strongly to publish, are drawn to study designs with low statistical power (47). Moreover, easy access can spur unqualified use as e.g., contextual constraints and quality caveats, may be ignored. Again, it is an empirical question whether easy access spurs low-quality research, but it calls for attention to the danger that easy access fosters easy designs, which fosters low quality evidence. New measures for data quality assurance may be needed as part of OS data use.”

Paragraph 5 (“Another challenge is handling…”) needs to be rewritten as it is difficult to understand as is.

RESPONSE: I fully agree. This now reads: “Another challenge is handling intellectual interests and intellectual property rights as OS threatens to undermine “cognitive capitalism,” which is explained to be a “parasitical and rentier exploitation of collective production, offering the conditions for its reproduction as in free platforms of access to digital networks. At the same time, it spoils this very dynamics of value creation with the toughening of mechanisms for protecting intellectual property” (48). Accordingly, to obtain epistemic efficiency and promote pluralism appears to be a substantial challenge: “This should be the greatest ethical challenge of open science: the dialogue with the other, the building of bridges and mutual fertilisation in the diversity of knowledge” (48). In practice, to avoid appropriation and unwarranted profiting from OS is identified as an important task.”

Citation 42 is very old (1998), and much has changed since.

RESPONSE: I agree, but the point still seems to be valid. Moreover, the substantial parts of this section is supported by other and newer references. If you insist, I will remove the reference, but age has not been an exclusion criteria for this study.

This assertion needs to be elaborated: “Without institutions verifying the quality and validity of its knowledge, OS may come to promote epistemic pluralism (44) and polarized research (45).” How should institutions engage differently than they already do and what would be the outcomes of that?

RESPONSE: Good point. This now reads: “Yet another challenge is that OS knowledge production depends on specific types of institutions and ogranization (49). Some have described the demand for a “new institutionality,” i.e., that OS requires or enforcces new institutional formats as well as new normative and legal frameworks for the production, circulation, appropriation, evaluation, and use of scientific knowledge (48). Correspondingly, OS may foster new epistemic cultures, with emerging local knowledge-producing practices (50). However, it is pointed out that without institutions verifying the quality and validity of its knowledge, OS may come to promote epistemic pluralism (51) and polarized research (52). While the validation problem is not unique to OS, it may become more pressing or enhanced in OS, and thus is important for maintaining the trustworthiness of science. Hence, OS may contribute substantially to scientific development by refining existing or elaborating new validation methods.”

This passage is taken direction from the citation, but is not quoted, is not analyzed, and is actually a critique of the academy in general (which is fine) as opposed to open science in particular: “Moreover, it is argued that traditional single-level epistemic approaches are inadequate. Virtue theorists focus on individuals, paternalists on environments, and collectivists on groups. Accordingly, it it is argued that we need to apply interactionist approach integrating individuals, environments and groups to reduce the distortion of bias in OS (55).”

RESPONSE: Again, I am thankful for this point. This section now reads as follows: “Moreover, OS may enhance (or reduce) bias in research, as bias is difficult to detect and address. While the openness and transparency of OS can reveal (and hopefully reduce) biases in research, it may also enhance them, e.g., in parts of OS that are hidden (such as data procurement, see below). Guidelines, such as the Transparency and Openness Promotion (TOP) Guidelines for the reporting of scientific research, may be used, adapted, or developed to detect and avoid bias. On a general level it has been argued that traditional single-level epistemic approaches are inadequate. Virtue theorists focus on individuals, paternalists on environments, and collectivists on groups. Accordingly, it it is argued that we need to apply interactionist approach integrating individuals, environments and groups to reduce the distortion of bias in OS (62).”

Citation 56 is misunderstood and inaccurately used here. The source describes the ethical problems of using social media platform data, which is different than open data.

RESPONSE: Thank you for discovering this. Due to text editing, this reference was misplaced. It is corrected in the revised manuscript.

In section 3.3: Many claims in this section are unsupported. For instance, in the second paragraph the author claims that open science data can’t be easily checked, but offers no evidence for this. In fact, the opposite is true.

RESPONSE: I very much agree that this was too short. This section now reads: “First, there exist risks of scientific misconduct in various phases of OS knowledge production. While open data facilitates verification of analyses, algorithms, and models, the entrance and quality assurance of the data themselves may not be as easily checked and quality assured. The reasons for this may be multiple, e.g., that the data user is remote from tha data source and has limited information about the context of its retrieval. It may be easy to fabricate or falsify data. Hence, while OS may reduce scientific miscondict in several parts of OS knowledge production, it may not reduce the chance of fabrication, falsification, and plagiary in data provision and procurement (65). While this problem certainly is not unique to OS, the point is that we should pay special attention to data procurement and provision in OS.”

In section 4: The ethical implications were not really explored at all, so this statement can’t be made without adding that exploration to the article: “Ethical implications of the epistemological challenges have been revealed, such as injustice, deception and manipulation (reduced autonomy), reduced benefit, and increased harm.”

RESPONSE: I fully agree. This has been revised accordingly: “While I have focused on the epistemological challenges and ethical implications of OS and briefly discussed the ethical issues with various parts of OS, each of the issues deserve more discussion in detail than can be included in this article. There are also a wide range of ethical issues related to OS in general that are not directly related to epistemic challenges, for example that OS in qualitative research may violate privacy rights (68). The objective here has been to provide an overview of the epistemological challenges and their ethical implications so that the specific issues can be addressed in a more detailed manner in further research.”

RESPONSE: This table has been deleted and a new Table 3 summarizes the potential negative aspects of OS (and what we can do about them).

Reviewer 2 Report

Brief summary

This author clearly states that their objective is is to provide an overview of the epistemological challenges with OS and their ethical implications through conducting a literature review. This is done through three distinct areas: benefits that do not occur; negative effects; and epistemological challenges with OS processes. While this approach has value given there isn’t a huge amount of analysis on epistemological challenges, the article lacks sufficient analysis of the literature. Instead, it compiles brief points from a long list of sources. This makes it extremely difficult for the reader to understand the strength of the arguments presented unless they are already familiar with that aspect of the topic, or are willing to consult the corresponding reference.

Contribution and quality summary

In order to ensure that Open Science can be successful, there is significant value in exploring the epistemological and ethical challenges of Open Science in order to raise awareness of issues that need to be addressed. The article is clear about what is out of scope and its purpose in providing a broader overview of the challenges rather than diving into the specifics of each component of Open Science. This would therefore make it a great starting point for researchers exploring the topic. However, while the aims of the article are valuable, the execution does need improvement in order to improve the clarity of argument.

Methods

Methods were very clearly explained in this article. The author states that this literature review is not meant to be exhaustive but highlight the main challenges. Given that the article is heavily referenced, there is value in this approach. However, I do think there should have been an attempt to address the fact that not all countries use the term Open Science – some articles may have been missed by not looking for articles on Open Scholarship or Open Research. Typically, literature reviews are either exhaustive or exemplary. The author is clear that it is not an exhaustive review but there does not appear to be enough analysis around the arguments for it to be considered an exemplary review.

References
A very extensive list of current references, covering a diversity of aspects of open science.

For specific comments on the article, please see the annotated PDF (please get in touch if there is any issue viewing this). A summary of each section is provided below.

Introduction:

Clear about definition and components of Open Science. Lays out the objective of the study as a way to further the argument of the need for an ethics of Open Science.

Initiation clarification:

More could be done to clarify the distinct connections between knowledge production stages and OS.

Author is clear about what is out of scope. Although what is out of scope is also extended in the discussion section of the article.

Acknowledging the envisioned benefits with OS:

First paragraph contains many benefits, more as a list. Second and third paragraphs are a little stronger as they provide a little more explanation behind the benefits.

Table 1 is a clear summary of benefits/implications.

Methods:

See comments above

Epistemological challenges regarding OS:

Fair summary

Concerns with benefits that do not occur:

My specific comments in the PDF on this section are against the arguments presented. This is what is lacking in a lot of the analysis – it presents arguments from the references, without any value-add from the author. Should we be taking all these as gospel? Particularly as a lot of the concerns arise from poorly executed OS practices that appear to have more in common with closed research practices. If that is the point: that if researchers are not sufficiently incentivised to pursue gold standard OS practice, then the benefits will not materialise, then that needs to be more clearly conveyed.

Table 2 is a clear summary of benefits/challenges/solutions

Concerns with negative effects

Similarly to the previous section, a lot of arguments feel like there is opportunity to refute them. As a reader, I want to learn about potential negative effects that are accompanied by robust arguments. Perhaps the original sources do present robust arguments, but this is not conveyed in the article. Instead, such high-level points are conveyed that a learned reader might see inherent flaws and choose to stop reading.

Epistemological challenges with OS processes:

This is the easiest section to understand because it’s focusing in on more details.

Discussion:

Rather than a discussion of the previous sections, which really do warrant some analysis, this section focuses a lot on putting more things out of scope. The comment on epistemic injustice is really fascinating and certainly warrants further research.

Summary and conclusion:

Clearly conveyed.

Table 3: introduced in section 4 but feels more relevant to the “Epistemological challenges with OS processes” section.

Comments for author File: Comments.pdf

Author Response

Dear reviewer,

I thank you for your very relevant comments and constructive suggestions (both below and in the manuscript). Following them have substantially improved the manuscript. Below follows a point-by-point response to the comments and suggestions. Please also see the attached pdf where I respond to every specific comment, for which I am very thankful.

RESPONSE: This is a very good and important point. I have fleshed out the arguments as much as possible without making the text unreadably long. As the aim is to provide readers with an overview I cannot go too deeply into each issue. Otherwise, the readers will lose the overview. Please, see also my response to your excellent comment on exemplary literature review below.

Contribution and quality summary: In order to ensure that Open Science can be successful, there is significant value in exploring the epistemological and ethical challenges of Open Science in order to raise awareness of issues that need to be addressed. The article is clear about what is out of scope and its purpose in providing a broader overview of the challenges rather than diving into the specifics of each component of Open Science. This would therefore make it a great starting point for researchers exploring the topic. However, while the aims of the article are valuable, the execution does need improvement in order to improve the clarity of argument.

RESPONSE: I hope that by elaborating more on the various issues, I provide more content, but not too much for the reader to lose the overview, which is the aim of the article.

Methods: Methods were very clearly explained in this article. The author states that this literature review is not meant to be exhaustive but highlight the main challenges. Given that the article is heavily referenced, there is value in this approach. However, I do think there should have been an attempt to address the fact that not all countries use the term Open Science – some articles may have been missed by not looking for articles on Open Scholarship or Open Research. Typically, literature reviews are either exhaustive or exemplary. The author is clear that it is not an exhaustive review but there does not appear to be enough analysis around the arguments for it to be considered an exemplary review.

RESPONSE: I thank you for this opportunity to clarify and discuss the methods. The parts of the introduction that address these issues now read: “To provide an overview of the epistemological challenges and related ethical implications, a basic literature search was performed in GoogleScholar with the search string “("open science” AND epistem* AND ethic* AND (problem* OR challeng* OR implication OR issue)).” The aim of the search was not to get an exhaustive overview of the literature, but to identify the main challenges and issues. As such it follows a method for identifying ethical issues in Health Technology Assessment (HTA) (17-19). Hence, only articles discussiong epistemological challenges and ethical implications were included. Articles only mentioning that such challenges and implications exist without any substantial analysis were excluded. So were articles mentioning epistemological challenges and ethical implications already identified, but not adding new content.” In the Discussion I now state that “the aim was not exhaustiveness and completeness of references, but rather to provide an overview of the challenges and implications, a more elaborate search may not extend the content of the results. While not an exhaustive literature review, it looks like an exemplary literature review as far as it “presents only key references to reacquaint the reader with representative work that relate to the research study” (65= Rubin, R. B., Rubin, A. M., & Piele, L. J. (2009). Communication research: Strategies and sources. Wadsworth Cengage Learning.). More precisely, it is in line with an ethics review (in Health Technology Assessment) where it aims to provide the reader with an overview of the main issues to consider (66= Hofmann, B., Droste, S., Oortwijn, W., Cleemput, I., & Sacchini, D. (2014). Harmonization of ethics in health technology assessment: a revision of the Socratic approach. Int J Technol Assess Health Care, 30(1), 3-9. https://doi.org/10.1017/S0266462313000688).”

References: A very extensive list of current references, covering a diversity of aspects of open science.

RESPONSE: I am most thankful for this response.

For specific comments on the article, please see the annotated PDF (please get in touch if there is any issue viewing this). A summary of each section is provided below.

Introduction: Clear about definition and components of Open Science. Lays out the objective of the study as a way to further the argument of the need for an ethics of Open Science.

RESPONSE: I am most thankful for this comment.

Initiation clarification: More could be done to clarify the distinct connections between knowledge production stages and OS.

RESPONSE: This is a good point. This now reads: “As OS is defined in terms of “processes of scientific knowledge creation” it is interesting to investigate the modes and steps of the provision or production of this knowledge. Accordingly, when I investigate epistemological challenges with OS knowledge production, the later is here broadly defined as a systematic approach to provide information elements with which one has obtained familiarity through specific processes, such as information acquisition, individual and group learning, (knowledge) claim formulations, (knowledge) claim evaluations. See for example (3).”

Author is clear about what is out of scope. Although what is out of scope is also extended in the discussion section of the article.

RESPONSE: I am most thankful for this comment.

Acknowledging the envisioned benefits with OS: First paragraph contains many benefits, more as a list. Second and third paragraphs are a little stronger as they provide a little more explanation behind the benefits.

RESPONSE: I agree. I have elaborated on this, and this part now reads as follows: “Very broadly, OS is envisioned to increase scientific efficiency as more people can access research data, analyses, and results. Accordingly, OS contributes to reach the productivity goal of science (4). OS is thereby expected to democratize knowledge (the democratization goal) (4). Relatedly, it is predicted to promote open culture and solidarity through fostering an attitude of sharing(5), and thereby to equitize and remedy global historical injustices in scientific knowledge-production (6, 7). By this OS is envisioned to contribute to reaching the global cognitive justice goal of science. By facilitating active participation of previously excluded groups and promoting the co-production of knowledge OS aims at addressing sustainable development goals (8). In particular, OS has been envisioned to produce locally relevant knowledge, particularly for developing countries and to be a driver of innovation (9, 10) and socioeconomic growth (8, 11).” At the end of this section, I explicitly state the purpose and limitation of this section: “The point here has not been to provide an in-depth analysis of these advantages, but only to present an overview of the elements to be discussed below.”

Table 1 is a clear summary of benefits/implications.

RESPONSE: I am most thankful for this comment.

Epistemological challenges regarding OS: Fair summary

RESPONSE: I am most thankful for this comment.

Concerns with benefits that do not occur: My specific comments in the PDF on this section are against the arguments presented. This is what is lacking in a lot of the analysis – it presents arguments from the references, without any value-add from the author. Should we be taking all these as gospel? Particularly as a lot of the concerns arise from poorly executed OS practices that appear to have more in common with closed research practices. If that is the point: that if researchers are not sufficiently incentivised to pursue gold standard OS practice, then the benefits will not materialise, then that needs to be more clearly conveyed.

RESPONSE: This is a very profound comment. I have now elaborated on the issues identified in the literature and commented on their implications. In order not to divert from the objective of the article (to provide an overview of the challenges and its implications) I have been careful to avoid polemics against the various authors or their arguments. However, I hope that the elaboration has provided more content and clarity, making the epistemological challenges and their ethical implications clearer.

Table 2 is a clear summary of benefits/challenges/solutions

RESPONSE: I am most thankful for this comment.

Concerns with negative effects: Similarly to the previous section, a lot of arguments feel like there is opportunity to refute them. As a reader, I want to learn about potential negative effects that are accompanied by robust arguments. Perhaps the original sources do present robust arguments, but this is not conveyed in the article. Instead, such high-level points are conveyed that a learned reader might see inherent flaws and choose to stop reading.

RESPONSE: I fully agree. However, as the objective is to review the literature and provide an overview, I have tried to be careful not to engage in personal argumentation with the various issues pointed out. However, due to your advice, I have elaborated more and presented alternative views for the various points and arguments found in the literature.

Epistemological challenges with OS processes: This is the easiest section to understand because it’s focusing in on more details.

RESPONSE: Thank you. I have elaborated more on the other issues (and sections) using this as a model as I realize what was written was too brief.

Discussion: Rather than a discussion of the previous sections, which really do warrant some analysis, this section focuses a lot on putting more things out of scope. The comment on epistemic injustice is really fascinating and certainly warrants further research.

RESPONSE: I fully agree. However, this topic merits a separate article.

Summary and conclusion: Clearly conveyed.

RESPONSE: I am most thankful for this comment.

Table 3: introduced in section 4 but feels more relevant to the “Epistemological challenges with OS processes” section.

RESPONSE: Table 3 is totally revised/rewritten. See also comments from Reviewer 1.

Author Response File: Author Response.pdf

Reviewer 3 Report

Thank you to the author for submitting this paper. The paper aims to provide an overview of the epistemological challenges and ethical implications of Open Science (OS). Three challenges are derived from the literature:

Concerns with benefits that do not occur (consider rephrasing to “unrealized benefits”?)
Concerns with negative effects
Epistemological challenges with OS processes

These themes are then discussed briefly, with many different threads and issues raised. I would have liked to have seen a bit more explanation – many concepts and ideas are introduced with a lot of assumed knowledge, and the reader is required to trawl through the references to find what is being discussed. I have identified some of these in my minor comments below.

For example: “While the literature identifies and discusses many crucial challenges of OS and its ethical implications, the scope of this article does not allow a detailed analysis of each of them.” What, then, is the contribution of this article?

I believe that the paper could be strengthened through elaborating on the tables, so that it might provide a more helpful starting point for readers engaging with these issues. For example, in Table 1, we see a summary of the epistemological advantages of OS. I would like a little more detail here - it feels as though terms are being thrown around and either a) require further reading to understand or b) are ill-defined. For example, one epistemic advantage is broader participation, and I believe the implication is that researchers should pursue broader goals - but what does this actually mean?

I had a similar issue with table 2, which is likely even more useful as the themes are derived from the literature (as opposed to the general discussion leading to table 1). For example: the epistemic advantage of “alternative impact metrics”, is challenged by “looping effects”, not defined, and the solution is “adaptive adjustments” - it’s unclear to me what this means. There are also no references - consider adding a column for the sources of these themes, so that we know where they have come from?

While I acknowledge that the paper is intended to provide a broad overview, I still think a bit more detail/elaboration is required to ensure that the paper provides a contribution in itself.

Minor points

Page 1 - typo – “transparent” should be “transparent”

Page 2 - typo - “while detailed analyses the epistemological challenges” should be “while detailed analysis of the epistemological challenges”

Page 3 - typo - “Additionaly” to “Additionally”

Perhaps 12 articles were identified (not “included”) through snowballing?

Page 5 - what are “looping effects”? “Open display of knowledge can be used for bad purposes” - colloquial, rephrase

Page 6 - “(see also below)” - see what?

“Cognitive capitalism” - what is this?

Page 9 - connecting money to the metrics - I’m not sure why this would help?

Author Response

Dear reviewer,

I am most thankful you for your very relevant comments and constructive suggestions. Following them have substantially improved the manuscript. Below follows a point-by-point response to the comments and suggestions. Please also see the attached revised manuscript with track changes.

Concerns with benefits that do not occur (consider rephrasing to “unrealized benefits”?)
Concerns with negative effects
Epistemological challenges with OS processes

RESPONSE: I thank you very much for the acknowledgement and fully agree that more elaboration is needed. This has been provided for each identified item/topic/challenge. However, I have tried to avoid going to deep into each point as this may make the reader lose the overview and be beyond the objective of the article.

Discussion points which are outside the scope are mentioned numerous times, but this left me wondering, as a reader, what is actually in scope? I think the contribution of the article could be made clearer, beyond the promise of a discussion of broad issues. For example: “While the literature identifies and discusses many crucial challenges of OS and its ethical implications, the scope of this article does not allow a detailed analysis of each of them.” What, then, is the contribution of this article?

RESPONSE: To address this important issue, I explicitly state the following in the Discussion: “As OS is a very broad field with a wide range of merits and challenges it has been necessary to delimit the scope of this study. The objective has been delimited to provide an overview of the epistemological challenges with OS knowledge production, and their ethical implications in terms of a) factors undermining the envisioned benefits, b) direct negative epistemic effects, and c) epistemological challenges with the various phases of the OS process. The objective appears relevant and warranted as OS is elaborate enough to critically assess its benefits and challenges in order to adjust and improve OS in the future. Although the objective is not to discuss the details of each of the challenges and implications, the important contribution of this article is the identification, gathering, and synthesizing a wide range of information, which otherwise is very labor intensive to get and to providing an overview and inspiration for further and more in-depth studies.”

RESPONSE: Good point. Table 1 (and 2) have been revised.

RESPONSE: Good point. The Table has been revised. The looping effect is now defined in the following section: “While OS is envisioned to spur new science assessments and metrics (35), there are reasons to consider looping effects, i.e., the interaction between a classification and the persons (or kinds) that are classified (36). Scientists may adapt to new metrics in manners that distort fair accreditation and undermine scientific research quality. This is of course a general problem to science metrics that does not vanish with OS. However, as it may be continued and enhanced in OS, it is crucial that we pay careful attention to it. As pointed out above, alternative metrics may reduce adaptation, but it may not eliminate the looping effect.”

While I acknowledge that the paper is intended to provide a broad overview, I still think a bit more detail/elaboration is required to ensure that the paper provides a contribution in itself.

RESPONSE: I fully agree and have elaborated on the content – hopefully without spoiling the overview.

Minor points

Page 1 - typo – “transparent” should be “transparent”

RESPONSE: Sorry, but I do not see the difference. In the manuscript is written: “Open Science is defined as “transparent and accessible knowledge that is shared and developed through collaborative networks” (1) Accordingly, OS has four key characteristics: OS is transparent, shared, accessible, and collaborative-developed.”

Page 2 - typo - “while detailed analyses the epistemological challenges” should be “while detailed analysis of the epistemological challenges”

RESPONSE: This has been corrected.

Page 3 - typo - “Additionaly” to “Additionally” Perhaps 12 articles were identified (not “included”) through snowballing?

RESPONSE: This has been corrected.

Page 5 - what are “looping effects”? “Open display of knowledge can be used for bad purposes” - colloquial, rephrase

RESPONSE: The looping effect is explained in the following section: “While OS is envisioned to spur new science assessments and metrics (35), there are reasons to consider looping effects, i.e., the interaction between a classification and the persons (or kinds) that are classified (36). Scientists may adapt to new metrics in manners that distort fair accreditation and undermine scientific research quality. This is of course a general problem to science metrics that does not vanish with OS. However, as it may be continued and enhanced in OS, it is crucial that we pay careful attention to it. As pointed out above, alternative metrics may reduce adaptation, but it may not eliminate the looping effect.” “Open display of knowledge can be used for bad purposes” is rephrased: “Open disclosure of knowledge can be used for bad purposes”

Page 6 - “(see also below)” - see what?

“Cognitive capitalism” - what is this?

RESPONSE: For simplicity “(see also below)” has been removed. “Cognitive capitalism” has been explained: “Another challenge is handling intellectual interests and intellectual property rights as OS threatens to undermine “cognitive capitalism,” which is explained to be a “parasitical and rentier exploitation of collective production, offering the conditions for its reproduction as in free platforms of access to digital networks. At the same time, it spoils this very dynamics of value creation with the toughening of mechanisms for protecting intellectual property” (49). Accordingly, to obtain epistemic efficiency and promote pluralism appears to be a substantial challenge: “This should be the greatest ethical challenge of open science: the dialogue with the other, the building of bridges and mutual fertilisation in the diversity of knowledge” (49). In practice, to avoid appropriation and unwarranted profiting from OS is identified as an important task.”

Page 9 - connecting money to the metrics - I’m not sure why this would help?

RESPONSE: I agree. This is an open question. This has been rephrased as follows: “To address these epistemological challenges, we may need to change the epistemic governance structure, actively to adapt the incentive systems and funding requirements, and make adaptive adjustments to the impact metrics and peer review system. Connecting money to the metrics is but one way of doing so acknowledging the importance of reward. Improve infrastructure, counter-frame, and compensation may be ways to address skewed infrastructure effects.”

Round 2

Reviewer 1 Report

Dear Author,

Thank you for your thoughtful engagement with my feedback on your original submission; the review process can be a challenging one and so I appreciate your openness. I have read through both your responses to my comments and this second iteration of the manuscript. This version is definitely an improvement, but there are two remaining issues that I feel still need to be addressed:

1. Tables: Tables 1 and 2 don't have adequate specificity and don't capture the narrative content well. I actually don't think they are necessary, and instead feel that the text of the sections those tables are in should be sufficient to get these concepts across clearly. The original Table 3 has been eliminated, which I think is a good decision, but is still referenced in the last sentence of section 4. The new Table 3 has the same issues as Tables 1 and 2, and, as with those, either needs to be revised or simply eliminated, as the text in the related section should be sufficient. I see from your comments that other reviewers have a different opinion, so this may be a topic to discuss with the editors.

2. Overall clarity: The text you have added throughout that more clearly provides your own summary and analysis is an important, positive change. In general, though, more could be done to achieve greater clarity and readability, both in the organization of the points and their articulation.

On a more minor point, regarding citation 42, which I had noted as somewhat old, I have no concern with retaining it.

Finally, reading this second version surfaced an important point threaded throughout the manuscript, but not identified distinctly in the Discussion or Summary/Conclusion sections and that is the call for empirical exploration of the issues identified in the paper, many of which are described as being of possible concern, as evidenced by the repeated use of the word "may" in referring to a potential impact. Returning to that set of risks and posing to the community the importance of investigating in concrete ways the extent and manner in which those risks have or have not materialized could be a powerful component of the conclusion.

Author Response

Dear reviewer,

Again, I must thank you for your engagement and constructive suggestions and for your acknowledgement of the improvements of the manuscript. An Acknowledgment section has been added i to express my appreciation: “I am most thankful to two anonymous reviewers and the Editor for valuable comments and constructive suggestions which have been key in revising and improving the manuscript.” Below follows a point-by-point response to the comments and suggestions. Please also see the revised manuscript (with track changes).

Tables: Tables 1 and 2 don't have adequate specificity and don't capture the narrative content well. I actually don't think they are necessary, and instead feel that the text of the sections those tables are in should be sufficient to get these concepts across clearly. The original Table 3 has been eliminated, which I think is a good decision, but is still referenced in the last sentence of section 4. The new Table 3 has the same issues as Tables 1 and 2, and, as with those, either needs to be revised or simply eliminated, as the text in the related section should be sufficient. I see from your comments that other reviewers have a different opinion, so this may be a topic to discuss with the editors.

RESPONSE: The references to the old Table 3 has been removed. Regarding Table 1 and 2 Reviewer #2 finds them helpful and for (the new) Table 3 writes: “I also want to add that Table 3 has been vastly improved in the second version of this article. It contributes a much better summary.” I therefore have been hesitant to remove all tables. I fully agree that the tables do not provide additional substantial content to the text, but as this is a review containing much and diverse information, my thought has been that readers may want summarizing overviews. I will therefore leave it to the Editor to decide which tables to keep (if any). To convey the purpose of the tables, I have also added text in the legends, such as “The table is not exhaustive but only provides a summarizing overview of the advantages” in Table 1. I also refer to the summary of the tables in the text, e.g., in 3.1.

Overall clarity: The text you have added throughout that more clearly provides your own summary and analysis is an important, positive change. In general, though, more could be done to achieve greater clarity and readability, both in the organization of the points and their articulation.

RESPONSE: The manuscript has been revised in order to increase clarity. Several linguistic errors and shortcomings have been corrected.

On a more minor point, regarding citation 42, which I had noted as somewhat old, I have no concern with retaining it.

RESPONSE: Thanks.

RESPONSE: I am most thankful for this comment. A new section has been added to the Discussion which reads: “While many epistemological challenges and ethical implications have been identified, we have very limited evidence on how they play out in the practice of OS. This is of course because it is still early days in OS knowledge production. This means that we should pay attention to the challenges to obtain the envisioned benefits of OS (as pointed out), but also that we need empirical research on whether these challenges occur and how they play out. Hence, we need more research on the epistemological and ethical aspects of OS.”

Reviewer 2 Report

I am pleased with the changes that have been made as a result of the peer review process. Clarifying sentences have been well used to qualify the arguments made in the referenced literature. I also believe that the overview approach has been better justified through the comparison to the HTA approach.

Author Response

Dear reviewer,

Again, I must thank you for your constructive engagement and for your acknowledgement of the improvements of the manuscript. An Acknowledgment section has been added i to express my appreciation: “I am most thankful to two anonymous reviewers and the Editor for valuable comments and constructive suggestions which have been key in revising and improving the manuscript.” Below follows a point-by-point response to the comments and suggestions. Please also see the revised manuscript (with track changes).

RESPONSE: I am most thankful for this comment.

My only comment on the changes made is in relation to the myopia and sclerosis paragraph on page 9. It is primarily a collection of quotes from the one source. The source presents quite dense academic language. I am conscious that open science scholarship has quite a broad audience and I believe this paragraph could benefit from the inclusion of a brief lay summary by you. This would enhance the accessibility of the this particular argument.

RESPONSE: This is a good point. This paragraph now reads: “While it is envisioned that OS can contribute to open contact and fruitful influence of different theoretical perspectives, OS may also enhance group think, polarization, and theoretical “myopia and sclerosis.” For example OS is predicted to promote “open theorizing,” i.e., “when loosely coordinated researchers realize they can draw on one another’s empirical, methodological, or theoretical material to develop theoretical contributions” and in technical terms that “enactment of the social epistemological principles of free criticism and diversity fosters the concentration, extension, reinvigoration, and procreation of theoretical vocabularies, which may promote theoretical deepening, expansion, rejuvenation, and generativity, respectively” (63). At the same time, it is pointed out that open theorizing “may escalate the field’s already existing tendencies toward theoretical myopia, dilution, shallowness, and faddishness” due to theoretical narrowmindedness, theoretical branching out, superficial analogies, and “theoretical cul-de-sacs.” (63) Hence, whether OS will contribute to open theory building is an open question that deserves attention (both epistemically and ethically).”

I also want to add that Table 3 has been vastly improved in the second version of this article. It contributes a much better summary.

RESPONSE: I am most thankful for this comment.

Finally, as these changes were turned around quite quickly, I would advise a very thorough proofing process as I have noticed more (but small) grammatical errors in the revised text.

RESPONSE: I apologize for the many errors in the previous revision. The manuscript has been carefully revised and proof edited.

Reviewer 3 Report

Thank you for your submission. I am pleased to see your responses to my reviewer feedback.

I have no further comments to make aside from a few typos throughout, e.g. "indorsing", "particition", "democtratizing".

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

The paper gives an well structured overview of the issues around open science. However, the discussion of possible solutions is not very extensive.More work on this would have made the paper much stronger.

The manuscript gives an overview of the phases related to Open Science. Then it describes – based on the literature – which aspects are prone to potential problems.

Strength of the manuscript: The paper gives an well-structured overview of the issues around open science.
Weakness: the issues are only briefly described, and the discussion of possible solutions is not very extensive. More work on this would have made the paper much stronger.

Provide a point-by-point list of your major/minor recommendations for the improvement of the manuscript;

Major recommendations:

Either:

Describe more clearly that this manuscript only is a short introduction of the issues regarding Open Science

Find a 'real world' example of the issues described and what the consequences of that issue have been.

If the latter is chosen, describe in detail:

The problems encountered
Consequences
Mitigating measurements, and the effects on Open Science in general

Author Response

I am most thankful for the opportunity to revise this manuscript, and I am most thankful for the wise comments and constructive suggestions by the reviewers. Their reviews have guided the revision (or rewriting) of the manuscript, which has undergone major revisions. All changes are visualized in the attached file (with track changes). You also find a point by point response to the reviewers’ comments in the attached file where my responses are highlighted in red color in order to facilitate reading.

Author Response File: Author Response.pdf

Reviewer 2 Report

Dear Professor Hofmann,

thank you very much for the opportunity to read your manuscript. I think you have identified an interesting research idea: Presenting an overview on the various sources of distortion of knowledge production in the different phases of Open Science and their ethical implications. I truly got excited after reading your abstract and having a structured and comprehensive overview would be much needed to further advance our understanding of open (and collaborative) scientific knowledge production! Kudos to this timely and relevant idea.

With this being said, I have some concerns related to how the article tries to answer this research question. In the following I will point towards some issues that I consider critical for pursuing this idea further as well as some suggestions for alternative approaches. Please note that all my comments are meant to be purely constructive and grounded in my own (biased) epistemic culture as organization of science scholar.

After reading your manuscript several times, the most critical aspects to me are:

I was missing a broader conceptual contextualization in the front end, clearly outlining your implicit assumptions towards OS. As you correctly say in the very beginning, there are many definitions of OS and openness and collaboration in scientific research is subject to many fields including sociology, STS, management, or philosophy scholars. I think the article (and the research endeavor itself) would substantially increase its value, if you would consider these different approaches towards OS as they influence how the terms you use are understood.
Related to this, key terms are not properly defined. This is particularly important regarding epistemological advantages / challenges. It is not clear to me what is considered an epistemic advantage (vs. an advantage) and on the other hand, what is NOT considered an epistemic advantage / challenge. Likewise, what OS practices do you consider? You refer to different practices such as data sharing and OA publishing, but a more structured or conceptual approach towards what practices are considered in the article would be helpful.
What is your approach towards your research goal? Based on the information provided in the abstract, I assumed you would do a literature review. However, the main part of the article does not confirm this impression because a) there are no signs for a methodological approach towards screening the literature (structured or unstructured); and b) the literature covered seems to be very narrow and, apologies, random (i.e., borrowing from policy reports, educational textbooks, and articles from very diverse fields spanning the continuum from medical sciences, to open innovation and sociology, etc.). I would be eager to learn more about the concrete scientific approach you took for deciding what literature to consider and what deliberately not to consider. This is also important behind the background that I perceived several sections in the article almost dogmatic advocating pro open science. While I share the enthusiasm for making the scientific knowledge production process more open (and collaborative) for the sake of scientific productivity as well as democratization goals (see Sauermann et al., 2020), I think it is important to consider openness not as an end in itself but a means to achieve these goals.
The conclusions you draw would very much benefit from more carefully crafted arguments. This includes, for example, the consideration of contradicting findings in the literature and the consideration of boundary conditions. I think the article would benefit a lot from a more balanced and in-depth discussion of the literature. This goes hand-in-hand with the necessity to get a more comprehensive overview on what has been published on challenges across the scientific knowledge production process so far (which is certainly more than 23 sources).
Similarly, you present potential remedies in Table 2, but without discussing them at all. On what grounds do you consider them to be helpful?

Please allow me to share a suggestion on how to proceed. Again, please take all my suggestions and comments with a grain of salt given my own background. If that would be my research project, I would conduct a structured literature review covering and distinguishing the different scientific fields talking about OS, the OS practices, and some conceptual aspect (e.g., whether the OS practice includes inbound-, outbound, or coupled knowledge/data/information flows). I think it would be very promising to analyze the resulting body of empirical work on this matter in a structured way to identify valid conclusions (and potential remedies) on the advantages and challenges associated with applying OS practices along the scientific knowledge production process.

Again, I can only encourage you to continue your work on this topic and I hope to see some of your work published soon. Best of luck!

References and potentially interesting readings:

Sauermann, H., Vohland, K., Antoniou, V., Balázs, B., Göbel, C., Karatzas, K., ... & Winter, S. (2020). Citizen science and sustainability transitions. Research Policy, 49(5), 103978.

Knorr-Cetina, K. D. (1991). Epistemic cultures: Forms of reason in science. History of Political Economy, 23(1), 105-122.

Cetina, K. K. (2007). Culture in global knowledge societies: Knowledge cultures and epistemic cultures. Interdisciplinary science reviews, 32(4), 361-375.

David, P. A. (1998). Common agency contracting and the emergence of" open science" institutions. The american economic review, 88(2), 15-21.

Chan, L., Okune, A., Hillyer, R., Posada, A., & Albornoz, D. (2019). Contextualizing openness: Situating open science University of Ottawa Press

Brown, J. S., & Duguid, P. (2001). Knowledge and organization: A social-practice perspective. Organization Science, 12(2), 198–213. https://doi.org/10.1287/orsc.12.2.198.10116

Author Response

Please see the attached file for response to the wise comments and constructive suggestions from the reviewer.

Please also find a manuscript with track changes to make it easier to identify the changes. In general, substantial parts of the manuscript have been rewritten in accordance with the suggestions by both reviewers.

Author Response File: Author Response.pdf

Reviewer 3 Report

A typology of unintended or negative consequences of open science practices could be useful. However, there are several errors in the author’s framing of the question that fundamentally undermine his conclusions.

Most importantly to a reader not familiar with this literature, it is unclear why attributes of open science (many of which are attributes of all science) are epistemological challenges and/or have ethical dimensions in the first place. If epistemology is “what distinguishes justified belief from opinion” then that is certainly relevant to consumers of knowledge, especially with increased access thanks to the internet and open science. But how it relates to the production of knowledge is less evident. Expanded access to knowledge or data does not per se imply there would be any change in researchers’ ability to apply established methods of inquiry to distinguish justified belief from opinion.

Introducing “ethical implications” implies agency on the part of actors in the system, actors who could foresee and address negative ethical consequences. Science and knowledge production involves many such actors who are concerned with ethical behavior, and with building incentives and other mechanisms to increase ethical behavior and dis-incentivize unethical behavior. This paper discusses ethical consequences at such a high level, isolated from specifics about the actors who could influence them, and also from specifics as to how open science is different than traditional science, that it is hard to understand what is even meant by ethical consequences of the author’s observations about open science.

Many of the examined attributes of open science are shared by science in general. Thus, unrealized benefits or “negative effects” are not unique to open science. For example, “undermining quality (lack of replicability, bias, polarization), lack of norms, institutions, and regulations to scientific activity” are all characteristic of both traditional and open science. Thus the conclusion that “we need to find new ways to verify OS elements and results and develop new institutions inciting new norms and regulations” is non-sensical because it does not relate specifically to aspects of open science that are distinct from traditional science. A statement like “The policies behind OS are not neutral, but evoke key actors’ profound values, beliefs and paradigmatic assumptions to promote their interests and institutional demands” begs the question, How is that different from traditional science? It is mostly not different. In fact, it could be argued (but isn’t here) that open access publishing weakens institutional (publisher) power.

It is a logical fallacy to characterize the failure to realize the promised advantages of open science as “negative effects” of open science. Benefits that did not occur are not negative effects. The more useful questions are, How can open science better realize the advantages that are claimed for it? How can negative consequences of open science practices be mitigated? Recognizing that the author may not be interested in those questions, it could be useful to delve into the specifics of the one area where possible epistemological and ethical issues pertain according to the author’s analysis (data procurement, knowledge quality and resulting impact on trust).

There are many statements throughout that repeat this fallacy, e.g., “openness does not assure quality.” “There are valid arguments that some types of data and information are kept secret. Accordingly, OS may not become completely epistemically open, democratic, and just, having ethical implications for those who produce and use the knowledge.”

Open Science (or traditional science) does not have an “epistemic governance structure.” Science is a complex system with what could be considered many epistemic structures, some of which may have formal governance (research funding allocations, university hiring, publishing in journals) but others are governed by professional norms (data collection, research practices).

Bottom of p. 6 states (without citation) that “OS may not reduce the chance of fabrication, falsification, and plagiary in data provision and procurement” and “OS makes it easier to verify and detect errors and flaws in data analysis…” But the author then concludes (without citation), “Data procurement remains to be a source of errors and flaws compared to traditional science.”

In Discussion, it is stated “OS violates or undermines the concept of copyright.” There is a citation for that claim, but it is unclear and illogical.

All of the “challenges” of OS in the final paragraph are also challenges of traditional science.

There are a lot of typos and missing or incorrect words that add to the confusion of reading this paper. E.g., “post-normal literacy,” “collaborate OS,” STS.

Round 2

Reviewer 2 Report

Thank you very much for revising your manuscript. The more structured approach towards your literature search certainly improves the quality of your manuscript and I very much appreciate the effort you have invested into the revision.

On a general level, I continue to think that the idea you are pursuing is very important and that an overview of the challenges related Open Science practices along the scientific knowledge production process would be valuable for multiple fields interested in openness in science in general. But I also still think that the realization of the idea does not live up to its full potential. There is no solid theoretical foundation laid out to contextualize your approach to research, there are opinions expressed that are not backed-up by literature nor clearly deducted from theory. You say that you would like to be broad, but any researcher is influenced by her/his own background, which is why we need scientific methods. For example, how you approach prior literature on that matter: you do not equally consider literature streams like Responsible Research and Innovation (RRI), Open Innovation in Science (OIS), Citizen or Crowd Science, public engagement, technology transfer, or third mission activities, which all discuss challenges related to openness in the scientific knowledge production process but may not use the term “Open Science” specifically). As a remedy and to make assumptions more explicit, our research in the social sciences should as well be guided by theories allowing for conceptual clarity. This helps to contextualize your research and offers connections to future work while avoiding to getting lost in the void.

Anyhow, leaving these high-level comments aside, I have the following, more specific comments:

Definition: Thank you for making it clearer. If I understand you correctly, you define the epistemological challenges of OS as such challenges that refer specifically to the knowledge production occurring while implementing OS practices? I am putting a question mark here, because I concluded this based on your state in the discussion section that “This article has focused on the epistemological challenges, and not problems with OS in general.” If this is indeed the case, please specify how you define “scientific knowledge production”. This may seem simple, but it has strong implications for the steps of the research process you are considering. For example, is publishing an article still part of the knowledge production process? This is not a simple question and there is not only one answer to that question but many. So, I would urge you to be very specific because it will also allow you to structure your insights more clearly. Similarly, what do you mean by ethical implications?
In Table 1 you introduce different “OS school of thought” but you don’t discuss these in the text. I see that they are partly derived from Fecher & Friesike (2014), but what is the “quality improvement school”? Do these different “school of thought” inform your research? I am asking because you prominently place this overview in the introduction which creates expectations for their role in your research piece. What is the value of the table for answering your research question? Do you need it?
It is confusing that most of your introduction praises OS, but then only in the last couple of sentences you point towards the challenges (which is the main story of your article). My suggestion would be to focus earlier on the challenges and why it is relevant to understand these in a systematic way.
I very much appreciate your more structured approach towards searching the literature and including a section on Methods. I also appreciate a lot that you reflect upon your approach in the limitations. I ran your search string and ended up with 21.600 articles. While I see that several articles are included, there are others missing that specifically address challenges related to OS such as Hensel (2021) or Garcia-Alvarez & Lopez Sintas (2012) to name just two. What I am pointing at is that google scholar is fantastic for many things including to find literature, but it is not ideal for structured literature reviews given its lack of reproducibility. I would recommend (for this and/or for future literature reviews) to use (a combination of) other databases such as Web of Science, EBSCO, etc. This also allows you to select more specific search criteria (e.g., exclude grey literature).
Please provide more information on how many articles were excluded at each step of your literature review. Again, this serves the purpose of making the process more reproducible.
Readers of literature reviews always appreciate a table of the articles incl. core statements, in your case, the specific challenges, the ethical consequence (if mentioned in the article), maybe also the specific OS practice. I know it is a lot of work, but the value of such an overview would be substantial (and I guess you have already done much of the work anyways already). Even more, I would suggest to combine this new table with Table 3 (see comment below for more details).
The three categories of challenges suggested by the article are somewhat difficult because they are not mutually exclusive. For example, the looping effects and the open display of knowledge described in section 3.1. could also be classified as negative outcomes. Vice versa, the lack of replicability could also be associated with 3.1 instead of 3.2. Please either provide arguments for choosing these three categories or you could consider an alternative (see next comment).
While you say on p. 8 that you do not consider the different practices associated with Open Science (such as Open access, etc.), you distinguish between them indirectly in the third category of your challenges (section 3.3.). I think that there is value in distinguishing them because the challenges associated with INPUT knowledge flows (e.g., re-use of data; data collection) may be very different to OUTPUT knowledge flows (e.g., OA publishing). Using a (simplified) research process to structure your findings (as you have done in Table 3) could be a nice and insightful way to structure your results and identify overarching patterns. Hence, I would suggest to combine table 3 with the literature you have identified in your literature search, so that it becomes visible how the literature informs Table 3.
There is not a single reference in section 3.3. This is not per se critical, but it is very critical given the fact that section 3.3. is supposed to summarize challenges related to category 3 identified in the literature.
As a consequence of the last four comments, here is what I would do if it was my paper: I would produce the structured overview of the articles reviewed and their epistemological challenges, the research step, the OS practice, ethical implications, and (if applicable) remedies. Then I would structure the challenges (and ethical implications if you find something in this regard) along the research process, maybe a simplified version with fewer steps compared to your current table 3 (e.g., conceptualization, exploration/testing, and documentation). Based on this, you might be able to see emerging patterns and derive propositions or research gaps. But let me emphasize, that this is what I would do and that I don’t expect you to do this of course. Please consider it a well-intended suggestion.
However, if you would not distinguish the different OS practices, I am not sure I would start the abstract by distinguishing them as it may create wrong expectations.
I appreciate that you refer more to the “what can we do” column of table 2 in the text. I think it would be more insightful, however, if you could be more precise in your suggestion. How exactly would you change the governance structure? How should the funding requirements look like to avoid biased knowledge production? How exactly should the infrastructure be improved?
Thank you for including limitations and being very frank about them.

Further references I would recommend (please don’t feel obliged to cite them, but they might be interesting for you for various reasons):

Beck, S. et al., (2022). The Open Innovation in Science research field: a collaborative conceptualisation approach. Industry and Innovation, 29, 136 - 185.

Chan, L., Okune, A., Hillyer, R., Posada, A., & Albornoz, D. (2019). Contextualizing openness: Situating open science University of Ottawa Press.

Coles, N.A., Hamlin, J.K., Sullivan, L.L., Parker, T.H., & Altschul, D.M. (2022). Build up big-team science. Nature, 601 7894, 505-507 .

Fecher, B., S. Friesike, M. Hebing, and S. Linek. 2017. “A Reputation Economy: How Individual Reward Considerations Trump Systemic Arguments for Open Access to Data.” Palgrave Communications 3 (1): 1–10.

García-Álvarez, E., & López Sintas, J. (2012). Open science, e-science and the new technologies: Challenges and old problems in qualitative research in the social sciences. Intangible capital, 8(3), 497-519.

Hensel, P. G. (2021). Dissecting the tension of open science standards implementation in management and organization journals. Accountability in Research, 1-26.

Vicente-Sáez, R., and C. Martínez-Fuentes. 2018. “Open Science Now: A Systematic Literature Review for an Integrated Definition.” Journal of Business Research 88 (7): 428–436.

Article Menu

Open Science Knowledge Production: Addressing Epistemological Challenges and Ethical Implications

Further Information

Guidelines

MDPI Initiatives

Follow MDPI