Reproducibility: A Researcher-Centered Definition
Round 1
Reviewer 1 Report
The manuscript's topic is of significance for open science, as transparent and accessible knowledge is more and more shared and developed through collaborative networks in different disciplines. The subject is treated with detail and in considerable, almost surprising length.
In the eyes of the Reviewer who has gained experience in collaborative networks of engineering projects and has met similar problems of terminology and re-use of data, the treatment of the topic is somewhat tedious and therefore dissatisfying. The text seems not well structured: no figures, no tables or lists, no collection of different proposed phrases from the literature review (like 'established', 'incorrectly used', 'proposed', 'proposed here for the first time' etc.) are used to break open the lengthy text.
1) Major issues:
The term 'researcher-centered' is extensively used, implying a subjective perspective - from an amateur or professional or both? Wouldn't 'research-centered' be more appropriate?
Early in the text, a new definition is proposed, based on the phrases 'efficient and effective interaction', 'ease of access', 'completeness'. The latter two phrases are explained only in the Discussion, the former phrase is never explained.
At the end of Section 4.3 it is stated that the 'central anchor point' to the novel definition of reproducibility is that accessibility and resource completeness are dependent on the purpose of resource use. In reality, however, accessibility and completeness are often given first, framing or even restricting the later purpose of the re-use of data. Maybe the authors could elaborate on that arguable contradiction.
What are the practical implications of the proposed definition?
Bottom-up practices and top-down expectations are never again mentioned outside the Abstract.
For ease of reading and highlighting the core findings, it is proposed to rename Section 4.4. into 'Conclusion' and edit the text accordingly.
2) Use of language:
major ongoing (?) shifts
This requirements research (?) (line 236)
may be impacted -> affected
science data -> research data
Author Response
We thank Reviewer 1 for the effort in reviewing our manuscript and for providing thoughtful suggestions. Below, we react to the suggestions provided by Reviewer 1:
The term 'researcher-centered' is extensively used, implying a subjective perspective - from an amateur or professional or both? Wouldn't 'research-centered' be more appropriate?
We acknowledge that we need to better introduce terms that stem from fields like Human-Computer Interaction (HCI), interaction design, and user experience research. Here, the terms „human-centered“ and „user-centered“ are established to describe research and design approaches that place the user in the center of any activity (i.e., mapping requirements, designing frameworks, and evaluating those frameworks). Within this context, we refer to "researcher-centered" as activities and definitions that focus on the researchers’ needs and socio-technical frameworks. We added a subsection to the introduction to describe those concepts. Here, we also referred to some example reports that describe researcher-centered perspectives or studies.
Early in the text, a new definition is proposed, based on the phrases 'efficient and effective interaction', 'ease of access', 'completeness'. The latter two phrases are explained only in the Discussion, the former phrase is never explained.
We note that while we discuss „efficient and effective interaction“ in Section 4.3, a clearer presentation is needed to inform about the individual meanings and implications of „efficiency“ and „effectiveness“. We added such details in 4.3.
At the end of Section 4.3 it is stated that the 'central anchor point' to the novel definition of reproducibility is that accessibility and resource completeness are dependent on the purpose of resource use. In reality, however, accessibility and completeness are often given first, framing or even restricting the later purpose of the re-use of data. Maybe the authors could elaborate on that arguable contradiction.
We agree with this from a systemic point of view. However, our empirical research with scientists showed that researchers demonstrate higher tolerance towards investing more time into repeating a data process to make up for inaccessibility if this effort is proportional to the expected outcome. The key statement of this paragraph is intended to stress that, in contrast, researchers are not willing to spend much effort on overcoming reuse barriers if they do not expect a tangible benefit from such actions. This is echoed in the example of adapting plots for presentations. We rephrased parts of Section 4.3 to address this perceived contradiction.
What are the practical implications of the proposed definition? Bottom-up practices and top-down expectations are never again mentioned outside the Abstract.
We expanded Section 4.4 „Adoption of Researcher-Centered Definitions“ to provide better access to our findings and the proposed definition. We also reframed Section 4.4 into the Conclusion section (see next item).
For ease of reading and highlighting the core findings, it is proposed to rename Section 4.4. into 'Conclusion' and edit the text accordingly.
As suggested, we created a dedicated ‚Conclusion‘ section that expands Section 4.4.
Language
We made changes that correspond to the reviewer’s remarks and suggestions.
Reviewer 2 Report
This paper describes a researcher-centered definition of reproducibility in current open data movements over a scientific community in the digital age. It is a very important topic and have a good contribution to start a discussion. However, it lacks background or context in the title and introduction part, too difficult to understand a merit of this work in the context of this journal and potential readers (not only scientists but also engineers).
Major Comments:
1) Provide sufficient title and introductory background for this journal.
“MTI is inherently multi- and interdisciplinary, covering both fundamental and applied research dealing with the design, analysis, evaluation, and use of technologies that support multimodal interaction and their impact.” (https://www.mdpi.com/journal/mti/about)
There are a few topics on multimodal interaction, although well written about socio-technical research about the scientific community in the HCI research context.
2) Provide a clear target, limitations or issues of this research.
I strongly agree that “Reproducibility” is a very important requirement of scientific research.
There are many types of researchers, physician, biologist, anthropologist, engineer, inventor, philosopher, artist, sociologist, economist, etc. Some excellent researches have produced new concepts or new philosophy or new design or new thinking style, without no evidence data or very difficult to share the resources. Therefore, your description must take care of the whole process of research work or issues of your research ideas in not scientific community.
How about computer science? Donald Knuth believes artistic aspect is very important to use computing machinery. Herbert Simon thinks artificial science (including design science) is different from natural science.
3) Add discussion of ethical sharing resources
Collecting data of natural science research will be easily shared but those data of social science research will be considered human right protection. Today, research of Big data is required to keep an ethical manner [1].
[1] Zook M, Barocas S, boyd d, Crawford K, Keller E, Gangadharan SP, et al. (2017) Ten simple rules for responsible big data research. PLoS Comput Biol 13(3): e1005399. https://doi.org/10.1371/journal.pcbi.1005399 //
Author Response
We thank Reviewer 2 for the effort in reviewing our manuscript and for providing thoughtful suggestions. Below, we react to the three key suggestions provided by Reviewer 2:
1) Provide sufficient title and introductory background for this journal.
Based on this suggestion, we recognize that we need to provide more details related to the terms established in Human-Computer Interaction (HCI), to support the understanding of readers with a different background. We made corresponding additions to the text. In particular, we added a sub-section to the introduction that focuses on the HCI perspective. We note that missing references to multimodal interaction are partly grounded in the journal’s reorientation towards a wider coverage of topics relevant to the scientific community. In fact, we submitted this paper to a special issue on Reproducible Science that does not refer to multi-modal interaction in any form: https://www.mdpi.com/journal/mti/special_issues/Reproducible_Science
2)Provide a clear target, limitations or issues of this research.
We thank the reviewer for arguing for a more diverse view on reproducibility and data sharing. We added a discussion on resource diversity and needs for reuse that go beyond traditional forms of scientific resources. We also added references to creative coding as an example field in which the lines between computer science contributions and design become blurry.
3) Add discussion of ethical sharing resources
We added a subsection dedicated to the impact of privacy on sharing practices.
Round 2
Reviewer 2 Report
The revised paper answers my questions clearly, especially describes Human Computer Interaction context for other domain researchers. In addition, most researchers with digital technology will have relate to this described concept beyond the HCI context.
