**3. Results**

## *3.1. Results: Identification of KSAs*

As noted, a steward is someone who "will creatively generate new knowledge, critically conserve valuable and useful ideas, and responsibly transform those understandings through writing, teaching, and application". Based on this definition, with consideration of its utility *outside* of doctoral education (i.e., earlier as well as for those who will not pursue a doctorate) as well as within it, the KSAs for stewardship were determined to be:


Seven of the eight KSAs (all except the first item in the list above) were derived directly from the original definition of stewardship. Following the two-phase approach shown in Figure S1 in the Supplementary Materials, while the PLDs for those seven KSAs were being articulated, it became clear that knowing when to exhibit which aspect of stewardship is, itself, something that needs to be taught and practiced explicitly. In the words of the CID, stewardship encompasses both a set of roles and skills, and a set of principles: "the former ensure competence, and the latter provide the moral compass" [1] (p. 9). Thus, we determined that an additional KSA was needed to describe the responsibility to *recognize when* these behaviors need to be applied or modeled, which can represent a concrete and observable version of a "moral compass" (see also [27], p. xxi). We have called this KSA "requisite knowledge/situational awareness" to capture the attention that would be given to standards of professional practice (if they exist) during education or training, or when orienting new employees in the workplace. The applications of the other KSAs of stewardship are contingent upon this situational knowledge. An individual who learns and grows all the other KSAs, but cannot recognize *when to use* them, is less likely to act in a stewardly manner when it is needed. As an individual becomes increasingly inculcated into the habits of mind and practice of a discipline or profession, their abilities to recognize situations in which stewardship is needed should similarly increase. Formally including this KSA in a curriculum is essential for wider exposure to, and greater likelihood of demonstrating, stewardly behaviors.

One of the defining features of the scholarship is the generation of new knowledge; however, modern scientific scholarship, in and outside of academia, can also emphasize new methods or techniques. Thus, we added "generate new methods", which also includes the development of software; computational, mathematical, and statistical techniques, as well as new methods for cross-disciplinary work. Augmenting the KSA this way both broadens the scope of behaviors to which a stewardly approach is important and specifies professional activities (where techniques and methods are developed for industry and non-academic venues) as meaningful sources of evidence of stewardship.

We also broke one of the features of the steward into two separate KSAs: "the critical conservation of valuable and useful ideas" was separated into "critical evaluation of extant knowledge" (KSA 2) and "conserve ideas and justify rejection in non-conservation of ideas" (KSA 3). In fact, critical evaluation of extant knowledge is essential to both the creation of new knowledge and to the conservation (or non-conservation) of the ideas of the discipline. For the scholar, both of these KSAs invariably require a critical evaluation of extant knowledge. However, for the independent practitioner who is not primarily a scholar, conservation (or non-conservation) of ideas may not require a critical evaluation of the literature supporting the idea or an intention to generate new knowledge. One current (2017) example is the data scientist who creates a new algorithm and, recognizing that the dominant or prevailing computational paradigm creates bias in the algorithm's output (e.g., [28]), rejects that paradigm and seeks to create a di fferent algorithm that does not have that bias. Including two separate KSAs instead of one makes these features of stewardly practice more widely applicable. The di fferences in performance of these separated KSAs may be less obvious for scholars than for others, but the fact that di fferent performance level descriptors were generated for each KSA a ffirms the suggested separation. The distinction was further reinforced by recognizing fundamental di fferences in the two KSAs. Critical evaluation entails an obligation to contribute to the vetting (e.g., by performing peer review) that is essential to promote the vitality and rigor of the literature or knowledge base. Conservation/non-conservation is distinguished with a dimension of understanding *the process* of vetting, and potential influences on vetting by cultural as well as extra-disciplinary forces.

Similarly, we took the "transformation" feature of the steward ("responsibly transform those understandings through writing, teaching, and application") and divided it into four fundamental KSAs, namely: responsibly write; responsibly teach/mentor/model; responsibly apply disciplinary knowledge; and responsibly communicate. In particular, many faculty members are stewardly specifically because they responsibly teach and apply disciplinary knowledge (but do not primarily write scholarly papers), while many non-faculty stewards (e.g., those in industry) responsibly teach informally (e.g., modeling professional practice and mentoring junior members of a team or work group) and communicate outside of scholarly venues. We distinguished "communication" from both teaching and scholarly writing, because some discipline-specific communication (with collaborators, team members, or the public) is *not* considered scholarly, but communication outside of the discipline must be as stewardly as that within the discipline. All of these are key features of stewardly practice, although all practitioners may not have opportunities to develop or demonstrate all four. By separating the components of this stewardship feature, we sought to make each learnable and improvable throughout the development of both scholars and professionals in any discipline.

Thus, these eight KSAs represent the definition of stewardship with subtle refinements that make stewardship achievable and demonstrable by a wider range of practitioners and professionals across disciplines.

## *3.2. Results: Trajectory definition*

As shown in Table 1, four stages of development were articulated based on the European guild structure.

**Novice**: The novice is just beginning to engage with the discipline. This individual is focused on their own performance of any given KSA, which tends to be at Bloom's levels of cognitive complexity 1–2 (understand and summarize). They do not recognize that, or act as if, failures to act in a stewardly manner have ramifications beyond themselves. The novice stage represents the individual embedded in the acquisition of discipline-specific content. This could be an undergraduate declaring the major or an early-stage graduate student.

**Apprentice:** The apprentice is actively engaged in study of the profession or discipline and developing the full range of Bloom's cognitive abilities. She is developing the capacity to practice independently, but has not ye<sup>t</sup> demonstrated ability qualification to do so. However, compared to the novice, her work and KSA performance exhibits greater reflection, and thus, awareness of her own limitations. Additionally, by learning the "tacit responsibility for the quality and integrity of their own work and that of colleagues", and the "responsibility to the larger public for the standards of practice associated with the profession" that Golde and Walker (2006) [1] assume "all professionals" have, the apprentice is aware that professional and disciplinary independence will require stewardship. This stage, which can be the longest period in one's formation, might describe the advanced undergraduate in a major or the later graduate student who has articulated research problems (or their thesis) with some support. However, the stages are not time-dependent, so any individual at any point in a career could be or become an apprentice steward.

**Journeyman**: The journeyman is an independent scholar or practitioner—a steward of the discipline. Depending on the field, this individual could be a doctoral student, an independent scholar (with or without a PhD in the field), or baccalaureate holder who is prepared for independent work in a profession or practice. The journeyman steward ("steward") is uniformly stewardly in their interactions with others in the disciplinary or professional community. They may seek new opportunities to reinforce less-well developed skills. Performance is reflective, and includes analysis and synthesis of their experience with their knowledge (Bloom's 4–6).

**Master**: The Master steward is recognized by evidence and consensus as one who teaches effectively. The distinction between the journeyman and the Master is the Master's demonstrated ability to teach e ffectively, comprising evidence of successful diagnosis and remediation of the thinking or work of practitioners at earlier stages. (This characterizes performance at the master level in every Mastery Rubric that includes this level. Many independent practitioners (journeymen) have responsibilities to mentor or instruct, but have not accumulated evidence of qualification and the focus on the diagnosis and remediation of challenges that are encountered earlier in development, that distinguishes the Master.) The Master steward is therefore an expert in the KSAs themselves (journeyman) and also as someone to whom apprenticeship in stewardship can be entrusted. Evidence of successful diagnosis and remediation of earlier-stage performance of the KSAs, rather than a listing of the jobs or funding that one's trainees/students have gone on to obtain, is an essential feature of master level stewards. This can include the development and evaluation of assessments that are specific to the KSAs and the progressive evolution of their performance, plus methods by which these cognitive skills can be elicited by, or developed in, those who are less-expert. These would be most clear from a formal educational context but are also important in the workplace: rather than focusing on student work, Master stewards in the workplace might focus on transparency in promotional processes and supporting the creation of evidence that mentees and junior practitioners need in order to be recognized as "developing professionals" who are making progress towards career goals. Simply *modeling* stewardship is not su fficient to train others to become stewardly or to promote active, critical, engagemen<sup>t</sup> in the profession or the discipline; thus, achieving the master level requires evidence of the ability to support the development of these KSAs in those whose stewardly behaviors and attitudes are still in development.

## *3.3. Results: Performance Level Descriptors (PLDs)*

In the academic or workplace context, the novice steward deals mostly with facts and pre-defined problems, but is also focused on their own actions and is only starting to learn about professional practice standards, such as the fact that standards exist (and why). Novices lack awareness of many or all of the dimensions of stewardship and professionalism, and also of their own development or place in the continuum. In many fields, this corresponds to performance at Bloom's levels 1–2 (understand and summarize), moving towards developing levels 3 (apply) and 4 (analyze/predict) on the stewardship KSAs. The PLDs for the novice are consistently representing these Bloom's levels for all KSAs. Depending on motivation as well as structural support, an individual might spend 1–2 years in this stage (e.g., first two years of college before declaring a major), or longer if they have not begun to identify with a profession or discipline.

The apprentice steward is someone who is actively learning about practice standards as they begin to align themselves and their emerging professional identity with these standards. Their abilities to engage with stewardship KSAs are improving, so they can engage with less-structured problems and less sca ffolding. Apprentices perform the KSAs at Bloom's levels 3–4, and because they are learning more about standards of practice for their domain, they would be developing abilities at Bloom's levels 5–6 (evaluate and synthesize) and also demonstrating a growing awareness of their own limitations and opportunities for growth (i.e., developing metacognitive skills). The PLDs for the apprentice represent more confidence with Bloom's levels 3–4 than the novice would be expected to demonstrate for all KSAs, but for those whose sense of professional identity is actively being shaped, engagemen<sup>t</sup> with less structured challenges—requiring Bloom's levels 5–6—would be sought and practiced. In an academic context, this individual collects evidence of their apprenticeship of stewardly behaviors and habits of mind over the period of time "in training" for a job or role. For example, this might be the time spent "in the major" for undergraduates, a master's program, or the majority of the time in a doctoral program. In the workplace, individuals may train for, and learn to identify with, new jobs. However, once an individual leaves the apprentice steward performance level, they will be adaptable to new contexts and might enter this stage briefly, as needed, for refinements to professional identities they are on the way to forming.

The journeyman is a steward of the discipline. They have concrete awareness of professional practice standards and understand how these apply to themselves, to others, and to interactions within and outside of the profession or discipline. The journeyman steward performs the KSAs with understanding, analysis, and synthesis of their experience with their knowledge, functioning at the highest Bloom's levels as they perform all the KSAs that comprise stewardship in their working life. In an academic context, this individual is recognizable as someone to whom the integrity of the field can be entrusted. In the workplace, this individual is recognized as someone whose integrity, and whose commitment to their field or profession, is visibly and concretely demonstrated. One can be considered a journeyman steward when one has evidence of this level of performance on all of the stewardship KSAs that are relevant for their practice; if new dimensions of practice are added (e.g., adding responsibility for junior team members or supervising duties), development of stewardly attributes of those new professional responsibilities can follow the articulated trajectory.

The Master is confirmed, by evidence, as both independent in their performance of the KSAs themselves and also as someone to whom apprenticeship can be entrusted. Master level performance of all stewardship KSAs includes understanding, analysis and synthesis, *and* an understanding of mechanisms by which these cognitive skills can be elicited by those less-expert stewards within their discipline. Evidence of master level performance in stewardship comes from diagnosis (the identification of weaknesses in KSA performance) and remediation (the recommendation of methods to address those weaknesses in the KSAs), including the development and evaluation of assessments that are specific to the KSAs and their progressive evolution. The Master can train others to begin to be, and to be, steward*ly* (novice/apprentice), or to be (journeyman) stewards; they may also train new Master stewards. This individual directly and indirectly supports the development and recognition of stewardly behaviors in those he/she works with whether in academic or workplace contexts. Examples of evidence of master level achievement include the use of individual learning plans that incorporate the development of stewardly behaviors in academic settings, or by encouraging continuing professional development in the workplace. In addition to the explicit "instructional" performance by a Master steward, the Master is also entrusted to support a stewardly context in which the apprentice is trained/prepared for practice. For example, in the academic setting, as the steward creates new knowledge, they understand that the goal of advancing knowledge diverges from the worst characteristics of the "publish or perish" culture, which does not prioritize the discipline and may actually be detrimental to the discipline *and* to public trust in the academic enterprise more generally (see [29]). Outside of the academic setting, where the "publish or perish" attitude may seem absent, there may still be a "rush to results", or emphasis on short-term tasks that can ultimately weaken the discipline or profession. The journeyman steward seeks to ensure that their own work prioritizes the discipline or profession, while the Master steward goes further, seeking to support a culture where the integrity of the discipline is not undermined by pressures such as publication or pushing to complete a project simply for completions' sake or because publication and completion are deliverables.

With these features and characteristics in mind, the specific PLDs were crafted for the KSAs and are presented in the Mastery Rubric for Stewardship in Table 1.

## *3.4. Results: Validity Evidence for KSAs of the MR-S*

The Mastery Rubric supports professionals, practitioners, and scholars envisioning themselves as stewards, "committed to the foundation ("heart and essence") of one's field, but also to thoughtful and innovative forward momentum, and development of the future of one's field". To test more empirically whether stewardship can be extended beyond the scholarly disciplinary domain into professional practice (i.e., to test the validity of this extension), we applied the degrees of freedom analysis method to study alignment of the stewardship KSAs with the guidelines for professional conduct—which guide professional and disciplinary practice—in three disparate domains, History, Statistics and Data Science, and Neuroscience. For history, the guidelines are publicly available in narrative form. The key principles for Professional Practice of History were extracted and summarized for the case analysis. For the Professional Practice of Statistics and Data Science, and Neuroscience, the publicly available text for each standard or principle was copied directly from the then-current website (in July 2018). If there is alignment between the Guideline principle and a stewardship KSA, we indicate this with an asterisk (\*)—we made no attempt to quantify "how well-aligned" a KSA and Guideline principle is, partly because this will depend on individual training programs' (in schools or the workplace) specific emphasis on either stewardship or their professional standards. Our indications of alignment are, at their most foundational, signals that (or whether) stewardship is consistent with professional practice guidelines.

## **Case 1. American Historical Association (AHA) Statement on Standards of Professional Conduct (2018) [30]**

First published in 1987 and most recently revised in 2018, the AHA Standards of Professional conduct recognize that the discipline of history does not belong exclusively to historians. It is an area of "shared human fascination" that is accessible to and produced by people outside of the narrow body of professionals. The Standards thus serve not just as a guide to proper historical inquiry, but also as a way of distinguishing professional historians from others. Professionals are defined by "a self-conscious identification with a community of historians who are collectively engaged in investigating and interpreting the past as a matter of disciplined learned practice". The historian's task begins with discovery. In the words of the AHA, "Scholarship—the discovery, exchange, interpretation, and presentation of information about the past—is basic to the professional practice of history". From here, seven principles describe the "discipline of learned practice":


interpretations express a point of view, and that no mortal mind can ever aspire to omniscience. Because the record of the past is so fragmentary, absolute historical knowledge is denied us."


Table 2 presents the Degrees of Freedom Analysis matrix to examine the alignment between the Principles of the AHA Guidelines (1–7; columns) and the KSAs of the Steward (Rows). Alignment is indicated with an asterisk.


**Table 2.** Alignment of seven AHA Guideline Principles (columns) with Stewardship KSAs (rows).

Table 2 shows that all of the MR-S KSAs are consistent with at least three of the AHA Practice Standard elements. The KSAs of responsible writing, communicating, and teaching/mentoring/modeling have the strongest alignment (with 5–7 AHA standards); this is not surprising given the fact that History is a profession that is practiced mostly within the academy. According to a study of the 10-year cohort of History PhD recipients in the US published in 2018 [31], roughly a third of PhD graduates between 2004 and 2013 work outside higher education of any sort, while just over half hold tenure track positions in four-year colleges/universities.

When it comes to historians working in non-academic roles, the question of their demonstrating stewardship is more complex. There are a few popular historians, such as David McCullah and Ken Burns, whose work serves an important role in promoting historical literacy within our society. There are also public historians, who work in places such as museums, and institutional historians, who serve on sta ff at places ranging from the US senate to small organizations and associations. Historians also serve an important role in think tanks or in agencies that maintain or establish archives; including the European Molecular Biology Laboratory (which has outposts, associates, and partner organizations hosting cutting edge science all over the world). These roles, though not in the academy, certainly allow one to bring their historical knowledge, skills, and abilities to bear in their areas of service; therefore, historians both in and outside academia are expected to follow the AHA Practice Standards. The "Alignment" column in Table 2 shows that training the historian in the stewardship KSAs will create opportunities to demonstrate 3–7 of the seven AHA Standards; training individuals to follow the AHA Standards in their practice will create opportunities to demonstrate 5–7 of the eight KSAs of stewardship. The strong alignment of each AHA Practice Standard with the stewardship KSAs supports the claim that stewardship can be exercised by historical professionals, whether or not they complete a PhD.

## **Case 2. The American Statistical Association (ASA) Ethical Guidelines for Statistical Practice (2018) [32]**

The Ethical Guidelines were first endorsed by the ASA Board in 1995 and the latest revision was endorsed by the Board in 2018. These Guidelines include seven general principles and 52 specific elements. The ASA Ethical Guidelines for Statistical Practice describe the professional habits of all practitioners in statistics and data science; practice in governmen<sup>t</sup> (e.g., Bureau of Labor Statistics; Census), industry (e.g., business; pharmaceutical; biomedical), social science fields (decision making/marketing; industrial/organizational), and scholarship.

## **A. Professional Integrity and Accountability**

The ethical statistician uses methodology and data that are relevant and appropriate, without favoritism or prejudice, and in a manner intended to produce valid, interpretable, and reproducible results. The ethical statistician does not knowingly accept work for which he/she is not su fficiently qualified, is honest with the client about any limitation of expertise, and consults other statisticians when necessary or in doubt. It is essential that statisticians treat others with respect.

## **B. Integrity of data and methods**

The ethical statistician is candid about any known or suspected limitations, defects, or biases in the data that may impact the integrity or reliability of the statistical analysis. Objective and valid interpretation of the results requires that the underlying analysis recognizes and acknowledges the degree of reliability and integrity of the data.

## **C. Responsibilities to Science**/**Public**/**Funder**/**Client**

The ethical statistician supports valid inferences, transparency, and good science in general, keeping the interests of the public, funder, client, or customer in mind (as well as professional colleagues, patients, the public, and the scientific community).

## **D. Responsibilities to Research Subjects**

The ethical statistician protects and respects the rights and interests of human and animal subjects at all stages of their involvement in a project. This includes respondents to the census or to surveys, those whose data are contained in administrative records, and subjects of physically or psychologically invasive research.

## **E. Responsibilities to Research Team Colleagues**

Science and statistical practice are often conducted in teams made up of professionals with di fferent professional standards. The statistician must know how to work ethically in this environment.

## **F. Responsibilities to Other Statisticians or Statistics Practitioners**

The practice of statistics requires consideration of the entire range of possible explanations for observed phenomena, and distinct observers drawing on their own unique sets of experiences can arrive at di fferent and potentially diverging judgments about the plausibility of di fferent explanations. Even in adversarial settings, discourse tends to be most successful when statisticians treat one another with mutual respect and focus on scientific principles, methodology and the substance of data interpretations.

## **G. Responsibilities Regarding Allegations of Misconduct**

The ethical statistician understands the di fferences between questionable statistical, scientific, or professional practices and practices that constitute misconduct. The ethical statistician avoids all of the above and knows how each should be handled.

An 8th Guideline Principle is specific for *employers*:

**H. Responsibilities of Employers, Including Organizations, Individuals, Attorneys, or Other Clients Employing Statistical Practitioners**

Those employing any person to analyze data are implicitly relying on the profession's reputation for objectivity. However, this creates an obligation on the part of the employer to understand and respect statisticians' obligation of objectivity.

Table 3 presents the Degrees of Freedom Analysis matrix to examine the alignment between the Principles of the ASA Guidelines relating to the practitioner (A–G, columns) and the KSAs of the steward (rows). Alignment is indicated with an asterisk.


**Table 3.** Alignment of ASA Professional Statistics Guidelines (columns) with Stewardship KSAs (rows).

Table 3 shows that professional behaviors that are consistent with the ASA Guidelines (columns) can also generate evidence of stewardship of the discipline or profession (rows). The row marginals show that between three and seven of these seven core ASA Ethical Guideline principles are aligned with each stewardship dimension. Thus, although stewardship was originally conceptualized for PhD level practitioners whose primary objective is scholarship, it can also generally support practitioners who are not "scholars first and foremost", which describes many, if not most, practicing statisticians and data scientists. Statisticians and practitioners in quantitative sciences (to whom the ASA Ethical Guidelines pertain, see [32–34]) practice in every discipline, and not solely as scholars; thus, finding alignment between stewardship dimensions and ASA professional practice guideline elements strongly supports the claim that stewardship can be relevant for those who are not principally scholars. Master's level preparation is also sufficient for "professional preparation" as a statistician, and with new undergraduate degree programs in data science being developed across the U.S., it is possible that baccalaureate preparation may suffice for professional preparation in some cases. With the role of data increasing in priority in so many science and technology fields, it is essential to engage all those preparing for statistical practice in active consideration of how they may be as stewardly as possible.

Not every statistician or data scientist who is (or wishes to be) stewardly will have opportunities to teach or create new knowledge; therefore, these "typical" stewardly behaviors cannot be required of every steward of statistics and data science. However, every ASA Ethical Guideline Principle *is* relevant for every practitioner [32–34]—and practicing these principles is clearly relevant for stewardship of the profession of statistics and the practice of data analysis/data science. Importantly, while some students in statistics and data science programs must complete general training in the responsible conduct of research at their universities, that general training, especially if it focuses on principles and practices of research subject protections, can seem unrelated to data intensive applications (see [33,34]). The "Alignment" column in Table 3 shows that training the statistician/data scientist in the stewardship KSAs will create opportunities to demonstrate 3–7 of the seven ASA Guideline Principles relating to the practitioner (A–G); training individuals to follow the ASA Guidelines in their practice will create opportunities to demonstrate 5–7 of the eight KSAs of stewardship. The alignment of each ASA Guideline Principle with the stewardship KSAs supports the claim that stewardship can be exercised by the full range of professionals in statistics and data science (irrespective of degree completion).

#### **Case 3. Society for Neuroscience Guidelines for Ethical Practice (2010) [35]**

The Society for Neuroscience (SfN) issued its Guidelines for Ethical Practice in 2010. These Guidelines include nine general principles. The Guidelines describe the scope of practitioners to whom guidelines—and the integrity of its scientific mission—pertain. Importantly, many individuals are included in this scope who are not actually producing scholarship themselves, but are supporting its dissemination (e.g., as editors or reviewers). While most neuroscience is carried out specifically for scholarly purposes, neuroscientists also work in industry (e.g., pharmaceutical; biomedical) and in some social applications (networks), as well as scholarship.


Table 4 presents the alignment of the Guidelines from the Society of Neuroscience (1–9, Columns) with the KSAs of the steward (rows). Alignment is indicated with an asterisk.


**Table 4.** Alignment of Neuroscience Guidelines (columns) with Elements of Stewardship (rows).

Table 4 shows considerable overlap between the KSAs of stewardship and the guidelines of the Society for Neuroscience. The weakest areas of overlap are "conservation of ideas" and "responsibly teach/mentor/model". It is possible that the reason "responsibly teach" is not more explicit in the SFN Guidelines is because most science research is done in academic settings (including research-intensive and medical school contexts), and teaching is a part of the more general "practice" of academia—and as such, might not be included in the specific neuroscience community practice standards (being more akin to "workplace" standards). The conservation of ideas might have less resonance for Neuroscience than for Statistics and Data Science and History if innovation and discovery are higher priorities for neuroscience; in fact, several of the guidelines themselves relate to treating new/breaking knowledge in a stewardly and responsible manner (which is not considered in the other two sets of guidelines).

The "Alignment" column in Table 4 shows that training the neuroscientist in the stewardship KSAs will create opportunities to demonstrate 2–8 of the nine Neuroscience Guideline Principles; training individuals to follow the Neuroscience Guideline Principles in their practice will create opportunities to demonstrate 4–7 of the eight KSAs of stewardship. The alignment of each Neuroscience Guideline Principle with the stewardship KSAs supports the claim that stewardship can be exercised across the professional contexts for the neuroscientist.
