Exploring Sustainable Measurements of Academic Research: How Do Faculty Members in Teaching-Oriented Universities of China Evaluate Good Research in Tourism and Hospitality?

Abstract

Academic research contributes to the achievement of sustainable development goals (SDGs) of teaching-oriented universities in many ways, such as providing quality education, promoting the healthy and diverse development of academia, and satisfying the sustainable career development of faculty. In essence, research can facilitate knowledge advancement, dissemination, and innovation, while existing metrics are outcome-based, quantitatively measured, and have methodological and invalidity problems in achieving these orientations. This exploratory study adopted a qualitative research approach guided by constructivist grounded theory to explore the fundamental question of how to evaluate good research in tourism and hospitality. By conducting 32 in-depth interviews with full-time faculty members in 16 teaching-oriented universities in mainland China, the perceptions of good research were captured. The perceived measurements that evaluate good research were interpreted, including counting publications and research projects, citation analysis and peer review, contribution to teaching, contribution to societal service and industrial interaction, and building research teams. Finally, this study provided discussions about the orientations of academic research as well as authorships in research evaluation from the perspective of HEI sustainability.

1. Introduction

The understanding of the UN 2030 Agenda of Sustainable Development Goals (SDGs) has been improved. Education for sustainable development (ESD) is increasingly emphasized (in Goal 4) because education has great potential to shape worldviews and values that determine the future of sustainability. The sustainable assessment tools and indicators for higher education institutions (HEIs) have been extensively discussed, demonstrating HEIs’ movements towards sustainability [1,2,3]. Creatively, a range of elements facilitating the emergence of sustainability was analyzed, incorporating “safe operating space, just operating place, resilient sustainable behaviors, health and wellbeing, collaboration, alternative economic models, diversity and inclusion, and transparency and governance” [4]. These elements involve all aspects of campus life in universities. Academic research, as a theme of the campus life of faculty members, generates either negative pressure or positive momentum and impacts social dimensions including health and wellbeing, collaboration, resilient sustainable behaviors, and diversity and inclusion. Consequently, despite disciplinary discrepancies, academic research determines the quality of education, sustainable career development of faculty, and the future of academia.

In the academic world of tourism and hospitality, generations of scholars exert effort to produce good research, revealing the truth, pursuing research aspirations, satisfying recruitment, and achieving career development. Universities and institutions endeavor to promote research achievements, compete for resources, ascend in position in academic league tables, and improve reputations. Both sides place academic research in a critical position, thus the development of academic research undoubtedly influences the sustainable future of universities. Scarce studies illustrate the criteria of good research from the perspective of human rights (academics as the research conductors and audiences) and few studies reveal how these criteria or attributes can be associated with HEI sustainability.

The field of tourism and hospitality has grown into a wide and multidisciplinary field, with increasing recognition inside and outside the academic community. Scholars in this field have contributed to the understanding, measurement, and improvement of academic research. Related studies have developed into three branches. The first branch involves rating academic journals in tourism and hospitality, addressing the problem of where to publish. Extensive studies focused on measuring journals via indexes such as citation analysis, impact factor, influential ratios, and authorship [5,6,7,8,9,10]. The second branch explores potential indicators representing the levels of academic research, such as leadership, network analysis, and university rankings [11,12,13,14,15,16,17]. The third branch focus on the art of improving research achievement by investigating the scope, depth, methodologies, and applications of academic research and by examining the referee perspectives [18,19,20]. However, the fundamental questions of what good research is and how to measure it remains undiscovered.

As the benchmark of knowledge advancement, research assessment measures (RAM) determine the direction of academic research. However, lacking solid criteria and principles in determining good research leads to a range of issues. The existing RAM of tourism and hospitality are outcome-based and quantitatively measured, with methodological and invalidity problems [7,8,9]. First, research outputs are assessed only after their publications. No consensus has been met about what type of research outputs can be accredited as good research, by which global research can be oriented. Specifically, existing RAM is considered narrow-scoped, English language biased, and vulnerable to gamesmanship and unethical tactics. Second, the evident publication-orientation and citation-orientation are closely tied with academic journals. Notably, all these measures, regardless of the amount of publications, citations, impact factors, or other indexes, have a strong link with academic journals. Researchers are largely led by the orientation, quality, and preference of journals. Despite the irreplaceable significance of academic journals of tourism and hospitality, the scarcity of alternative measures that evaluate good research impedes the progression of tourism and hospitality. Readers might be misled when reading published findings based on current assessments; more seriously, the unreasonable or even distorted research assessment measures have not only led to the tremendous pressure of publication for academics but might also mislead the entirety of academia in its future development.

Taking a reverse approach, this study discovers orientation-based measures by examining scholars’ perceptions of evaluating good research in tourism and hospitality. Considering the career landscape and academic context influencing peoples’ perceptions, this exploratory study aims to investigate perceptions of full-time faculty members in the teaching-oriented universities with tourism and hospitality programs in China. The research questions are as follows:

1.

What are the perceptions of good research?

2.

What measurements evaluate good research?

In China, the development of academic research of universities has undergone dramatic changes. Since the founding of the People’s Republic of China, academic research had been subordinated to teaching in the universities of China. In 1999, the focus of university evaluation shifted from a period of “teaching-only” to a period of “paying equal attention to teaching and scientific research” [21]. In the past 20 years, with huge investment and rapid development in higher education, the position of research has overweighed teaching in the functions of universities in China. The research evaluation of universities in China has been gradually bought in line with international standards. Consequently, the quantitative assessment system of research has been well constructed in China. Counting papers in selected lists, computing citations, peer review, and assessing representative achievements dominate the existing RAM of China [22]. In 2020, the Chinese government proposed an initiative involving breaking the negative guidance of “paper only” and “SCI only” when evaluating research performance [23]. However, principles and concepts that guide the evaluation of academic research remain unexploited. It is important and demanding to explore the understanding of expectations of academics who are evaluated, to construct a scientific and sustainable evaluation system [24].

The proposal of “teaching-oriented universities of China” is valuable for at least three reasons. First, the demand for educators with strengths in academic research is considerably rising. Up to 2020, 1272 universities (where about 1.83 million teachers are employed) in China are offering bachelor’s degrees [25,26] and larger efforts will be made to build a high-caliber research workforce. Teaching-oriented universities of China take the majority responsibility of educating the huge enrollment. Hence the question of how to evaluate the research performance of academics in teaching-oriented universities of China becomes ever more important. Second, teaching-oriented universities of China seriously lack clear orientations of academic research. Owing to the lack of official classification and positioning for different categories of universities, many universities are confused about the determination of research responsibilities for faculty members. Third, unlike research-oriented universities, faculty members of teaching-oriented universities need to strike a balance between teaching and research because of the heavy teaching workload and research requirements. This situation has influenced research activities in teaching-oriented universities. Finally, debates remain on whether and to what extent research should take place in teaching-oriented universities. Thus, teaching-oriented universities of China should be highlighted, and the improvement of RAM is increasingly critical for teaching-oriented universities of China.

Teaching-oriented universities were specified in this study with the following characteristics: (a) the predominant schooling objectives and the vision of university incorporate cultivating talents; (b) the universities enroll undergraduate students who should take up the largest proportion of enrolment; (c) teaching workload should be essential for academic faculty members, indicating that the compulsory teaching workload should account for more than 50% of total workload, or should be above seven teaching hours per week; and (d) the top 10 universities of China in the Chinese University Ranking of ARWU are excluded in this study because most of the top universities are research-oriented universities [27].

2. Related Studies

2.1. Good Research

In the academic world, the underlying values of academic research involve beliefs and principles that guide university life and research activities [28]. Good research is the cornerstone of academic value because it decides the research orientations for which the whole of academia is fighting. However, a universal definition of good research cannot be found in the literature. In practice, the term “excellence research” has replaced good research or quality research, explaining a range of implementations that encourage “research excellence,” such as the Research Excellence Framework (REF) launched in 2008 in the UK [29], the Initiatives d’ Excellence started in 2010 in France, Clusters of Excellence in Germany [30], and “Double World-class” planning initiated in 2015 and “Double ten thousand” planning in China [31].

Excellence is perceived as “a quintessential polymorphic term,” and different standards of excellence exist due to the differences in “intellectual habitus across disciplinary clusters” [32] (p. 159). Even though “excellence” is considered complex and can be variedly interpreted in different fields, it cannot represent the concept of “good research.” Limiting good research to excellent research excludes complete meanings and diverse values, resulting in a misleading of the evaluation of research performance. Quantitative techniques (including publications, citations, impact factors, H-index, and i10-index) have been extensively developed to assess the research productivity of individuals, organizations/institutions, and universities as well as the performance of academic journals. In contrast, solely depending on quantitative indicators is problematic because “academia is a highly variegated world, one where qualitatively incommensurate proposals cannot be subsumed under a single standard” [32].

In the domain of tourism and hospitality, studies examining the meanings and characteristics of good research are scarce. Good research was illustrated by Chin such that research starts with an important inquiry and pushes the boundary further, on the basis of his investigation in six tourism universities of Hong Kong [33]. A few scholars started to explore the perspectives of important persons in evaluating research performance. Law and Chon [34], and Tung, Law, and Chon [35] consecutively explored the global university heads’ perspectives on evaluating research performance 10 years apart. The general finding of the above two studies presented diverse proxies of evaluating research performance including publication in referred journals, books or monographs, editorial service, conference presentations/publications, research grants, and supervision of students. These two studies represent a constructive start in examining perceptions of the evaluation of research performance in tourism and hospitality, discussing non-quantitative indicators, and discovering underlying academic values.

As an essential academic value, good research is an ambiguous concept that is largely determined by subjective and individual choices. Therefore, more extensive investigation probing into people’s subjective world is necessary to explore this concept. Before starting the main study of examining academics’ perceptions, understanding the existing research assessment measures is a prerequisite, and is demonstrated in the following section.

2.2. Research Assessment Measures

Synthesizing literature, social conventions, and research policies, the existing RAM can be broadly divided into three categories: publication-based approach, citation-based approach, and academic leadership and network analysis.

Academic publications have served as an important platform for disseminating academic findings, a communication channel for academics and practitioners, and a system for identifying contributions to knowledge development [36,37]. As a straightforward way of evaluating and interpreting research performance, manually calculating the sum number of publications for individual scholars and institutes [5] has been widely used around the world. Several studies argue that the usage of quantitative counting of publications represented and encouraged productivity in research [38]. Recognizing scholars’ academic achievement was materialized by counting their publications, whereby comparisons and scientists’ rankings could be made across all disciplines [39]. The limitations of the publication-based approach are also evident, including being rather “narrow and geographically-based” [34], “milking publications” [40] (p. 658), output-oriented research activities [11], and discouragement in collaboration and teamwork under the policy of sole authorship [41].

Citation analysis refers to the number of times that a particular publication has been cited by other publications, indicating the contribution of certain publications to another published research. The citation-based approach incorporates citation analysis by ISI, Google Scholar, Scopus, China National Knowledge Infrastructure (CNKI), and Baidu Scholar. High-quality publications and journals are commonly more cited than low-quality ones [36]. Assessing citation frequency can be used to indicate the effect of individuals’ academic output and rank individuals [6,7,42], journals [40], and institutes [43]. For individuals, the significance of citation has gradually turned the notion of “publish or perish” into the prediction of “be cited or vanish” [6]. The shortcomings of the citation-based approach are in the following aspects. First, the authors might intentionally produce papers that easily attract citations, leaving other areas unexplored; second, citation counting does not distinguish the work perceived positively, negatively, or neutral; third, citation cannot be verified to be a contribution to knowledge advancement [38]. Fourth, the citation tools incorporating citation analysis by ISI, Google Scholar, Scopus, CNKI, and Baidu Scholar are not free from issues [7,44,45,46]. Thus, the findings based on counting citations also need careful interpretation.

Academic leadership was proposed as a complementary metric to measure research performance and identify excellent contributions. First, academic leaders demonstrated their leading roles by frequent “involvement” with the editorial services of tourism and hospitality journals. Editorial services for journals, including being chief editors, associate editors, or editorial board members in tourism and hospitality journals, are respectable and could be a proxy of leadership in academic journals [11]. Journal editors and reviewers fundamentally determine the quality and reputation of journals by evaluating transcripts for the journals and by offering advice for their improvement or refinement. Consequently, journal editors and reviewers decide the content of journals, trends, and strategies of development of academic journals [47] as well as affect the disciplines constructively [11]. Second, leading scholars showcase their extraordinary capabilities and contributions to build research networks and promote collaborations. Some scholars indicate that contributions to the co-author network were positively associated with research performance [27,48]. Multi-author and multi-university were revealed as the most dominant pattern of authorship structure, and the average collaboration intensity and the number of collaborators were verified as two key determinants of the four types of collaborative strategies [48]. Representative researchers were characterized by extroversive and introversive collaborating, using data analysis over 20 years (1991–2010) with component analysis [27]. A dynamic network analysis on co-authorship of tourism research in China was conducted by presenting author rankings based on the centrality metrics and the structural holes index of the tourism research co-authorship network in China (TRCNC) [49]. A noticeable shift is observed from solely focusing on the productivity of researchers to the productivity and connectivity of the academic network.

3. Methodology

3.1. Research Design: Constructivist Grounded Theory

Considering the nature of this study: discovering the perceptions of academics’ of the state of academia in tourism and hospitality, constructivist grounded theory is appropriate to achieve the research objectives. Grounded theory (GT) is a systematic method that emphasizes several flexible strategies for constructing theory, serving qualitative inquiries. From the perspective of a philosophical stance, constructivist GT assumes the ontology of realism and the epistemology of relativism, which approves that reality exists independently from people’s minds but acknowledges their subjectivities in understanding the world, indicating that our experiences shape our knowledge.

Constructivist GT could be characterized by the following aspects: the co-producing role of the researcher in the “interactive proceeding” of treating data [50] (p.178), a specific literature review that is organized in a short section before data collection but a comprehensive literature review after data collection, the “iterative” and “comparative” process of developing categories [51] (p. 300), the “flexible” [52], and interpretative character of coding, and the immersive nature of interpreting the findings. Under the tenet of constructivist GT, this exploratory study is designed to unveil the mystery of research questions. The design of research questions involves what good research of tourism and hospitality is, what measurements evaluate good research in tourism and hospitality, and if and how the current research measurement affects the career development of participants.

The 32 valid semi-structured interviews were conducted from July 2020 to November 2020 via WeChat calls. The participants were recruited from 16 teaching-oriented universities of China, which are scattered in most parts of mainland China except for Northeast and Northwest China. All interviews were conducted by the researchers in Mandarin.

3.2. Sampling and Profile of Participants

Purposive sampling and snowball sampling were adopted to identify the group of faculty members of teaching-oriented universities in this study. Instead of emphasizing statistical representativeness in probability sampling, the selection of cases in purposive sampling depends on research questions [53]. Purposive sampling supports the strategies of heterogeneous variation and homogeneous variation sampling. Heterogeneous variation sampling requires identifying the sample selection criterion before selecting the sample [54]. Homogeneous variation sampling concentrates on the similarities in sub-groups. Therefore, purposive sampling is appropriate to initially target the objective, combined with snowball sampling to further expand the participant group.

Considering that this study aims to uncover an unexploited area involving the perceptions of academics, participants were selected from reachable teaching-oriented universities that are defined in the introduction section. The selection criteria of participants were also regulated as follows: (a) a full-time faculty member working in a department of tourism or hospitality in a teaching-oriented university; (b) “research activeness,” in which the proxies of research activeness refer to any experiences in conducting research (incorporating publications, directing or participating in research projects, presenting in conferences or other research activities) or research aspirations in career planning; (c) a faculty member with job responsibilities in both teaching and research, excluding academics in teaching-only positions or research-only positions or administration-only positions; and (d) the default qualified participants include tenure track professors at different seniorities. Two screening questions were asked to all potential participants before the main interviews, aiming to confirm the satisfaction of participants. A translation check of the interview questions was conducted by three Chinese native speakers who have strengths in academic writing in English.

Originally, ten universities were selected from the researchers’ personal and business networks. Then, some participants in these ten universities introduced their friends working in other universities. The researchers inspected the new additions to the university list and used their judgment to determine the other six universities. Thirty-eight potential participants were investigated and six were excluded due to their unfitness for this study. The authors were informed during the interviews that the six exclusions could not satisfy the selection criteria of participants. Three of the six claimed that they were not willing to focus on academic research in their future career life, contrary to selection criterion (b); two switched to research universities at the beginning of 2020 after receiving their doctoral degrees, making them unqualified for the selection criterion (a); and one shifted her job to an administrative position after six months of teaching, contradicting the selection criterion (c). Finally, the selected 16 teaching-oriented universities (including six private universities and ten public universities) were determined, and 32 usable interviews were included in the data analysis.

The 32 interviews collectively lasted 1992 min, and the average duration of each interview was 62.25 min (with a maximum of 135 min and a minimum of 40 min). These participants varied extensively in terms of professional title, degrees, research areas, personality traits, age, gender, and working experience (including participants’ employment in universities and industries). All participants were very supportive during the interviews. Associate professors (average 64.08 min) and lecturers (average 63.47 min) are the most talkative groups, followed by the assistant lecturers (average 57.5 min). Full professors investigated in this study share their viewpoints concisely and comprehensively.

Table 1. Profile of Participants.

Types of Interviewees	Number of interviews	Duration of Interviews (min)	Average Age	Education (Degree)			Gender		Industrial Experience of Full-Time Employment
				Doctor’s	Master’s	Bachelor’s	Male	Female	2–9 Years	Above 10 Years
Full/Chair Professor	3	52	57	2	1	0	2	1	2	0
Associate Professor	12	64.08	42.92	6	5	1	5	7	4	2
Lecturer	15	63.47	33	3	12	0	4	11	2	0
Assistant Lecturer	2	57.5	38	0	1	1	0	2	2	1
Total	32	62.25	39.28	11	19	2	11	21	10	3

shows the breakdown information of participants according to seniority in terms of professional titles. The participants of lecturer and associate professor dominated the sample, accounting for 53.12% and 37.50% of participants, respectively, while the proportions of full/chair professors were small, standing at 9.38%. These proportions are consistent with the personnel situation of full-time academics in regular HEIs of China (as reflected in the 2020 Statistics), which strengthens the representativeness of participants. According to the 2020 Statistics Bulletin of Education Development, which was released in August 2021 by the Ministry of Education in the People’s Republic of China, the subtotal percentage of lecturers and below was 56.70%, whereas the percentages of associate professor and full professor were 30.04% and 13.25%, respectively [26]. The gender imbalance of the sample (11 male participants and 21 female participants) does not produce many differences in the findings, for this exploratory study collected and analyzed perceptions of participants adopting constructivist grounded theory instead of counting statistical variations.

Two characteristics of the participants are noteworthy. The first one is the education background: Eight of the 19 participants holding master’s degrees are currently doctoral candidates, presenting their research aspiration; secondly, 14 of the 32 participants achieved their degrees outside of mainland China, showcasing their overseas learning experiences. The second noteworthy characteristic is the industrial experiences of participants. Three participants had more than ten years of full-time employment in industries, while the number of participants with two-nine years of full-time employment in industries is ten. In a word, this structure of participants enhances the convincingness of findings of the study, indicating that some participants have a global understanding of education and that some have both industrial insights and educational intelligence. To protect the confidentiality of individual participants, their institutions were not reported in this study; their ages and years of work were showcased in brackets.

3.3. Validity and Credibility

On the basis of constructivist GT, this study conducted the iterative process of data analysis including coding, memo writing, and sorting [55], accompanied by comprehensive content analysis. The first analytic step is coding, which is the process of noticing, naming, and labeling concepts. This step requires researchers to ask analytic questions to elevate understandings of study and “help direct the subsequent data-gathering toward the analytic issues” [56] (p. 109). This study pursues the martinet guideline of constructivist GT in the coding process, following a logical sequence from “initial coding” to “focused coding [57] (pp. 12–13). Memos are relatively short notes developed by researchers to capture the immediate and emerging reflection of data. Memos might vary in length and form based on the research process and personal styles [58]. Memo writing accompanies the researcher’s journey of data analysis. Sorting involves reading and rereading codes and memos to discover categories and their interrelationships through the comparative and inductive processes [55]. In contrast to how coding suggests researchers break apart the data into small and scattered chunks, sorting directs the researchers to put the chunks back, during which new concepts are recorded as new memos. NVivo 12 was used to produce and visualize findings.

On an operationalized level, triangulation is implemented to promote the quality of research, including data, theoretical, investigator, and methodological triangulation [59]. Data triangulation was first implemented in the sampling of participants, combined with collecting written research policies in selected universities. The 32 participants were recruited from 16 universities in 12 provinces and municipalities across the majority of areas in mainland China. The 16 universities comprise ten public universities and six private universities. In other words, these participants are under different contexts, where local research policies, regulations, and academic atmospheres are varied. Second, iteratively reading the transcripts of interviews, research policies and regulations of the selected universities, official reports released from the governments, and literature can be considered multiple tunnels of data collection. Theoretical triangulation was completely presented in the procedure of data analysis. Initial coding (including word-for-word, line-by-line, and incident-by-incident coding) was used to separate the data into scattered pieces before focused coding refined the initial codes. These steps were followed by memo writing and sorting for reconstruction of the data. This process is called “slicing data” which was deemed as the special form of triangulation in theoretical sampling [60] (pp. 66–68). From the theoretical triangulation perspective, the notion of constructing theories per se is a discovery path of theoretical argument. The investigator triangulation can be realized by the diversely corporative teamwork of the three co-authors. The selection of teaching-oriented universities of China was decided by the consensus of all authors. The coding process was conducted independently by two authors and a comparison of findings was made. The inter-rater agreement level was at 96.5%. To reach 100% in the coding process, inconsistencies were discussed and moderated by a third author not involved in the initial coding. In addition, before sending the interview guide to participants, we undertook the step of translation check, which is considered investigator triangulation and helps improve the effectiveness and precision of communications in the semi-structured interviews. For methodological triangulation, the in-depth interview, observation, and analysis of documents were simultaneously adopted to be congruent with the requirements of quality and rigor.

4. Findings

4.1. Perceptions of Good Research on Tourism and Hospitality

The answer to the question of the definition of good research on tourism and hospitality is fundamental to the whole of academia. The sense of territory was manifested by all participants. They believe that the properties of the discipline and industrial development pattern the perceptions of good research, indicating the phenomena of “small worlds” and “academic tribes” created by specific areas [61,62].

Good research is that research that contributes to knowledge inheritance and innovation. Good research starts with insightful inquiries into research gaps, is problem-oriented in the process of exploration, and completes with research contribution. The term research value refers to dealing with a scientific problem that is valuable from both a theoretical and a practical perspective. Instead of rephrasing and restating previous research achievements, knowledge inheritance and innovation signify pushing the boundary line further to fill the research gap as well as benefiting the future development of exploration.

“Research value can be explained as theoretical value, social value, and social effect. Discovering problems, analyzing problems, and solving problems consist of the essential procedure of problem-oriented research.”

-P11 Associate Professor

“Good research is excellent at knowledge inheritance and intergenerational transformation, highlighting the future research direction.”

-P26 Associate Professor

“The discussed research questions should be science questions worth exploring. Therefore, good research achieved a theoretical breakthrough by a dialogue with existing theories.”

-P10 Associate Professor

Good research contributes to academia or industries. The theoretical contribution and industrial implication can be equally emphasized; basic research and applied research can be advanced side by side. Unlike some basic disciplines of social science like economics and management that can be applied in a variety of industries and fields, research on hospitality and tourism management better targets the corresponding industries. These industries range from extensive providers of tourism attractions, tourism products, hotels, catering services, transportation, and related public services. These industries have turned into the real backbones and strong drivers of research into hospitality and tourism. The field of hospitality and tourism management is considered emergent with the nature of being interdisciplinary or multidisciplinary, and little previous experience exists to develop the research system.

“The social contribution involves addressing related social problems of the industries and serving for the people.”

-P17 Associate Professor

“Good research can promote productivity and produce technological reform.”

-P11 Associate Professor

Good research adopts scientific methodologies. The strengths in research methods of good research are demonstrated by its reasonable and rigorous research design that matches research questions and the quality of the data (including the accuracy, comprehensiveness, and timeliness of data).

“The research design should be rigorous and logical. The used data should support your research. Specifically, the accessibility, traceability of the adopted data, and coordination between research methods and research questions collectively produce good research.”

-P10 Associate Professor

Good research explores laws of nature and society, stands the test of time, or forecasts future development tendencies. Laws and regulations enlighten people to effectively understand the world and set standards for human behaviors. Good research can help people to discover the mysterious world and improve the cognition of human beings by either advancing knowledge about the laws of nature and society or explaining phenomena and behaviors. Furthermore, contemporaneity and foresightedness are the extraordinary characteristics of good research. Time tells the truth.

“In essence, discovering interesting laws and regulations from some phenomena might be essential in tourism research, which can be perceived as good research.”

-P16 Associate Professor

“Some research outputs can be reconfirmed in several years of implementation by the practitioners in industry or the governments. This is good research.”

-P31 Full Professor

“Good research is prospective and forward-looking, which is demonstrated by withstanding the test of time.”

-P22 Lecturer

Good research opts for good topics. First, a good topic should be perceived as an interesting topic by researchers, who achieve self-affirmation by exploring it. Given that the success of research is closely associated with the subjective efforts of researchers, only concentrating on favorite areas produces good research outputs. Second, good research does not necessarily follow official policies and the public mainstream or cater to the tastes of journals and fund sponsors. Academics should also concern themselves with some topics that benefit the industry and disciplinary development but are not very popular. Third, the proposal of good research might be developed from personal experience or insights.

“Good research should be interesting, which is the impetus to complete the research, I think.”

-P10 Associate Professor

“For me, good research depends on if I could put effort, passion, and emotion into this research. It is a process evaluation with self-affirmation.”

-P28 Lecturer

“The topics of good research must come from practical problems or personal insights. If you are not a tourist lover and are not willing to go out, you cannot observe the subtle emotions when studying tourists’ behaviors.”

-P26 Associate Professor

4.2. Metrics that Measure Good Research in Tourism and Hospitality

4.2.1. Counting Publications and Projects at High Levels

Publications and project counts have prevailed in the research assessments of global academia, including but not limited to teaching-oriented universities of China. Publishing in top journals not only helps academics achieve high esteem but is also considerably weighted in the reward system in China. Most of the interviewees agree that publishing papers in selected journals showcase research capabilities and academic reputation; therefore, the pursuit of good papers is necessary for their research aspirations. Moreover, writing articles is a fundamental activity that paves the way for further development such as writing monographs, book chapters, edited books, and translations of works [63]. According to the evidence of this study, publications in top-tier journals are perceived as a prerequisite for achieving research grants, presenting in conference proceedings, and expanding academic reputation.

“Counting papers published in good journals is necessary to evaluate academic research performance.”

-P31 Full Professor

“The most direct indicator is publications in the selected journals. We have some metrics, such as SSCI and CSSCI journal lists.”

-P26 Associate Professor

Conducting research projects and foundations is perceived as a significant research activity, not only due to the requirement of existing RAM but also to financial support. A clear hierarchy of research projects has been established in China, which defines the ways of assessing research projects. In other words, the levels of sponsors usually determine the corresponding credits of research projects. This comprehensive hierarchy consists of multiple levels of public-supported or government-planned projects (called longitudinal projects) as well as private or commercialized projects (called horizontal projects). The longitudinal projects range from the national level (including the influential National Natural Science Foundation of China and National Social Science of Funds of China), ministerial or provincial/state level, and city department level. In particular, national-level (i.e., the highest level) projects are required as a baseline to achieve tenures for associate professors and full professors.

“The outputs of research projects in high-levels are more influential, such as research projects in the National and Ministerial/Provincial levels.”

-P24 Associate Professor

“Publication in quality journals can bring high recognition and good articles are necessary for applying to longitudinal projects.”

-P28 Lecturer

4.2.2. Citation Analysis and Peer Review

Citation analysis has been accepted as an important indicator of evaluating research quality by worldwide academics. Citation represents discussions, developments, or recognitions on previous results, which play a significant role in research development in terms of knowledge advancement, alignment, and theoretical inheritance. Through positive confirmation or negative criticism, scholars cite previous viewpoints to either justify their works or develop new frames. Citation references provide rich information to trace the theoretical source for scholars and therefore identify the evolutionary relationships between bodies of literature. High citations in well-known databases such as ISI (Web of Science), Scopus, Google Scholar, and CNKI, have turned into important barometers that measure the academic influence of research outputs. Notably, the period is critical in citation analysis. The pendulum of citation analysis swings in favor of research outputs published a long time ago. Thus, senior researchers with many publications are likely to gain more citations. This point is in line with the literature.

“Citation and peer review could be good indicators that are measurable and representative.”

-P17 Associate Professor

“Obtaining high citations is not easy for young scholars because high citations and academic reputation take a long time to accumulate.”

-P14 Lecturer

Citations are not only very helpful to measure research values of publications, but also recognizes academic leaders in certain fields. Thus, citation analysis has become a popular instrument that measures the lifetime achievement of individual scholars. The reputation-based reward system has been well established by peer recognition and simultaneously, researchers as the “intellectual proprietors” of knowledge achieve individual prestige and other rewards [64] (p. 121). In China, the widespread CNKI (https://www.cnki.net, accessed on 23 September 2021) serves the largest audience from a variety of disciplines. The citation service includes the citation times, download times, and cited reference networks for each output. CNKI does not offer indicators for accumulated achievement of academics, but Baidu Scholar (https://xueshu.baidu.com, accessed on 23 September 2021) fills the gap, displaying the number of citations, the number of outputs, H-index, and G-index.

In many practices of research policies, the citation indexes were not officially put in the assessments of individuals. Some participants claimed that the citation indexes influence research policy in terms of stipulating lists of selected journals of tourism and hospitality. This claim is attributed to the importance of research performance in indicator systems of university rankings. Specifically, the citation contributes more than 20% in the indicator systems of the three mainstream university rankings including THE, QS, and ARWU Shanghai Ranking [65,66,67]. Officials and university executives tend to put those journals with high citation scores in their lists of selected journals to improve research performance and finally university rankings. In this situation, citation analysis should be put in the RAM of evaluating individuals to maintain the consistency of evaluation regulation.

4.2.3. Contribution to Teaching

Teaching, research, and public service are perceived as the three main duties of higher education. Tension, conflict, symbiosis, and reciprocity, the relationships between research and teaching universities have been fully discussed in the literature [68,69]. In this study, all participants acknowledge that teaching should dominate the education responsibilities of teaching-oriented universities, while the reality is that research is central in the academic reward system of universities. At present, contribution to teaching is not explicitly indicated in the RAM of investigated teaching-oriented universities. From the narrative of this study, some faculty members were asked to provide proof involving what research findings were presented in class. This step is an effective start to realizing the significance of research contribution to teaching.

Some academics creatively proposed that the contribution of research to teaching and the cultivation of students can be a good indicator of research performance in teaching-oriented universities. By conducting research, on the one hand, faculty members made efforts to update teaching materials and widen their academic scopes, potentially enhancing teaching quality and raising students’ learning interest. Students are attracted and inspired by newly generated things and cutting-edge technologies. On the other hand, undergraduate students can be encouraged to participate in research activities, by which students can apply what they learned in class and benefit from practical assignments in research projects. Undergraduate students can achieve fundamental research techniques that might be required in their career positions in the foreseen future. Therefore, research practices could be developed into good platforms for training students. In turn, undergraduate students can also be good helpers for research. This potential role is evident in teaching universities where postgraduate students are rarely recruited or cannot be recruited.

“Merging research with teaching refers to how much research findings or knowledge frontiers are utilized in teaching and what research programs serve in training students. I think this could be a good category of indicators to measure research performance. From my perspective, research should serve, support, and promote teaching, especially in a teaching-oriented university.”

-P19 Associate Professor

“Teaching and research could be perfectly blended. When conducting research, we guide or cooperate with undergraduate students to complete research projects with applied value. This form is really meaningful for the cultivation of students.”

-P23 Associate Professor

4.2.4. Contribution to Societal Service and Industrial Interaction

Societal service is an ultimate orientation of academic research and the whole of higher education. Fundamental and applied research should be closely associated with societal development. First, from the perspective of academic development and sharing talent resources, societal services demand the active participation of experienced academics. Some outstanding academics are invited to be editors and reviewers of journals, conference proceedings, and academic competitions as well as research projects initiated by companies or governments. These works not only represent the excellence and academic prestige achieved by these academics but also their contributions to academic development and society. However, the existing RAM and incentive system in universities neglect the section of societal service, impeding the harmonious development of academia. The invitation of social service per se is a way of social recognition, demonstrating a certain status and honor in the field. Thus, social service could be a parameter of research assessment.

“Participation in social operation can be a good measurement of research performance.”

-P18 Lecturer

Second, from the perspective of industrial interaction, as some participants claimed, the existing research and industrial implications go separate ways, speaking their languages and booming respectively in their tracks without sufficient overlaps and communications. Practitioners in the industries of hospitality and tourism do not read academic journals, while academics do not mind if the industrial audience likes their research outputs but rather care about the number of publications and citations. Some participants believe that this vicious circle should be broken by building research–industry links, in which the first step can be taking the industrial implications of research as an indicator of research performance. The faster a certain industry develops, the stronger the demand for research achievements to solve industrial problems.

“The measurement of industrial implication is a more important evaluation index of research. I think the term ‘application-oriented’ explains how to explore the application value of research.”

-P23 Associate Professor

“Horizontal projects should be able to solve practical problems. In many cases, theoretical problems are extracted from practical problems, which can effectively define scientific research achievements.”

-P22 Lecturer

Societal service and industrial interaction can be measured and serve as good measurements to emphasize the industrial applications of academic research. A condition of this is ensuring that the explored effective indicators are measurable and comparable.

4.2.5. Contribution to Building Research Teams

Building research teams is critical to creating a favorable academic atmosphere, encouraging young academics to grow, uniting the power of individuals, and complementing the advantages of team members. The key figure of a good research team is its academic leader or mentor. With the guidance of the academic leader and cooperation between team members, academics can be largely encouraged to explore their research potentials and improve their capabilities. This point is obvious among academics without systematic research training. In other words, for those without a doctorate or who have not experienced professional research training, teamwork and academic leadership might be more important.

“Team members complement each other’s advantages of research capabilities; team building can promote mutual recognition and self-recognition. In a research team, we can do what we like. I would feel very happy to do it.”

-P15 Associate Professor

“Research teams with high yield always must have teamwork and clear collaboration, such as the situation in ** university. There are limitations for individuals to fight alone.”

-P2 Lecturer

The literature proves that academic leadership and academic network construction are positively associated with research performance in hospitality and tourism by facilitating inter-institutional and intra-institutional collaboration [27,48]. This process is how some research giants grow by the cluster effect of collaboration. In reality, academia in hospitality and tourism in China sees “loose collaborations” and “immatureness” [49], suggesting that team building is a must-do for further research development. The evidence of this study indicates that existing research governance and incentive policies neither explicitly reward the contribution to building research teams nor count the workload of academic leaders. This situation discourages experienced professors from being academic leaders.

“Professors’ contribution to the guide application of research project and team-building are not considered workload in universities, which is disappointing. Conducting this job without any financial reward is unreasonable.”

-P32 Full Professor

As an underdeveloped and promising discipline, tourism and hospitality call for sufficient investment in academic talents. The building of research teams is a driving force of disciplinary development and can be developed into a measurement of research performance.

5. Discussions: What Can Be Emphasized in Research Evaluation?

The orientation of academic research directly links the operational-social and governance dimensions of sustainable assessment tools in HEI sustainability [3]. At first, the orientation of the RAM of a university illustrates the vision of sustainability, commitment, policy, and strategic plan, reflecting the sustainable responsibility of academic research in the governance dimension. Secondly, the stipulation of RAM decides the human resource management, i.e., staff governance policies including hiring and promotion coordination. Thirdly, regarding the operation-social dimension, issues about staff employment, occupational health and safety, and employee satisfaction are associated with the human rights of staff. Collaboration is interpreted as a sustainability attribute for assessing the alignment of university programs to sustainability [4]. Specifically, collaboration refers to the conditions or environment that facilitate competence. For instance, “working in interdisciplinary teams, empathy, sharing responsibility, motivating self and others to participate, and effective communication” are typical elements of collaboration promoting good research [2,4]. The following discussions focus on the orientation of academic research guiding RAM and the authorships involving cooperation.

5.1. Publication-Orientation vs. Contribution-Orientation

Although the hot debate on “publish or perish” or “publish or prosper” did not reach a consensus, in reality, the measurable indicators of publication have dominated the RAM and reward system. Professors are forced to publish by the pervasive competition of publication regardless of whether scientific progress has been made. “Non-performance has been replaced by the performance of nonsense” [70] (p. 30). When academics conducted research in the pursuit of discovering the world and revealing truths, they published their work for knowledge advancement, providing new explanations and solutions. During that time, publishing was autonomous behavior without any external forces. Until the emergence of systematic research assessment, academia had been involved in the competition of “academic excellence” [71]. The ideal “excellent” professor is believed to be a mixture of the good publisher, project manager, and fundraiser [70]. Professors are so busy and are distracted by these “responsibilities” from the track of the “noble” orientation of advancing knowledge.

Increasing academic publications can considerably benefit academics’ career development and academic reputation under the existing metrics and reward system. Publications in top journals are not only officially recognized by existing research assessments of universities but are also widely accepted by peers. The classical concepts of research orientations proposed by Hakala and Ylijoki [72], include academic orientation, civil society orientation, state–governmental orientation, and entrepreneurial orientation. Producing publications is not on this list of research orientations. Specifically, in nature, publishing articles is simply a form of research assessment and a tool of distributing knowledge, rather than a research orientation.

The worst shortcoming of the overemphasis on the orientation of publication refers to the negative influence on the intrinsic motivation of progressing knowledge that is essential for creativity and innovation. The demotivation effect on scholars is profound. The “puzzle,” the “taste for science” [73], is substituted by the principle of “publish or perish” [74]. Furthermore, a range of perverse practices is associated with the competition of publications, such as strategic citation, salami slicing, co-authorship strategies (including guest authorship and ghost authorship), catering to reviewers, playing with models, becoming more specialized, and engaging in fraud. In this case, filtering the good output from the sea of publication is more difficult because an excessive number of mediocre and nonsensical work is also published.

Academia should not be misled by the orientation of publication and competition. Instead, the progression of knowledge, discovering the world, and creating a better future should be reemphasized.

5.2. Sole Authorship vs. Co-authorship/Cooperation

The last half-century has seen the evident trend of co-authorship and collaboration in the research of tourism and hospitality. From a global scope, the percentage of sole authors has considerably decreased, whereas co-authorship and fractional authorship has been conspicuously raised [9,10]. However, surprisingly, in this study, some investigated universities do not encourage co-authorship. In regular research assessment and tenure evaluation, only the sole author and first author can be counted in individuals’ research performance. The existing regulations that discourage co-authorship hinder academic cooperation and communications.

“My university only confirms research outputs of the first author or corresponding author. The articles signed as the second or the third author are useless to me. Such implementation excludes cooperation and forces the teachers to fight alone without teamwork.

-P26 Associate Professor

Whether and how sole authorship or co-authorship should be implemented in research policy is a controversial issue. Sole authorship aims to prohibit the phenomena of gift authorship, guest authorship, and nominal authorship. Consequently, it is conducive to constructing an ethical academic atmosphere. The fatal weak point of sole authorship refers to the possibility of dampening academics’ enthusiasm in cooperation and team building. Conversely, co-authorship encourages sufficient communication and interpersonal learning. Teamwork complements the advantages of researchers. Cooperation significantly facilitates the production of research achievements, self-development, and research team building. However, the disadvantage of co-authorship policy involves inescapable “fabrication authorship” which violates research ethics. To reach the assessment requirement, some academics are likely to invite nominal authors in their research without substantive contribution to their research. Some “real” authors disappear from the author list (known as the ghost author) to help their friends to reach the RAM.

Although striking a balance between research productivity and academic ethics is not easy, academic cooperation should be encouraged. Research cooperation and communication are particularly significant for the domain of tourism and hospitality due to its disciplinary nature and unique industrial links.

6. Conclusions

This study started by investigating academics’ perceptions of good research before probing into the metrics that measure good research in tourism and hospitality, followed by a comparison of the existing RAM (by reviewing literature and collecting research policies in the selected teaching-oriented universities) and expected RAM. Constructivist GT was adopted, and the perceptions of good research were elaborated as follows. Good research is that which contributes to knowledge inheritance and innovation, it should benefit either academia or industries, adopt scientific methodologies with rigorous research procedures, explore laws, have explanatory power in explaining social phenomena and academic behaviors, should stand the test of time or predict the future, and discover good topics.

Figure 1 compares the existing RAM and expected RAM from the evidence of this study. When discussing the expected RAM that evaluates good research, this study confirms the pervasive key performance indicators that effectively serve the field of tourism and hospitality. At first, counting publications and projects at high levels dominate the measurement systems. Then, citation analysis is widely acknowledged by academics; citation indexes have potentially affected the list of selected journals by university rankings. Academic leadership and network analysis have been studied as complementary methods to measure research achievements of individuals in literature by some researchers but were not adopted as research policies in investigated universities. Academic leadership and network analysis are of a broader scope that incorporates academics’ editorial service, building research networks, promoting collaborations, and other efforts on societal development.

The novel perceptions of RAM discovered in this study are in three aspects: contribution to teaching, contribution to societal service and industrial interaction, and contribution to building research teams. Contribution to teaching points to the sustainable attribute of teaching quality that benefits from academic research. Contribution to societal service and industrial interaction bridges academic research with social and industrial development in a harmonious way, showcasing the diversity and inclusion trend of developing academic research. Building research teams is a powerful catalyst that helps academics grow up and signifies humanistic care for academic staff; it is associated with academic leadership but is a fairly narrow and concrete concept that concentrates on developing research teams.

7. Contributions and Implications

This empirical study adds academic value in the following two areas. First, the concept of good research in this domain is elaborated. The findings echoed the core concepts that have been fully discussed in the literature. For instance, Lamont [32] (pp. 160–161) concluded that excellent research should have “clarity,” “quality,” “originality,” “significance” (scholarly and social/political), “methods,” and “feasibility.” His study also mentions the evanescent qualities of “excitement” and being “elegant,” and “intelligent.” This study reconfirms the clarity, originality, method, significance (especially in social and industrial aspects), and feasibility. This study expanded the notion of “significance” by emphasizing “teaching value” and “industrial link.” Interestingly, the statement of “good research should be interesting” is in line with the wording of “excitement,” demonstrating the emotional elements in defining good research. This point emphasizes the importance of social sustainability while conducting academic research, in terms of psychological satisfaction, occupational health of faculty members. The attributes reveal the sustainable value of academic research that fundamentally promotes knowledge advance.

Second, this exploratory study examined perceived RAM in tourism and hospitality. The findings verified the fact that the most widely used research assessments, counting publications and research projects, have achieved a consensus in China. The findings also reconfirmed the results in previous studies about the advantages and disadvantages of counting publications and citation analysis. Three emerging measurements were proposed from the survey, namely the “contribution to teaching,” “contribution to societal service and industrial interaction,” and “contribution to building research teams.” These measurements enrich the theory of research assessment and indicate that the research contribution should emphasize the transition function towards teaching and industrial implication. This finding improves the understanding of sustainability in academic research, highlighting the integrating role of academic research with teaching practices, and social and industrial development.

In addition to academic contributions, this study also provides manifold practical implications for teaching-oriented universities. Under the circumstances of the prosperity of higher education and the tourism and hospitality industries as well as the rising demand for educators in this field, the sustainable value of good research and measurements is critical in developing HEI sustainability. Decision-makers and stipulators are inspired to think over the real meaningfulness of academic research and research orientation for faculties. In detail, many universities in China are confused in the determination of research responsibilities for faculty members. This is partly due to the lack of official classification and positioning for universities and partly due to the lack of a unanimous value and guidance for academic research from the perspective of HEI sustainability. It is time for universities and scholars to reflect on the existing epidemic of publication orientation and the practice associated with a sole focus on university ranking. It is also time to create a virtuous research environment where a range of sustainable elements are supported, including resilient sustainable behaviors, positive competition and collaboration, health and wellbeing, diversity, inclusion, equity, and transparency. Besides, the expected RAM “contribution to teaching” and the “contribution to societal service and industrial interaction” imply the insufficient consideration of transition abilities of research in teaching-oriented universities, while “contribution to research teams” calls for emphasis on academic teamwork and mutual assistance. These three additions embody the direction of future efforts on academic research.

To improve the limitations of existing RAM, the corresponding suggestions for teaching-oriented universities are proposed in the following five directions. The first and fundamental suggestion is that the contribution orientation of academic research should be emphasized in the university practices regarding research policies. In addition to counting publications/research projects and counting citations, diverse metrics should be explored and then implemented to assess output. Second, considering the significance of journals, the list of selected journals in the domain of tourism and hospitality should be improved. Academics of China continue to face the question of where to publish due to the disciplinary status of tourism and hospitality (that affiliated with first-level disciplines) and historical exclusion from citation indices. Good journals of hospitality and tourism in Chinese should be well developed. Third, academic ethics should be advocated in academia. The gamesmanship and unethical tactics in publication, citation, and authorship can be reduced only when academics realize that these behaviors are contradictory to the original intention of knowledge advancement and HEI sustainability. Moreover, education in academic ethics should be started from graduate school, which is an important part of HEI sustainability, helping scholars set up correct research values.

This study discussed sustainable measurements in tourism and hospitality in teaching-oriented universities in China but also shed light on other disciplines and other countries. The characteristics and homogeneity of the field of tourism and hospitality with certain disciplines determine the research value of generalization. The modern patterns of tourism and hospitality research development incorporate interdisciplinary nature [75,76], geographic and industrial links [77], sophisticated applications of diverse research methods [76], and the trend of co-authorship [9]. These patterns describe a popular trend of research development for certain disciplines and areas. Therefore, the findings of this study may be extended to other disciplines or countries with similar contexts.

This is the first comprehensive investigation to widely explore the topic of sustainable measurements of academic research in HEIs of China. The sample of this study provides rich information and valuable insights. The participants with reasonable structure in seniorities, age, education background, and industrial experience, were from 16 teaching-oriented universities (six private universities and ten public universities) in 12 provinces of mainland China. This study identifies the sustainable attributes of good research and sustainable measurements through rigorous research design and points out the direction of measurement sustainability. This study starts from teaching-oriented universities, but the proposed sustainable attributes of good research and measurements also provide comparisons for research universities, therefore paving the way for further studies.

8. Limitation

This study is not free from limitations. First, this qualitative study adopts constructivist GT. The research paradigm and research method are inevitably subject to the question of subjective bias. The findings are co-produced by participants and the researchers before being interpreted by the researchers. The results are tentative and need to be further improved. The proposed framework needs further examination to verify the feasibility of being supplemented into research policies. Second, the sample representativeness could be questioned. The 32 participants are recruited from 16 teaching-oriented universities in most parts of mainland China, without universities from Northeast and Northwest China. Third, this study targeted academics working in departments of tourism and hospitality in mainland China. It might lack generalization when applied to different disciplines and different countries under different regulations.

This study concentrates on the perceptions of academics working in teaching-oriented universities, where strong contextual specialties are influential to their research concepts. Studies toward academics in research-oriented universities might provide a meaningful comparison with the findings of this study. This study excludes the top universities, and examining the top universities is a good direction of future studies. Then, the proposed elements need to be further proved by using quantitative research methods. Finally, future studies can expand the geographical scope of participants to verify and improve the emerging framework.