Next Article in Journal
Developing Transportation Livability-Related Indicators for Green Urban Road Rating System in Taiwan
Next Article in Special Issue
Nexus between Financial Education, Literacy, and Financial Behavior: Insights from Vietnamese Young Generations
Previous Article in Journal
The Effects of Temper Traits and Study Method (Full-Time vs. Extramural) on Polish Students’ Adaptability to Online Learning as a Result of COVID-19. A Pilot Study
Previous Article in Special Issue
Knowledge Management and Business Education
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

SDGs: A Responsible Research Assessment Tool toward Impactful Business Research

by
Kathleen Rodenburg
1,*,
Vinuli De Silva
2 and
Julia Christensen Hughes
1,3
1
Gordon S. Lang School of Business and Economics, University of Guelph, Guelph, ON N1G 2W1, Canada
2
Department of Mathematics and Statistics, University of Guelph, Guelph, ON N1G 2W1, Canada
3
Yorkville University, Toronto, ON L4K 4N1, Canada
*
Author to whom correspondence should be addressed.
Sustainability 2021, 13(24), 14019; https://doi.org/10.3390/su132414019
Submission received: 1 December 2021 / Revised: 17 December 2021 / Accepted: 17 December 2021 / Published: 19 December 2021
(This article belongs to the Collection Knowledge Management and Business Education)

Abstract

:
An alternative research assessment (RA) tool was constructed to assess the relatedness of published business school research to the United Nations’ 17 Sustainable Development Goals (SDGs). The RA tool was created using Leximancer™, an on-line cloud-based text analytic software tool, that identified core themes within the SDG framework. Eight (8) core themes were found to define the ‘spirit of the SDGs’: Sustainable Development, Governance, Vulnerable Populations, Water, Gross Domestic Product (GDP), Food Security, Restoration, and Public Health. These themes were compared to the core themes found in the content of 4576 academic articles published in 2019 in journals that comprise the Financial Times (FT) 50 list. The articles’ relatedness to the SDG themes were assessed. Overall, 10.6% of the themes found in the FT50 journal articles had an explicit relationship to the SDG themes while 24.5% were implied. Themes generated from machine learning (ML), augmented by researcher judgement (to account for synonyms, similar concepts, and discipline specific examples), improved the robustness of the relationships found between the SDG framework and the published articles. Although there are compelling reasons for business schools to focus research on advancing the SDGs, this study and others highlight that there is much opportunity for improvement. Recommendations are made to better align academic research with the SDGs, influencing how business school faculty and their schools prioritize research and its role in the world.

1. Introduction

The United Nations’ 17 Sustainable Development Goals (SDGs) were adopted in 2015 as a ’universal call to action’ in support of the elimination of poverty, ensuring a healthy planet, and ‘peace and prosperity’, by 2030 [1]. Supporting the implementation of these goals are 169 targets and 231 indicators. While all countries and sectors of society are expected to contribute to this ambitious agenda, higher education has been singled out for its unique potential to align the scholarly work of faculty and students, and the education and career ambitions of graduates, with its achievement. The Higher Education Sustainability Initiative (HESI) [2], represents global networks and higher education institutions committed to supporting these outcomes, including the Global Compact’s Principles for Responsible Management Education Initiative (PRME) [3]. Comprised of over 850 business school signatories from around the globe, PRME’S vision is to realize the SDGs “through responsible management education” [3] (p. 1).
PRME’s Blueprint for SDG Integration into Curriculum, Research and Partnerships [4] provides compelling reasons for business schools to focus on research that investigates “both the practices and policies for implementing SDGs’ and the theory underpinning these goals to help solve ‘wicked world problems’” [4] (p. 5). “Beyond a moral obligation” [4] (p. 5), research funding sources (government, private industry, foundations, professional organizations) require in most cases an assessment of the anticipated impact and effects from research output and outcomes [5,6] within the application process. Business school and program accreditation bodies (AACSB, EFMD, MMBA, CEEMAN) now require evidence of an authentic and meaningful commitment to sustainability, ethics, and responsible leadership.
Concomitantly, business schools and faculty are compelled to focus research on publications in ‘top ranked’ journals, to provide evidence of research productivity, quality and to contribute favourably to reputation, through published rankings. Faculty research productivity and quality can be measured based on a number of factors, including the number of publications and consequent citations (H-index) in high ‘impact factor’ journals. ‘High impact’ journals contain articles judged to be of high quality, that are numerously cited within other research publications, especially for citations in papers that are in high impact journals [7]. The difficult task of measuring how research impacts society is not well represented through this mechanism. Business schools and other disciplines have largely adopted a simplistic proxy that is quantifiable but succeeds only in measuring outputs within the publishing forum, not impacts or outcomes on society [7].
With research continuing to be the more valued aspect of faculty work (versus teaching and service) [8,9,10], this measure of research ‘productivity’ is routinely assessed by peers (both within and beyond the institution), when making decisions concerning promotion and tenure (P&T) [11]. While academics are encouraged to participate in a variety of scholarly activities (publishing books and book chapters, making presentations at academic conferences and industry events, securing research grants, and participating on editorial boards and in peer review, etc.), at many business schools, these activities are valued far less than journal publications [11]. Rankings have had an inordinate influence on business school prestige, enrollments, and hence, priorities; including “incenting” research “behaviours that may be at odds with the achievement of the United Nations Sustainable Development Goals (SDGs)” [12,13], [14] (p. 1). Research in other disciplines has also identified perverse effects on institutional and faculty research priorities caused by this measurement proxy for ‘research with impact’ and the conflation of its inclusion in published rankings [15,16,17,18].
Despite these shortcomings, the reputational and/or financial gains, coupled with the need to maintain institutional legitimacy, can make participation in rankings unavoidable. As long as students, prospective faculty, future employers, and funding agencies continue to use rankings to inform their decisions on where to study, where to work, whom to hire, and what research should be funded [19], business schools will be incented to pursue rankings and the metrics employed.
Recently, the potential for using the SDGs as a tool to measure the impact of academic research has been identified [20]. Arguably, given their influence on business school priorities, published rankings’ adoption of such a metric could play a significant role in aligning research agendas with the SDGs. Hence, HESI’s recent “Rankings Roundtable”, which called for a better alignment between university rankings’ criteria and the advancement of the SDGs (see workshop [21]). Notably, sustainability and social impact criteria have begun to be included in some business school rankings. For example, the Financial Times FT50 ranking (long considered the gold standard amongst business schools), recently layered in a new corporate social responsibility (CSR) criterion in addition to equity, diversity, and inclusion (EDI) criteria. Times Higher Education (THE) launched its new impact rankings in 2019 to assess and promote the progress of universities toward the SDGs. Despite such initiatives, business school rankings continue to face criticism [12,22]. While not focused on research output directly, the Positive Impact Rating (PIR) endeavours to assess to what extent students feel their business schools are committed to positive impact [14].
For business schools to respond meaningfully to the call to action put forth by PRME regarding their scholarly contributions, a responsible research assessment (RRA) measurement tool is needed that incents research that is open, inclusive, and impactful, helping to advance the SDG agenda [15,20]. Curry et al. described five different ways that stakeholders have been responding to the need for RRA; “cosmetic appropriation”(acknowledging there is a problem and taking superficial steps to address the issue), “calibrating the machine”(productive incremental improvements to the assessment criteria), “can openers” (changing what and the way we measure), “advocacy coalitions”(groups forming to respond to the need for change), and “institutional cultural changers” (changes to institutional policies and guidelines) [20] (pp. 16–19). Most recently academics, representatives from ranking organizations (2019 THE), and scholarly analysts (see [23,24]) have had some success working on “calibrating the machine”. These efforts have focused on using the SDGs as a framework to measure the positive impact on society of research publications. More precisely, bibliometric and machine learning techniques [23,24,25,26,27,28,29] have been used to help track progress, identify gaps, and suggest interventions in support of the United Nation’s 2030 agenda. The many studies and methods that have attempted to map academic projects and publications related to the SDGs have shown varied and diverse results. See for example, Bergen University [25] Elsevier [26], Aurora Network [27], SIRIS Academic [28], NESTA and Dimensions [29], Rotterdam School of Management AI Dashboard (RSM Emasmus University) [23], and Cabell’s and SDG Dashboard (SJU) [24].
This study contributes to the “calibration” of the research assessment “machine” by testing an alternative approach for identifying the relatedness of business school research to the SDGs, published in journals listed in the FT50, during 2019. Arguably, a more valid and reliable RA instrument for measuring research relatedness to the SDGs, used in prestigious business school rankings (i.e., FT50), or identifying journals well aligned with the SDGS, that should be included in ‘high impact’, prestigious publication lists could have a significant impact on business school research priorities. The literature suggests that the SDG’s 17 goal framework, while an effective communication tool that inspires countries and organizations to get on board with the agenda and helping to track progress against the 17 goals, is an imperfect research measurement tool, as it includes over-lapping concepts between individual SDGs, which may complicate assessment. This may explain varied results across previous studies, in particular differences found in which SDGs align with particular academic articles.
To investigate the potential extent of overlapping themes a network analysis was performed from the textual description of the SDGs, including its 169 targets and 231 indicators. For this, we used LeximancerTM, a machine learning software tool that identifies concepts, themes, and relationships within documents. Additional researcher augmented terms were added, where these arguably represented similar ideas and synonyms, to improve the robustness of the themes generated. Next, the over-lapping themes that emerged from the SDG framework were used to construct a holistic RA tool. In this way, we constructed an assessment tool that arguably represented the ‘spirit of the SDGs’. It was this tool that we used to assess articles published in the 2019 FT50 journals.
Eight (8) high-level distinct concepts or themes found to define the ‘spirit of the SDGs’: Sustainable Development, Governance, Vulnerable Populations, Water, GDP, Food Security, Restoration, and Public Health. When applied to the FT50 articles, we found that, overall, 10.6% of themes found in the FT50 journals had an explicit relationship to the SDGs, while 24.5% were implied. Furthermore, we found that a subset of journals within the FT50 list were more closely aligned with the SDG themes than others. Those journals found to be most closely aligned included the Journal of Business Ethics, Human Resource Management, and the Journal of Management. Those furthest removed were the Journal of Financial Economics and the Journal of Finance. Assessing research outcomes and impacts on society is challenging. It can take years for published research to be applied, e.g., messenger DNA (mRNA) used in the development of the COVID19 vaccine was first published in 1961 and later debunked in 1990 [30], before reemerging at a time of urgent social need. However, despite the SDG framework’s limitations as an RA tool, its use in this and in other studies have highlighted a glaring gap in the ‘elite business journal’ literature on topics exploring sustainable business concepts and theory, and their practical applications. This gap in the literature should inspire stakeholders to “open the can” [20] (p. 17) on ‘business- as usual research’, toward a research agenda that would be better positioned to tackle pressing and future world challenges. As it stands today, without multi-stakeholder alignment towards this goal, faculty who are interested in contributing to the achievement of the SDGs will likely do so at their own expense, potentially risking jobs, and promotion. Arguably, urgent global matters require a more efficient means of facilitating academic contribution beyond publishing in the current list of ‘elite’ business journals. To this end, for future research we suggest that it is progress towards the “spirit of the SDGs” that should be assessed, using the distinct underlying themes identified through the current analysis. This approach leverages the framework’s overlapping and incomplete concepts to allow for a more generous and inclusive interpretation of evidence of where progress is being made—or not. Furthermore, we recommend future research that provides a deeper understanding of the publishing culture underpinning each of the FT50 journals, and the trajectory of future research as well as isolate embedded levers that could be used to redirect this focus—where appropriate—toward the urgent global matters acknowledged through the SDGs.
This paper is organized as follows. In Section 2, we review the literature that discusses challenges with current measures of research productivity and both the benefits and limitations of using the SDG framework as a research assessment tool. Additionally, two methods of analysis for mapping academic publications to the SDGs are reviewed, highlighting the plurality of perspectives and interpretations that can lead to varied results. Similar methodological approaches utilized in two different studies are then compared to demonstrate additional challenges associated with the SDGs as a RA tool. In Section 3, we describe our methodology. Section 4 presents the results. Following, in Section 5, we discuss how using the SDGs as a framework to measure the social impact of business school research could be better designed and integrated into ranking systems, such as the the FT50. Next, limitations to the current research and future research enhancements are identified. Finally, next steps and suggested changes to help push business research output toward impact are presented.

2. Literature Review

2.1. Challenges with Current Measures of Research Productivity and Their Impact on Published Rankings

For business school researchers, H-index scores and the ‘impact factors’ of the journals where their papers are published have become important metrics for measuring the productivity and quality of scholarly work [31]. Alperin, Nieves, Schimanski, Fischman, Niles and McKiernan [11] reviewed P&T documents from 129 NA universities and noted tensions in these documents between the public’s view of what constitutes high value scholarship and the internal focus on ‘high prestige’ or ‘high impact’ publications by committees. The inclusion of these same metrics in published rankings have intensified the perceived relevance and influence of these benchmarks.
Curry et al. [20] in their working paper highlighted long-standing problems associated with how published rankings assess research with impact, given their influence on institutional outcomes: “First, …there are narrow criteria and indicators of research quality or impact... that distort incentives, creates unsustainable pressures on researchers, and exacerbate problems with research integrity and reproducibility. Second, this narrowing of criteria and indicators has reduced the diversity of research missions and purposes, leading institutions, and researchers to adopt similar strategic priorities, or to focus on lower-risk, incremental work. Third, the systemic biases against those who do not meet—or choose not to prioritize—narrow criteria and indicators of quality or impact, or to conform to particular career pathways have reduced the diversity, vitality and representative legitimacy of the research community. Finally, there has been a diversion of policy and managerial attention towards things that can be measured, at the expense of less tangible or quantifiable qualities, impacts, assets and values—a trend exacerbated by the rise of flawed university published rankings” [20] (pp. 3–4).
As an example, the Financial Times published ranking [32], which claims to rank the ‘world’s best’ 100 full-time MBA programs, includes six criteria that arguably align with specific SDG targets. However, individually, and collectively they represent only a small fraction of the overall score (16% weight collectively). For example, linked to SDG 5 “Achieve gender equality and empower all women and girls” there are scores for the percentage of female faculty (2% weight), percentage of female students (2% weight), and percentage of women representation on the board of directors (1% weight), for a total of 5%. Linked to Goal 10, “Reduce inequality within and among countries”, there are scores for the percentage of international faculty (4% weight), the percentage of international students (2% weight), the percentage of international representation on the board of directors (2% weight), for a total of 8%. There is also a score which reflects the overall spirit of the SDGs, which is CSR related topics in course curricula (3% weight). Although, the FT50 includes these criteria as proxy indicators of institutional commitment to corporate social responsibility and EDI, they are arguably insufficient to offset the acknowledged perverse effects caused by other measures, such as the focus on salary increases of students and the focus on the number of publications by the institution’s faculty in 50 pre-determined journals known as ‘the FT50’ (a 10% weight). In other words, the layered-on SDG criteria may be undermined by other metrics, influencing faculty priorities with respect to both curriculum and research.

2.2. Challenges with SDGs, Targets, and Corresponding Indicators to Measure Research with Impact

The universal set of goals, targets, and indicators developed by UN member states, as represented by the SDGs, and with its overarching goal of achieving sustainable development in all its dimensions [33] was intended as a communication tool, to inspire countries and organizations to get on board with the intended societal “paradigm shift” as well as a tracking device against the 17 goals, allowing the UN to report annual on the progress being made. The effective communication of the framework between the many actors involved is critical in developing a mutual understanding of priorities and which actions to take [33]. Despite their many shortcomings, sustainability indices—such as the SDGs—can be effective tools for inspiring and communicating progress to various stakeholders, including practitioners and the public. However, using the SDGs inconsistently or in an uncoordinated manner to measure progress towards achieving these goals is problematic.
There are several challenges associated with using the SDGs as a framework to measure the social impact of research. Firstly, and most importantly, evidence suggests that there is a potential conflict between the socio-economic development and environmental sustainability goals [34,35]. Conducting research that helps to advance a country’s manufacturing base, for example, could support the achievement of Goal #8, Economic Growth, yet be at odds with Goal 12, Responsible Consumption and Production, or other environmental-related targets, such as Goal 11, Sustainable Cities and Communities. In short, if the goals are ‘cherry picked’ or pursued independently of each other, without a superordinate focus on ‘sustainability’. This may result in contradictions or perceived alignment, when the SDGs ‘in spirit’ are actually being undermined.
Secondly, the SDGs have been described as inconsistent, difficult to quantify, implement and monitor [36]. Both the Sustainable Development Solutions Network [34] and the International Science Council [35] have raised issues about the wording of the goals, claiming that a number of the targets could be constructed more clearly. The Center for Global Development (CGD) dedicated an entire blog series on how many of the targets could be improved if small changes in language were made [37]. Furthermore, the SDGs cover several broad themes and contain interrelated targets [38], making it difficult to separate goals as unique ideas and concepts. Additionally, the UN global indicator framework for SDGs, when reviewed by the Statistical Commission in March 2020, noted that 12 of the indicators used to measure progress were not unique to any one goal, but rather appeared under two or three different targets [38]. For example, SDG 7.b.1. indicator of progress toward Goal 7—“Ensure access to affordable, reliable, sustainable and modern energy for all”—was “installed renewable energy-generating capacity in developing countries (in watts per capital)” and this was identical to the SDG 12.a.1. indicator of progress toward Goal 12—“Ensure sustainable consumption and production patterns” [38] (pp. 7&12).
Additionally, depending on the version and information source, the SDG framework can include a variety of titles (short and long versions) and has been translated into many different languages, that are often then translated back to English, introducing additional challenges. These include, but are not limited to, language structure differences, multiple different meanings of key words, and different idioms and expressions peculiar to any one language.
Spangenberg [39] argued that the messages of the SDGs are ambiguous. Many in the scientific community support the notion that the SDG’s targets are weak, vague, or even meaningless [40]. Leblanc [41] found some goals and interpretations as straight forward, and others more difficult for non-experts to interpret. Beyond clarity, also concerning is that the indicators do not explicitly address barriers that can block a country’s progress toward achievement of the goals such as poor planning, poor and slow implementation, and lack of government or public involvement and support. Finally, the SDGs do not capture the subtle effect of research collaboration and inter- and multidisciplinary research as relevant and necessary conditions for progress in sustainability [42,43].

2.3. A Review of Different Methods and Approaches for Assessing SDG Relevance in Research

Despite the challenges just reviewed, several studies have attempted to map scholarly publications related to the SDGs. Their diverse results have been attributed to the different methodologies employed, different data sources used, the large number of indicators ascribed to each of the 17 SDGs, and the similarity of concepts across the goals, leading to various interpretations despite extensive user knowledge [33]. Rafols et al. [36] suggested that the differences were not due to technical issues, but rather were a result of different methodologies employed and varying interpretations of the SDGs. Bergen University [25], Elsevier [26] Aurora Network, SIRIS and STRINGS each completed this task using bibliometric techniques searching the content of research articles for strings of key words found in UN SDG policy documents. In contrast, NESTA and Dimensions [29], the Rotterdam School of Management (RSM) [23], and Cabell’s AI dashboard used the same SDG policy documents to train machine learning (ML) algorithms as opposed to using strings of key words to identify articles related to the SDGs [36].
Below, we highlight two of these studies to illustrate the differences between bibliometric and machine learning techniques.

2.3.1. Elsevier (Bibliometric Techniques)

Elsevier launched a report in 2020 entitled “The power of Data to advance SDGs” that provided valuable insight into the role published scholarly work plays in supporting societal change [44]. The report included SDG related publications from 2015–2019 over multiple disciplines, producing a word cloud highlighting the key terms extracted from the articles that captured specific SDGs. Additionally, they reported on the percentage of publications from high-income versus low-income countries, and the percentage of publications with corporate and/or international collaborations (identifying both high- and low-income international collaborations). Elsevier relied mainly on Boolean search queries containing SDG related terms on SCOPUS (an abstract and citation database) to identify SDG related publications. Developing the search queries for the SDG goals in Scopus involved two stages. In the first stage, an audit was conducted on a subset of SDG relevant articles by knowledge experts to determine accuracy of the Boolean search. The second stage involved updating the search query if warranted, given the results of the first audit, and re-running the data. A second audit was then conducted with the search queries updated as needed.

2.3.2. RSM AI Dashboard (Machine Learning)

The RSM [23] developed an SDG mapper that relied on machine learning, incorporating a natural language processing algorithm designed by Google (BERT). This approach is classified as using artificial intelligent (AI) technology as accuracy improves through continuous iterative processes. In addition to ranking business schools by the ratio of SDG publication to total publication output, the AI model reviewed the abstracts of all FT50 published articles from 2010–2019 to determine their SDG relatedness. The compilation of SDG keywords used open data sets published by multiple research groups. The resultant dataset was validated several times to maintain its robustness. Additionally, SDG identification was updated as new datasets became available. The FT50 abstracts of the articles published from 2000–2019 were analyzed through SCOPUS. A confidence interval of >50% was required for the abstract to be positively acknowledged as SDG related. Furthermore, the model identified individual articles as aligned with multiple SDGs.
While both studies used automated methods and relied on similar information to identify the SDG relevance of published articles—e.g., SCOPUS and SDG key words—each method has its own strengths and limitations. Both approaches allow for the assessment of large data sets and do so efficiently. The RSM AI dashboard, in particular, learns from the data, continuously improving its accuracy with each iteration. The speed in which it conducts its analysis is also superior to human methods. That said, bibliometric methods are more exposed to researcher biases when determining which key words should be included in the Boolean search [45]. Translation of interpretations into search terms is clearly challenging [25]. Interpretation skills are open to human judgement as they involve “…explaining, reframing, or otherwise showing your own understanding of something” [46] and that understanding could vary greatly between analysts. For ML, there is an accepted understanding by practitioners of the lack of transparency within the iterative processes of what the machine is actually learning (existence of a black box) [47]. While some differences in results are attributed to different methodologies, studies employing the same methodologies on the same data sets also arrived at varied results.

2.3.3. Comparison of Two Bibliometric Techniques Using Same Data Sets

Applying similar methods and same data sources (Web of Science (WOS) Core Collection between 2015–2018), Armitage, Lorenz, and Mikki found few similarities between the Bergen and Elsevier bibliometric approach. Bergen found 39,507 articles relating to SDG 1—No poverty—whereas Elsevier found 6793 articles, with only 13% of the Bergen articles shared with Elsevier [25]. These incongruent results were consistent across several articles for many of the SDGs. For bibliometric techniques researchers must decide what strings of words should be included in the Boolean searches. Armitage et al. [25] suggested that differences in results could be accredited to the research teams’ “(1) interpretation of the themes and concepts of the SDGs, (2) decisions around how publications must discuss these concepts to be considered as a “contribution” to the chosen interpretation of the SDG, (3) translation of concepts into a search query that will find contributing publications, and (4) data source (i.e., abstracts versus complete articles)” [25] (p. 3).
Interpretive processes are required for all methodologies given that in most cases societal contributions are not made explicit within the research analyzed. Subjective decision making is required to determine how a publication should discuss the themes to be considered relevant to an SDG. Does the SDG need to be explicitly stated, or can it just be implied? Should indirect contributions be counted?
In other literature, Harley and Fleming [48] reviewed and analyzed 5500 articles published in “top-tier” management journals between 2008 and 2018 and “found that only 2.8% of articles critically addressed global ‘grand challenges…’” identified by the SDGs “—such as inequality, climate change, racism, and gender discrimination” [48] (p. 1). Similarly, Steingard and Linacre, in their forthcoming chapter entitled “Transforming Academic Journal Assessment from ’Quality’ to ‘Impact’” applied an ‘SDG impact intensity algorithm’ to a selected list of 50 journals focused on sustainable development topics (SDG50) and to the FT50 journals from 2015–2020. They found that the FT50 had an extremely low SDG impact intensity score in stark contrast to the SDG50. [49].
The aforementioned literature highlights both the limitations associated with the SDG framework as a RA tool and the need to encourage academic research that is more explicitly aligned with the SDGs.
In an effort to help this important work evolve, an alternative method for mapping SDGs to published scholarly work is tested, which leverages both ML and human interpretation skills.
The study was designed to test explicitly the following three hypotheses:
Hypothesis 1 (H1).
The SDG framework has several overlapping concepts between each designated SDG.
Hypothesis 2 (H2).
FT50 journal themes for articles published in 2019 are not in close proximity to the overlapping themes identified in H1.
Hypothesis 3 (H3).
Some FT50 journals lie closer to the SDG themes than others.

3. Methodology

To test all three hypotheses, a network analysis was performed from the text that informed the SDGs, its 169 targets and 231 indicators and the 4576 FT 50 journal articles published in 2019. For this, we used Leximancer™’s Lexiportal V5, an online, cloud-based text analytics software tool that analyzes collections of text (up to two sentences at a time) and visually displays the extracted information in a concept map outlining the main themes and concepts that informed these themes [50].
Leximancer™ conducts its analysis in multiple phases. The software first utilizes both conceptual and relational analysis to create a list of concepts by measuring the presence and frequency of concepts, as well as the relationship between different concepts in texts [50]. Concept seeds are the starting point of each identified concept, with more terms being added to the seeds as the software continues the analysis, identifies like terms, and expands on the concept definition. The list of concepts is then used to generate a thesaurus of terms related to each concept seed. Leximancer™ groups the concepts into themes based on how often they appear together and how strongly related they are to each other. The software presents the themes on a concept diorama using a heat map where the strength of themes within the data are indicated by its colour. Red indicates the highest significance, followed by other warm colours, while themes with lesser significance appear in cooler tones such as purple and blue. The order of significance depicted by the heat map corresponds to the order of themes by number of hits or narratives found within the text that relate to this theme. The location of the themes on the map indicates the strength of the relationships between the themes, with some themes sharing similar narratives illustrated by overlapping theme circles. The size of the diameter of themes shown in the map represent the number of different narratives that informed the theme. Therefore, a larger theme circle has a larger number of diverse narratives, contained within the theme, whereas a smaller circle may have the same number of hits as a larger circle, but with more repeating narratives.
To test Hypothesis 1, the verbatim document of the UN’s 17 SDGs, including each goal’s title (excluding the title “Sustainable Development Goal” and its accompanying number), its targets, and indicators were fed into Leximancer™ allowing the software to automatically generate the first concept heat map. No manual settings or user-defined concepts were included. To over-come a limitation of AI text-analytics software, specifically its inability to capture the many nuances of the English language, our research team added ‘like’ terms and included alternative definitions and ideas that were similar to the ideas and concepts generated by the automatically generated concept seeds. For example, the term ‘food-bank’ was added to the auto-generated concept entitled ‘food insecurity’. The autogenerated list along with the ‘researcher defined concepts’ (AG+RD) were re-input into Leximancer™ generating a second concept heat map, that cast a wider net of narratives to better capture the SDG related themes. This second iteration of the heat map was used to create an assessment tool to test Hypothesis 2 and 3, specifically, to measure the relatedness of FT50 journal articles to the themes and concepts captured from the SDG framework. This tool we defined as representing the ‘spirit of the SDGs’.
To test Hypothesis 2 and 3, all 4576 FT50 articles published in 2019, and sorted by journal, were fed into Leximancer™ to generate a concept heat map for each journal. The journals were also organized by discipline category. The top themes for each journal and each journal category were documented and compared against the combined ‘automatically and human researcher defined’ (AG+RD) SDG concept map created in step 1. All heat maps created to test Hypothesis 1, 2, and 3 restricted the theme size to the auto-specified by the Leximancer™ program to generate eight major themes which also allowed for a commensurable analysis between concept maps. Note, restricting the theme size to eight does not remove concepts or key words from the heat map, rather it reorganizes the key concepts into new themes that connect their narratives. An SDG relatedness ‘incidence score’ was calculated for each journal based on the number of explicit and implicit narratives, concepts and theme hits from the SDG heat map found within the journal article set.

4. Results

4.1. Creating the Responsible Research Assessment (RRA) Tool

Figure 1 and Figure 2, present the auto-generated (AG) heat maps and auto-generated plus researcher defined (AG+RD) heat maps respectively. Both figures illustrate each SDG’s relationship to the eight themes. These figures illustrate the emerging themes captured by the 17 SDGs heatmaps as non-distinct, and frequently overlapping. This result confirmed past observations that SDGs cover several broad themes and contain interrelated targets and indicators [25].
The eight core themes that emerged from the automatically generated heatmaps (Figure 1) included (from most to least significant): Development, Governance, Vulnerable Populations, Education, Management, Water, GDP, and Food Security. The location of the SDG relative to the theme bubble indicates the closeness of that relationship. For example, SDG 11—Sustainable Cities and Communities—was closer to the Governance theme (contained within the theme bubble), than SDG 15—Life on Land—was to the theme entitled Development. Notably, the GDP theme bubble is separate from all other themes.
Figure 2 enhanced the autogenerated heat map with the inclusion of researcher defined concepts to help capture the many nuances peculiar to the English language. Similar themes emerged for the AG+RD heat map as the AG map. However, there were incidences of increases in the number hits per theme, verifying the logic for adding these ‘like’ terms. Researcher-defined concepts only showed up on a heat map if they existed and were significant within the text. For example, (Figure 1 and Figure 2) included a theme entitled Vulnerable Populations. In Figure 1, some of the key terms that generated this theme included: sex, age, persons, population, children, women, as well as several underlying phrases. Some of the researcher-defined concepts that were added expanded on these key terms (see Figure 2) and included words and phrases such as female board of director, people of colour, indigenous, BIPOC, etc. Consequently, Leximancer was able to identify four additional locations within the SDG targets and indicators framework that related to the theme of Vulnerable Populations.
The Eight (8) core themes that emerged from the AG+RD heatmap (Figure 2) included (from most to least significant): Sustainable Development, Governance, Vulnerable Populations, Water, GDP, Food Security, Restoration, and Public Health. In comparison to the AG heat map found in Figure 1, five themes were identical (Governance, Vulnerable Populations, Water, GDP, Food Security). The Development theme picked up the pre-qualifying tag of ‘Sustainable’, and Management and Education were replaced by the more significant themes of Restoration and Public Health. Education appeared as a concept, but now fell within the Governance theme. The Management theme spread across a variety of themes given the ‘user defined’ requirement that the broad ‘Management’ concept be qualified with a more detailed descriptor—i.e., ‘water’ management, ‘sustainable management’, etc.
Figure 2, like the autogenerated map in Figure 1, showed many of the SDGs located near each other with many connected to multiple themes (see Table 1). A direct link between a SDG tag and a theme represented at least one instance where the text that emerged within the seed was verbatim text from the tagged SDG.
Sustainable Development was the most significant theme with 201 hits, having a direct connection with all SDGs. The concept seed size indicated there were many diverse narratives informing this theme. These included ideas connected to sustainability, sustainable management, practices and policies, sustainability information, reforestation, and recycling. The SD theme shared concepts with both the Governance and Restoration theme. SDG 12—Responsible Consumption and Production—was located directly within both the Sustainable Development and Restoration theme.
The second most significant theme was Governance, with 131 hits and a direct relationship with all 17 SDGs. This theme had shared narratives with Sustainable Development, Food Security, Vulnerable Populations, and Water. Within the Governance theme, there were concepts related to information, access, service, education, rights, and equality. Where Governance overlapped with the Vulnerable Populations theme, concepts related to trafficking, torture, well-being, and health appeared. Both Health and Public appeared as separate concepts within the Governance theme, however, the Public Health theme resided outside the Governance bubble. SDG 4—Quality education—was located directly within both the Governance and Vulnerable Populations themes.
Next, the Vulnerable Populations theme had 104 hits. This theme overlapped with Governance, Food Security, and the Public Health theme. This theme included concepts characterizing different vulnerable populations—such as women, indigenous people, and children—as well as concepts discussing exploitation, such as torture and trafficking. The Vulnerable Populations theme had direct connections to 15 of the 17 SDGs. The two SDGs it did not have a direct connection to were SDG 12—Responsible Consumption and Production—and SDG 14—Life Below Water.
The Water theme overlapped the Governance theme only. The concepts in this concept seed included reduction and prevention, and threats to clean water such as pollution and extinction, as well as issues surrounding water management, and access. The Water theme had a direct connection to 14 of the 17 SDGs. The three SDGs it did not have a direct connection to were SDG 5—Gender Equality; SDG—9, Industry, Innovation, and Infrastructure; and SDG 13—Climate Action.
Following Water, the GDP theme had 20 hits. The GDP theme did not overlap with any other theme yet had direct connections to 9 of the 17 SDGs. These included SDG 1—No Poverty; SDG 7—Affordable and Clean Energy; SDG 8—Decent Work and Economic Growth; SDG 9—Industry, Innovation, and Infrastructure; SDG 10—Reduced Inequalities; SDG 11—Sustainable Cities and Communities; SDG 12—Responsible Consumption and Production; SDG 14—Life Below Water; and SDG 17—Partnerships for the Goals.
The Food Security theme had three (3) hits. Food Security overlapped with the Governance and the Vulnerable Populations themes. The concepts in this theme were related to populations that would be vulnerable to food insecurity, and barriers that occur due to lack of rights and inequalities. The Food Security theme had a direct connection to 15 out of the 17 SDGs. The two SDGs it did not have direct connections to were SDG 7—Affordable and Clean Energy, and SDG 15—Life on Land.
The next theme was Restoration, with the second lowest significance and a hit number of 2. This theme overlapped with Sustainable Development. The concepts found in this theme were related to restoration and the availability of information to support restoration. The Restoration theme had direct connections to 4 out of the 17 SDGs (SDG 11—Sustainable Cities and Communities; SDG 12—Responsible Consumption and Production; SDG 14—Life Below Water; and SDG 15—Life on Land).
Finally, the least significant theme shown with one hit, was Public Health. This theme overlapped with the Vulnerable Population theme and had a direct connection to 1 out of the 17 SDGs, SDG 3-Good Health and Well Being.
Given the difficultly in separating SDG goals as unique ideas and concepts, these heatmaps suggest that a more robust research assessment tool for measuring SDG relevance might focus on the extent to which the research analyzed is related to core themes captured across the 17 SDGs, rather than any specific SDG goal.
Thus, Hypothesis 1 is supported, confirming that there are several overlapping themes that interconnect the 17 SDGs.

4.2. Measuring FT50 Using the Newly Created RRA

Concept Maps were auto generated using Leximancer™ for each of the FT50 journals, based on papers published in 2019. Supplementary Material includes a table that captures the eight themes in order of importance that emerged for each journal (50) from this analysis. The top eight themes by journal are compared against the top eight themes generated for the SDG (AG+RD) heat map (see Figure 2 and Table 1).
Applying the same theme size restriction as the SDG heatmaps, Leximancer software identified 2,991,649 unique concepts from the 4576 articles contained within the 50 Financial Times journals for 2019. Across all journals, 10.6% of the top eight theme hits (303,294) were explicitly related to the eight core themes of the SDGs. Next, implied relationships were estimated using Leximancer™ software features that allowed the investigator to closely examine the sentences that informed the concept and add like ideas and statements. This additional method uncovered 641,140 implicit hits (+24.5%). Therefore, 34.9% of the themes found within the 50 FT journals had an explicit or implicit connection to the AD+RD themes found within the SDG framework. Hypothesis 2 was supported, as only 10.6% and 24.5% of the articles had an explicit and implicit connection, respectively, to the themes that emerged from the SDG framework.
The auto-generated heatmaps for the Journal of Business Ethics and the Accounting, Organizations and Society Journal are presented in Figure 3 and Figure 4 as two examples of the output generated from the network analysis conducted for each journal.
From the heat mapping, the eight themes that emerged from the Journal of Business Ethics were (from most to least significant): CSR (Corporate Social Responsibility), Ethical, Development, Governance Organizations, People, Law, Workers (see Figure 3). In contrast, Figure 4 shows the eight themes, the significance and the connections between themes that emerged from the articles contained within the Accounting, Organizations, and Society journal (from most to least significant): Management, Accounting, Firm, Governance, Auditors, People, Control, Users.
Results were next organized by five journal topic categories; Economics; Ethics; Finance and Accounting; Management, Human Resources and Organizations; Marketing and Consumer Behaviour; Operations and Entrepreneurship; and Magazines published by academia for practitioners (see Table 2).
Total explicit hits (304,636) broke down as follows: 75.6% were from the category Ethics, 11.5% from Management, HR, and Organizations, 11.2% from Operations and Entrepreneurship, 1.2% from Finance and Accounting, and 0.4% from Practitioner-Focused Magazines (see Figure 5). There were no explicit theme hits from the Economics, and Marketing and CB categories. Figure 6 compares the explicit journal hits by category. This figure captures the explicit theme hit rate by category relative to the percentage of all articles captured. For example, Ethics had approximately 76% of the explicit theme hits, and these came from 12% of all category articles analyzed
Total implicit hits (706,442) broke down as follows: 43.0% from Management, HR and Organizations, 18.1% from the category Ethics, 14.7% from Operations and Entrepreneurship, 10.1% from Finance and Accounting, 6.7% Marketing and Consumer Behavior, 6.0% from Economics, and 1.5% from Magazines (see Figure 7). Figure 8 compares the explicit SDG theme hits and article distribution by category.
Hypothesis 3 is supported, HR, Organizations, and Ethics journals have more related themes to the SDGs than Marketing and Consumer Behaviour, Economics, and Finance.

5. Discussion

The SDGs—while arguably a remarkable accomplishment, representing a global commitment to a set of targets to be reached by 2030—have been described as unclear, abstruse, and repetitive. Unfortunately, their inconsistent ‘content’ characteristics, suggest that when used as a RA instrument, the SDG framework can result in both unreliable and invalid findings [51]. This study attempted to address this shortcoming, by contributing to a common interpretation of the SDGs as a framework to help measure research with positive social impact. Utilizing Leximancer™ software and researcher defined concepts we found eight (8) high level distinct concepts or themes that define the ‘spirit of the SDGs’: Sustainable Development, Governance, Vulnerable Populations, Water, GDP, Food Security, Restoration, and Public Health.
Other researchers have conducted various analyses demonstrating the interconnectivity of ideas and concepts contained within the SDG framework. Joshi, Hughes, and Sisk, 2015 [52], found that the three dimensions of governance (security, capacity, and inclusion) were on the top of the importance list of concepts that crossed all SDGs. Other studies [53,54,55] clustered the 17 SDGs under more than three themes, using different methods, further corroborating the inter-connectivity between the goals, targets, and indicators. Dalampira and Nastis [56] conducted a networking analysis, to assess empirically how the 17 SDGs mapped to the three forced themes or pillars of economy–environment–society. Distances between the three themes on the network graphs revealed the significance of the 17 goals relative to the 3 themes. Like the results from this study, the three pillars found to be highly interconnected [57]. Specifically, ‘’15 of 17 SDGs were multidimensional, combining two or all three themes of Sustainable Development” [56] (p. 53). SDG 8—Decent Work and Economic Growth and SDG 9—Industry Innovation and Infostructure were the only two that sat at the intersection of society and economic development with no connection to the environment pillar [57]. Our study showed the GDP core theme separate from all other core themes, with SDG 8 and 9 in close proximity, and SDG 13—Climate Change with no connection to GDP. Arguably, economic growth in and of itself is not what was intended, as unchecked economic growth has historically been associated with the exploitation of the environment and of people. Progress on SDG 8, pursued and assessed in isolation, could therefore undermine other goals, such as SDG 6–Clean Water and Sanitation; SDG 11–Sustainable Cities and Communities; and SDG 15–Life on Land. The results from this study and others [34,35] suggest that there is a potential conflict between the socio-economic development and environmental sustainability goals, reinforcing the need to consider these goals in an integrated way.
Once the SDG RRA tool was established, auto-generated core themes for all FT50 2019 journals were compared to determine their relatedness to the SDG themes generated. Once theme connections were made, features within Leximancer™ enabled the identification of research trajectories within the themes and identification of the closeness of these themes to specific SDG numbers. Both the overall explicit and implicit theme related hits were low, 10.6% and 24.5%, respectively. Journal theme connections to the SDG themes were restricted to a subset of categories—e.g., Ethics and Management, Human Resources, and Organizations combined accounted for 87.1% of explicit and 61.1% of the implicit theme hits. There were no explicit relationships found in any FT50 journal for the themes of Vulnerable Populations, Water, Food Security, Restoration, and Public Health. The FT50 journal themes, except for the Journal of Business Ethics, found to have little overlap with the eight main themes generated from the SDG framework, and more than half of these themes were undetected within the journal articles analyzed. This result corroborates the previous studies mentioned [48,49].
This finding suggests that articles published in FT50 journals may be contributing little to the advancement of the SDGs. Furthermore, those that are, are contributing in a limited way, with the core themes of GDP and Governance being most prevalent. It is interesting to note that in 2019, no one paper published by a FT50 journal, explicitly mentioned the SDGs. Perhaps this is somewhat understandable, given the time it takes for faculty to conduct research, combined with the fact that these journals have a slow publication rate (on average these journals in 2019 took 19–20 months until acceptance, and 6–7 months to publish [58]). This of course is in and of itself concerning, as urgent global matters require a more efficient means of facilitating academic contribution.
When we looked for implicit evidence of contributions to the SDGs (based on an analysis of concepts), however, there remained scant evidence. This suggests that the SDGS—explicit or otherwise—did not figure prominently in FT50 journals in 2019. Whether these topics were not of interest to these journals, and/or not of interest to the contributing authors is unclear, yet journals that did publish related work (such as the Journal of Business Ethics, with 541 publications representing 12% of articles published in 2019 FT50), do not seem to have been short of content. Given this observation, business school researchers aspiring to publish in the FT50 may be being discouraged from pursuing themes associated with sustainability and positive social impact. Unfortunately, this suggests that faculty who were interested in contributing to the achievement of the SDGs, may be doing so at their own expense, potentially risking jobs and promotion. It also suggests that SDG related work was relegated to the domain of lower ranked journals, as it was simply not a priority for the vast majority of so called ‘elite’ journals or faculty working in ‘top ranked’ business schools. The fact that few such concepts were present in the practitioner-oriented magazines, indicates that little guidance was being provided to help business leaders adapt to the expectations of the SDGs as well.
Mio et al. [59], in collaboration with other researchers, noted the limited academic literature focused on helping businesses integrate SDGs into their business models [60]. Specifically, they found that the role of business in achieving the SDGs had yet to be explicitly highlighted in the literature [59]. Overall, they found through an investigative study of ABS journal rankings from 2015–2020, which provides a wide-range of journals stretching across fields central or relevant to business and management studies, that most academic papers on the role of business in achieving the SDGs were mainly published in low-ranked journals dealing with Business Management and Accounting. Specifically, they found 6 of 101 published SDG related papers in 3-star journals, 29 in 2-star and, 12 in 1-star journals (note, top rank journals in ABS are designated 4-star,3-star,2-star,1-star, with 1-star being the lowest in the rank). Furthermore, they showed that the majority of the published papers on the SDGs (54 out of 101) had appeared in non-ABS-ranked journals belonging mainly to the academic discipline of Energy.
We suggest that the lack of SDG related research in so called ‘top journals’ is a serious shortcoming that deserves addressing. Following a discussion of the limitations of the current study, we discuss current progress and propose actions that can be taken to address this pressing need.

5.1. Limitations

Assessing research outcomes and impact is challenging. There are different types of research—from purely theoretical, to integrative and applied—some more difficult to measure impact given the length of time for bodies of research to evolve. Additionally, research can have a significant positive impact on one small community, and hardly any impact when you look at it from a global perspective. Therefore, some research could easily be misjudged as not having research impact potential by bounded rationale analysts (anyone who cannot see the future with this task is bounded rationally). All types of research can have value, but increasingly society—and funders—are expecting an assessment of at least the potential for positive social impact. Reed et al. [61], in a review of past studies designed to capture research impact, concluded that there was no one best assessment to critically evaluate the impact of research on society.
The SDG framework, while directionally inspirational as a paradigm shift, falls short as a tracking tool or assessment device. Yet, that is in large measure its intended purpose, with countries and organizations expected to report on their progress against the goals each year. Indeed, the 2021 report underscores this purpose, reporting on areas where advances are being made, and where they are not; “Progress had been made in poverty reduction, maternal and child health, access to electricity, and gender equality, but not enough to achieve the Goals by 2030. In other vital areas, including reducing inequality, lowering carbon emissions and tackling hunger, progress had either stalled or reversed” [62] (p. 1). The use of the SDG framework and its accompanying policy documents as an assessment tool is not statistically valid or reliable, given the interpretative processes required for its use. While this study attempted to overcome the inter-connectivity and vague and ambiguous language contained in this framework, identifying eight core themes, this involved yet another interpretative process.
Although machine learning technologies offer an explicit, systematic approach, the limitations of the SDG framework, the AI and human interpretation methodologies employed in this study, amongst others, may have missed concepts that were implied and not plainly expressed. While we tried to overcome AI limitations by including ‘researcher added themes and like concepts’ to further describe the goals, the Bergman team [25] warns that given these limitations, the SDG rankings and associated tools should be used with caution.

5.2. Next Steps

Despite the SDG framework’s limitations as a RA tool, the SDGs—originally introduced in 2015, as a 15-year plan for global achievement—continues to be the blueprint for world leaders, helping countries and institutions frame agendas and engage in policy development for a specified time span and to measure progress. Although its use as an assessment tool provides varied results amongst researchers, this and other studies have agreed that there is a glaring gap in the ‘elite business journal’ literature on topics exploring sustainable business concepts and theory, and their practical applications. This gap in research is concerning as it could stand in the way of, or work against, the achievement of the SDGs given that the business community has been identified as being essential to the achievement of the UN’s 2030 agenda (UN2015) [63].
The Business and Sustainable Development Commission (WBCSD) recommended companies “incorporate SDGs into business strategy, to rethink entire organizations from this perspective, and collaborate with peers to move towards SDG achievement faster” [64]. There are unique capabilities that business can contribute toward achieving these goals, including “financing-specific expertise and knowledge, management and enforcement capabilities” with “a higher willingness to take risk” [65] (p. 16). However, although willing to contribute, organizations have highlighted the lack of knowledge available on how to activate their role. Price Waterhouse Coopers (PWCs) recently noted in their “SDG Challenge Report”, that “while there is a general acknowledgement of the importance of the goals, there is still not enough understanding of what concrete action should be or is taking place” [66] (p. 6). Previous researchers have noted that the role business should play in the attainment of the SDGs was unclear [39] and as such, research was needed more than ever to help navigate the role of business in achieving these goals [67].
We have reflected on what steps might be taken in support of changing the trajectory of research by business school faculty toward helping achieve the goals. Much of these suggestions come from our own research experience and networks as ‘change makers’.
First, to overcome the aforementioned limitations of the SDGs as a RA tool, for future research we suggest that it is the progress of research towards the ‘spirit of the SDGs’ that should be considered, with assessments that track progress against the distinct underlying themes and concepts identified through the current analysis. This approach leverages the framework’s overlapping and incomplete concepts to allow for a more generous and inclusive interpretation of evidence of where progress is being made and ensures that sustainability concepts are considered alongside GDP. Next, we recommend research that provides a deeper understanding of the publishing culture underpinning each of the FT50 journals to determine the trajectory of future research, as well as isolate embedded levers that could be used to redirect research toward the urgent global matters acknowledged through the SDGs.
Other suggestions include rewarding the occurrence of research publications that align with the core themes contained within the SDGs when assessing the value of research for P and T and incorporating this same criterion in established and revered business school published rankings (i.e., FT50), or adding more relevant journals to the ‘prestige list’. Academic publishers, editors, and reviewers also have a role. Julia Christensen Hughes presented a “top 10 list” at the UN Publishers Compact [68], of simple steps that would help enhance the validity and reliability of the SDG framework as a responsible assessment tool, motivate potential authors (through the peer-review process) to think about how their research would contribute to this area, and ensure a diversity of research trajectories that have greater possibilities of positive societal change.
Finally, for an academic community looking for meaningful work, research is needed to better understand the role business plays and how the SDGs could be incorporated in corporate strategies to accelerate the impact of their actions taken [66]. Additionally, more research is required to determine the contradictory incentives within an organization that work against realizing these targets [67,69].

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/su132414019/s1.

Author Contributions

Conceptualization, K.R. and J.C.H.; Methodology, K.R. and V.D.S.; Software, J.C.H.; Validation, K.R., V.D.S., and J.C.H.; Formal analysis, V.D.S.; Resources, J.C.H.; Data curation, V.D.S.; Writing—original draft preparation, K.R.; Writing—review and editing, J.C.H. and K.R.; Visualization, V.D.S.; Supervision, K.R.; Project administration, K.R. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Data Availability Statement

Data used for this study can be found in Supplementary Materials.

Conflicts of Interest

One of the authors, Julia Christensen Hughes, is a former dean, who prioritized becoming “ranked” as part of her efforts to build the global brand of her business school. Following an analysis of various rankings, she identified Corporate Knights Better MBA ranking, as the one most aligned with the aspirations of her school to “develop leaders for a sustainable world”. She also contributed to the development of the Positive Impact Rating and formerly served as member on its board. At the invitation of Corporate Knights and the United Nations (UN) Global Compact and Principles for Responsible Management Education (PRME) initiative, Julia also facilitated several “Deans Dialogue” events at Davos during the World Economic Forum on business school rankings. More recently, under the auspices of the Globally Responsible Leadership Initiative (GRLI) she was invited to engage with the UN’s Higher Education Sustainability Initiative (HESI), Rankings Roundtable and Publisher’s Compact events. While Julia provided input to the design of the current survey and its implementation, as well as discussions on the analysis of the results, to guard against any potential bias, she had no direct contact with the raw data.

References

  1. The SDGs in Action. Available online: https://www.undp.org/sustainable-development-goals (accessed on 20 August 2021).
  2. Higher Education Sustainability Initiative. Available online: https://sustainabledevelopment.un.org/sdinaction/hesi (accessed on 20 August 2021).
  3. Principles for Responsible Management Education. Available online: https://www.unprme.org/ (accessed on 31 July 2021).
  4. Blueprint for SDG Integration into Curriculum, Research, and Partnerships. Available online: https://www.academia.edu/43370734/BLUEPRINT_FOR_SDG_INTEGRATION_INTO_CURRICULUM_RESEARCH_AND_PARTNERSHIPS (accessed on 31 July 2021).
  5. Discussion Paper on Mission-Oriented Research. Available online: https://www.globalresearchcouncil.org/fileadmin//documents/Library/Discussion_Paper_Mission_Oriented_Research_for_the_2019_2020_RM.pdf (accessed on 16 August 2021).
  6. Bornmann, L. What is social impact of research and how can it be assessed? A literature survey. J. Am. Soc. Inf. Sci. Technol. 2013, 64, 217–233. [Google Scholar] [CrossRef]
  7. Re: What is High Impact Factor Journal? Available online: https://www.researchgate.net/post/What-is-high-impact-factor-journal#:~:text=A%20’high%20impact’%20journal%20is,’Impact%20Factor’%20of%20journals (accessed on 12 September 2021).
  8. Acker, S.; Webber, M. Discipline and Publish: The Tenure Review Process in Ontario Universities. In Assembling and Governing the Higher Education Institution; Palgrave Macmillan: London, UK, 2016; pp. 233–255. [Google Scholar]
  9. Green, R.G.; Baskind, F.R. The second decade of the faculty publication project: Journal article publications and the importance of faculty scholarship. J. Soc. Work. Educ. 2007, 43, 281–291. [Google Scholar] [CrossRef]
  10. Macfarlane, B. Defining and rewarding academic citizenship: The implications for university promotions policy. J. High. Educ. Policy Manag. 2007, 29, 261–273. [Google Scholar] [CrossRef]
  11. Alperin, J.P.; Nieves, C.M.; Schimanski, L.A.; Fischman, G.E.; Niles, M.T.; McKiernan, E.C. How significant are the public dimensions of faculty work in review, promotion and tenure documents? Elife 2019, 8, e42254. [Google Scholar] [CrossRef] [PubMed]
  12. Business School Rankings for the 21st Century. Available online: https://www.gmac.com/news-center/gmac-press-releases/business-school-rankings-for-the-21st-century-report (accessed on 30 March 2021).
  13. Badat, S. Global rankings of universities: A perverse and present burden. In Global Inequalities and Higher Education: Whose Interests are We Serving; Entwistle, N., King, R., Eds.; Palgrave Macmillan: London, UK, 2010; pp. 117–141. [Google Scholar]
  14. Rodenburg, K.; Rizwan, T.; Liu, R.; Christensen Hughes, J. Enhancing the Positive Impact Rating: A New Business School Rating in Support of a Sustainable Future. Sustainability 2021, 13, 6519. [Google Scholar] [CrossRef]
  15. San Francisco Declaration on Research Assessment. Available online: https://sfdora.org/read/ (accessed on 2 October 2021).
  16. Hicks, D.; Wouters, P.; Waltman, L.; De Rijcke, S.; Rafols, I. Bibliometrics: The Leiden Manifesto for research metrics. Nat. News 2015, 520, 429. [Google Scholar] [CrossRef] [PubMed]
  17. Global Young Academy Working Group on Academic Excellence. Available online: https://globalyoungacademy.net/ (accessed on 21 October 2021).
  18. Human Metrics Initiative. Available online: https://humetricshss.org/ (accessed on 17 September 2021).
  19. Wedlin, L. Ranking Business Schools: Forming fields, Identities and Boundaries in International Management Education; Edward Elgar Publishing: Cheltenham, UK, 2006. [Google Scholar]
  20. Curry, S.; de Rijcke, S.; Hatch, A.; Pillay, D.; van der Weijden, I.; Wilsdon, J. The changing role of funders in responsible research assessment: Progress, obstacles and the way ahead. RoRI Work. Pap. 2020, 3. [Google Scholar] [CrossRef]
  21. Driving Collective Action for the SDGs: The Role of Further Education—Building a Sustainable and Resilient Recovery. Available online: https://sdgs.un.org/events/driving-collective-action-sdgs-role-further-education-building-sustainable-and-resilient (accessed on 1 November 2021).
  22. Athavale, M.; Bott, J.; Myring, M.; Richardson, L. Deans’ perceptions of published rankings of business programs. J. Educ. Bus. 2017, 92, 81–88. [Google Scholar] [CrossRef]
  23. Rotterdam School of Management Erasmus University. Available online: https://www.rsm.nl/ (accessed on 30 September 2021).
  24. SDG Dashboard. Available online: https://sdgdashboard.sju.edu/?page_id=22 (accessed on 19 September 2021).
  25. Armitage, C.S.; Lorenz, M.; Mikki, S. Mapping scholarly publications related to the Sustainable Development Goals: Do independent bibliometric approaches get the same results? Quant. Sci. Stud. 2020, 1, 1092–1108. [Google Scholar] [CrossRef]
  26. Jayabalasingham, B.; Boverhof, R.; Agnew, K.; Klein, L. Identifying research supporting the United Nations Sustainable Development Goals. Elsevier Doc. 2019, 1. [Google Scholar] [CrossRef]
  27. Aurora Universities Network. Available online: https://aurora-network.global/ (accessed on 14 October 2021).
  28. SIRIS Academic. Available online: https://sirisacademic.com/ (accessed on 6 September 2021).
  29. Wastl, J.; Porter, S.; Draux, H.; Fane, B.; Hook, D. Contextualizing Sustainable Development Research. Digit. Sci. 2020. [Google Scholar] [CrossRef]
  30. The History of mRNA Applications. Available online: https://the-dna-universe.com/2021/04/15/the-history-of-mrna-applications/ (accessed on 20 November 2021).
  31. Richmond, G. From the president: Measuring research impact. Am. Sci. 2020, 108, 125. [Google Scholar]
  32. FT Global MBA Ranking 2021: Methodology and Key. Available online: https://www.ft.com/mba-method (accessed on 19 September 2021).
  33. Swain, R.B. A Critical Analysis of the Sustainable Development Goals. In Handbook of Sustainability Science and Research; Leal Filho, W., Ed.; World Sustainability Series; Springer: Berlin/Heidelberg, Germany, 2018. [Google Scholar] [CrossRef]
  34. Spaiser, V.; Ranganathan, S.; Bali Swain, R.; Sumpter, D. The Sustainable Development Oxymoron: Quantifying and Modelling the Incompatibility of Sustainable Development Goals. Int. J. Sustain. Dev. World Ecol. 2016, 24, 457–470. [Google Scholar] [CrossRef]
  35. Review of Targets for the Sustainable Development Goals: The Science Perspective. Available online: https://council.science/wp-content/uploads/2017/05/SDG-Report.pdf (accessed on 15 October 2021).
  36. Rafols, I.; Noyons, E.; Confraria, H.; Ciarli, T. Visualising plural mappings of science for Sustainable Development Goals (SDGs). SocArXiv 2021. [Google Scholar] [CrossRef]
  37. SDG Targets: Here′s How to Make Them Stronger. Available online: https://www.cgdev.org/blog/sdg-targets-heres-how-make-them-stronger (accessed on 25 September 2021).
  38. SDG Indicators. Available online: https://unstats.un.org/sdgs/indicators/indicators-list/ (accessed on 16 August 2021).
  39. Spangenberg, J. Hot Air or Comprehensive Progress? A Critical Assessment of the SDGs. Sustain. Dev. 2017, 25, 311–321. [Google Scholar] [CrossRef]
  40. Holden, E.; Linnerud, K.; Banister, D. The Imperatives of Sustainable Development. Sustain. Dev. 2017, 25, 213–226. [Google Scholar] [CrossRef]
  41. Le Blanc, D. Towards Integration at Last? The Sustainable Development Goals as a Network of Targets. Sustain. Dev. 2015, 23, 176–187. [Google Scholar] [CrossRef]
  42. Filho, W.L.; Ulisses, A.; Alves, F.; Pace, P.; Mifsud, M.; Brandli, L.; Caeiro, S.S.; Disterheft, A. Reinvigorating the sustainable development research agenda: The role of the sustainable development goals (SDG). Int. J. Sustain. Dev. World Ecol. 2018, 25, 131–142. [Google Scholar] [CrossRef]
  43. Reid, W.; Bréchignac, C.; Lee, Y.T. Earth System Research Priorities. Science 2009, 325, 245. [Google Scholar] [CrossRef]
  44. The Power of Data to Advance the SDGs: Mapping Research for the Sustainable Development Goals 2020. Available online: https://www.elsevier.com/__data/assets/pdf_file/0004/1058179/Elsevier-SDG-Report-2020.pdf (accessed on 30 October 2021).
  45. Szomszor, M.; Adams, J.; Fry, R.; Gebert, C.; Pendlebury, D.A.; Potter, R.W.K.; Rogers, G. Interpreting Bibliometric Data. Front. Res. Metr. Anal. 2021, 5, 30. [Google Scholar] [CrossRef]
  46. Definitions of Interpretataion. Available online: https://www.vocabulary.com/dictionary/interpretation (accessed on 7 October 2021).
  47. Felzmann, H.; Fosch-Villaronga, E.; Lutz, C.; Tamò-Larrieux, A. Towards Transparency by Design for Artificial Intelligence. Sci. Eng. Ethics 2020, 26, 3333–3361. [Google Scholar] [CrossRef]
  48. Harley, B.; Fleming, P. Not Even Trying to Change the World: Why Do Elite Management Journals Ignore the Major Problems Facing Humanity? J. Appl. Behav. Sci. 2021, 57, 133–152. [Google Scholar] [CrossRef]
  49. Steingard, D.; Linacre, S. sTransforming academic journal assessment from “quality” to “impact”: A case study of the SDG Impact Intensity academic journal rating artificial intelligence system. In Responsible Management Education and the Digital Transformation Challenge; Hauser, C., Amann, W., Eds.; Palgrave Macmillan: New York, NY, USA, 2022. [Google Scholar]
  50. Leximancer User Guide. Available online: https://doc.leximancer.com/doc/LeximancerManual.pdf (accessed on 15 June 2021).
  51. A Primer on the Validity of Assessment Instruments. Available online: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3184912/ (accessed on 12 December 2021).
  52. Joshi, D.; Hughes, B.; Sisk, T. Improving Governance for the Post-2015 Sustainable Development Goals: Scenario Forecasting the Next 50 years. World Dev. 2015, 70, 286–302. [Google Scholar] [CrossRef]
  53. Ali-Toudert, F.; Ji, L. Modeling and measuring urban sustainability in multi-criteria based systems—A challenging issue. Ecol. Indic. 2017, 73, 597–611. [Google Scholar] [CrossRef]
  54. Moldan, B.; Janoušková, S.; Hak, T. How to Understand and Measure Environmental Sustainability: Indicators and Targets. Ecol. Indic. 2012, 17, 4–13. [Google Scholar] [CrossRef]
  55. Mirshojaeian Hosseini, H.; Kaneko, S. Causality between pillars of sustainable development: Global stylized facts or regional phenomena? Ecol. Indic. 2012, 14, 197–201. [Google Scholar] [CrossRef]
  56. Dalampira, E.S.; Nastis, S. Mapping Sustainable Development Goals: A network analysis framework. Sustain. Dev. 2019, 28, 46–55. [Google Scholar] [CrossRef]
  57. Dhahri, S.; Omri, A. Entrepreneurship contribution to the three pillars of sustainable development: What does the evidence really say? World Dev. 2018, 106, 64–77. [Google Scholar] [CrossRef]
  58. Rodenburg, K.; Rowan, M.; Nixon, A.; Christensen Hughes, J. Embedded Biases in Top Tier Journals: Influencing the Trajactory of Business School Research. Work. Pap. 2021, in press. [Google Scholar]
  59. Mio, C.; Panfilo, S.; Blundo, B. Sustainable development goals and the strategic role of business: A systematic literature review. Bus. Strategy Environ. 2020, 29, 3220–3245. [Google Scholar] [CrossRef]
  60. Witte, C.; Dilyard, J. Guest editors’ introduction to the special issue: The contribution of multinational enterprises to the Sustainable Development Goals. Transnatl. Corp. 2017, 24, 1–8. [Google Scholar] [CrossRef]
  61. Reed, M.S.; Ferré, M.; Martin-Ortega, J.; Blanche, R.; Lawford-Rolfe, R.; Dallimer, M.; Holden, J. Evaluating impact from research: A methodological framework. Res. Policy 2021, 50, 104147. [Google Scholar] [CrossRef]
  62. The Sustainable Development Goals Report. Available online: https://unstats.un.org/sdgs/report/2021/ (accessed on 15 December 2021).
  63. UN Forum Highlights ‘Fundamental’ Role of Private Sector in Advancing New Global Goals. Available online: https://www.un.org/sustainabledevelopment/blog/2015/09/un-forum-highlights-fundamental-role-of-private-sector-in-advancing-new-global-goals/ (accessed on 8 October 2021).
  64. CEO Guide to the SDGs. Available online: https://www.wbcsd.org/Overview/Resources/General/CEO-Guide-to-the-SDGs (accessed on 8 October 2021).
  65. van Zanten, J.A.; van Tulder, R. Multinational enterprises and the Sustainable Development Goals: An institutional approach to corporate engagement. J. Int. Bus. Policy 2018, 1, 208–233. [Google Scholar] [CrossRef]
  66. PwC’s SDG Challenge 2019. Available online: https://www.pwc.com/gx/en/services/sustainability/sustainable-development-goals/sdg-challenge-2019.html (accessed on 21 September 2021).
  67. Schönherr, N.; Findler, F.; Martinuzzi, A. Exploring the interface of CSR and the Sustainable Development Goals. Transnatl. Corp. 2017, 24, 33–47. [Google Scholar] [CrossRef]
  68. HESI Webinar: SDG Publishers Compact: Aligning Scholarly Activity with the Sustainable Development Goals. 2021. Available online: https://www.iau-hesd.net/events/5309-hesi-webinar-sdg-publishers-compact-aligning-scholarly-activity-sustainable-development (accessed on 15 November 2021).
  69. Kolk, A.; Kourula, A.; Pisani, N. Multinational enterprises and the Sustainable Development Goals: What do we know and how to proceed? Transnatl. Corp. 2017, 24, 9–32. [Google Scholar] [CrossRef]
Figure 1. Auto-generated Leximancer map of the SDG targets and indicators, showing placement of each SDG on the map. There are detailed description and concepts underlying each theme.
Figure 1. Auto-generated Leximancer map of the SDG targets and indicators, showing placement of each SDG on the map. There are detailed description and concepts underlying each theme.
Sustainability 13 14019 g001
Figure 2. Leximancer ,ap of the SDG targets and indicators with auto-generated and researcher defined concepts, showing placement of each SDG on the map.
Figure 2. Leximancer ,ap of the SDG targets and indicators with auto-generated and researcher defined concepts, showing placement of each SDG on the map.
Sustainability 13 14019 g002
Figure 3. Auto-generated Leximancer map of the Journal of Business Ethics.
Figure 3. Auto-generated Leximancer map of the Journal of Business Ethics.
Sustainability 13 14019 g003
Figure 4. 4 Auto-generated Leximancer map of Accounting, Organizations, and Society.
Figure 4. 4 Auto-generated Leximancer map of Accounting, Organizations, and Society.
Sustainability 13 14019 g004
Figure 5. Category: Ethics has noticeably higher % of explicit SDG theme relationships.
Figure 5. Category: Ethics has noticeably higher % of explicit SDG theme relationships.
Sustainability 13 14019 g005
Figure 6. Explicit hits and article distribution by category.
Figure 6. Explicit hits and article distribution by category.
Sustainability 13 14019 g006
Figure 7. Category: Management, HR, and Organizations has noticeably higher % of implicit SDG theme relationships.
Figure 7. Category: Management, HR, and Organizations has noticeably higher % of implicit SDG theme relationships.
Sustainability 13 14019 g007
Figure 8. Implicit hits and article distribution by category.
Figure 8. Implicit hits and article distribution by category.
Sustainability 13 14019 g008
Table 1. SDG’s direct connections to concept themes found in Figure 2.
Table 1. SDG’s direct connections to concept themes found in Figure 2.
Sustainable DevelopmentGovernanceVulnerable PopulationsWaterGDPFood SecurityRestorationPublic Heath
SDG 1
SDG 2
SDG 3
SDG 4
SDG 5
SDG 6
SDG 7
SDG 8
SDG 9
SDG 10
SDG 11
SDG 12
SDG 13
SDG 14
SDG 15
SDG 16
SDG 17
The “✓” indicates a direct connection between the SDG and the theme, grey shaded area represents no connection. The shades of blue represent the location of each SDG on the concept map relative to the theme. SDG is located directly within the theme. SDG is located near the theme. SDG is located near more than one theme.
Table 2. FT50 journals categorized by journal discipline.
Table 2. FT50 journals categorized by journal discipline.
Journals by CategoriesJournal TitlesNo. of Articles for EachProportion of Total No. Articles
Economics
  • American Economic Review
  • Econometrica
  • Journal of Financial Economics
  • Journal of Political Economy
  • Quarterly Journal of Economics
  • Review of Economic Studies
50911.1%
Ethics
  • Journal of Business Ethics
54111.8%
Finance and Accounting
  • Accounting, Organizations, and Society
  • Contemporary Accounting Research
  • Journal of Accounting and Economics
  • Journal of Accounting Research
  • Journal of Finance
  • Journal of Financial and Quantitative Analysis
  • Review of Accounting Studies
  • Review of Finance
  • Review of Financial Studies
  • The Accounting Review
65814.4%
Practitioner-focused Magazines
  • Harvard Business Review
  • Sloan Management Review
2455.4%
Management, HR, and Organizations:
  • Academy of Management Journal
  • Academy of Management Review
  • Administrative Science Quarterly
  • Human Relations
  • Human Resource Management
  • Journal of Management
  • Journal of Management Studies
  • Management Science
  • Organization Science
  • Organization Studies
  • Organizational Behaviour and Human Decision Processes
  • Research Policy
  • Strategic Management Journal
138130.2%
Marketing and Consumer Behaviour
  • Journal of Applied Psychology
  • Journal of Consumer Psychology
  • Journal of Consumer Research
  • Journal of Marketing
  • Journal of Marketing Research
  • Journal of the Academy of Marketing Science
  • Marketing Science
4429.6%
Operations and Entrepreneurship
  • Entrepreneurship Theory Practice
  • Information Systems Research
  • Journal of Business Venturing
  • Journal of International Business Studies
  • Journal of Management Information Systems
  • Journal of Operations Management
  • Manufacturing and Service Operations Management
  • MIS Quarterly
  • Operations Research
  • Production and OperationsManagement
  • Strategic Entrepreneurship Journal
80017.5%
Total504576100%
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Rodenburg, K.; De Silva, V.; Christensen Hughes, J. SDGs: A Responsible Research Assessment Tool toward Impactful Business Research. Sustainability 2021, 13, 14019. https://doi.org/10.3390/su132414019

AMA Style

Rodenburg K, De Silva V, Christensen Hughes J. SDGs: A Responsible Research Assessment Tool toward Impactful Business Research. Sustainability. 2021; 13(24):14019. https://doi.org/10.3390/su132414019

Chicago/Turabian Style

Rodenburg, Kathleen, Vinuli De Silva, and Julia Christensen Hughes. 2021. "SDGs: A Responsible Research Assessment Tool toward Impactful Business Research" Sustainability 13, no. 24: 14019. https://doi.org/10.3390/su132414019

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop