Next Article in Journal
Knowledge, Attitude, and Acceptance of Sinopharm and AstraZeneca’s COVID-19 Vaccines among Egyptian Population: A Cross-Sectional Study
Next Article in Special Issue
The Impact of Migration Experience on Rural Residents’ Mental Health: Evidence from Rural China
Previous Article in Journal
Catch-Up Growth as a Risk Factor for Rapid Weight Gain, Earlier Menarche and Earlier Pubertal Growth Spurt in Girls Born Small for Gestational Age (SGA)—A Longitudinal Study
Previous Article in Special Issue
Promotion of Parenting and Mental Health Needs among Chinese Women Living in Japan: A Qualitative Study
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Agent-Based Modelling of Health Inequalities following the Complexity Turn in Public Health: A Systematic Review

1
MRC/CSO Social and Public Health Sciences Unit, School of Health and Wellbeing, University of Glasgow, Glasgow G3 7HR, UK
2
School of Health and Related Research, University of Sheffield, Sheffield S1 4DA, UK
3
College of Medical, Veterinary and Life Sciences, University of Glasgow, Glasgow G61 1QH, UK
4
School of Social and Political Sciences, University of Glasgow, Glasgow G12 8RT, UK
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2022, 19(24), 16807; https://doi.org/10.3390/ijerph192416807
Submission received: 18 November 2022 / Revised: 11 December 2022 / Accepted: 12 December 2022 / Published: 14 December 2022

Abstract

:
There is an increasing focus on the role of complexity in public health and public policy fields which has brought about a methodological shift towards computational approaches. This includes agent-based modelling (ABM), a method used to simulate individuals, their behaviour and interactions with each other, and their social and physical environment. This paper aims to systematically review the use of ABM to simulate the generation or persistence of health inequalities. PubMed, Scopus, and Web of Science (1 January 2013–15 November 2022) were searched, supplemented with manual reference list searching. Twenty studies were included; fourteen of them described models of health behaviours, most commonly relating to diet (n = 7). Six models explored health outcomes, e.g., morbidity, mortality, and depression. All of the included models involved heterogeneous agents and were dynamic, with agents making decisions, growing older, and/or becoming exposed to different health risks. Eighteen models represented physical space and in eleven models, agents interacted with other agents through social networks. ABM is increasingly contributing to our understanding of the socioeconomic inequalities in health. However, to date, the majority of these models focus on the differences in health behaviours. Future research should attempt to investigate the social and economic drivers of health inequalities using ABM.

1. Introduction

Systematic socioeconomic inequalities in health persist and continue to widen within many economically prosperous countries across the globe [1,2]. The socioeconomic gradient in health remains one of the main challenges for public health as socioeconomically disadvantaged individuals have a lower life expectancy and a higher risk of developing life-limiting illnesses, such as diabetes and cardiovascular disease, compared to their advantaged counterparts [3,4].
The theories and frameworks developed to understand the causes of and solutions to the socioeconomic gradient in health are undoubtedly complex. For example, the World Health Organization’s (WHO) Commission on the Social Determinants of Health (CSDH) developed a conceptual framework to illustrate the relationship between the social determinants of health and equity in health and wellbeing, which was multi-level and contained feedback loops [5]. The CSDH framework highlights the multi-faceted nature of inequality from the impact of the socioeconomic and political context to psychosocial factors and biology. Thus, there is an increasing recognition that health inequality is a complex or ‘wicked’ problem and systems simulation models are a useful tool to understand the underlying causes and mechanisms [6].
Complex systems are systems which consist of interacting parts or subsystems. Some key characteristics of complex systems are dynamic, resulting in adaptation to change, non-linear relationships, feedback loops, tipping points, and the emergence of macro-phenomena from interactions at the micro level (see, e.g., CECAN 2018) [7]. It is difficult to capture these relationships using a traditional epidemiological “risk factor” approach which uses linear reductionist models to test the relationships between decontextualised dependent and independent variables [8]. Agent-based modelling (ABM), a well-established methodological approach used widely in the field of social science, has been highlighted as a methodological approach that can be used to address this problem [6]. ABM involves simulating the actions and interactions of individual agents with other agents and their environment based on a set of specified rules and observing emergent phenomena [9]. Agents may adapt their own behaviour in response to previous behaviour, their social network, or environmental stimuli [9]. Not only can ABM be used to understand complex phenomena, but they can also be used to test the impact of policy interventions and inform policy decisions and have been successfully applied in other areas of public health, particularly for the control of infectious diseases [10].
ABM has been used successfully to understand the causes of inequality more broadly outside the field of public health. Famously, the Schelling model of segregation which identified residential segregation is generated in the presence of relatively simple nearest neighbour preferences and could be used to understand the racial segregation patterns in the USA [11]. Additionally, the Sugarscape model developed by Epstein and Axtell has offered insights into the generation of wealth inequality using a relatively simple model which simulates the land in which sugar is grown and can be harvested by individuals to become their wealth [12,13]. Individuals in the simulation are programmed to harvest the sugar closest to them; strikingly, even when the wealth available to all individuals at the beginning of the simulation is equal, trends in wealth inequality are produced even after a short simulation period. Additionally, only a very small proportion of individuals have high levels of wealth, while a much larger proportion have low levels of wealth. These models, alongside many others developed in the field of social science, have illustrated the benefits of using ABM to understand complex observable phenomena.
A review by Speybroeck and colleagues, covering research published before January 2013, explored how simulation models had been used in the field of socioeconomic inequalities in health specifically [14]. They found only four ABM studies, which focused on understanding differences in health behaviour or infectious disease transmission between socioeconomic groups. Speybroeck and colleagues concluded that ABM is the most appropriate computational modelling method to examine health inequalities as they can incorporate all the characteristics of a complex system such as the heterogeneity, interactions, feedback, and emergence [14]. However, while the four identified models contained many of the expected features of ABM (e.g., multi-level, dynamic, and stochastic), the Speybroeck review concluded that to better understand the complex mechanisms underlying health inequalities, more ABM that features feedback loops, temporal changes, and agent–agent and agent–environment interactions are required.
Since the Speybroeck review, there has been a methodological shift towards using complex system methods in public health and public policy, much supported by large investments in data accessibility and computing power. In the UK, this is also reflected in the Medical Research Council’s updated guidance for the development and evaluation of complex interventions [15] and the Her Majesty Treasury’s Magenta Book Annex “Handling Complexity in Policy Evaluation”, both published in 2021 [16]. This methodological turn has resulted in a significant increase in computational modelling papers in the public health literature in recent years; therefore, it is now timely to update and deepen the previous review. Here, we focus on the contribution of ABM to understand the socioeconomic inequalities in health specifically, by reviewing the application area (e.g., the inequality mechanisms studied, the choice of the health outcome(s), and the measure of socioeconomic position), and the details of the ABM approach (e.g., the represented complexity features and whether models have been validated). The aim of this review was to synthesise the growing evidence based on the use of ABM in the field of health inequalities research.

2. Materials and Methods

We followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [17]. The protocol for this review was developed and registered on the International Prospective Register of Systematic Reviews (protocol registration PROSPERO 2022 CRD42022301797). PubMed, Scopus, and Web of Science were searched from 1 January 2013 to 15 November 2022. The Scopus search was limited by subject area to Medicine; Social Sciences; Computer Science; Multidisciplinary; Mathematics; Nursing; Economics, Econometrics and Finance; Neuroscience; Health Professions; Psychology; Decision Sciences; and Engineering. For Web of Science, searches were made of the editions of Science Citation Index Expanded and Social Sciences Citation Index. For both Web of Science and PubMed, only the titles and abstracts were searched. An extensive list of search terms was used (see Table S1 in Supplementary Materials) to capture the themes of simulation modelling, socioeconomic inequality, and health. The search strategy was validated against that used in the Speybroeck review [14], confirming that all ABM studies included in that review also appeared using our search strategy.

2.1. Eligibility Criteria

Table 1 lists the inclusion criteria for this review; this criterion includes the population, exposures, comparisons, outcomes, and study designs (PECOS) required for a study to be eligible for inclusion. Studies were included if they: (i) were full papers published in English, and (ii) the paper described an ABM study with the purpose to understand the emergence and/or persistence of health inequalities in relation to either non-communicable disease or the differential response of different socioeconomic groups to health-related interventions. Papers were only included if they simulated human individuals or groups and investigated within-country socioeconomic inequalities (using measures such as the socioeconomic position, income, and education) in health, restricted to the differences in the health status, health behaviour, or access to healthcare. Papers in which healthy food access was modelled as a proxy for the consumption of healthy food were also included. Studies that developed ABM in combination with system dynamics or population-based models were included. There were no geographical restrictions.
Papers that modelled communicable diseases or water or food access/security as the health outcomes were outside the scope of this review and were therefore excluded. The studies published before 2013 were also excluded as these studies were covered in the Speybroeck review [14].

2.2. Screening

Searching returned a total of 2533 records. All the records were downloaded to EndNote X9 and imported to the EPPI-Reviewer. The total records were reduced to 1436 following the removal of duplicates. An initial screening was carried out by one reviewer (RW). Following title screening, 477 records were identified for abstract screening. A second reviewer (JB) independently double-screened a randomly selected subset of abstracts (20%). After title and abstract screening, 51 records were selected for full-text screening and 18 of these met the eligibility criteria for data synthesis (Figure 1). The second reviewer (JB) also independently screened all the selected full-text studies to validate that the included papers met all the eligibility criteria. Any disagreements were recorded and discussed to ensure consistency. Two further reviewers (CE and AH) assisted with the screening for papers queried on methodological grounds (n = 29), in instances where it was uncertain whether a simulation model met the inclusion criteria. Manual reference searching identified two additional papers which met the inclusion criteria, giving a final sample of 20 included studies.

2.3. Data Extraction

Data from the papers were extracted by one reviewer (RW). A second reviewer (JB) assessed the accuracy of the data extraction for all the included studies. In the case of a disagreement, both reviewers referred to the paper in question, and a consensus was reached. A data extraction matrix was developed which included the basic characteristics of the studies (the year, location, and study’s aims), variables modelled (socioeconomic measure and health outcome), model characteristics (multi-level, dynamic, feedback loop, stochastic, spatial, heterogenous, agent–agent interaction, and adaptation to environment), if and how the model was validated, the model’s function (framework development and/or to test an intervention/scenario), and the relevant findings. The model’s characteristics were not always explicit but could be derived from the methods section. The relevant findings were defined as those related to health or intervention outcomes stratified by a measure of the socioeconomic position.

2.4. Quality Assessment

Given the lack of an appropriate quality assessment or a risk of bias assessment tool to assess ABM, a quality assessment was not conducted, but we recorded the compliance with the reporting guidelines of the ODD (the overview, design concepts, and details) [18].

2.5. Analysis

Descriptive summary statistics were used to describe the search results and study’s characteristics. We describe the specific modelling details of the included studies using a narrative synthesis in which we group models based on the health outcome.

3. Results

3.1. Descriptive Analysis

The study characteristics for the 20 included papers are displayed in Table 2. The most common geographical settings for the models were the USA (n = 7) and the UK (n = 4). The other models were set in the Netherlands, Mexico, India, South Korea, Canada, and Japan. Only two models were abstract and did not have a geographical setting. Most of the included models were set at the city level (n = 10), other settings included the national (n = 5), state (n = 2), and district level (n = 1).
Most of the included papers described the ABM of the socioeconomic differences in health behaviour (n = 14). Three papers focused on explaining the socioeconomic differences in the physical health outcomes and three papers modelled a mental health outcome. The measures of the socioeconomic position covered the income (n = 14), educational attainment (n = 4), social grade (n = 2), and wealth (n = 1).
All of the included models were multi-level (they represented both individuals and structural entities), dynamic (captured changes over time), stochastic (based on probabilities), and had heterogeneous agents. Most models represented both the individuals and the environment with environmental features (e.g., shops, green spaces, and public transport). Often, in the models, agents could age, die, and change their behaviour over the course of the simulation. Only three papers used the ODD reporting guidelines when writing descriptions of their ABM [18].

3.2. Health Behaviours

Most models with a focus on the health behaviour modelled dietary behaviours (n = 7). Four of the models were concerned with physical activity and the access to green space, and three modelled substance use, specifically the consumption or purchase of alcohol and tobacco as a proxy for consumption.

3.2.1. Dietary Behaviour

Papers that used ABM to model the socioeconomic differences in dietary behaviours tested the impact of interventions on the consumption of sugar-sweetened beverages [19], the purchase of ultra-processed food [20], the consumption of fruits and vegetables [21,22], and the access to healthy food outlets [23]. The interventions were educational campaigns (e.g., nutrition warnings and school-based programmes), advertising campaigns, changes to tax, increasing access to vegetables, and reducing the cost of vegetables. However, two papers focused on the impact of residential segregation on the access to healthy food outlets as an explanation for the socioeconomic differences in dietary behaviours [24,25]. All models used the level of income of the individual or household, educational achievement, or both as the measure of the socioeconomic position.
The only paper that did not include a spatial component to the model was set at a national level, and explored the impact of tax, nutrition warnings, and advertising on the purchase of ultra-processed food in Mexico [20]. The other six models used artificial grid space [24], a 1-dimensional linear township [25], a raster map to represent the spatial distribution of income [21], or actual geographic space, including GIS modelling of real-life cities [19,22,23]. Six of the models included agent–environment interactions which often captured how individual agents engage with food outlets [21,22,23,24,25]. Only two of the included papers modelled agent–agent interactions through dietary social norms operationalised via a social network which influenced the taste preferences and health beliefs [22], and the purchasing of ultra-processed foods [20]. Five of the models featured feedback loops, these included the update of social norms based on behaviour over the course of the simulation [20,22], and the food outlet responses to the agent’s behaviour by closing and opening outlets [23], changing the type of food available for sale [24], or an increasing appetite and overeating following the consumption of foods high in fat, sugar, and salt [21]. Only two of the papers had attempted validation by comparing the simulated outcomes to the ‘observed’ outcomes in real world data [19,22].

3.2.2. Physical Activity and Use of Urban Green Space

All the models that investigated the socioeconomic differences in physical activity simulated intervention scenarios. These scenarios included additional physical education in schools, the promotion of active travel, educational campaigns, increasing the availability and affordability of sports activities, improving neighbourhood safety, and increasing the expense associated with driving [26,27,28]. All the models focusing on physical activity used the level of income of the individual or household as the measure of the socioeconomic status and explored a range of physical activity-related outcomes including the minutes of physical activity per day [26], sports participation [27], and walking [28]. Models concerning physical activity involved a spatial component operationalised as either a representation of the actual geographical space [26,27] or an artificial grid [28]. All the models simulated both agent–agent interactions (e.g., social interactions modelled via a social network that impacts behaviour) and agent–environment interactions (e.g., playing outdoors or engaging with sports facilities in the environment). Two models contained feedback loops, including the updating of social norms regarding exercise and travel preferences [27,28] and environmental feedback, including the safety and traffic levels of travel routes on the attitudes towards transport methods [28]. Two models were validated by comparing the simulated outcomes to the outcomes observed in the pre-existing data [26,28].
One paper modelled intra- and inter-city inequalities in visiting urban green spaces, specifically testing the mechanism that the decision to visit these spaces is influenced by an individual’s assessment of who had previously visited the space [29]. Given conflicting evidence, the model explored two possibilities: (1) that agents visit spaces that people like themselves to visit (homophilic preference) and (2) that individuals with a lower SES (socioeconomic status) prefer to spend time in areas which those of a high SES visit (heterophilic preference). This model used the occupational grade to classify the agents into either a high or low SES. The model spatially represented the cities of Edinburgh, Glasgow, Aberdeen, and Dundee, and simulated both agent–environment interactions in the form of visiting urban green spaces and agent–agent interactions via agents assessing the similarity of other agents visiting the green space. The feedback loop in this model was the update of who visited green spaces, which was a function of the update to whether ‘in-’ or ‘out-’ group members were present in those spaces. Given a lack of observed data, the model was validated using a pattern matching approach; the model could reproduce the observed patterns of urban green space visitation in a spatial microsimulation of Glasgow.

3.2.3. Substance Use

Two of the models that focused on the socioeconomic differences in substance use tested the impact of interventions, including alcohol taxation [30] and the restriction of menthol cigarette sales and tobacco retailer density [31]. One paper simulated several counterfactual scenarios which varied the degree of socioeconomic disparity and gender-related susceptibility to social influence in the context of smoking [32]. All the models used the income level of the individual or household as the measure of the socioeconomic position and investigated substance use in the form of smoking prevalence [32], tobacco purchasing [31], and the average number of alcoholic drinks per day [30].
Two models simulated agent–agent interactions including the influence of gender and socioeconomic social norms on an individual’s own smoking [32], and social network influences on drinking behaviour [30]. Two models were spatial; one represented the city of New York [30] and the other an abstract town called ‘Tobacco Town’ [31]. Two models simulated agent–environment interactions, such as travelling to and from locations and engaging with tobacco and alcohol retail outlets [30,31]. One paper not only focused on the consumption of alcohol but also examined the interaction between neighbourhood characteristics, social networks, sociodemographic characteristics, drinking, and violence [30]. Two models featured feedback loops in the form of updates to norms based on drinking and smoking behaviour [30,32]. One model validated the simulated outcomes by comparing these to the outcomes observed in real-world data on the prevalence of smoking in Japan [32].

3.3. Physical Health

Of the three models that focused on physical health outcomes, one examined the incidence of severe neonatal morbidity and deaths per 1000 live births averted [33], one looked at the health status and care need [34], and the other investigated the impact of an exposure to air pollution on the health status [35]. Two of the papers modelled the effect of potential interventions on the physical health outcomes [33,34]. In one, the intervention was the alteration of the eligibility criteria for government-funded social care, in the other increasing the responsibilities and coverage of community health workers. All the models used a different measure of the socioeconomic position including wealth quintile [33], approximated social grade [34], and educational attainment [35].
All three models included the individual and household levels and two included additional levels such as kinship networks and the regional level. One study represented space using a grid based on the geography of the modelled country [34] and two represented the actual geographic space [33,35]. An interaction with the environment was in the form of migration, seeking treatment at facilities, and the exposure to pollution. Only two models included a feedback loop, including feedback between the parental income level and childhood educational attainment and feedback between the level of disease and the probability of developing a further disease [34,35]. Only one model involved agents interacting with each other, in the form of a kinship network, which consisted of familial relationships [34]. None of these models validated their results using real world data. One of the models was used to create a complex theoretical framework to represent the social care system. The geographical and population data inputted into this framework could then be adjusted to model and understand the drivers of the unmet social care need in different countries [34].

3.4. Mental Health

Two of the three papers focusing on a mental health outcome examined the impact of transport on depression among older adults. The first examined the impact of multiple transport interventions [36], and the second examined that of a free bus policy on public transit use and depression [37]. An individual’s income was used as a measure of the socioeconomic status in both papers.
One model carried out three experiments: increasing the walkability and safety of neighbourhoods to promote walking; decreasing bus fares and bus waiting times; and adding bus lines and stations [36]. While the second model focusing on transport carried out four experiments: altering mean attitudes towards the bus; bus waiting times; the cost of parking; and fuel prices; each experiment was also carried out with and without the inclusion of the free bus policy [37]. Both models captured the individual and neighbourhood level. A feedback loop resulted in improved attitudes towards a certain mode of travel following the positive experience of that mode. The spatial element was applied to income segregation patterns. In one model, the agents interacted with each other in the form of social networks influencing the travel behaviour [37]. In both models, agents interacted with the environment by using transport. Both models were validated against empirical data on the prevalence of depression in the United States by gender, age, and income level.
The third paper examined the impact of reducing income inequality on depression among expectant mothers [38]. Four interventions to increase income were tested: two child benefit programs (ACB and CCB), universal basic income (UBI), and increasing minimum wage. This model focused on individuals, and while it captured the neighbourhood characteristics for each individual (e.g., a sense of safety and the prenatal services available), the environment was not spatially represented in the model. Agents could decide to make or break social connections with other agents, and whether to break ties with other agents with depression. This model was not validated.

3.5. What Can ABM Tell Us about Socioeconomic Inequalities in Health?

Studies investigating the explanations for the socioeconomic differences in health found that those of a higher socioeconomic position were more likely to be exposed to healthier environments and therefore engage in healthier behaviours and have better health outcomes. For example, one model found that a greater income segregation in communities led to a decreased access to healthy food for lower income households [25]. Another study which modelled agents’ movements from work to home found that regardless of the level of air pollution, those with a lower level of education consistently had the highest risk of developing an illness [35].
Models which tested the impact of interventions on the socioeconomic inequalities in health found that some interventions increased inequalities. For example, those of a high socioeconomic position improved their health behaviour more in response to educational campaigns concerning nutrition [22,23]. It was argued that nutritional education campaigns may be ineffective for those of a lower socioeconomic position due to a sensitivity to food prices and a lack of access to healthy alternatives [22]. Similarly, it was found that the promotion of active travel had greater benefits for those of a high socioeconomic position, as they are more likely to travel by car and travel by car more often to extra-curricular activities prior to the intervention [26].
However, there were some modelled interventions that decreased the socioeconomic inequalities in health. For example, one model tested the impact of a sugar-sweetened beverage tax and found that at 25% tax, the reduction in the consumption of sugar-sweetened beverages was greater in those from low-income populations [19]. This finding was largely the result of increases in price which made sugar-sweetened beverages less affordable to low-income households. Another study, which modelled the expanded responsibilities and increased coverage for accredited social health activists who perform postnatal check-ups, found that these interventions resulted in greater decreases in the neonatal morbidity and mortality among those of a low socioeconomic position [33]. Yang and colleagues also showed that, in older adults when attitudes towards bus use improved and the waiting time decreased, decreases in depression were estimated to be greater among low-income groups [37]. This larger increase was because those of a low income are less likely to own cars and are therefore more susceptible to an intervention to increase the uptake of public transport, which increases the number of non-work trips they take, which was beneficial to their mental health.

4. Discussion

This review included 20 papers that described the ABM of the socioeconomic inequalities in health that have been published since January 2013, the end point of the Speybroeck review which found only four ABM studies on the topic [14]. Using ABM in the context of socioeconomic health inequalities was most common in the USA and UK (n = 11). The included studies illustrated that ABM is a useful tool to understand complex problems and has been used flexibly to represent dynamic, multi-level processes, often in physical space, and to capture the interactions between individuals and interactions with their environment. These models can tell us about the causes of health inequalities, potential interventions to reduce health inequalities, and which interventions may inadvertently increase health inequalities.
Typically, ABM has been used to explore socioeconomic differences in health behaviours (n = 14) including diet, physical activity, access to green space, and substance use, but few have approached the socioeconomic differences in physical and mental health outcomes. Additionally, only one paper modelled access to healthcare as a potential explanation for socioeconomic inequalities in health [33]. To an extent this is unsurprising, given a historic focus on health behaviours in public health [39] coupled with the fact that ABM as a method captures how behaviours at the micro-level give rise to emergent phenomena at the macro-level [40].
Most ABM were used to test a range of interventions (n = 14), from educational campaigns to taxation, and were underutilised for other purposes, such as testing the explanatory value of the theory or mechanisms to explain the generation or persistence of socioeconomic inequalities in health. This is consistent with the Speybroeck review which found that all ABM studies were used to test an intervention or scenario [14] and highlights that a valuable feature of ABM is the ability to experiment and test a range of interventions in silico [40].
Less than half of the included studies (n = 9) attempted to validate their models and, to varying degrees, some using observational data or pattern matching methods. However, none of the included studies used structural validation techniques which would ensure that it is the intended “structure of the model that drives its behaviour” [41]. This finding is consistent with the Speybroeck review which found that only one ABM had been validated using observational data [14]. Additionally, only three of the included papers explicitly used and referred to the ODD protocol, the guidelines with the purpose of ensuring that ABM is described fully to facilitate its replication [18].
It is clear from the findings of this review that most existing ABM studies investigating the socioeconomic inequalities in health have focused on health behaviour. This individualistic focus on health inequalities in ABM efforts on this topic so far is not reflective of ABMs in the field of social science more generally. ABM has been used to understand broader social phenomena such as racial segregation and the generation of wealth inequality at the societal level [11,12,13]. While these patterns are generated by individual-level behaviours, these models do not seek to explain these behaviours. Reducing health inequality to understanding the differences in health behaviour is problematic given that research has shown that for the same level of any given behaviour, the health outcomes remain worse for the most socioeconomically deprived [42].

4.1. Limitations

Currently there is no available tool to assess the quality of ABM studies, and therefore we could not ensure that the models included in this review were of a high quality. There are a variety of quality assessment tools available to assess other study types, for example, the appraisal tool for cross-sectional studies (AXIS) which can be used to assess a study’s design, reporting quality, and risk of bias [43]. Given the increase in ABM studies in public health, it is critical to consider how we will assess the quality of these studies going forward.
While the Speybroeck review considered a breadth of simulation models [14], we chose to focus on ABM only, given the particular promise of ABM applied to health inequality research and the rapid increase in the use of simulation modelling techniques since 2013 [10]. The application of alternative simulation modelling techniques (e.g., microsimulation and system dynamics) to the socioeconomic inequalities in health in recent years awaits a further examination.

4.2. Future Research

Efforts thus far to use ABM to understand socioeconomic inequalities in health have focused on the contribution of health behaviour. However, this focus on health behaviour is at odds with calls from researchers to “move beyond bad behaviours” [44] and the position of influential public health organisations. For example, the WHO concluded that it is the underlying social and economic factors that determine health and health inequalities as opposed to health behaviours [45]. We are increasingly aware that health inequalities are not only the result of differences in health behaviour, yet little has been done using ABM to attempt to understand the complex relationships between the social and economic environment people live in and the influence on their health via pathways other than health behaviour.
There are explanations for socioeconomic inequalities in health that shift the focus from individual-level behaviours to the social determinants which themselves determine health and to an extent behaviour (e.g., the social determinants of health) [45]. Existing ABMs have started to look at the social drivers of health behaviours (e.g., the role of social network and social norms) [20,22], however they avoid alternative pathways through which social and economic factors directly or indirectly impact health. It has been argued that ABM could be used to investigate the mechanisms specified in social and economic explanations for health inequality [46]. An existing hypothetical example of how this may be done is the operationalisation of psychosocial theory [46]. Instead of a focus on health behaviours, operationalising psychosocial theory would involve simulating support seeking and giving among friendship networks which mediates health outcomes via stress pathways. Future research should consider how we can use ABM to simulate alternative mechanisms which could explain the socioeconomic inequalities in health that are not exclusively focused on health behaviour.

5. Conclusions

In recent years, ABM has increasingly been used to explain socioeconomic inequalities in health. ABM allows us to develop a deeper understanding of the complex consequences of individual heterogeneity, spatial settings, feedback, and adaptation resulting from agent interactions with each other and their environment. However, to date, much of the focus has been on understanding the role of health behaviours. The features of ABM provide the opportunity to investigate alternative, more complex explanations for socioeconomic health inequalities. Therefore, an important next step in public health is to attempt to operationalise explanations for the causes and consequences of health inequalities beyond representations of health behaviour.

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph192416807/s1, Table S1: Systematic Search Strategy.

Author Contributions

Conceptualization, C.E., A.H. and P.M.; methodology, J.B., R.W. and P.M.; validation, J.B.; formal analysis, J.B. and R.W.; investigation, J.B. and R.W.; data curation, R.W.; writing—original draft preparation, J.B. and R.W.; writing—review and editing, C.E., A.H. and P.M.; visualization, R.W.; supervision, C.E., A.H. and P.M. All authors have read and agreed to the published version of the manuscript.

Funding

This work was funded by the MRC Research Council [grant number MC_UU_00022/5], the CSO [grant number SPHSU20], and supported by the University of Sheffield and The Wellcome Trust [108903/B/15/Z].

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

References

  1. Mackenbach, J.P.; Bos, V.; Andersen, O.; Cardano, M.; Costa, G.; Harding, S.; Reid, A.; Hemström, Ö.; Valkonen, T.; Kunst, A.E. Widening socioeconomic inequalities in mortality in six Western European countries. Int. J. Epidemiol. 2003, 32, 830–837. [Google Scholar] [CrossRef] [PubMed]
  2. Mackenbach, J.P. The persistence of health inequalities in modern welfare states: The explanation of a paradox. Soc. Sci. Med. 2012, 75, 761–769. [Google Scholar] [CrossRef] [PubMed]
  3. Marmot, M.; Bell, R. The socioeconomically disadvantaged. In Social Injustice and Public Health; Oxford University Press, INC: New York, NY, USA, 2006; pp. 25–45. [Google Scholar]
  4. Bartley, M.; Sacker, A.; Clarke, P. Employment status, employment conditions, and limiting illness: Prospective evidence from the British household panel survey 1991–2001. J. Epidemiol. Community Health 2004, 58, 501–506. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Solar, O.; Irwin, A.A.; Conceptual Framework for Action on the Social Determinants of Health. Social Determinants of Health Discussion Paper 2 (Policy and Practice). 2010. Available online: http://apps.who.int/iris/bitstream/10665/44489/1/9789241500852_eng.pdf?ua=1&ua=1 (accessed on 10 August 2022).
  6. Silverman, E.; Gostoli, U.; Picascia, S.; Almagor, J.; McCann, M.; Shaw, R.; Angione, C. Situating Agent-Based Modelling in Population Health Research. Emerg. Themes Epidemiol. 2021, 18, 1–15. [Google Scholar] [CrossRef]
  7. CECAN. Policy Evaluation for A Complex World Version 2.0. 2018. Available online: https://www.cecan.ac.uk/wp-content/uploads/2020/09/CECAN-Manifesto-Version-2.0-April-2018-online.pdf (accessed on 10 August 2022).
  8. El-Sayed, A.M.; Scarborough, P.; Seemann, L.; Galea, S. Social network analysis and agent-based modeling in social epidemiology. Epidemiol. Perspect. Innov. 2012, 9, 1–9. [Google Scholar] [CrossRef] [Green Version]
  9. Galea, S.; Riddle, M.; Kaplan, G.A. Causal thinking and complex system approaches in epidemiology. Int. J. Epidemiol. 2010, 39, 97–106. [Google Scholar] [CrossRef]
  10. Hunter, E.; Mac Namee, B.; Kelleher, J. A Comparison of Agent-Based Models and Equation Based Models for Infectious Disease Epidemiology. In Proceedings of the 26th AIAI Irish Conference on Artificial Intelligence and Cognitive Science in AICS, Dublin, Ireland, 6–7 December 2018; pp. 33–44. [Google Scholar]
  11. Singh, A.; Vainchtein, D.; Weiss, H. Schelling’s segregation model: Parameters, scaling, and aggregation. Demogr. Res. 2009, 21, 341–366. [Google Scholar] [CrossRef] [Green Version]
  12. Epstein, J.M.; Axtell, R.L. Growing Artificial Societies: Social Science from the Bottom Up; Brookings Inst. Press: Washington, DC, USA, 1996. [Google Scholar]
  13. NetLogo Group. NetLogo 3.1.3 Software, Center for Connected Learning and Computer-Based Modeling; Sugarscape 3 Wealth Distribution; Northwestern University: Evanston, IL, USA, 2006. [Google Scholar]
  14. Speybroeck, N.; van Malderen, C.; Harper, S.; Müller, B.; Devleesschauwer, B. Simulation models for socioeconomic inequalities in health: A systematic review. Int. J. Environ. Res. Public Health 2013, 10, 5750–5780. [Google Scholar] [CrossRef] [Green Version]
  15. Skivington, K.; Matthews, L.; Simpson, S.A.; Craig, P.; Baird, J.; Blazeby, J.M.; Craig, N.; French, D.P.; McIntosh, E.; Petticrew, M.; et al. A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ 2021, 374, n2061. [Google Scholar] [CrossRef]
  16. HM Treasury. Handling Complexity in Policy Evaluation. Magenta Book 2020; Supplementary Guide; HM Treasury: London, UK, 2020. [Google Scholar]
  17. Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021, 372, n71. [Google Scholar] [CrossRef]
  18. Grimm, V.; Railsback, S.F.; Vincenot, C.E.; Berger, U.; Gallagher, C.; DeAngelis, D.L.; Edmonds, B.; Ge, J.; Giske, J.; Groeneveld, J.; et al. The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism. J. Artif. Soc. Soc. Simul. 2020, 23, 7. [Google Scholar] [CrossRef] [Green Version]
  19. Langellier, B.A.; Lê-Scherban, F.; Purtle, J. Funding quality pre-kindergarten slots with Philadelphia’s new “sugary drink tax”: Simulating effects of using an excise tax to address a social determinant of health. Public Health Nutr. 2017, 20, 2450–2458. [Google Scholar] [CrossRef] [Green Version]
  20. Langellier, B.A.; Stankov, I.; Hammond, R.A.; Bilal, U.; Auchincloss, A.H.; Barrientos-Gutierrez, T.; Cardoso, L.D.O.; Diez Roux, A.V. Potential impacts of policies to reduce purchasing of ultra-processed foods in Mexico at different stages of the social transition: An agent-based modelling approach. Public Health Nutr. 2021, 25, 1711–1719. [Google Scholar] [CrossRef]
  21. Salvo, D.; Lemoine, P.; Janda, K.M.; Ranjit, N.; Neilsen, A.; van der Berg, A. Exploring the Impact of Policies to Improve Geographic and Economic Access to Vegetables among Low-Income, Predominantly Latino Urban Residents: An Agent-Based Model. Nutrients 2022, 14, 646. [Google Scholar] [CrossRef]
  22. Li, Y.; Zhang, D.; Pagán, J.A. Social Norms and the Consumption of Fruits and Vegetables across New York City Neighborhoods. J. Urban Health Bull. N. Y. Acad. Med. 2016, 93, 244–255. [Google Scholar] [CrossRef] [Green Version]
  23. Blok, D.J.; de Vlas, S.J.; Bakker, R.; van Lenthe, F.J. Reducing income inequalities in food consumption: Explorations with an agent-based model. Am. J. Prev. Med. 2015, 49, 605–613. [Google Scholar] [CrossRef]
  24. Auchincloss, A.H.; Garcia, L.M. Brief introductory guide to agent-based modeling and an illustration from urban health research. Cad. Saude Publica 2015, 31, 65–78. [Google Scholar] [CrossRef] [Green Version]
  25. Suresh, S.S.; Schauder, S.A. Income segregation and access to healthy food. Am. J. Prev. Med. 2020, 59, e31–e38. [Google Scholar] [CrossRef]
  26. Almagor, J.; Martin, A.; McCrorie, P.; Mitchell, R. How can an agent-based model explore the impact of interventions on children’s physical activity in an urban environment? Health Place 2021, 72, 102688. [Google Scholar] [CrossRef]
  27. Blok, D.J.; van Lenthe, F.J.; de Vlas, S.J. The impact of individual and environmental interventions on income inequalities in sports participation: Explorations with an agent-based model. Int. J. Behav. Nutr. Phys. Act. 2018, 15, 1–10. [Google Scholar] [CrossRef]
  28. Yang, Y.; Auchincloss, A.H.; Rodriguez, D.A.; Brown, D.G.; Riolo, R.; Diez-Roux, A.V. Modeling spatial segregation and travel cost influences on utilitarian walking: Towards policy intervention. Comput. Environ. Urban Syst. 2015, 51, 59–69. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  29. Picascia, S.; Mitchell, R. Social integration as a determinant of inequalities in green space usage: Insights from a theoretical agent-based model. Health Place 2022, 73, 102729. [Google Scholar] [CrossRef] [PubMed]
  30. Keyes, K.M.; Shev, A.; Tracy, M.; Cerdá, M. Assessing the impact of alcohol taxation on rates of violent victimization in a large urban area: An agent-based modeling approach. Addiction 2019, 114, 236–247. [Google Scholar] [CrossRef] [PubMed]
  31. Combs, T.B.; McKay, V.R.; Ornstein, J.; Mahoney, M.; Cork, K.; Brosi, D.; Kasman, M.; Heuberger, B.; Hammond, R.A.; Luke, D. Modelling the impact of menthol sales restrictions and retailer density reduction policies: Insights from tobacco town Minnesota. Tob. Control 2020, 29, 502–509. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  32. Chao, D.; Hashimoto, H.; Kondo, N. Dynamic impact of social stratification and social influence on smoking prevalence by gender: An agent-based model. Soc. Sci. Med. 2015, 147, 280–287. [Google Scholar] [CrossRef]
  33. Nandi, A.; Colson, A.R.; Verma, A.; Megiddo, I.; Ashok, A.; Laxminarayan, R. Health and economic benefits of scaling up a home-based neonatal care package in rural India: A modelling analysis. Health Policy Plan. 2016, 31, 634–644. [Google Scholar] [CrossRef] [Green Version]
  34. Gostoli, U.; Silverman, E. Modelling social care provision in an agent-based framework with kinship networks. R. Soc. Open Sci. 2019, 6, 190029. [Google Scholar] [CrossRef] [Green Version]
  35. Shin, H.; Bithell, M. An agent-Based assessment of health vulnerability to long-Term particulate exposure in Seoul Districts. J. Artif. Soc. Soc. Simul. 2019, 22, 12. [Google Scholar] [CrossRef]
  36. Yang, Y.; Langellier, B.A.; Stankov, I.; Purtle, J.; Nelson, K.L.; Diez Roux, A.V. Examining the possible impact of daily transport on depression among older adults using an agent-based model. Aging Ment. Health 2019, 23, 743–751. [Google Scholar] [CrossRef] [Green Version]
  37. Yang, Y.; Langellier, B.A.; Stankov, I.; Purtle, J.; Nelson, K.L.; Reinhard, E.; Van Lenthe, F.J.; Roux, A.V. Public transit and depression among older adults: Using agent-based models to examine plausible impacts of a free bus policy. J. Epidemiol. Community Health 2020, 74, 875–881. [Google Scholar] [CrossRef]
  38. Benny, C.; Yamamoto, S.; McDonald, S.; Chari, R.; Pabayo, R. Modelling Maternal Depression: An Agent-Based Model to Examine the Complex Relationship between Relative Income and Depression. Int. J. Environ. Res. Public Health 2022, 19, 4208. [Google Scholar] [CrossRef]
  39. Baum, F.; Fisher, M. Why behavioural health promotion endures despite its failure to reduce health inequities. Sociol. Health Illn. 2014, 36, 213–225. [Google Scholar] [CrossRef]
  40. Bianchi, F.; Squazzoni, F. Modelling and social science: Problems and promises. In Modelling Transitions; Routledge: Abingdon, UK, 2019; pp. 60–74. [Google Scholar]
  41. Qudrat-Ullah, H. Structural Validation of System Dynamics and Agent-based Simulation Models. In Proceedings of the 19th European Conference on Modelling and Simulation, Riga, Latvia, 1–4 June 2005; Volume 94. [Google Scholar]
  42. McCartney, G.; Collins, C.; Mackenzie, M. What (or who) causes health inequalities: Theories, evidence and implications? Health Policy 2013, 113, 221–227. [Google Scholar] [CrossRef]
  43. Downes, M.J.; Brennan, M.L.; Williams, H.C.; Dean, R.S. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016, 6, e011458. [Google Scholar] [CrossRef] [Green Version]
  44. Katikireddi, S.V.; Higgins, M.; Smith, K.E.; Williams, G. Health inequalities: The need to move beyond bad behaviours. J. Epidemiol. Community Health 2013, 67, 715–716. [Google Scholar] [CrossRef] [Green Version]
  45. Marmot, M.; Friel, S.; Bell, R.; Houweling, T.A.; Taylor, S. Closing the gap in a generation: Health equity through action on the social determinants of health. Lancet 2008, 372, 1661–1669. [Google Scholar] [CrossRef]
  46. Boyd, J.; Bambra, C.; Purshouse, R.C.; Holmes, J. Beyond behaviour: How health inequality theory can enhance our understanding of the ‘alcohol-harm paradox’. Int. J. Environ. Res. Public Health 2021, 18, 6025. [Google Scholar] [CrossRef]
Figure 1. PRISMA Flow Diagram.
Figure 1. PRISMA Flow Diagram.
Ijerph 19 16807 g001
Table 1. Population, exposures, comparisons, outcomes, and study design criteria for study inclusion.
Table 1. Population, exposures, comparisons, outcomes, and study design criteria for study inclusion.
CriteriaDefinition
PopulationHuman individuals or groups
ExposuresSocioeconomic position (any measure including income, educational status, occupation, and employment status)
ComparisonsCompares individuals to other individuals with a different level of socioeconomic position (e.g., high vs. low)
OutcomesHealth (any measure that captures health status, health behaviour or access to healthcare)
Study designOnly ABM studies were included
Table 2. Characteristics of included papers.
Table 2. Characteristics of included papers.
Author, Year Country of Authors Population Aim of Study SE Factor Modelled Health Outcome Modelled Characteristics of the Model Validation and Utilisation of Model
MLDStFLSpHtIAIEIVFI
Almagor et al., 2021 UKGlasgow To explore the potential impact of interventions on physical activity of children in an urban environment.SEP divided into 4 levels representing a gradient of household income: AB-high, C1, C2, DE-low. Minutes of moderate-to-vigorous physical activity/day.
Auchincloss and Garcia, 2015 USA, BrazilAbstract SpaceIntroduce guide for agent-based modelling and explore impact of urban segregation on inequalities in diet.Urban segregation by household income—location and income of households (divided into low or high-income).Average proportion of times the household shopped at a healthy food store (depends on household income, proximity to stores, and food preferences).
Benny et al., 2022CanadaCalgaryTo simulate the effects of government transfers and increases to minimum wage on depression in mothers. Individual Income categorised into CAD 39,999 or less, CAD 40,000 to 79,999, and CAD 80,000 or more. Education categorised into high school or less, some or completed university/college, and some or completed graduate school.Depression measured using the Edinburgh Postnatal Depression Scale (EPDS).
Blok et al., 2015 The NetherlandsEindhoven To explore the impact of 3 interventions (eliminating residential income segregation, reducing prices of healthy food, health education) aimed at reducing food consumption inequalities between low and high-income households.Household Income divided into high (>USD 31,777/year) and low (<USD 31,777/year).Average proportion of times a household visited a healthy food outlet.
Blok et al., 2018 The NetherlandsEindhoven Explore impact of 5 interventions (health education, lowering prices of sports facilities, increasing availability of sports facilities, improving neighbourhood safety, combining all these interventions) on reducing income inequalities in sports.Individual Income divided into low, middle, and high.% of individuals participating in sport annually.
Chao et al., 2015 JapanJapan Explore how socioeconomic disparity between and within gender groups affects changes in smoking prevalence.Socioeconomic Status divided into 1–9 according to distribution of income.% of each gender group who were smoking.
Combs et al., 2020 USATobacco Town, Minnesota Project the impact of menthol cigarette sales restrictions and retailer density reduction policies on tobacco sales for low income, African American and LGBTQ+ populationsIndividual Income, divided into two groups: low-income (<USD 42,500) and high-income (>USD 42,500).Costs to consumers per pack of cigarettes as proxy for tobacco consumption.
Gostoli and Silverman, 2019 UKUK Provide theoretical framework to understand drivers of unmet social care need and test policies.Approximated Social Grade, a socioeconomic classification produced by the Office for National Statistics (six categories A, B, C1, C2, D, and E).Health status and care need (weekly hours of care required); death (affected by agents’ level of unmet care need).
Gouri Suresh and Schauder, 2020 USAUSAExplore how income segregation impacts food access for poor, when preferences and knowledge of healthy foods are equal among different income groups.Household Income randomly generated on the basis of 2016 income distribution reported by US census bureau.Distance to nearest grocery store and whether healthy food was reliably available at nearest grocery store.
Keyes et al., 2019 USANew York City Estimate the impact of alcohol taxation on drinking, violence and homicide.Household Income—stratified into 5 quintiles.Average number of alcoholic drinks per day.
Langellier et al., 2017 USAPhiladelphia Evaluate impact of beverage tax and pre-kindergarten programme on children’s SSB consumption.Household Income—categorised as low-income (≤100% of Federal Poverty Level) and modest-income (≤300% of FPL) households.Sugar Sweetened Beverage consumption in drinks/week.
Langellier et al., 2021 USA, Australia, BrazilMexico Develop a simulation framework to assess how tax, nutrition warning and advertising impact ultra-processed food purchasing.Individual Income, divided into low-income (<1890 pesos/week) and high-income (>1890 pesos/week). EA, divided into low- (less than high school education) and high-education (at least high school).Ultra-processed food purchased, measured in kcal (energy intake) purchased per week.
Li et al., 2016 USANew York City Simulate how mass media and nutrition education change fruit and vegetable consumption in NYC.Educational Attainment, categorised by less than high school, high school, some college and college and above.Proportion of the population in a given neighbourhood who consume on average >2 servings of fruit and vegetables per day.
Nandi et al., 2016 USA, India, UKIndia Estimate reduction in disease burden by scaling up home-based newborn care in rural India. Wealth quintile. Incidence cases of severe neonatal morbidity averted and deaths per 1000 live births averted.
Picascia and Mitchell, 2022 UKEdinburgh, Dundee, Glasgow, Aberdeen Investigate intra- and inter-city inequalities in Urban Green Spaces visiting by SES.SES divided into 4 categories based on occupational grade: AB-high, C1, C2, DE-low.Median number of visits to urban green space/year.
Salvo et al., 2022USAAustin, TexasTo simulate the food environment and test the impact of different food access policies on vegetable consumption.Annual Household Income categorised into: Under USD 25,000, USD 25,001–USD 45,000, USD 45,001–USD 65,000, and >USD 65,000, and educational attainment in four categories: <High school, High school or GED, Some college, and Full college or more.Fruit and vegetable intake.
Shin and Bithell, 2019 UKGangnam and Gwanak districts, Seoul Understand cumulative effects of PM10 exposure on population vulnerability by education level and age.Educational Attainment in 8 categories: primary-school dropout, primary-school graduate, middle-school dropout, middle-school graduate, high-school dropout, high-school graduate, college or university student, over a bachelor’s degree.Health status: starts with 300 and drops when exposed to pollution; categorised into <100, 100–150, or 150–200.
Yang et al., 2015 USAUS city Explore how travel costs and educational interventions can alter income differentials in walking.Household Income segregation—income divided into quintiles (1 to 5).Proportion of trips to destinations made by walking.
Yang et al., 2019USAAbstract spaceInvestigate how transport interventions may affect depression in older adults.Individual Income—divided into quintiles (1 to 5). Depression status yes/no, where having depression is a score of >/=4 on CESD Scale-8.
Yang et al., 2020 USA, UK, The NetherlandsEnglish city Examine the impact of a free bus policy on public transit use and depression among older adults.Individual Income—divided into quintiles (1 to 5).Prevalence/ percentage of agents with depression.
ML—multi-level. D—dynamic. St—stochastic. FL—feedback loop. Sp—spatial. HtI—heterogeneous individuals. AI—agent–agent interactions. EI—agent–environment interactions. V—validation. F—framework. I—test an intervention.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Boyd, J.; Wilson, R.; Elsenbroich, C.; Heppenstall, A.; Meier, P. Agent-Based Modelling of Health Inequalities following the Complexity Turn in Public Health: A Systematic Review. Int. J. Environ. Res. Public Health 2022, 19, 16807. https://doi.org/10.3390/ijerph192416807

AMA Style

Boyd J, Wilson R, Elsenbroich C, Heppenstall A, Meier P. Agent-Based Modelling of Health Inequalities following the Complexity Turn in Public Health: A Systematic Review. International Journal of Environmental Research and Public Health. 2022; 19(24):16807. https://doi.org/10.3390/ijerph192416807

Chicago/Turabian Style

Boyd, Jennifer, Rebekah Wilson, Corinna Elsenbroich, Alison Heppenstall, and Petra Meier. 2022. "Agent-Based Modelling of Health Inequalities following the Complexity Turn in Public Health: A Systematic Review" International Journal of Environmental Research and Public Health 19, no. 24: 16807. https://doi.org/10.3390/ijerph192416807

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop