Applying the Delphi Approach to Incorporate Voiceless Stakeholders in Community Planning

Abstract

In the planning process, it is crucial to involve every key stakeholder for effective collaborative participation. However, in real-world practice, planners often face difficulties engaging stakeholders who remain passive. The authors argue that the Delphi method, when modified to enhance planners’ roles as a communication channel, can bridge this gap, serving as a tool to facilitate communication with these passive stakeholders and fostering consensus between those who actively voice their opinions and those who do not. Therefore, this study introduces a ‘planner-aided’ policy Delphi method, which is a revised version of existing Delphi methods tailored to better serve these goals. The authors then explore the effectiveness of this proposed method in engaging voiceless stakeholders in the participatory planning process through a case study of the Seohak-dong Art Theme Street design project in Jeonju, South Korea. This study finds that the PAP Delphi technique integrated into participatory planning effectively reflected the thoughts of voiceless stakeholders. This research contributes by proposing a practical and easy-to-use method for urban planners to engage less-vocal groups and demonstrating its effectiveness.

1. Introduction

In participatory planning, fostering authentic dialogue among diverse stakeholders is paramount [1,2]. Such dialogue not only facilitates information exchange but also promotes mutual understanding and trust, leading to consensus [3,4]. Face-to-face interactions are often the preferred method of communication in the participatory planning process [5,6,7,8]. However, factors such as social or power imbalances among stakeholders can make these interactions uncomfortable for some, and this discomfort can lead them to become ‘voiceless’ [9,10,11]. In this study, ‘voiceless’ stakeholders are defined as those who, due to such discomfort, lose the will or interest to participate and voice their opinions in the planning process. The existing literature emphasizes the importance of supporting these less-vocal groups by empowering them through education and information sharing, as well as technical and financial assistance [5,12,13]. Nevertheless, due to the significant time and effort required by such methods [12], practicing planners face challenges, particularly when administrative support for community involvement is weak or when project constraints limit time and resources [10,14]. Therefore, it is vital to identify practical strategies that practicing planners can adopt within given constraints. Delphi methods may be one such strategy to address these challenges. They offer a way for both outspoken stakeholders and those who are less vocal to find common ground more efficiently than through traditional community empowerment approaches [15,16]. By employing Delphi methods, all stakeholders including the less vocal can maintain anonymity and engage in a form of two-way communication, avoiding the potential conflicts that often arise from face-to-face interactions. This is achieved through repeated participation in surveys and receiving feedback on the survey results [17,18]. Particularly, the policy Delphi approach, which is generally better suited for heterogeneous groups, may be more appropriate for participatory planning compared to the classical Delphi approach, which is intended for homogeneous groups. However, because policy Delphi processes often fall short in reflecting the voices of less-vocal stakeholders, the approach needed to be modified to strengthen the role of planners as communication channels among stakeholders. The authors revised it and have termed this approach the ‘planner-aided’ policy Delphi method (henceforth PAP Delphi).

This study aims to explore the key question of whether applying the PAP Delphi method to the participatory planning process effectively captures and reflects the perspectives of stakeholders who are less vocal. This main question is further divided into two sub-questions: (1) ‘Do stakeholders who were reluctant to engage in face-to-face interactions participate in the PAP Delphi survey?’ and (2) ‘Does the PAP Delphi method contribute to building consensus among conflicting stakeholder groups?’ To address these questions, we conducted a case study focused on the design project for Seohak-dong Art Theme Street in Jeonju, South Korea. In this project, practicing planners employed the PAP Delphi technique to engage stakeholders who had previously been passive in the planning process. Serving as planners responsible for designing and executing the Delphi surveys, the authors primarily used participant observation as the data collection method. The data collected included planning documents, meeting transcripts, and records from informal interviews.

The structure of this research is as follows: Section 2, the literature review part, covers two main areas. First, it outlines the key elements and limitations of participatory planning, focusing especially on collaborative participation. This serves as the theoretical framework for discussing the research findings in Section 5. Second, it contrasts classical Delphi methods with policy Delphi methods, indicating that the latter are more suitable for participatory planning processes than the former. It also identifies areas for enhancement in policy Delphi methods to better represent the voices of less-vocal stakeholders. Then, it introduces the PAP Delphi method that makes up for their shortcomings. Section 3 serves as the introduction to the case study. It first outlines the specific contexts of the site that led planners to use the PAP Delphi method in participatory planning. Then, it provides an overview of how the PAP Delphi method was integrated into the participatory planning process, as well as the methods used for data collection and analysis. In Section 4, which covers the description and analysis of the findings, the participatory planning process is presented chronologically. This is followed by an analysis of changes in stakeholders’ participation and perspectives. Section 5, the discussion part, evaluates the potential benefits and challenges of the PAP Delphi method in participatory planning shown in the case study through the lens of the three key elements in participatory planning. Finally, Section 6, the conclusion part, summarizes the findings and discusses the limitations of the study.

2. Literature Review

2.1. Key Elements and Limitations of Participatory Planning

In a world where society is becoming more diverse, participatory planning may help fix some of the flaws of representative democracy. When done right, it ensures that the different needs and interests of various groups are not overlooked [19]. However, merely adopting this approach does not guarantee success. If planning only relies on one-way communication such as public hearings or surveys and leaves out important stakeholders, then public involvement might just remain surface-level. This shallow engagement could potentially erode trust in the resultant policies and the wider administrative system [1,20,21].

Consequently, contemporary planning practices emphasize the value of collaborative participation, which often involves face-to-face meetings among a diverse group of stakeholders [1]. The principles of collaborative participation, have been propounded in both collaborative and communicative planning literature [22,23]. Judith Innes and Patsy Healey are key figures in the field of collaborative and communicative planning theory [24]. Both theorists emphasize communication, social relationships, and human resources as core elements of communicative planning while they differ in how they articulate these concepts. In this study, the authors adopt these elements as a theoretical framework for further discussion about the case study results. Specifically, the authors follow Innes’s articulation of the concepts, as she has tailored them for participatory planning.

Innes and Booher [1] identify three foundational elements of collaborative participation: authentic dialogue, networks, and institutional capacity. Firstly, authentic dialogue ensures that all stakeholders, regardless of their stature, have an equal voice, access to uniform information, and actively engage in listening to one another [1,12,25]. Such dialogical interactions promote accurate information sharing and mutual understanding among stakeholders, and often lead to a re-evaluation of initial stances, thereby aiding in building consensus [1,3]. Authentic dialogue is crucial for genuine collaborative participation, while networks and institutional capacity arise as outcomes of the participatory planning process [12]. Secondly, networks refer to the relationships established among stakeholders in the planning process [1]. In the participatory planning process, stakeholders encounter new individuals, resolve conflicts with those holding divergent interests via authentic dialogue, and build trust [26,27,28]. Once established, these networks create a foundation that facilitates improved outcomes in subsequent collaborations [29]. Thirdly, institutional capacity encompasses intellectual capital, social capital, and political capital [30]. Intellectual capital relates to the knowledge resources underpinning decision-making, social capital speaks to the relational resources that bolster collaboration among stakeholders, and political capital signifies the ability to define and execute agendas [31]. Through authentic dialogue, stakeholders become better informed, collaborate with a diverse set of individuals to address issues, and gain confidence in the process [1,32]. The networks and institutional capacities formed in this manner not only enhance the current planning mechanisms but also equip stakeholders to tackle forthcoming local challenges beyond the immediate planning framework [12,25,33].

2.2. Applications of the Delphi Method in the Fields of Policy and Planning

Here, the authors outline the key features of classical Delphi methods and contrast them with policy Delphi methods, which are more suitable for participatory planning processes than the classical Delphi methods. The authors also identify areas in which policy Delphi methods could be enhanced to better represent the voices of less vocal stakeholders. Finally, the authors introduce the PAP Delphi method.

Initially developed in the 1950s, the Delphi technique was designed to facilitate agreement among experts regarding future forecasts [34]. Known as ‘classical Delphi’, this original form aims to organize group interactions in a way that minimizes the drawbacks commonly associated with face-to-face discussions, such as biases related to authority, hasty conclusions, and inflexibility in initial viewpoints [18,35]. In the classical Delphi approach, experts from relevant fields are typically selected as panelists. Anonymity is maintained among these panelists, who individually respond to questions. These responses are then collected, and statistical feedback summarizing the group’s collective opinions is provided to the panelists [35,36]. Upon reviewing this group feedback, panelists have the opportunity to re-evaluate and adjust their initial answers for the subsequent round of feedback [37]. The classical Delphi process usually involves two to three iterative rounds and concludes when researchers determine that a state of equilibrium has been reached in participant responses [36,38].

While classical Delphi methods were primarily developed for future forecasting, policy Delphi methods were introduced by Turoff [15] as a tool to clarify varied perspectives on policy topics. Similar to the classical Delphi process, policy Delphi methods involve iterative rounds where participants compare their views with others and reconsider their initial opinions. However, policy Delphi methods differ from classical ones in the following ways: First, the panels for policy Delphi methods are generally composed of stakeholders involved in the policy issues, whereas those for classical methods consist of experts in related fields for future forecasting. Second, due to the influence of various beliefs and values in policy fields, the panels for policy Delphi methods often comprise heterogeneous groups with diverse viewpoints, while those for classical methods are made up of homogeneous expert groups [15,39,40,41]. Finally, unlike classical Delphi methods, which focus on panel consistency, policy Delphi methods may adjust their panel composition to better represent diverse interests [38].

However, in planning practices, the Policy Delphi methods have often been used simply for building consensus among stakeholders, even though they were originally designed to both clarify diverse stakeholder views and build consensus [42,43]. For example, Schneider [44] empirically validated the capability of the Delphi technique to achieve consensus among stakeholders. Morgan et al. [45] highlighted its use in establishing unified objectives among community leaders for a comprehensive plan in the early stages. In the study by Baumann et al. [46], the Delphi approach was once again used as a one-way communication tool; however, Baumann also highlighted its potential as a tool for consensus-building, particularly when face-to-face dialogues are impractical in participatory planning. Such research emphasizes the Delphi method’s viability for achieving consensus in participatory planning. However, the policy Delphi processes in the above studies failed to reflect the diverse views of different stakeholders, particularly those of less vocal ones, mainly for the following two reasons: First, the panels were made up of stakeholders already involved in the planning process who were vocal about their interests. As a result, those who had not participated in the planning process were excluded from the Delphi process. Second, the entire policy Delphi process was largely document-based. In other words, panelists read and responded to questionnaires, and their answers were then summarized and delivered back to the panel in written form. This document-centric approach could pose challenges for some panelists who may have difficulty understanding the questionnaires or articulating their opinions without expert guidance [47,48].

To address these limitations of policy Delphi methods, the authors introduced a PAP Delphi approach in this study’s case project. In this modified approach, planners expanded the panel to include individuals who had not previously participated in the planning process. Additionally, planners actively engaged with panelists in person to assist with their understanding of the questionnaires and to help articulate their opinions more effectively.

3. Materials and Methods: Case Study Synopsis

This section first delineates the context of the case study, which provides background information needed to understand the case. Subsequently, it provides a detailed overview of the participatory planning process using the PAP Delphi method, along with the methods used for data collection and analysis. Participant observation was used as the primary research method for data collection. The authors participated in the project as researchers to design and execute PAP Delphi surveys.

3.1. Context of the Case Study

This section provides an overview of the policy background, site details, and the sequence of events that led to one of the key stakeholder groups becoming voiceless. Additionally, it delves into the characteristics and roles of the primary stakeholder groups involved in the case.

In response to the nationwide decline of old downtowns, the South Korean government enacted the “Special Act on Promotion of and Support for Urban Regeneration” in 2013 [49]. Following this, they launched the “Urban Regeneration New Deal Project” program in 2014. This program has two key features. First, it mandates the creation of a comprehensive master plan, often referred to as the “Strategic Plan for Urban Regeneration”. This plan consolidates various initiatives aimed at revitalizing declining areas. Local governments are responsible for developing this strategic plan, which serves as a tool for identifying areas showing signs of urban decay. Once such areas are pinpointed, they are designated as “Urban Regeneration Revitalization Areas”. After this designation, local governments must then develop a specific plan for these areas, known as the “Urban Regeneration Revitalization Plan”. This plan outlines future urban regeneration projects, detailing elements such as budget, timeline, and execution plans. Second, the program emphasizes community engagement throughout the revitalization process. To prioritize local communities in regeneration efforts, the program mandates the formation of a “Resident Council”, which serves as a representative body for the local community. This council is intended to play an active role in both formulating the “Urban Regeneration Revitalization Plan” and executing individual urban regeneration projects included in the plan. To support the Resident Council’s activities throughout the duration of the “Urban Regeneration New Deal Project”, local governments are mandated to establish an “Urban Regeneration Support Center [49]”.

The focus of this case study is the Seohak-dong Art Theme Street Development Project, which is part of the “Urban Regeneration Revitalization Plan” for Seohak-dong Art Village (Figure 1). Seohak-dong Art Village is a low-rise residential area with a main commercial street, which is located near the old downtown of Jeonju City in South Korea (Figure 2). While the overall population of Jeonju City has grown to approximately 640,000 in 2023, the old downtown area, including Seohak-dong Art Village, has been in steady decline since the early 2000s [50]. To counteract this decline, Jeonju City implemented policies in 2002 to support Jeonju Hanok Village, an area concentrated with hanoks—traditional Korean houses. This village is located near Seohak-dong Art Village [51]. As a result, from the late 2000s, there was a significant increase in tourists visiting Jeonju Hanok Village, driven by its unique architectural landscape. The surge in tourist activity led to increased land prices and subsequent gentrification, causing an exodus of artists from the village [52]. Seohak-dong Art Village emerged as an alternative for these displaced artists. Despite its proximity to Jeonju Hanok Village—being just a bridge away—Seohak-dong Art Village remained relatively overlooked due to its unremarkable low-rise residential landscape. This lack of attention resulted in stable land prices, making it an attractive option for artists forced out of Jeonju Hanok Village. An artist duo moved to the area in 2010, initiating a gradual influx of artists [53]. By 2018, the village had become home to approximately 50 artists, leading to its official designation as Seohak-dong Art Village [50].

Despite the influx of artists, Seohak-dong Art Village continued to experience a steady decline in its overall population. To address the decline, Jeonju City initiated the “Urban Regeneration New Deal Project” for the village in 2017, covering a total area of 153,555 m². As an initial step in April of that year, a preliminary Resident Council was formed [50]. Comprising around 100 stakeholders, including artists, shop owners, and residents, the council was established to guide the regeneration efforts. Throughout 2018, a comprehensive “Urban Regeneration Revitalization Plan” was drafted for the Seohak-dong Art Village. This plan included various regeneration projects such as housing repair support, street environment enhancement, the creation of community facilities, and community empowerment programs. In total, 13 distinct projects were planned for completion by 2021 (Figure 1).

In 2019, the Urban Regeneration Support Center for Seohak-dong Art Village was inaugurated, marking the project’s transition to its implementation phase. Subsequently, the Resident Council was officially established. During this period, the council selected its leadership and enhanced community capacity for active participation in the revitalization efforts. Starting in 2020, a series of projects were launched, including the Seohak-dong Art Theme Street Development Project (Figure 1), all aimed at enhancing the area’s physical environment. The Resident Council organized several subcommittees equivalent to these projects.

Among the various projects, the Seohak-dong Art Theme Street Development Project, which is the focus of this study, serves as a pivotal element in the overall revitalization plan. With a budget of KRW 1.9 billion, this project represents approximately 11.2% of the total KRW 17.2 billion allocated for the entire revitalization plan. This is the second-largest allocation, following the parking lot development project, which has a dedicated budget of KRW 2.9 billion primarily for land acquisitions. Although Seohak-ro serves as the main street of Seohak-dong Art Village and is concentrated with small businesses such as artists’ ateliers, cafes, and restaurants, it has attracted limited visitor traffic (Figure 3). To address this, both Jeonju City and the Resident Council have sought to vitalize Seohak-ro by enhancing its walkability and establishing a distinctive streetscape.

Within the Resident Council, a subcommittee takes charge of the Seohak-dong Art Theme Street Development Project. Most buildings facing Seohak-ro operate as ateliers or shops. Artists’ ateliers are predominantly located on the western side of the street, while the eastern side mainly features general commercial establishments such as cafes and restaurants. This makes artists and shop owners the primary stakeholders (Figure 4). However, the subcommittee lacked representation from shop owners on the eastern side of Seohak-ro. Its nine members consisted of eight artists and one resident. Shop owners were passive in participating in the Resident Council meetings for the following two reasons. First, the use of ‘art’ in titles such as ‘art village’ and ‘art theme street’ led community members to perceive the project as primarily focused on artists. Second, unlike artists, shop owners were often too busy with their businesses to regularly attend Resident Council meetings. This limited engagement resulted in shop owners having diminished influence and less access to information compared to artists. Consequently, no shop owners were included in the subcommittee for the Art Theme Street project.

During Resident Council meetings, city officials stayed back and did not demonstrate their opinions strongly. They often communicated indirectly with the Resident Council through the Urban Regeneration Support Center, rather than directly participating in meetings. This approach aligns with the focus on community engagement in urban regeneration projects. The center, established to aid the Resident Council, concentrated on administrative responsibilities such as venue reservation and schedule announcements, rather than expressing their own views during discussions.

A private urban planning firm became involved in the project following the formation of the subcommittee. Hired by the city to design the streetscape, the firm’s practical planners were responsible for developing a street design through participatory planning processes. Initially, the team for this project consisted of an urban planner, an urban designer, two architects, and a graphic designer. However, the need to add the PAP Delphi method to supplement the participatory planning process later emerged, so the authors engaged in the project as researchers specialized in community participation. The main role of the researchers was the design and execution of the PAP Delphi surveys to promote communication among stakeholders (Figure 4).

3.2. Participatory Planning for Seohak-Ro Street Design

The streetscape design for Seohak-ro was launched on 14 January 2020 as part of the Art Theme Street Development Project. Shortly after the project began, the COVID-19 pandemic imposed strict limitations on public gatherings. Even small group meetings required masks and social distancing. Because online meetings were not used widely at that time [54], the practical solution was to reduce the number of attendees. Consequently, interactions were limited to subcommittee members, practicing planners, and staff from the support center, ensuring adequate social distancing during discussions.

In the first subcommittee meeting, four members were in attendance. The practicing planners presented their assessments of the target area, along with sample images of potential streetscape designs, with the aim of gathering feedback (Figure 5a). After the meeting, the support center informed the planners that the subcommittee mainly consisted of artists. They also highlighted that the shop owners, the other crucial stakeholder group with views contrasting those of the artists, were hesitant to actively participate in the project. To address this issue, the support center advised the planners to find ways to include the perspectives of these reluctant stakeholders. In response, the planners integrated the PAP Delphi method into the existing participatory framework.

The Delphi method, which was not originally designed for public participation, had to be adapted for this project. First, we broadened the scope of the panel for our PAP Delphi surveys to include all direct stakeholders. Specifically, this comprises artists and shop owners operating businesses on Seohak-ro, rather than limiting it to just the subcommittee members. Second, since subcommittee meetings were already underway, a parallel approach was developed that combined these meetings with PAP Delphi surveys. The same images presented in the subcommittee meetings were provided to survey respondents to aid their understanding. The aggregated results were then shared with artists and shop owners in subsequent survey rounds. Third, throughout all three survey rounds, the practicing planners personally visited the stakeholders to request their participation in the surveys and also conducted unstructured interviews with respondents. While conventional Delphi surveys often repeat similar questions, the PAP Delphi survey used in this study differed by presenting different questions alongside evolving designs, allowing the practicing planners to track shifts in perspectives among artists and shop owners (Appendix A). Fourth, while it is ideal to maintain a consistent panel throughout all rounds of a Delphi survey, achieving this was challenging in the context of this project. Initially, the practicing planners explained the entire Delphi survey process to the respondents and encouraged them to participate in all subsequent rounds. However, despite the planners’ detailed explanations, some respondents failed to grasp the need for repeated surveys and chose not to participate again. Others were occasionally unable to take part in subsequent surveys due to time constraints. Additionally, in keeping with the emphasis on inclusivity in the participatory planning process, newcomers who had not participated in previous surveys were also allowed to join subsequent survey rounds (Figure 5b).

Following the completion of the second subcommittee meeting, the PAP Delphi survey was initiated. The first round of the survey evaluated preferences for design examples relevant to each space (

Table A1. First PAP Delphi survey.

Questions	Response Rates for Each of the Questions
What design techniques do you think are appropriate for roadways and sidewalks?	Shared Use (No Separation)	Maintain as is (Elevation Separation)	Physical Separation (Bollards, Benches, Planters, etc.)
	20%	20%	60%
What design techniques do you think are appropriate for on-street parking?	No Parking	Maintain as is (Allow Fully)	Allow partially
	15%	15%	70%
What design do you think is appropriate for streetlights?	Traditional form	Maintain as is	-
	76%	24%	-
What design techniques do you think are appropriate for the west entrance?	Installation of a gate	Maintain as is	Installation of objets
	50%	15%	35%
Do you think it is appropriate to install a digital information kiosk at the west entrance?	necessary	unnecessary	-
	76%	24%	-
What design techniques do you think are appropriate for the east entrance?	Installation of objets	Maintain as is	-
	75%	25%	-
Do you think it is appropriate to install a digital information kiosk at the east entrance?	necessary	unnecessary	-
	62%	38%	-
What design techniques do you think are appropriate for the north entrance?	Installation of objets	Maintain as is	-
	81%	19%	-
Do you think it is appropriate to install a digital information kiosk at the north entrance?	necessary	unnecessary	-
	71%	29%	-

of Appendix A). In the second survey round, perspective views of various alternatives were presented based on the initial survey results, and preferences for these alternatives were assessed (

Table A2. Second PAP Delphi survey.

Questions	Min	Max	Mean	Standard Deviation
Are you satisfied with the parklet design for Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	2	5	3.71	0.98
Are you satisfied with the streetlight design for Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	3.38	1.25
Are you satisfied with the overall street design for Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	2	5	3.45	0.94
Are you satisfied with the gate-type design for the west entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	2	5	3.57	0.90
Are you satisfied with the overall street design for the west entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	2	5	3.62	0.79
Are you satisfied with the objet-type design for the west entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	3	5	3.95	0.80
Are you satisfied with the overall street design for the west entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	3	5	3.95	0.83
Are you satisfied with the objet-type design for the east entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	2	5	3.75	0.77
Are you satisfied with the overall street design for the east entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	3	5	3.65	0.68
Are you satisfied with the objet-type design for the north entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	3.71	1.12
Are you satisfied with the overall street design for the north entrance? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	2	5	3.94	0.91

of Appendix A). In the final round, more developed design variations with more details such as material choices were presented for the survey (

Table A3. Third PAP Delphi survey.

Questions	Min	Max	Mean	Standard Deviation
Are you satisfied with the pavement design for Alternative 1 on Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	4.03	1.10
Are you satisfied with the pavement design for Alternative 2 on Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	3.22	1.26
Are you satisfied with the parklet and parking space layout for Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	3.10	1.56
Are you satisfied with the parklet design for Alternative 1 on Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	2.81	1.33
Are you satisfied with the parklet design for Alternative 2 on Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	3.34	1.54
Are you satisfied with the trash can design for Alternative 1 on Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	4.10	1.04
Are you satisfied with the trash can design for Alternative 2 on Seohak-ro? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	2.35	1.30
Are you satisfied with the streetlight design for the east and north entrances? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	3.97	1.08
Are you satisfied with the digital information kiosk for the west, east, and north entrances? (1 = Very dissatisfied, 2 = Dissatisfied, 3 = Neutral, 4 = Satisfied, 5 = Very satisfied)	1	5	4.00	1.14

of Appendix A). In the third round, the two groups finally reached a consensus, eliminating the need for further subcommittee meetings and surveys (Figure 6).

3.3. Data Collection and Analysis

From January to mid-June 2020, the authors conducted participant observations throughout the entire community engagement process (Figure 6). The authors participated in both formal events—including subcommittee meetings, PAP Delphi surveys, and discussions with Jeonju City officials and experts—and informal discussions wherein artists and shop owners conveyed their views to the practicing planners. Discussions and activities during these sessions were documented in detail. Data collection for this study was focused solely on the feedback directly related to the project, avoiding any potentially sensitive personal information.

Due to the COVID-19 pandemic, attendance at the subcommittee meetings was limited. Typically, three or four artists consistently participated, while no shop owners attended any of the meetings. Additionally, one or two residents, who were not members of the subcommittee and thus not formally invited, sporadically attended the meetings. The scope of the PAP Delphi surveys was set to include all direct stakeholders, encompassing both artists and shop owners doing business facing Seohak-ro, not just the subcommittee members. Among them, those who responded to the surveys were actually included in the surveys.

Data gathered were systematically analyzed for the following street design elements: ‘Street Parking and Parklets’, ‘Pavement Types’, ‘Street Entrance Design’, ‘Curbless Street Design’, and ‘Street Furniture Design’. Among these elements, regarding the ‘Street Furniture Design’, there was a large agreement between artists and shop owners, with no significant conflict between the two groups. Therefore, the analysis and discussion in this study focused on the remaining four elements.

4. Results

This section presents a description and analysis of the participatory design process for the Seohak-dong Art Theme Street project, based on six months of observation [55]. The authors begin by chronologically outlining the proceedings of subcommittee meetings and PAP Delphi surveys. This offers foundational information crucial for understanding the result analysis in Section 4.2 and the discussion that follows in Section 5. Subsequently, the authors focus their analysis on the research questions posed at the outset: ‘Do stakeholders who were reluctant to engage in face-to-face interactions participate in the PAP Delphi survey?’ and ‘Does the PAP Delphi method contribute to building consensus among conflicting stakeholder groups’?

4.1. Descriptions of the Participation Planning Process

4.1.1. The First Subcommittee Meeting

In the first subcommittee meeting, only four artists attended, along with the practicing planners and staff from the support center. During the meeting, the planners first presented various street design examples. The artists then actively discussed their design preferences. Given their deep understanding of aesthetics and their long-standing contemplation of these issues, the artists provided the planners with a variety of specific demands.

The artists envisioned a unique streetscape characterized by elegant pavement patterns made from high-quality stone materials. They also proposed a curbless street design to accommodate their annual festival. Concerned about the issue of illegal parking on the wide one-way road, they suggested narrowing the roadway, expanding the sidewalks, and intensifying enforcement against illegal parking. To enhance the street’s identity and attract tourists from the nearby Jeonju Hanok Village, they proposed installing symbolic structures at the three major entrances—east, west, and north. They also recommended the placement of digital information kiosks at each entrance. In conclusion, the artists told the planners, “After thorough deliberations, we have established the design direction for Seohak-ro and trust you to bring our vision to life”.

A few days after the first subcommittee meeting, the planners and the support center staff convened a separate meeting to discuss future planning steps. During this meeting, the staff informed the planners that the artists who had attended the previous subcommittee meeting were regular attendees and generally agreed with each other. However, they also noted that many shop owners, who were not members of the subcommittee, held views that differed from those of the artists. Although COVID-19 guidelines limited face-to-face decision-making to subcommittee meetings, the staff of the support center expressed concerns about relying solely on the subcommittee that did not include any shop owners as its members. They were concerned that favoring artists too heavily in the decision-making process could lead to dissatisfaction among shop owners. To mitigate the issue of one-sided decision-making, the planners chose to employ the PAP Delphi method, aiming to incorporate input from shop owners.

4.1.2. The Second Subcommittee Meeting and the First PAP Delphi Survey

In the second subcommittee meeting, three artists who were subcommittee members attended, as well as a resident who was not part of the subcommittee and had not been invited. Given that the resident’s home was not near the street, his direct interest in the project was limited. During the meeting, the practicing planners summarized the artists’ suggestions from the first meeting on topics such as street layout, on-street parking, pavement materials, and entrance design. They also presented sample images showcasing various design alternatives for these issues.

At the start of the second subcommittee meeting, the practicing planners proposed conducting PAP Delphi surveys alongside future meetings. This suggestion met with some resistance from the artists, who, while agreeing on the need for broader feedback, expressed concerns about being influenced by majority opinions. Given this, the planners decided to postpone the decision about whether to conduct the PAP Delphi surveys until after their scheduled presentation was complete. They then proceeded with the presentation.

Just after the presentation began, the uninvited resident raised questions about the priority of streetscape improvements over other concerns such as providing parking spaces and revitalizing local commerce. Despite this interruption, the planners carried on and concluded their presentation. However, the tension between the artists and the resident remained unresolved even after the presentation had ended. To address this, the practicing planners stepped in and recommended collecting broader community opinions via PAP Delphi surveys, with plans to resume discussions in the third subcommittee meeting. Initially hesitant, the artists ultimately gave their consent to move forward with the PAP Delphi survey.

After the second subcommittee meeting, the practicing planners, in collaboration with the support center, conducted the first PAP Delphi survey targeting shop owners and artists near Seohak-ro. While conducting the first survey, some shop owners questioned whether their opinions would be reflected in the plan. In response, the planners explained the unique characteristics of Delphi surveys as a two-way communication method. Specifically, they pointed out that Delphi surveys would be conducted in multiple rounds, and the results from each round would be shared with the respondents.

During the interviews conducted alongside the first survey, many shop owners expressed negative views of the project to the practicing planners. While they acknowledged its potential to boost commerce, they felt that the project was being disproportionately influenced by artists, leading to excessive budget allocation for street improvements. Additionally, shop owners were wary of artists’ preferences for reducing parking spaces, fearing this could lead to fewer customer visits. They also pointed to a recent traffic incident to underscore the need to reduce vehicle speeds.

The first PAP Delphi survey included questions about preferences for street light design, paving pattern design, curbless street design, on-street parking, and street entrance design (Table A1 of Appendix A). Among these questions, respondents showed significant interest in curbless street design, on-street parking, and street entrance design. Figure 7a–c present the results related to these three design elements. 60% of respondents favored a curbless street design with street furniture to delineate the boundary between sidewalks and the roadway (Figure 7a). On the topic of on-street parking—a point of divergence between artists and shop owners—70% of respondents were in favor of partial allowance (Figure 7b). Additionally, 64% of respondents supported the installation of objets at the three main entrances—west, east, and north—in the aggregated survey results. However, for the west entrance, the survey included an additional option for installing a gate, as only this entrance had sufficient space for such an installation. The results for this option diverged somewhat from the aggregated results, with 50% of respondents favoring the installation of a gate (Figure 7c).

4.1.3. The Third Subcommittee Meeting and the Second PAP Delphi Survey

In preparation for the third subcommittee meeting and the second PAP Delphi Survey, the practicing planners first created visual simulations of a street design scheme. This scheme was based on preferences expressed by respondents during the first PAP Delphi survey. The scheme includes several key features: a curbless street equipped with street furniture, limited permission for on-street parking, and a narrowed lane design intended to slow down vehicle speeds. Specifically, to reduce vehicle speeds, the planners introduced a zigzagged lane design accompanied by ‘parklets’ (Figure 8). These parklets are small community spaces featuring seating and planters, converted from existing parking spaces [56]. This design serves multiple purposes: it provides both parking and resting areas while also acting as a traffic-calming measure.

Next, the planners developed visual simulations for two design options for the west entrance: the ‘objet’ type and the ‘gate’ type. These simulations were prepared to clarify conflicting results from the first survey. Although respondents generally preferred the ‘objet’ design for entrances, they specifically favored the ‘gate’ type for the west entrance. The planners aim to double-check these preferences through the simulations.

In the third subcommittee meeting, three artists were present. The practicing planners began by sharing the results of the first survey. They then presented simulations for a street design option as well as two alternative designs for the west entrance. Following the presentation, the artists largely agreed with the proposed street design but expressed a desire for no parking in front of their ateliers. As for the west entrance design, they favored a design of the ‘objet’ type.

After expressing their satisfaction with the schematic designs presented, the artists shifted their focus to design details. Specifically, they strongly advocated for using stone as the road material, even if it would increase costs. Concerned about the potential for stone paving to significantly raise the overall project cost, the planners suggested a postponement. They recommended delaying the discussion on this matter until the next subcommittee meeting, which would be held after determining the costs associated with stone paving.

After the third meeting concluded and the artists had left the room, another resident, who had not attended the second meeting, arrived. Similar to the resident who had attended the second meeting, this person did not have a direct stake in the project and held negative feelings towards the artists leading the initiative. The planners briefly summarized the day’s presentation for the resident. Notably, the resident’s perspective shifted after learning that a PAP Delphi survey had been conducted. Recognizing that the broader community’s views were genuinely incorporated into the planning, the resident responded positively.

In the second PAP Delphi survey, the practicing planners first shared the results of the initial survey with respondents. They then asked the respondents to rate their preferences for a street design option and two alternative designs for the west entrance using a 5-point Likert scale. During interviews conducted concurrently with the second survey, respondents generally expressed support for plans to limit on-street parking and introduce parklets (Figure 8). However, shop owners among the respondents argued that the east and north entrances should receive more attention than the west entrance. They pointed out that most tourists coming from Jeonju Hanok Village are more likely to use the east and north entrances. This concern arose from a misunderstanding that the plan, seemingly led by artists, focused on improvements solely on the western side where ateliers are prevalent. The planners clarified that both the eastern and western sides of the street would be improved equally.

In the second survey, respondents gave an average preference rating of 3.71 for the overall street design (Figure 8). As for the design of the west entrance, the “Objet” type design option was favored, receiving a preference rating of 3.95, compared to the “Gate” type design option, which received a rating of 3.57 (Figure 9).

4.1.4. The Fourth Subcommittee Meeting and the Third PAP Delphi Survey

At the fourth subcommittee meeting, three artists were present. Additionally, two residents attended, one who had previously been at the first meeting and another who had been at the second meeting. A city official responsible for the entire Seohak-dong regeneration plan was also invited. The purpose was to hear the city’s opinion on whether to use stone paving, a decision that could significantly influence the overall cost of the project.

As in the previous two meetings, the planners began by summarizing the results of the second survey. They first presented the developed designs, showcasing detailed street layouts that included the locations of parklets and parking spaces (Figure 10). The discussion then moved to the detailed designs for the west, east, and north entrances. For the west and north entrances, small plazas featuring ‘objets’—symbols representing the street—were designed. Meanwhile, for the east entrance, a ‘road diet’ approach was utilized to create wider sidewalk spaces (Figure 10). Finally, they announced that within the current budget, the entire sidewalk area and half of the road could be paved with stone.

After hearing the presentation, the artists suggested replacing all proposed parking spaces with parklets on the western side of the street. They also requested that stone paving be secured for at least the western side, even if no additional budget could be allocated for the project. The residents were satisfied that the broader community’s opinions were continuously incorporated into the plan through repeated surveys. The city official clarified that there would be no increase in the budget. The official also stated that the plan appeared to adequately reflect community opinions and, therefore, it was time to proceed with formal review procedures for administrative approval. Finally, the planners announced that the participatory planning process would conclude with the conduct of the third survey.

The third PAP Delphi survey was conducted in the same manner as the second survey. The practicing planners first shared the results of the second survey with respondents. The planners then asked respondents to rate their preferences for the refined street design using a 5-point Likert scale (Figure 10). Recognizing that their feedback had been incorporated into the plan through repeated surveys, shop owners became more open to sharing their opinions in interviews conducted alongside the third survey. They requested that the same quality of street design be applied to the entire street, including the eastern side where their shops are located. Specifically, they emphasized the importance of both parking spaces and parklets, while considering stone paving to be less critical.

Reflecting the input from the fourth meeting and the third survey, the planners revised the design. On the western side, they replaced all parking spaces with parklets and planned stone paving for both the sidewalk and road areas. On the eastern side, they maintained the number of parking spaces and parklets as in the previous plan and planned stone paving only for the sidewalk area. During the formal review process, most design elements were retained; however, a small plaza planned for the north entrance and a widened sidewalk area at the east entrance were excluded, as shown in Figure 11.

4.2. Analyses of the Participatory Planning Process

This section analyzes the changes in stakeholders’ participation and opinions over the course of four subcommittee meetings and three surveys to address the two research questions posed in the introduction. As for the first question, ‘Do stakeholders who were reluctant to engage in face-to-face interactions participate in the PAP Delphi survey?’, the case study provides a positive answer. Although shop owners who did not want to participate in face-to-face meetings were unable to voice their opinions, they took part in the surveys, which allowed their viewpoints to be included in the plan. Furthermore, as more shop owners realized that their opinions were being incorporated through the first and second surveys, participation increased. In the third survey, the number of participants nearly doubled compared to earlier rounds (

Table 1. Number of participants from artists, residents, and shop owners for each stage of the subcommittee meetings and PAP Delphi survey.

	Artists (N)	Shop Owners (N)	Residents (N)	Total (N)
First Subcommittee meeting	4	-	-	4
Second Subcommittee meeting	3	-	1	4
First PAP Delphi survey	10	11	-	21
Third Subcommittee meeting	3	-	1	4
Second PAP Delphi survey	12	10	-	22
Fourth Subcommittee meeting	3	-	2	5
Third PAP Delphi survey	8	22	-	30

). In terms of the content of opinions, during the first survey, the shop owners mainly offered opinions on the on-street parking issue they had originally been concerned about. However, as they gained a better understanding of the overall plan through in-person communication with the planners, they also offered opinions on other issues such as street pavement and entrance design in subsequent surveys (

Table 2. Changes in the opinions of artists and shop owners.

		Before PAP Delphi	First PAP Delphi	Second PAP Delphi	Third PAP Delphi
Street parking and Parklets	Artists	No parking	No parking	Partial parking allowed, but prefer no parking on the western side	Partial parking allowed, but replace parking space with parklets on the western side
	Shop owners	-	Parking Allowed and reduced vehicle speeds	Partial parking allowed	Partial parking allowed, arranging parking spaces and parklets appropriately.
Pavement Types	Artists	Pave the entire street with natural stone	Pave the entire street with natural stone	Pave the entire street with natural stone by securing an additional budget	Pave the west side of the street with natural stone, considering the limited budget
	Shop owners	-	No opinion	No opinion	Pave the west side of street with natural stone (pavement material is not important)
Curbless Street Design	Artists	Necessary	Necessary	Necessary	Necessary
	Shop owners	-	Not necessary	Necessary	Necessary
Street Entrance Design	Artists	Installation of objets at every entrance	Installation of objets at every entrance	Installation of objets at every entrance	Installation of objets at every entrance
	Shop owners	-	No opinion	Installation of objets at east and north entrances	Installation of objets at every entrance

).

As for the second question, ‘Does the PAP Delphi method contribute to building consensus among conflicting stakeholder groups?’, the case study provides a positive answer, too. In the first survey, the artist group argued for several measures: a complete ban on on-street parking, paving the entire street with natural stone, and implementing a curbless street design. In contrast, the shop owner group advocated for fully allowing on-street parking and stated that a curbless design was unnecessary. They did not offer an opinion on the type of street paving. The conflicting opinions among stakeholders that were revealed in the initial survey converged into compromises, as can be seen in the results of the third survey (Table 2).

Such consensus among key stakeholders may be influenced by the following factors. First, the repeated surveys acted as a two-way communication channel, enabling shop owners and artists to exchange opinions—a particularly useful feature for those shop owners who felt uncomfortable meeting artists face-to-face. Without the PAP Delphi surveys, the shop owners’ opinions would have been minimally reflected in the plan, given that only artists were present at the subcommittee meetings. As the surveys were repeated, the shop owners increasingly came to trust the planning process, perceiving that their opinions were being incorporated into the plan [57]. Second, as the majority’s opinions were revealed through the PAP Delphi surveys, artists and shop owners were able to reach compromises on conflicting issues. For example, artists who initially argued against allowing on-street parking changed their stance to accept partial allowance of on-street parking after learning that the majority preferred this option in the initial survey results. Third, the practicing planners played a crucial role in facilitating consensus-building. They provided detailed visual simulations to show how each design would be implemented, as well as accurate information such as the expected costs associated with each design alternative [58].

5. Discussion

This section first explores to what extent the incorporation of the PAP Delphi method contributes to the participatory planning process in this project. The discussion is framed around core elements of collaborative participation as proposed in the existing literature, focusing particularly on the works of Innes, a key figure in the field of communicative planning. Subsequently, this section discusses challenges encountered in conducting the PAP Delphi surveys, comparing them with the case of typical Delphi surveys.

To begin with, Innes and Booher [1] argue that authentic dialogue, meeting the criteria of inclusivity, equal voice, access to uniform information, and mutual respect, is indispensable for achieving consensus among stakeholders. In this project, the PAP Delphi method largely fulfilled these criteria. Firstly, the perspectives of shop owners, who had previously been excluded from the planning process, were incorporated to some extent via the PAP Delphi surveys, thereby enhancing the inclusivity. Secondly, the criterion of equal voice was satisfied, as all participants’ responses in the surveys were weighted equally [35]. Thirdly, throughout the PAP Delphi surveys, planners provided information consistent with what had been shared in the subcommittee meetings, largely meeting the criterion for uniform information access [59]. Lastly, with respect to the criterion of mutual respect, the PAP Delphi surveys had some limitations. Conducted individually by planners without face-to-face interaction among stakeholders, they were less effective in fostering mutual respect [60]. However, artists, who had previously shown weak respect for individual shop owners’ opinions, demonstrated a greater regard for the majority opinions collected through the PAP Delphi surveys, suggesting an improvement in mutual respect. As discussed earlier, the PAP Delphi surveys enhanced the planning process in terms of the four criteria for authentic dialogue. Consequently, this improvement facilitated consensus-building between artists and shop owners, who had previously held differing views on the project.

Next, Innes and Booher [1] argue that authentic dialogue fosters mutual understanding and trust-building among stakeholders, thereby contributing to the formation of networks for future collaborations. In this project, planners actively served as communication channels among stakeholders, facilitating the exchange of information and opinions through the PAP Delphi surveys. These surveys, which featured ‘planner-aided’ and thereby indirect yet two-way communication, proved more effective in resolving misunderstandings and fostering mutual understanding among stakeholders compared to general Delphi methods that lack such interaction. However, the PAP Delphi surveys used in this project did not overcome the inherent limitations of indirect communication and were unable to facilitate trust-building or network formation.

Finally, Innes and Booher [1] argue that authentic dialogue also contributes to the formation of institutional capacity, which is composed of social, intellectual, and political capital. Similar to the case of network formation, the PAP Delphi surveys applied in this project scarcely contributed to the cultivation of stakeholders’ institutional capacity. As discussed earlier, the PAP Delphi process, which lacked direct communication among stakeholders, did not facilitate the formation of social capital. The information shared by planners during the surveys was restricted to content directly related to the project; as a result, it did not significantly contribute to the stakeholders’ development of intellectual capacity. Moreover, because the communication was conducted between planners and individual stakeholders without fostering collaborative experiences among the stakeholders, it did not contribute to the formation of political capital either.

In the next step, challenges encountered during the PAP Delphi surveys of this project are discussed in comparison with typical Delphi approaches. The first challenge is the time commitment; planners had to spend significant time communicating with each respondent individually. The second challenge is panel consistency; maintaining the same group of participants through repeated rounds of surveys proved difficult, particularly because the stakeholders involved were not familiar with Delphi methods, unlike experts who are usually targeted in typical Delphi surveys. The third challenge concerns the influence of planners; the results are significantly shaped by their capacity and ethical conduct, given their pivotal role as mediators among diverse stakeholders. The final challenge pertains to the quality of visualization; for instance, if one design alternative is visually more appealing than another, respondents are likely to favor it over the latter, regardless of the actual merits of the alternatives.

6. Conclusions

In the planning process, involving every key stakeholder is crucial for collaborative participation. However, real-world challenges often hinder planners from engaging voiceless stakeholders. This research evaluated an actual participatory planning process of the Seohak-dong street improvement project in which the PAP Delphi method was applied to reflect the opinions of stakeholders with a small voice. In total, three PAP Delphi surveys were conducted to address a situation where artists, one of the key stakeholder groups, were vocally active, while shop owners, another key stakeholder group, were largely silent and dissatisfied with the planning process. Because these surveys involved citizens unfamiliar with the Delphi method, a ‘planner-aided’ approach was employed. In this approach, planners provided project-related information to respondents and also conducted informal interviews to gather their opinions in greater detail. Consequently, the plan, which initially reflected only the intentions of the artists, came to also incorporate the perspectives of the shop owners through the consensus-building process between the two groups.

This research offers both theoretical and practical implications. On the theoretical side, the application of the PAP Delphi method offers an alternative for addressing issues related to inclusiveness. These issues, often stemming from conflicts between stakeholders or social and political imbalances, have commonly been overlooked in participatory planning. As detailed in the discussion, the method’s indirect communication approach can effectively address these challenges. Specifically, first, indirect communication via the PAP Delphi surveys was effective in allowing voiceless stakeholders to express their opinions without the discomfort of face-to-face confrontations with opponents. Second, beyond the actual incorporation of opinions from voiceless stakeholders, the mere availability of a communication channel alleviated the sense of alienation they had felt prior to the implementation of the PAP Delphi surveys.

On the practical side, while the PAP Delphi method effectively provided a basic communication channel for those who previously had no voice, it did not function as an advanced participatory planning tool for building community capacity. This outcome suggests that the PAP Delphi method could be more effective when used as a complementary tool alongside other participatory planning methods [61,62]. That is, the Delphi technique can be more effective when used in conjunction with other participatory planning tools, such as workshops, community education, committee meetings, and technical support, rather than solely on its own. In addition, the proposed PAP Delphi method, through subsequent improvements, can contribute more effectively to participatory planning. Firstly, we can go beyond merely sharing survey results. By also disseminating information gathered from informal interviews during the survey process with respondents, we can elevate the efficacy of the PAP Delphi survey. During this research, when planners engaged individually with target citizens to provide project details, many respondents often posed questions with similar content. By consolidating answers to these recurring questions into a Q&A format and sharing them in subsequent surveys, we can amplify the respondents’ understanding of the project. Additionally, this method minimizes the time and effort planners spend on repeated Q&A sessions. Secondly, by broadcasting both the survey findings and details about the planning progress in real-time via platforms such as community websites, we can bolster the transparency of the planning process. This approach fosters greater stakeholder engagement, motivating more of them to actively participate in the planning process [63].

Although this study has the implications described above, it also has the following limitations. First, the composition of the panels during the surveys could not be as rigorously controlled as in conventional Delphi surveys, mainly because the respondents were not experts familiar with the method. Therefore, interpreting the survey results required consideration of not just numerical data but also other factors, including detailed information about the respondents. Second, the street design project examined in this study may not be as contentious as other projects such as landfills, sewage treatment plants, or radioactive waste disposal facilities. In projects that can involve severe conflicts among stakeholders, the application of the PAP Delphi method might yield different outcomes. Therefore, future studies need to explore the method’s applicability in these more contentious domains. Finally, the impact of the PAP Delphi surveys was observed for only about a month, up to the beginning of the administrative process. As such, this study did not analyze the long-term effects, which should be the focus of future research.