Spatial Planning Implementation Effectiveness: Review and Research Prospects

Abstract

The question of how to implement spatial planning more effective is a fundamental but very difficult one and one that has been of great interest to both the academic and practical community. However, a comprehensive review of the conceptual models and methodological systems for evaluating spatial planning implementation, and the existing practical research results of various types of spatial planning implementation evaluation, are yet to be presented. The study systematically reviews the main research findings in the field of spatial planning implementation from four aspects: conceptual analysis, measurement methods, evaluation frameworks and evaluation methods. This study found three distinct evolutionary features of research in this field: (1) The evaluation concept changes from complete rationality to limited rationality; (2) research methodology changes from a simple closed system to a complex open system; (3) the research perspective shifts from the map to the main body of planning implementation behavior. It is suggested that an important part of future research lies in establishing a system for evaluating the effectiveness based on a single subject who is an actor in the planning implementation process. On this basis, the micro-influencing mechanisms of planning implementation effects will then be explored. Deepening the research to the individual level will help improve planning implementation’s effectiveness.

1. Introduction

The effectiveness of planning implementation is the basis of the existence of planning legitimacy, and the issue of the effectiveness of planning implementation has been widely emphasized in both practical and academic circles. Therefore, scientifically and objectively evaluating the implementation results of spatial planning scientifically and objectively has always been a hot issue of concern for planners, academics and managers. It is extremely difficult to assess accurately whether spatial planning has succeeded. It is because the conclusion involves not only a philosophical definition of values. It is also technically impossible to make a “final judgment” on the implementation results of relatively static planning in constant space and time. To this end, the researchers repeat that the purpose of planning implementation evaluation is not to argue the success or failure of a plan but to examine and evaluate the process and effectiveness of planning implementation comprehensively and effectively and to provide feedback based on relevant information, so as to propose amendments and adjustments to the content, policy, design, and operation system framework, bringing the plan into a virtuous cycle [1,2,3]. The precondition for improving control effectiveness in spatial planning is continuous evaluation and reflection on the actual implementation results [4].

In the early stage, the planning implementation evaluation drew more on policy implementation analysis and project evaluation. Pressman and Wildavsky’s landmark book Implementation is an essential guide to early planning implementation evaluation [5]. The book sets out the basic questions of planning implementation evaluation: (1) How is the plan implemented? (2) What are the reasons for the planning implementation results? (3) To what extent the planning has been implemented? Based on the concept and methodology of policy implementation as the main paradigm, planning evaluation was widely used in the 20th century to evaluate the implementation of economic, social and health planning (policy) [2]. However, spatial planning has its own specific characteristics, which are presented in physical or spatial form as a result of the implementation of the plan. As policy evaluation is not sensitive to the evaluation of physical and spatial forms, the updating and improvement of spatial planning need to be analyzed and determined based on the extent to which the previous round of planning has been implemented in spatial and physical forms [6]. As a result, since the Mid-1990s, spatial planning implementation evaluation studies have received increasing attention, mainly at the physical level [2,7].

Therefore, the key to evaluating the effectiveness of spatial planning is to answer whether the implementation has been successful or whether it has influenced land use decisions in the context of space-time. Scholars have attempted to answer this question by adopting different evaluation benchmarks, and the commonly used evaluation frameworks include (1) assessment frameworks based on spatial consistency [8,9]; (2) evaluation frameworks based on planning performance [10]; (3) evaluation frameworks of the combination of consistency and performance [11,12]. Therefore, it can be seen that the effective evaluation of planning implementation is a key issue of spatial planning evaluation. Whether the debate is about the success or failure criteria of planning implementation or the construction of frameworks for evaluating planning implementation, it has always been the focus of planning scholars worldwide [6,7,13,14,15,16,17]. However, a comprehensive review of the conceptual models and methodological systems for spatial planning’s implementation evaluation, and the existing practical research results of various types of spatial planning implementation evaluation, are yet to be presented.

This paper critically reviews recent studies on spatial planning implementation effectiveness to fill the research gap. The contributions of this article are (1) to sort out the research lineage of spatial planning implementation effectiveness evaluation by reviewing related studies; (2) and to summarize the transformational features of this research area; (3) as well as to propose potentially important future research topics.

This paper reviews the existing literature on evaluating the effectiveness of spatial planning implementation, following the logic of “Connotation-Criteria-Framework-Methods” (Figure 1). The review mainly includes four parts: (1) the connotation discrimination of the effectiveness of spatial planning implementation, (2) the evaluation criteria of the implementation effect of spatial planning, (3) the evaluation framework, and (4) evaluation methods. Following a systematic review of studies evaluating the effectiveness of spatial plan implementation, we have concluded the evolution of spatial planning effectiveness implementation evaluation. The final section briefly summarizes the findings and recommendations for future studies.

2. Connotation Discrimination of the Effect of Spatial Planning Implementation

2.1. The Core Concept of Spatial Planning Implementation Evaluation

Planners are naturally interested in planning success, which is also the core issue of planning implementation evaluation [6]. In planning implementation, success and effectiveness often have the same connotation. When planning is effective, or its implementation is effective, we can assume that planning or its implementation is successful. However, while studies have been conducted to evaluate the effectiveness of planning implementation [1,6,7,8,18,19], these studies have different understandings of what defines successful planning implementation.

Spatial planning is an externalized form of regional spatial composition and operation of the allocation or utilization intention of natural resources [20]. Physical planning, economic planning and public policy constitute the three basic levels of spatial planning [21]. Physical planning characteristics focus on the extent to which the implementation results match and deviate from the planning intentions at the physical level. Economic planning characteristics reflect the prominent role of spatial planning in allocating resources. The implementation of spatial planning is bound to be entangled with the game of interests of all parties [22]. When evaluating the implementation of spatial planning, it is necessary to consider the efficiency and rationality of actual resource utilization and planning allocation. The nature of public policy means that the implementation of planning is carried out in a specific political and institutional environment [23,24]. As part of public policy, the success of spatial planning is reflected in whether it is implemented to the public’s satisfaction and to enhance spatial equity. At the same time, the non-physical externalities such as fairness and democracy should be fully considered when evaluating planning implementation.

The success of planning implementation is in itself a very complicated proposition. To this end, planners only rely on vague and subjective value judgments to answer this proposition and even focus on the methods and processes of planning preparation to avoid this problem [14]. Although the tools and methodological system of policy implementation analysis are becoming more sophisticated, it is still necessary for planners to deeply explore whether planning is being implemented and how effective it is [1]. Therefore, it is as difficult to assess the effectiveness of spatial planning regulation as to evaluate the effectiveness of planning implementation. It is difficult to measure the impact of planning on outcomes and target achievement indicators [25]. Moreover, spatial planning is oriented to complex open systems with comprehensiveness, complexity and uncertainty characteristics. Therefore, although many studies have discussed the identification and measurement of the effectiveness of planning implementation, there has been no simple, direct, and unified definition of the concept [15]. Based on the prerequisites for the application of systematic comparative consistency and effectiveness evaluation methods, Faludi proposes four conditions and criteria for effective planning: (1) The actual development situation is subjectively matched with the planning and is fully consistent; (2) subject to objective factors, the actual development situation has to go against the planning subjectively, and this development decision has been proved; (3) planning provides certain reference and assistance to decision-making consequences on actual development; (4) although the actual development is contrary to the planning, the final goal of the planning has not changed and is in essence a constant revision of the planning [26].

In addition, the same planning implementation results can lead to different conclusions in evaluating the success or failure of planning implementation based on different analytical perspectives. If based on the control perspective, the goal of planning is to control future uncertainty. Then planning can only be considered to be successfully implemented if the actual land use corresponds exactly to the planning intention. If the goal of planning, based on an interventional perspective, is to guide land use and reduce the impact of future uncertainties, then as long as the decision-making process of land use is affected by the planning, even if the land use is not completely consistent with the planning intention, planning does not necessarily mean that the implementation has failed. From the perspective of management, the implementation of planning can be considered successful even if the actual land use is not consistent with planning, as long as the results of the implementation of planning help solve real problems [27].

2.2. Connotation of the Implementation Effect of Spatial Planning

The multiple amenities of land resources create a variety of potential possibilities for land use. In addition, spatial planning is a comprehensive land-use scenario that deals with future uncertainty and, therefore, requires a more complex goal orientation. As a result, spatial planning has multiple objectives, and the effectiveness of its implementation is often interpreted from multiple perspectives by various scholars.

2.2.1. Perspective of Value Realization

Land resources have six values: exchange, use, society, environment, image and culture, and planning implementation ensures the realization of the above land values [28]. In addition, some scholars have suggested that the effectiveness of planning should not only consider the economic objectives of land use but also the protection of important resources such as agricultural land, ecological environment and historical and cultural heritage, as well as the maintenance of social equity, the protection of infrastructure and the provision of adequate land for affordable housing. The effectiveness of planning implementation is a balance of health, safety, convenience, and aesthetics [29]. Furthermore, it cannot be overlooked that within the context of a sovereign state, spatial planning also reflects the government’s political intentions. That is, development goals and strategies for national security and ecological security are embedded in spatial planning, such as a development strategy for regional balance [30]. In this sense, political goals can be achieved through spatial planning implementation [31]. However, Evans maintains that achieving economic value and securing a livable environment are still important manifestations of the effectiveness of current planning implementation, as opposed to other planning objectives. In the late 1980s, urban planning was considered a political decision under the social, political, and economic structure and a rational behavior choice of the “economic man” [32]. Therefore, the interpretation of the effectiveness of planning implementation shifted to the value dimension of planning, that is, the impact of planning implementation on social costs and benefits. Land use is not only about creating social wealth and realizing its economic value through resource allocation but also about providing a livable and sustainable natural ecological environment for human beings and ensuring the realization of ecological and social values [29].

2.2.2. Perspective on Achieving Goals

The formulation and implementation of planning can provide a degree of certainty for the uncertainty of future development [33]. The long-term impact of the planned activities on the natural environment and the human-built environment is irreversible. Spatial planning guides and regulates land development behavior through zoning, regulation and other measures. From this perspective, the effectiveness of planning can be measured by the degree of conformity between planning policy results and their expected goals [34]. In addition, spatial planning also has regulatory characteristics. It defines the quantity and uses rules for various types of land through planning maps and zoning control maps [35]. The degree to which the planning implementation results conform to the planning expectation, that is, the implementation degree of the planning scheme, is an important reflection of the effectiveness of the planning implementation [36].

2.2.3. Transaction Cost Perspective

Alexander first introduced the concept of transaction cost into planning theory [37]. New institutional economics believes spatial planning is a public policy and institutional arrangement, and the reduction of total social costs is the effectiveness of planning implementation [38]. These social costs include the transaction costs of all stakeholders in the process of spatial development. Lai defined spatial planning as “institutionalized control of the spatial representation of human activities” [39]. Therefore, an important reflection of the effectiveness of planning implementation is whether the institutional arrangements of planning help planners reduce transaction costs. In the planning implementation process, its effectiveness will be greatly compromised if the relevant planning policies fail to reduce the transaction costs of land development [40]. In other words, planning can be implemented more effectively by reducing transaction costs [41].

2.2.4. Externality Perspective

Spatial planning involves all aspects of public interests. The purpose of spatial planning implementation is to provide public needs for survival, including food, housing, commuting, ecology, and other public interests, to ensure basic survival, maintain spatial fairness, and improve economic development [29]. Therefore, the results of planning implementation have significant externalities. Two aspects can characterize the effectiveness of planning implementation: one is to improve overall social benefits through planning implementation; the other is to reduce negative social impacts and curb negative public externalities. For this reason, public externalities brought by planning to society, economy, and environment through guiding and controlling land development can represent the implementation effect of planning [42]. Pennington revealed the effectiveness of government land use planning by comparing the changes of externalities before and after planning implementation [31].

3. Evaluation Criteria of the Implementation Effect of Spatial Planning

Spatial planning implementation evaluation is an important aspect of planning evaluation [22]. This approach essentially requires a set of standardized criteria to enable planners to make “real judgements” and reduce subjectivity in the planning evaluation process [6,16,43]. Shen proposes that evaluating the effectiveness of spatial planning implementation focuses on assessing the results, effects and their impact after implementation in terms of the social, economic and spatial environment [17].

The basic question of planning effectiveness evaluation is how to choose the basic criteria or reference norms for planning effectiveness. What is the benchmark of planning implementation effectiveness [44]? Moreover, to enhance the credibility of planning as a discipline or profession, planners must use evaluation criteria, and these evaluation criteria must be able to make real judgments about the effectiveness of planning: good plans must be distinguished from bad ones [6]. Moreover, these criteria serve as a sufficient basis for planners, the government, and the public to judge the implementation results of planning. In the existing planning research practice, although some researchers have put forward different criteria [18,45]. For example, Alexander and Faludi proposed a series of possible evaluation criteria, including the “consistency” between implementation results and planning objectives; the rationality of decisions made during planning implementation; the quality of planning solution; comparing the impact of planning before and after implementation decisions; and identifying whether planning affects decision making, etc. [6]. Baer formed a composite list of general criteria, which includes eight basic classifications, namely (1) Adequacy of Context, (2) “Rational Model” Considerations, (3) Procedural Validity, (4) Adequacy of Scope, (5) Guidance for Implementation, (6) Approach, Data, and Methodology, (7) Quality of Communication, (8) Plan Format [14]. According to the Plan-Process-Results (PPR) methodology for evaluating planning and planning implementation, Oliveria and Pinho have proposed nine evaluation criteria: (1) internal coherence, (2) plan relevance to the city’s needs and ambitions, (3) interpretation of the planning system, (4) external coherence, (5) public participation in plan-making and implementation, (6) plan utilization in decision-making, (7) commitment of human and financial resources, (8) effectiveness (plan results), (9) direction for the urban development process [16]. Thus far, there is still no consensus among planners on the exact set of criteria that can be used to judge the quality of planning and the effectiveness of the process [46].

3.1. Rational Results

Planning results from rational decision-making [47], and outcome rationality is an important value orientation in evaluating the effectiveness of planning implementation. Scholars who hold the concept of outcome rationality believe that the purpose of planning is to eliminate the influence of future uncertainty and that the extent to which the outcome of planning is consistent with planning expectations can be used as a criterion to evaluate the effectiveness of planning implementation [48]. Therefore, the evaluation criteria of planning implementation based on the consistency between planning implementation results and planning intentions has been widely used. Taking the implementation results of planning as the evaluation object, a consistent evaluation benchmark can simply and clearly describe whether the planning intentions have been implemented and realized [49].

First, the basic contents of spatial planning include the zoning of planned use and the corresponding control rules [35]. Therefore, the evaluation based on the consistency of implementation results needs to focus on two benchmarks: (1) the degree of coincidence between actual land use and planning expectations [8,49,50,51,52]; (2) to what extent planning-related supporting policies and institutional tools have contributed to the realization of planning objectives [36,53,54]. Sometimes, regulatory tools such as zoning, statutory plans and building/development control are used in the built environment, sometimes called “regulatory planning” [55]. Iban M.C. proposed an in-depth policy analysis with a historical framework to assess the performance of Turkey’s policy response to informal housing and non-compliant development. This assessment method has four main components: Inspection of non-compliant development, securing land tenure rights, land use and urban resilience and social inclusion [56]. Secondly, in addition to examining the consistency of plans and control measures, planning implementation evaluation also focuses on the implementation of specific control objectives [57,58]. Consistent evaluation criteria can help distinguish whether the planning results deviate from the targets [8].

Consistency evaluation criteria based on outcome rationality have been widely used both at the macro level to assess the effectiveness of urban sprawl control [59,60,61] and at the micro level to evaluate the effectiveness of land use control [9,12,51]. However, scholars have also questioned and criticized it due to its rather mechanical and even extreme evaluation philosophy [17], which tends to ignore the complexity of the planning implementation process. Wildavsky believed that planning would be doomed to be ineffective if only outcome rationality was adopted as the benchmark for measuring the effectiveness of planning implementation. The generation mechanism of planning implementation results is complex, and it is insufficient to reflect the effect and influence of planning on actual land use if only the consistency of results is used as the evaluation standard of planning implementation effectiveness [14,15,16].

3.2. Process Rationality

Process rationality holds that planning implementation is a process of a series of planning decisions and that the results of planning implementation are determined by the process [62]. Process rationality emphasizes that planning is not an end, but rather the right decisions are more important. Therefore, in the process of planning implementation, as long as the decision-making of the planning implementation subjects (including government departments, the market actors and other relevant stakeholders) are rational, the planning implementation can be considered effective even if the planning implementation results are inconsistent with expectations after full consideration of planning objectives and implementation conditions in the face of the current planning implementation environment [17].

It can be seen that the benchmark of process rationality is focused exclusively on planning decisions. Since planning has its inevitable limitations, it is impossible to fully qualify whether all the plan requirements are implemented, partially adopted or abandoned. Therefore, the correctness of planning decisions is the most important, and the planning or implementation is secondary [6,63]. Planners are conscious that planning schemes cannot reach a perfect state. In the face of development uncertainty, planning objectives may lag behind development needs.

Moreover, as planners and local government officials are not completely “rational people” like economic assumptions, planning schemes always have loopholes [64,65]. Therefore, it is also inappropriate to benchmark the effectiveness of planning based on the consistency of the results of its implementation. As a firm advocate of process rationality, scholars of the Dutch planning school believe that the basis of the effectiveness of planning implementation should be based on the planning implementation process and whether planning can guide and control the role and influence of subject land behavior [10,27,66]. Whether planning decisions achieve the expected results and whether decisions reflect the planning expectation are both measurements of the effectiveness of planning implementation [19]. Therefore, the criterion based on process rationality places greater emphasis on evaluating the effectiveness of planning implementation decisions.

The performance-based evaluation benchmark proposed by Alexander and Faludi has been endorsed and promoted by planning scholars from the Netherlands [6,10,27,67] and has also produced a wide range of influences [19]. However, the evaluation criteria based on process rationality also have their limitations. Does the planning process meet the requirements of rational decision-making? Given the information available to planners and decision-makers at the time, can it be reasonably judged that planning implementation decisions are feasible and optimal? The answers to these questions require a post-mortem reconstruction of decision makers’ prior perceptions of their situation, motivations and the context in which they acted. However, the key factors of these evaluations are difficult to observe, measure, and answer [6] and difficult to operate in practice [19,65].

3.3. Pragmatism

It is also inappropriate to judge the effectiveness of spatial planning only based on its conformity to the initial planning [5]. As a tool for public power to allocate public resources, the effectiveness of planning implementation is based on comparing the situation of a completely free market and a self-organization scenario without planning [68]. In contrast to the mechanistic and one-sided nature of the consistency criterion, Alexander and Faludi (1989) believed that the judgment of the effectiveness of planning implementation should be based on accurate results, that is, whether the actual effect of planning implementation is ideal [6]. Realism-based benchmarks for evaluating the effectiveness of planning are based on whether the planning is actually working [35].

A pragmatic-based approach suggests that the evaluation of the effectiveness of planning implementation should begin with a clear definition of planning objectives, further define the content of the planning based on these objectives, and then establish expedient indicators to assess the extent to which the objectives have been achieved after implementation [69,70]. Alexander believed that the criteria for judging the effectiveness of planning implementation should focus on execution and consider planning quality and strategic optimization [71]. After implementing the program, if the positive results of planning significantly outweigh the expected adverse effects, the planning implementation can be considered to be effective. In addition, rigid evaluation criteria are disconnected from their actual work for the subjects who supervise the operation of planning, such as the planning management department. The implementation results are the same as the requirements of the planning text and plans, and the implementation effect reflects the planning role [72]. In general, pragmatists emphasize that the evaluation of planning implementation needs to have a clear evaluation goal. If planning implementation is useful to achieve this goal, then it also indicates that planning implementation is effective. Even if the accuracy and systematicness of planning implementation results are affected to some extent, this criterion reduces the difficulty of implementing evaluation and the human, material and financial costs [18,73].

However, the evaluation criteria of pragmatism mainly come from the subjective judgments of planners and researchers, which also affects the objectivity and credibility of the benchmark of pragmatism [17]. Pragmatic criteria are often established after the planning implementation, and it is highly controversial to define whether this outcome is “useful.” Moreover, planning is oriented to multiple subjects. One of the biggest characteristics of multiple subjects is the heterogeneity of interest appeal, which also increases the “rationality” to determine the validity of the judgment benchmark of implementation results [35,74].

4. Evaluation Framework

4.1. Graph Superposition Method

Alterman and Hill were the first to evaluate the consistency between land use planning and results through a hand-drawn grid method, which was the forerunner of the GIS grid analysis widely used today [50]. With the popularity and widespread application of GIS technology, many researchers have also favored quantitative analysis of the differences between spatial planning control and actual land use situation through spatial superposition technology and evaluation of the implementation effect of planning [75]. The graphic superposition method is often used to test the extent to which specific control objectives are achieved to measure the effectiveness of planning implementation. Tian et al. used it to evaluate the compliance of actual land use during the implementation of the Guangzhou Urban Master Plan [65]. Dempsey and Plantinga used remote sensing data to evaluate the effectiveness of development boundary controls in 19 cities in Oregon, USA [54].

It can be said that the graphic superposition method is the core analytical framework for many planning consistency evaluations and also the most intuitive method to identify whether planning implementation results are consistent with expectations. Loh further subdivided the inconsistent parts of the planning implementation results based on applying the graphic superposition method [9]. Based on previous research results, Li et al. further distinguished the different characteristics between the planning implementation process and results by comparing the changes in land use types before and after planning implementation [22]. Under the analytical framework of graph superposition, the researchers also explore different representations of the effectiveness of planning implementation from different analysis perspectives by developing evaluation index systems. Based on superimposing the implementation results of planning and the planning map, Han et al. constructed three measurement indexes: boundary tolerance, boundary adequacy, and boundary adjacent development, to evaluate the control effect of planning on urban growth [59]. Moreover, this evaluation system has been further applied in evaluating the effectiveness of boundary control of planned construction land in Hangzhou’s General Land Use Planning (1997–2010) [23]. With the support of GIS spatial analysis, Liu compared the consistency and implementation of an urban master plan and land use plan in suburban Beijing [76].

4.2. The Policy-Plan/Programme-Implementation-Process Approach (PPIP)

The graphical overlay-based approach provides a very convenient analytical framework for evaluating the consistency of planning implementation. However, as the results of consistency analysis tend to be one-sided, the Dutch school has developed an implementation assessment system on “planning on what” to improve the limitations of planning implementation consistency evaluation [75]. Alexander and Faludi first proposed the Policy-Plan/Programme-Implementation-Process Approach (PPIP model) [6]. This model integrates policy, planning, development projects, project implementation decisions, and a series of implementation measures to serve the whole evaluation system. It also lists criteria, such as consistency, rational process, ex-ante optimality, ex-post optimality, utilization, etc., applied to ongoing policies and planning processes and their results [6].

The PPIP model combines the core perspectives and corresponding evaluation methods of policy evaluation, planning evaluation and project evaluation and is considered to be a complementary integration of the perspectives and criteria of the various evaluation methods. According to this series of criteria, the effectiveness of planning implementation can be classified as positive, neutral, or negative. Mastop and Faludi further modified and improved the PPIP framework and believed that evaluation should focus on the planning policy units and evaluate independent, clear, and operable policy objectives. Secondly, the planning assessment should focus on the subject of the planning implementation. That is, whether the subject of the planning implementation is clearly aware of the plan’s intentions and whether the information conveyed by the plan can be translated into a specific act of the planning implementation [10].

4.3. Means for Evaluating Actions of a Structural Nature (MEANS)

In the late 1990s, in the context of the allocation of structural funds, the European Commission proposed Means for Evaluating Actions of a Structural Nature (MEANS) as a comprehensive methodological guide for evaluating socio-economic projects. The methodology is based on four main criteria: relevance, efficiency, effectiveness, and utility, as well as three complementary criteria: clarity of objectives, internal consistency of objectives (within the project being analyzed), and external consistency between project objectives and other relevant public policies. MEANS has established indicators related to information processing, information comparability, information scope, plan completion stage, evaluation criteria, and information quantification and use patterns [77].

4.4. Plan Implementation Evaluation Approach (PIE)

The PIE framework (Plan Implementation Evaluation Approach, or PIE) has been developed by researchers from the USA and New Zealand. This evaluation framework is an approach that focuses on the degree of linkage among planning, related policies and planning permissions. It relies on the analysis of planning and permissions and provides a rigorous, quantitative and systematic method for assessing the degree of implementation of land use planning [8]. The PIE framework conceptualizes how planning realizes its policies by adopting relevant management technologies in the development permission. For this purpose, it determines whether planning has been implemented and to what extent by measuring “whether the designated management technologies have been used to implement the planning” [75].

Under the framework of PIE, the evaluation methodology focuses on the strength of the connection between planning control policies and planning permits and evaluates the effectiveness of planning implementation by measuring the adoption of relevant management techniques. A well-executed plan is defined as one that fully realizes its planning intentions using specific planning policies [8]. Two indicators, breadth and depth of implementation, were used to evaluate the implementation effect of planning administrative guidance documents in the management of planning permission. These include administrative mandatory laws and regulations to help implement relevant planning, planning implementation guidelines and guidelines standards, urban development management tools and methods, etc. [75].

4.5. Plans-Process-Results Model (PPR)

Oliveira and Pinho proposed a Plans-Process-Results model (PPR) for planning implementation and post-implementation [19]. This model is similar to the consistency-based assessment method and has much in common with PPIP, PIE and other models [75]. Unlike the other evaluation methods mentioned above, which focus only on one level of planning, the PPR framework is designed to evaluate the preparation, implementation and review results in planning practices and their contribution to the complex urban construction process. To this end, PPR evaluates against multiple criteria and data sources rather than just program-defined and focused indicators [19].

The PPR model was established based on integrating existing planning evaluation frameworks, policy analysis frameworks, strategy implementation analysis frameworks and evaluation standards, and referring to evaluation frameworks including PPIP, MEANS, PIE and evaluation benchmarks and methods proposed by many planning researchers [26,77,78,79,80,81,82] to construct ten measurement indexes. These include internal consistency, planning system interpretability, external heterogeneity, participation in planning compilation, planning application, resource coordination, public participation, planning efficiency and direction to comprehensively evaluate the effectiveness of planning implementation from the aspects of planning quality and planning application, respectively. Overall, the PPR framework integrates the evaluation of planning preparation and implementation [75,83].

4.6. Planning Goals—Planning Contents (PGs-PCs)

Planning goals (PGs) are the most basic content of planning [36], and Planning Contents (PCs) are detailed arrangements in the plan, such as regulations, measures, tasks and zoning. Therefore, the planning objectives play a leading role in planning implementation, and planning contents are the means to achieve the planning objectives. Therefore, measuring the effectiveness of planning implementation needs to combine the two organically. Shen et al. decomposed the planning implementation process as follows: (1) establishing planning objectives; (2) formulating planning content; (3) making a decision according to the plan; (4) making decisions and producing the corresponding results [15]. A comprehensive framework was built to integrate performance and consistency criteria and evaluate planned execution results. Under this framework, planning content evaluation includes five steps: (1) the impact of planning content on decision-making; (2) the consistency between decision execution results and planning; (3) satisfaction with the after-benefits of the results; (4) judging whether the planning goal is achieved by the deviation degree of the post-benefits and results from the planning; (5) reviewing the guiding power of planning objectives to decision-making by the degree of realization of planning objectives. The greatest contribution of the framework is to address a key difficulty in performance evaluation: identifying the impact of a plan on decision-making in non-conforming behavior in a case.

5. Evaluation Methods

5.1. Consistency Evaluation Method

According to the application of resultant rational evaluation benchmarks at different levels of planning implementation, consistency evaluation methods can be divided into consistency evaluation based on the implementation of planning indicators, consistency evaluation based on the control requirements of the plan and consistency evaluation based on the guidance of functional objectives.

Firstly, the consistency evaluation based on the implementation of planning indicators is the simplest and most widely used method [2,6,51,52,61,65]. From the perspective of pragmatism, the evaluation of planning effectiveness should clarify the purpose, content and targets of planning evaluation and reasonably select operational indicators that can achieve the evaluation targets of planning [44,69,70,78]. Due to the strong externalities of land use, spatial planning often takes land use as the main object of control. Moreover, spatial planning often sets clear control objectives and quantifiable evaluation indicators, such as ecological protection areas, the scale of new construction land and the plot ratio. Therefore, planning effectiveness evaluation based on consistency criterion measures the realization degree of planning control indicators [53,54,58].

Secondly, after the rapid development and popularization of GIS technology, the consistency evaluation based on plan control requirements has become a popular method in planning implementation evaluation. The consistency evaluation of spatial plans originated from Alterman and Hill’s hand-drawn cell grids to evaluate the consistency of planning implementation results and plans [50]. Subsequently, Loh further classified the plots with inconsistent planning implementation results into succession Type (Type A), inheritance Type (Type B) and deviation Type (Type C) [9]. Li et al. proposed that the spatial differences in the implementation and distribution of various types of use control in the evaluation year after the implementation of the planning scheme can be examined from a macro perspective to evaluate the spatial consistency of the planning implementation results [22]. In addition, based on the previous studies, the relationships between planning implementation results and planning control plans are reclassified and defined. That is, the part that conforms to the planning (including the same status quo use and the same location as the plan, and the status quo changes and conforms to the planning scheme), the part that violates the planning (that is, the status quo changes but does not conform to the planning scheme) and the part that is not implemented (that is, the planning requirements are adjusted but not yet implemented). Wu et al. applied remote sensing interpretation to construct a framework for evaluating the consistency of planning implementation based on three dimensions: spatial scale control effect, spatial result control effect, and spatial morphology control effect. This paper evaluates the spatio-temporal evolution characteristics of Beijing’s urban spatial form from 1978 to 2006 and the control effects of five plans from 1958 to 2004 [84].

Thirdly, as the time dimension of planning implementation is uncertain i.e., it is unpredictable what scenarios will be encountered during the implementation of the plan. Therefore, the effectiveness is tested by evaluating whether the results of planning implementation are consistent with the objectives of planning management and control through a consistency criterion based on a functional goal orientation [42]. The evaluation results test whether the so-called rational decisions in planning are implemented; whether reality works as planned, thus reducing uncertainty about future developments; and whether the planning objectives, i.e., the public interest, are achieved. In addition, most people only pay attention to planning preparation but know little about the implementation and actual results. Consistency evaluation can compensate for this deficiency to some extent and help improve the understanding of the planning implementation results [8,81,85]. Therefore, based on the consistency evaluation of atlas control, planning researchers have added the content of consistency evaluation guided by functional objectives according to specific planning control objectives. Yue and Zhang further decomposed spatial consistency into three dimensions: the change of spatial gravity center of urban development, the differentiation of regional spatial expansion, and the coincidence of functional land use [86]. They constructed a planning evaluation method system and comprehensively evaluated the spatial control effect of planning on urban development by taking the urban planning of Hangzhou as the evaluation object.

Finally, the issue of planning hierarchy often needs to be faced in evaluating the effectiveness of planning implementation. Top-down planning systems or between different plans with the same objectives at the same spatial scale also need to assess the effectiveness of plan implementation through the consistency of planning objectives. In order to figure out if and how strategic planning intentions trickle down to land-use planning and how ultimately zoning transforms them into legally binding regulations, Schmid et al. analyzed the compliance of land-use planning with strategic spatial planning in a multi-level planning system [83]. To identify how different plans from the same urban region relate and are externally consistent, Bacău et al. develop a framework to assess external consistency among plans [87].

5.2. Effectiveness Evaluation Method

Although consistency evaluation can directly reflect the degree of conformity between planning implementation results and intentions, there is no clear one-to-one correspondence between planning implementation results and planning. Planning implementation still needs to evaluate how much spatial planning, as part of the social interaction process, can influence actors to make decisions in accordance with the intentions of the plan [6]. Although efficiency evaluation focuses more on the impact of planning implementation on land use behavior, efficiency theory has been plagued by a lack of operational implementation methods since it was put forward [16]. Faludi and Altes proposed the following solutions to solve the operation difficulties of the efficiency evaluation method: (1) Effectively identify the decisions that play a role in planning; (2) relevant commitments in the planning decision-making process should be collected and sorted out, and the legitimacy of decision implementation should be fully demonstrated; (3) reasonable analysis of whether planning can guide the realization of planning objectives [88]. Mastop and Faludi clarified in their research that there should be three basic conditions for the realization of planning effectiveness: (1) Planning positioning should be a decision-making aid tool; (2) planning should be long-term in nature, especially with the late decision echo coordination; (3) planning should play a substantial guiding role in the decision-making process [10]. According to the differences in the assessment focus of effectiveness evaluation methods, it can be divided into the effectiveness evaluation based on process performance and the effectiveness evaluation based on measure performance.

Effectiveness evaluation based on process performance aims to influence planning on relevant decision-making processes [78]. Lyles et al. distinguished between what they called planning impact (the effect of planning on decision-making behavior) and planning performance (the effect of planning on implementation results) [52]. As Carmona and Sieh pointed out, a true understanding of the impact of planning on the development process in any given place, that is, the “outcome quality” of planning, can only be achieved when the planning process results are evaluated [74]. Available empirical studies mainly focus on qualitative analysis of recorded surveys, such as interviews, questionnaires, archives and meeting minutes. Most focus on some important aspects of the planning implementation process and lack comprehensiveness [15]. Lyles et al. evaluated the implementation compliance of more than 100 local disaster reduction plans formulated according to the national planning requirements of the United States and the performance of improved planning schemes using the above methods [52]. Millard-ball fully understood the role of planning in the decision-making process through in-depth interviews with local government workers, planning practitioners, developers and intermediaries and summarized the reasons for decision violation of planning [89].

The effectiveness evaluation based on measure performance focuses on the influence of specific policies and measures adopted on planning implementation decisions. Planning decisions have considerable impacts on both the natural and built environments of cities, and these impacts can last for decades, while many of them are irreversible. To better understand the long-term impact of planning implementation effectiveness, planners must systematically evaluate the policy tools used in planning implementation [41]. Performance-based evaluation requires planners to re-understand planning, planning participation and decision-making in the process of planning implementation. Planning itself is not a component of decision-making but should be a reference and auxiliary tool for decision-makers to improve the scientific nature of decision-making and reduce the uncertainty in the planning implementation process [6,25,74,90]. Therefore, planning effectiveness evaluation focuses more on the role and influence of planning implementation. Supporters of efficiency evaluation criteria believe that planning effectiveness does not require consistency between planning and reality but plays a guiding and referencing role in the decision-making process [2,91]. From the perspective of planning implementation performance, planning is more like a “decision-making framework,” and the evaluation of planning implementation focuses on whether and how planning guides and controls decision-making behaviors [19]. Planning implementation can be considered effective if planning plays a guiding role as an integral part of the implementation decision process [66]. Zhong et al. focused on analyzing the role of planning policies, regulatory rules and other basic elements in planning implementation decisions in the implementation process of planning. They took the second round of national land use master plans in China as a case study to evaluate the effectiveness of planning implementation measures [58].

5.3. Comprehensive Evaluation Method

The comprehensive evaluation method has been widely recognized and applied in the implementation evaluation of spatial planning. This method is mainly based on the comprehensive evaluation index system and is also called the multi-index evaluation method. Alexander and Faludi constructed a “policy-planning/program-program-application-process” model (PPIP) to evaluate the effectiveness of planning implementation based on five criteria, namely, conformance, rational implementation process, early-stage program optimization, late-stage result optimization, and applicability [6]. However, due to the complexity of its application, it is difficult to apply it in the practice of planning evaluation [17]. Oliveira and Pinho established ten specific index systems from three processes: planning preparation, implementation, and quality based on the planning process [19]. According to the role of land use planning in controlling the expansion of construction land and how it is achieved, the effectiveness of planning control can be evaluated from different perspectives. (1) From the perspective of the planning implementation effect, through temporal and spatial consistency of the planning visions and expansion practices to analyze visual planning control performance, called the temporal and spatial consistency perspective. (2) From the perspective of whether land-use planning has an impact on the expansion of construction land, the effectiveness of its control is evaluated through measurements, which is called the effectiveness perspective of planning control [22].

Consistency evaluation is the basis for linking planning intentions to actual results. But an objective evaluation of the effectiveness of the implementation of spatial planning is also necessary to explain the results of conformance evaluations and whether conformance successes or failures were due to deficiencies with the plan itself or how the plan was used by key actors charged with its implementation [92]. Therefore, the integration of consistency and performance evaluation methods is a trend in the comprehensive evaluation of the effectiveness of spatial planning implementation. Shen et al. built a framework for assessing the effectiveness of land use plans by integrating and widening the conformance and performance criteria that allow the different roles played by plans and the corresponding degrees of planning effectiveness to be distinguished [15]. To balance the consistency of the crucial evaluation element of spatial planning with the degree to which the plan is actually used, Feitelson et al. proposed an analytic framework combining conformance and performance in evaluating (regional) land use plans [12]. In order to reveal the gap between actual situations and planning goals and can be established by the degree of realization of planning goals, Li et al. seek to construct an integrated theoretical framework that integrates conformance and performance from a goal-oriented perspective [93].

6. Evolution of Spatial Planning Effectiveness Implementation Evaluation

6.1. The Evaluation Concept Changes from Complete Rationality to Limited Rationality

Conformance and performance are currently the most widely used measures to evaluate the effectiveness of program implementation [1,27,52]. Deviations in planning implementation may be a reasonable choice made according to changes in external factors, or they may result from problems with planning or planning implementation. However, consistency evaluation alone cannot make a necessary distinction between the abovementioned reasons. Consistency can be used to measure the effectiveness of the planning implementation, and its implicit premise is that the subjects of the planning implementation are completely rational. Its behavioral logic is 100% in compliance with the will of planning. Planning implementation is a long-term process full of uncertainties affected by multiple factors. Therefore, if the spatial evaluation of planning implementation only focuses on whether the final spatial form is consistent with the original spatial form or whether the implementation process has been carried out in accordance with the intention of planning. These completely rational evaluation criteria cannot accurately evaluate the implementation results of planning [17]. Therefore, the criteria of efficiency and reasonableness of results are proposed and applied to compensate for the bias in evaluating the effectiveness of planning implementation under the concept of total rationality. The implicit premise of these two evaluation criteria is that the planning implementation subject is bounded rational. As long as the outcome is beneficial, it is desirable to deviate from planning [88]. Especially in the case that the subject of planning implementation has independent initiative, under the influence of some irresistible external forces, and in the presence of strong disturbances in the planning environment, the subject of planning implementation may adopt behavioral strategies that are inconsistent with the planning intentions.

6.2. Research Methodology Changes from Simple Closed System to Complex Open System

With the deepening of research and the rapid development of science and technology, the methodology of planning implementation research has also changed. Traditional planning implementation researches abstract planning implementation as a simple closed system. Therefore, planning implementation evaluation can analyze and evaluate the effect of planning implementation through certain cross-sectional data. To a certain extent, this simple and closed research method runs counter to the evolution direction of the spatial planning system: a growing number of theoretical research and empirical evidence have led to the realization that planning compilation and planning implementation is highly open and has dynamic expediency. On the one hand, the completion of planning and the formal promulgation of a plan does not mean that it is once and for all due to the limited rationality of human beings and the uncertainty of the future. In the future implementation process, the plan may be constantly violated, reviewed, denied and revised to improve the plan’s adaptability and guiding power.

On the other hand, the compilation and implementation of planning should become an open decision-making platform, enabling relevant government departments, stakeholders and other various market subjects and social subjects to fully enjoy the right to participate and make planning a mechanism for public expression and communication and consultation. In addition, most existing studies set spatial planning as a rational and clear model that can guide future spatial development. Simultaneously, it regards spatial planning and spatial development as two relatively static and mutually causal states [75]. However, external factors also widely affect the compilation and implementation of planning. The research methods under the concept of the complex open system can adapt to the real needs of spatial planning implementation evaluation. However, they still face many theoretical and methodological dilemmas.

6.3. The Research Perspective Shifts from the Map to the Main Body of Planning Implementation Behavior

The traditional pattern-type spatial planning paradigm is being attacked by postmodernism. They argue that there are many problems in this rigid planning and implementation model. Based solely on comparing actual land use with the plan, the implementation evaluation only identifies the differences between reality and the plan’s vision. It does not objectively reflect the plan’s objectives which cannot be shown on the map to have been achieved. It is also not possible to explore the impact and effect the plan has had on the actors and why the actual land use deviates from the plan’s intentions.

Suppose planning is regarded as a tool to assist decision-making and reality due to people’s decisions and actions. In that case, it is more meaningful for planning implementation evaluation to focus on the effect of planning on relevant actors rather than the coincidence of results with planning drawings. This is not only because the main body of planning implementation behavior is the bridge connecting planning and actual land use but also because it is the real target of planning to influence and assist relevant main body decision-making and action. From the perspective of the evolution direction of the planning implementation evaluation framework proposed by the existing studies, the research perspective of planning implementation evaluation is shifting from the map to the main body of planning implementation. It focuses on the role of the subject’s initiative on spatial pattern evolution and the potential impact on implementing planning intentions. This shift deserves credit. However, due to the complexity of human beings and their behavior mechanisms, the basic problems of “what to evaluate” and “how to evaluate” have not yet been satisfactorily solved in the planning implementation of behavior subjects and need to be further promoted.

7. Conclusions and Future Prospects

The controversy of planning implementation effectiveness measurement lies in evaluating the impact of planning implementation deviation. The deviation of planning implementation in the general sense is the inconsistency between land-use selection and planning control boundary. In contrast, the deviation of planning implementation in the broad sense represents the conflict between planning intentions and actual land use. Scholars have different arguments about the impact of deviation results on the effectiveness of planning implementation. Scholars who take result rationality as the evaluation benchmark believe that no matter what kind of deviation of planning implementation, it will harm the effectiveness of planning implementation [8,9,49,51,81]. Scholars with a rationalist view believe that as long as land use behavior in planning implementation is guided and influenced by planning, even if the results are inconsistent with the original intention of planning, the effectiveness of planning implementation can be positive [10,27,88,91,94]. However, it should not be ignored that both the result rationality framework and rationalism have limitations in the evaluation perspective. Results rational evaluation methods are too simple and rigid and do not consider the potential diversity of planning implementation results caused by the multi-objective nature of spatial planning. A summary of the advatage and limitations of each category of criteria, framework and methods is shown in

Table 1. Summary of criteria, frameworks and methods for evaluating the effectiveness of planning implementation.

Category		Advantage	Limitations
Criteria	Rational results	Simple and clear Easy to identify	Ignoring the complexity of the planning implementation process
	Process rationality	Focus on the impact of planning on human decision-making behavior	Key elements of the evaluation are difficult to observe and measure Difficult to manipulate
	Pragmatism	Avoiding a disconnect between planning implementation and actual needs	Subjective Evaluation benchmarks are controversial
Framework	Graph superposition	Quantitative spatial analysis Intuitive	Neglect of the planning implementation process and the actual role of planning
	The policy-plan/programme-implementation-process Approach (PPIP)	Comprehensive content Broad impact	Lack of practical workability
	Means for Evaluating Actions of a Structural Nature (MEANS)	Synthesis	Need to collect a large amount of assessment information
	Plan Implementation Evaluation Approach (PIE)	Focus on the strength of the link between planning control policies and planning permissions	One-sidedness
	Plans-Process-Results model (PPR)	Include planning preparation in the assessment of plan implementation	Assessment data are difficult to obtain
	Planning Goals—Planning Contents (PGs-PCs)	Integration of planning objectives and planning content	Operational complexity
Methods	Consistency evaluation	Intuitive, objective and easy to use	Consistency evaluation is not sufficient to capture the actual impact and effect of plan implementation
	Effectiveness evaluation	Ability to assess the long-term impact of the plan Consider the impact of the planning implementation process	Lack of operability Ignoring the economic, social and ecological impact and effectiveness of the plan
	Comprehensive evaluation	Synthesis Ability to assess the effectiveness of planning implementation from multiple perspectives	Complex to operate and difficult to practice

.

Moreover, according to the law, land use activities are subject to planning constraints and controls. Therefore, the criterion of rationalism is too broad. Although some scholars try to organically integrate the two frameworks [12,15], existing research attempts failed to provide a systematic theoretical framework to explain the nature and composition of the effectiveness of planning implementation.

In addition, spatial planning clearly delineates the permitted scope of land use and is equipped with corresponding planning control rules. Although the effectiveness of spatial planning control varies, in theory, the implementing subject must carry out land use activities under the planning control rules within the control scope delimited by spatial planning [23]. However, many spatial planning practice results have shown that with the diversification of stakeholders in the planning implementation process, the effectiveness of planning implementation depends on the ability of the planning to guide and control land use behavior. Therefore, when evaluating the effectiveness of spatial planning implementation, it is necessary to take a single subject as the evaluation unit. However, most existing studies regard the results of planning implementation as an indivisible whole to evaluate the effectiveness of planning implementation and do not establish an evaluation system to evaluate the effectiveness of planning implementation of a single subject.

Despite their limitations, social media data will likely be used in future urban research. Social media provides crucial information on visitor behavior, interaction with the environment in real-time and spatial information to a certain degree. Each year, the growing number of social media platform users shows that the population is willing to share textual or visual information and opinions about their activities, places visited, beliefs and experiences. Given the limitations of the information provided, such as the residency status, the lack of socio-demographic information, and the lack of appropriate algorithms, the future application of social media data in research will require an improvement in existing methodologies. It will also require the development of a framework that improves data extraction and analysis so that also urban planning is supported by data extraction procedures. With the development of new technologies and their close integration with traditional methods, technologies such as remote sensing, GIS, the Internet of Things and machine learning will help us to deepen our knowledge of the uncertainty of future spatial development. Thus, it will reduce deviations in the preparation as well as in the implementation of plans. This will have important implications for the making and implementation of spatial planning.