**1. Introduction**

A smart city used to be defined in the 1990s [1] as a city that theoretically adopts newly invented technologies. However, for the past 20 years, its definition and perspectives have been diversified in line with its academic, technical and spatial applications [2] to better reflect infrastructural, economic and social changes and the subsequent needs. Despite such variation, there exists a consensus in that the smart city aims to promote a sustainable city by advancing it through the introduction and the utilization of Information Technologies (IT), and information and communication technologies (ICT) [3].

Such a consensus could have well been maintained because widely publicized projects for the smart city construction [4] have drawn attention for their introduction of large-scale advanced technology infrastructure in major countries and cities around the world. In particular, the smart city has emerged as a new alternative for resolving the side effects of sudden urbanization in such fields as transportation, water and waste, safety and security, and healthcare [5], on the back of the development of ICT, as well as the sustainable development advocated by multinational institutions [6,7].

As such, the smart city has sometimes served as a barometer of the competitiveness of a country [8] or a city in terms of its sustainability from the outset to date [9]. Accordingly, companies and multinational research institutes that lead the development and the distribution of the aforementioned technologies have devised and applied standardized indices to evaluate the smart city based on major social and economic indicators of various countries and cities [10]. This indicates that a smart city project has been a useful tool to

**Citation:** Oh, J.; Seo, M. Measuring Citizens-Centric Smart City: Development and Validation of Ex-Post Evaluation Framework. *Sustainability* **2021**, *13*, 11497. https://doi.org/10.3390/su132011497

Academic Editor: Harald A. Mieg

Received: 27 September 2021 Accepted: 13 October 2021 Published: 18 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

judge the competitiveness of an entity [11] and assess its development potential, as well as its marketing means.

However, despite the overall focus on competitive factors, there have been growing calls in the academic circle for the importance of citizens, who are members of a city and the end-users of technologies [12], rather than the technology itself and its usefulness as a marketing tool. Just as many scholars have mentioned before, including [13–15], that the smart city has been highlighted as a means to enhance city dwellers' quality of life, rather than for bureaucracy, who ultimately consume technology [16]. The authors of [17–19] and others also stressed the significance of both tangible and intangible end-products, including technology, facilities, and services (TFS) from citizens' point of view, while shedding the competitive comparison between technologies adopted by different cities.

It is noteworthy that the first-generation smart cities, armed with new technologies, became outdated, and the time has come to determine how to maintain and manage them; for example, by improving, replacing, or deleting existing technologies. As an alternative, some have stressed opinion gathering from citizens to solve the unequal supply of infrastructures and services of a smart city. However, quantitative research in this field, just as stated before, is still confined to the distribution of technologies among states or cities, or the performance of specific technologies. In other words, the methodology that puts a focus on citizens has not achieved much progress despite the recent emphasis on the people in this discourse, which has served as a stumbling block to drawing practical alternatives to discussing the sustainability of a modern smart city.

Against this backdrop, this study aims to evaluate technology, facilities, and services of an established smart city from citizens' perspectives and draw implications for smart city-related policy measures in the future, which is expected to be a practical alternative to a sustainable smart city. For the goal, the theoretical review on a citizens-centric ex-post evaluation system, including satisfaction levels, is carried out, and a research model is to be established so as to present a standardized evaluation system for existing smart cities. In addition, a survey is conducted involving citizens of two cities, both of which adopt smart city-related TFS but employ different ways in implementing urban planning projects.

After analyzing the results, the study explores the possibility of whether the evaluation model presented here could serve as a basis for a citizens-centric assessment model in the future. It also states the implications for urban planning policies by interpreting the differences between the two cities in line with their geopolitical locations and business methods.

#### **2. Literature Review**

#### *2.1. Paradigm Shift in the Smart City*

The smart city was defined in terms of the creation of the urban environments of a city, along with advanced information technologies, such as ICT technology and innovation [20], as well as changes in the urban paradigm of globalization [21]. Then, in the following decade, the concept expanded to include such keywords as integration, citizens, and public services. After it developed further, considering knowledge and community factors, among other things, the smart city gained a new goal of ensuring the wellbeing of citizens and the promotion of their quality of life [22].

More researchers in the field shed have light on smart cities' socioeconomic aspects, while criticizing current studies on smart cities that have failed to consider citizens as a key factor [23–26], though they acknowledge that centering on physical infrastructure is easier to draw tangible results.

Recent studies, including [18,27,28], also highlighted the need to devise a citizenscentric smart city and services, as well as to set up separate organizations in charge of its maintenance, and to devise an evaluation system, noting that citizen-centered services cannot exist without the consideration of the citizens who reside within it.

These views contributed to defining the smart city, even in the technical circle, as a consolidated infrastructure that provides various services to citizens in both technological and social terms based on ICT technology [29,30]. Moreover, the trend also led private companies, such as [31,32], to emphasize that urban dwellers are a key element of a smart city, calling them the "smart citizen". After a series of discussions, a "smart citizen" currently means an individual who consumes services installed in a city or via his or her individual devices, as well as a subject who evaluates and interprets those services and then directly affects the related decision making.

In this regard, the citizens-centric smart city, which is the key focus of the modern smart city discussions, is meant to supply technology, facilities and services for citizens. In addition, the public sector, as well as private firms and organizations, improve ways to produce, supply and sell related goods by monitoring citizens' consumption patterns and analyzing their feedback. A series of such acts allows those private entities to secure sustainability. Accordingly, citizens are direct consumers of goods in a smart city and, at the same time, are those who produce important information that can be used by suppliers. This paradigm shift indicates that the smart city can be maintained by seeking changes from its top-down development strategies to its bottom-up systems, as well as through proactive communication with smart citizens [33].

Likewise, supplies of technology, facilities and services of a smart city are required to establish a fresh bottom-up system. Various studies including [23,34,35] also suggest that the need is urgent to establish a system to measure and evaluate the value of smart city goods currently used, in order to advance the city from its development and infrastructure construction stage, which is expected to ultimately generate co-creation between users, i.e., citizens and stakeholders.
