An Analysis Matrix for the Assessment of Smart City Technologies: Main Results of Its Application

Abstract

The paper presents the main results of a previously developed methodology to better evaluate new technologies in Smart Cities, using a tool to evaluate different systems and technologies regarding their usefulness, considering each application and how technologies can impact the physical space and natural environment. Technologies have also been evaluated according to how they are used by citizens, who must be the main concern of all urban development. Through a survey conducted among the Smart City Spanish network (RECI) we found that the ICT’s that change our cities everyday must be reviewed, developing an innovative methodology in order to find an analysis matrix to assess and score all the technologies that affect a Smart City strategy. The paper provides the results of this methodology regarding the three main aspects to be considered in urban developments: mobility, energy efficiency, and quality of life after obtaining the final score for every analyzed technology. This methodology fulfills an identified need to study how new technologies could affect urban scenarios before being applied, developing an analysis system to be used by urban planners and policy-makers to decide how best to use them, and this paper tries to show, in a simple way, how they can appreciate the variances between different solutions.

1. Introduction

Several authors, companies, or public institutions generate, day-to-day, more and new definitions for the concept of the Smart City. Many of them have provided a critical review of the literature so far, either from the socio-logical perspective, as in the case of Hollands [1], or from that of economists specialized in urban development, as in the case of Shapiro [2] and Caragliu et al. [3]. Others, from a technological approach, as in Lee et al. [4], have commented that advances in Information and Communication Technologies open up new opportunities for a more efficient urban management, as the easy access to networks and embedded sensors in the environment allows extending the concept of the ubiquitous city (u-city) to the ubiquitous eco-city. Or, as stated by Gilman et al. [5], who predicted that increasingly interactive spaces can learn from users and their environment, by focusing on the concept of Smart Cities, blinded by new technologies and just on their positive effects, it could lead to underestimating its potential negative effects, or to ignore other aspects that do not necessarily involve the application of technologies for a city to truly become intelligent. In 1997, Aurigi and Graham [6] warned of the possible effects of digitization in the cities, and strongly urged the emergence of local policies to ensure an inclusive democratic support to lead the use of e-infrastructures for the proper cyberspace urban planning, which today has been called Smart Cities. However, to trivialize the concept of Smart City could be risky. As Hollands [1] said, a Smart City (or Progressive City as he calls it) cannot be labeled as “smart” for simply adopting sophisticated ICT or having created websites for self-promotion, even when the involvement and contributions of a large group of actors is required. Cities are more than cables, smart offices, or trendy bars, and the large number of people living in cities deserve more than this. Consequently, a label should not obscure the real underlying changes that must occur in a city through a progressive urban change. Furthermore, Sassen [7] commented that the first phase in the definition of a smart city is exciting: it becomes a living laboratory that installs and experiences new urban technologies. However, this stage is preceded by numerous negative potentials, passing from experimentation and discovery to the hyper-managed space, where “sensorised” can become “censored”, a place where those new technologies have not been sufficiently contrasted in different urban scenarios. A new technology cannot just be launched in any urban space, and the challenge should be that the Smart City could be able to urbanize the developed technologies, making them accessible and useful to the people it directly affects. Taking into account the difficulty to obtain a clear definition of Smart City, Neirotti et al. suggested [8] that there is not a shared definition of SC because it is difficult to identify common global trends. Actually, they have explored the diffusion of smart initiatives via an empirical study aimed at investigating the ratio of domains covered by a city’s best practices versus the total of potential of smart initiatives.

In Smart Cities, new standards for social organization and the role of information and communication technologies are essential [9], with the duty to guarantee the ability for citizens, urban managers, and designers to understand these new systems and avoid any socio-technological exclusion. They require new tools in order to find more information for real-time solutions. Considering the potential of these new instruments, it can be argued that while technologies may be helpful in producing new spatial data, voluntary activities may be a suitable method of bringing such data up-to-date while describing it in an informal, more accessible manner to citizens [10]. With new technologies, the new urban priorities are mainly relationships and movements, where everything is interconnected, requiring new logics, new analytical tools, and new concepts [11]. In the digital-city, new technologies must help to improve the management of communication networks, mobility, and energy. However, the implementation and monitoring of those systems remains as the “Achilles heel” of urban planning [12]. The use of simulation tools provides a fair assessment for designing strategies to support designers from early stages of decision-making [13]. New methodologies need to be aligned with the main elements of sustainability: environmental, social, and economic [14], and must be processed with a systemic vision. Those aspects should be designed in order to identify emerging features and improve every area of opportunity, through technological, economic, social, and political points of view [15]. All of those sentences led us to raise the development of a methodology for the technologies and systems analysis, prior to being applied in urban environments, developing a useful tool to previously assess the effect of those systems in SC strategies.

The study of urban realities and the potential impact of systems or technologies on the city should not be oversimplified. Many initiatives can arise in a SC strategy, but their real impact is virtually impossible to foresee until the strategies are fully implemented. Using a dynamic tool that provides assessments through a series of coefficients and indicators for corrections could significantly improve the planning during the long road that involves all urban strategy and can avoid pitfalls stemming from fashions and interests [16]. We have already presented, in several publications, a methodology attempting to capture the largest number of relevant criteria and variables that should be considered in SC analysis. For the development of this methodology, analytical principles were used in order to obtain an objective score based on the sum of indicators, both quantitative and qualitative, considering the various overlapping elements which interact in a city [17]. This methodology was useful to assess that some systems work better than others in diverse areas, while some of them may have a greater overall impact on the strategy because they affect more than one issue of a SC strategy. Here we present some results of those technological evaluations obtained with this abovementioned methodology, regarding all of the aspects to be considered in the main Smart Cities’ strategies. At the end, we will show in different comparative charts how some technologies are able to cover just some aspects, but not others, taking into account that complementary systems will be necessary to obtain a well-balanced strategy.

2. Experimental Section: Methodology Development

To approach the different technologies to be applied in urban areas, the main aim of this methodology is to not forget the ultimate goal of any technology: respond to the needs of citizens. As presented in various works made in the course of this research [18], this assessment model should take into account the relationships that take place in a spatial area, which are the three main issues that technology should attend: environment, city, and citizen, in a scheme of three hundred and sixty degrees (Figure 1). Among them, there will exist different relational effects, such as the specific needs of the citizen that must be covered by the city, their systems, and infrastructures. The city will take and share resources from the environment. Finally, technology should interact through these three elements, working in both directions, carrying the information of citizens’ needs from cities to the environment, managing resources, and returning solutions to meet those needs [19].

At the same time, technologies should be considered to have a direct influence both in people and cities. This could be defined as impact, through the following variables: utility, functionality, applications, expected results, and consequences. Therefore, the matrix to be developed should include—among the purposes of the technology application—the impact it can produce on the different variables that constitute what can be called environment: social, urban, environmental, economic, and energy requirements. Thus, a necessary analytical deployment occurs when applying the technologies, studying in a chained way each of those aspects and their influence. As shown in the following diagram (Figure 2), the technology is only one element of this chain, and citizen must be the centre at any time, since he is who has several needs to be solved, and the only one able to evaluate the tools to be used, which should be analyzed regarding their usefulness, functionality, applicability, infrastructure needs, and expected outcomes.

When studying each technology available to them, citizens should evaluate the inevitable consequences, and they must be characterized by advantages and disadvantages, or risks and benefits. This analysis should be done for society as a whole. In other words, for the city and people, but also for the different elements that permit their progress: the environment, the economy, energy use, configuration, and livability of cities. All those different concepts must be entered into a double-entry table called the technologies analysis matrix (TAM), defining indicators to manage the different variables [20]. Allowing the combination of each factor in every line and column, we can determine sub-total scores by calculating the average of different elements that influence Smart Cities planning. This will allow a quick analysis of the impact and effects of the analyzed technology over each element that configures the smart city strategy. On the other hand, the final results on every line and column are the average data of different evaluated issues, and will set the final score of the assessed technology, considering a high range of inter-relationship, regarding all factors that occur in the city, obtaining the TT score (Figure 3).

At the same time, the combination of different systems may pursue the goal of a well-balanced Smart City strategy. If all technologies are focused on mobility, the city could lose the focus on other parameters as energy efficiency or quality of life, just to mention a few. That is the reason why the single assessment of technologies or systems may give comparable information. This information could be considered as sufficient to evaluate different implementation approaches, both at the moment of planning or implementing strategies, as well as considering the percentage of territory where the system is implemented: this is because it is not the same to apply a new technology as a pilot project in a street, or in the whole city. As can be seen in

Table 1. Technology assessment matrix (TAM): qualification process [17].

	Technology (Name)	IMPACT			Total
		1. Functionality	2. Expected Results	3. Consequences
BACKGROUND	a. Citizen	1.a	2.a	3.a	Ta
	b. Social	1.b	2.b	3.b	Tb
	c. Urban	1.c	2.c	3.c	Tc
	d. Environment	1.d	2.d	3.d	Td
	e. Economic requirements	1.e	2.e	3.e	Te
	f. Energetic requirements	1.f	2.f	3.f	Tf
Total		T1	T2	T3	TT

, by combining background and impact issues, a value from 1 (lowest impact) to 5 (highest impact) will be obtained in each box, according to a number of predefined indicators. Each line and column generates subtotals of the global balance of each individual criterion in the sum of elements of the environment (T1, T2, T3), and the full impact of that technology (Ta, Tb, Tc, ...) that can be assessed. By averaging each row and column subtotals, the final result of a particular system will be obtained as a whole.

After that, it is mandatory to define the link between the technology and the different elements of SC strategies. Correction values must be applied to the score obtained on the TAM depending on mobility (m), energy efficiency (e), and quality of life (q). The tool must be capable of assessing the overall work of all variables that affect a SC. Therefore, in these three areas, each factor is combined with the other criteria (

Table 2. Technology Assessment Matrix (TAM): Complete [17].

	Mobility Coef. (m)	Energy Eff. Coef. (e)			Quality of Life Coef. (q)
	Technology (name)	IMPACT			Total
		1. Functionality	2. Expected Results	3. Consequences
BACKGROUND	a. Citizen	1.a	2.a	3.a	Ta
	b. Social	1.b	2.b	3.b	Tb
	c. Urban	1.c	2.c	3.c	Tc
	d. Environment	1.d	2.d	3.d	Td
	e. Economic requirements	1.e	2.e	3.e	Te
	f. Energetic requirements	1.f	2.f	3.f	Tf
Total		T1	T2	T3	TT
TTm		TTe			TTq
					TTg

). If a technology gets a specific TT scoring in the TAM, upon the application of these correction values the impact can vary significantly in the strategy. A technology can be highly valued by the TT, but have only a positive impact on one area (m, e, or q) of the overall SC strategy. The TT score, once the mobility coefficient is applied, will be called TTm; the energy efficiency will be designated as TTe, and the one for quality of life, TTq. The new overall scoring adjusted to these three concepts will be named TTg. [17]. When the proposed technology or system generates a greater impact on the three main axes (m; e; or q), the rate will be increased.

Along our research process, the TAM matrix was tested in different cities of RECI (Spanish Smart Cities Network) [21]. Firstly, we conducted a survey of all 65 RECI members, which was divided into eight sections, in order to obtain as much information as possible to define the indicators that will be deployed in the TAM. Those sections were:

0-

Municipality identification data: asking for basic information, such as the number of inhabitants and the urban area.

1-

Smart City Strategy: which should reflect whether they had drafted a strategy and how it was made

2-

Dissemination of SC Strategy: to see if the councils had made some type of dissemination of the strategy.

3-

Mobility Issues: asking if their strategy addresses issues related to mobility and, in that case, what results could be obtained.

4-

Energy Efficiency issues: same as above, but in terms of energy efficiency

5-

Quality of Life issues: same as above, but in terms of the quality of life.

6-

Additional aspects: where was it possible to extend with other issues that could be over-looked.

7-

Technology and Systems Assessment Tool: regarding their interest in relation to having a tool for technology assessment.

8-

Assessment Tool for Smart Cities: same as above, but in terms of an assessment tool for the Smart Cities’ global strategies.

The result of this research work, carried out together with the different members of the RECI network, has confirmed the interest and the need for assessment tools for such complex urban and technological systems as those that arise in the development of a Smart City. In turn, it has served to corroborate which key aspects are essential to city managers in this process of defining policies and strategies, so the indicators that should be considered in developing those instruments could be defined, grouping them in the abovementioned three main issues: mobility, energy efficiency, and quality of life.

After all of this work with RECI, we considered the development of an instrument that can be useful for the evaluation and classification of different technologies that impact on the urban environment. A methodology that allows, in a simple way, to obtain comparable and easily identifiable data combining objective and subjective indicators. City managers, both who conduct and define the policies—policy-makers—or the urban designs—urban planners—should begin their strategies by evaluating the different technologies available to fit their requirements. Then, taking into account all technologies and systems that could be applied, they should be able to weigh them together with the same balancing mechanism, contrasting different options in a second analysis matrix to set a rating protocol for Smart City strategies as a whole. This new tool could be dubbed the Smart City matrix (SCM) and we are now working in its better definition in order to be presented in further articles in the near future (Figure 4). Regarding that, the last question of the survey was referred to the interest in having an overall assessment tool for Smart City strategies, on which 44% of respondents answered that they already have a system of self-assessment for their strategy, and only 50% reported knowing any kind of assessment tool for this issue. However, all survey respondents said that there should be some kind of assessment tool for Smart Cities, and 50% of them did not know if any tool was being developed in this regard. Ninety-four percent of RECI members answered that would also be interested in having a rating system for Smart Cities, in order to allow for city ranking and learn about different experiences and results, which seems adequate for 100% of respondents, and 89% believe that this tool can lead to an increased competitiveness among different cities, encouraging entrepreneurship for their citizens and companies who want to offer new technologies and systems to be applied in urban area developments or in regeneration processes.

After all this information was collected and processed, we defined 105 indicators to be deployed in the TAM and took into account the further users’ final interest in our ongoing SCM methodology, which were contrasted with all of the RECI members, presenting and discussing them in one of their annual meetings in Pamplona [22].

3. Results and Discussion

In this section, we present the case study of the different RECI members’ technologies and systems evaluation, especially taking into account the Pamplona Smart City strategy [23], after we signed a covenant between the UPNA and city council [24] for the data collecting and information sharing.

After the application of the TAM, a comparative graph of the results is presented (Figure 5), with a ranking from ‘high’ to ‘low’ for TTg scoring of all 56 evaluated technologies and systems.

After the first analysis, we are able to confirm that the methodology allows for an objective and comparable scoring, regardless of their nature and complexity, in the cumulative value for the technology (TT) and also in its weighting when applied in each SC area (TTm, TTe, TTq), as well as in an overall score for an SC strategy (TTg). The results showed a pattern of TT scores ranging between 3.00 and 4.00 (88%), which are relatively high scores and, perhaps, too homogeneous. This may be the result from the fact that all of the systems analyzed were from SC strategies already being implemented. However, by applying the (m, e, and q) correction coefficients, the TTg score provides more variability, even dropping below 1.00 in some examples (16% of evaluated systems), which demonstrates that this methodology could be useful for decision-making.

As has been verified, almost 20% of the implemented initiatives could have been saved because of their limited impact. On the other hand, the remaining systems can be considered as complementary, covering almost equally (around 30% each) the three fundamental aspects (m, e, q). However, it should be noted that some of the evaluated systems have obtained high marks in all three aspects, such as the case of the management of alternative transport systems through public bicycle rental and in an efficient cycle track network [25] (Figure 6).

Other analyzed technologies have reached good performance levels, especially on the issues for that which they have been designed and installed. That is the case of Pamplona’s car-sharing system [26]. It has good scoring on mobility and quality of life, but taking into account this technology still has some energy consumption (less than fossil energies, but it has some), its Smart City performance (SC score) is good, but not excellent (Figure 7). Additionally, regarding that Pamplona has installed only eight vehicles in a city with more than 200,000 journeys a day, this system is capable of covering just 0.003% of mobility needs.

Another example regarding mobility-focused technologies is a different and less complex system: a mobile application that aims the public to transport route information. As can be seen (Figure 8), it provides good results with respect to mobility aspects, but in regard that the use of a smartphone (with it consequent electric consumption) is mandatory, it does not have very good results on energy issues. At the same time, taking into account that not all of the citizens have access to smartphones (like children, elderly or handicapped people, poor citizens, etc.) its scoring in quality of life is limited to a C.

As can be seen in the next graphic (Figure 9), if the information is ordered regarding those technologies that better cover mobility issues (TTm scores), we find that just 30% of the implemented systems reach 2.5 points or more in mobility, even when their global TT scores are significantly better (3.5–4 points).

When the obtained results are ordered by energy efficiency scores (TTe), just 28% of the systems and technologies have scores better than 2.5 (Figure 10). That is probably due to the energy requirements that all new technologies need, especially those based on ICTs.

Finally, we analyzed the results by quality of life scoring (TTq), and those systems which reach 2.5 or more points represent the 38% of total evaluated technologies (Figure 11). This may indicate that the majority of implemented systems are focused on user needs more than any other aspect, with the consequential quality of life improvement. On the other hand, it is imperative to take into account that any strategy that considers mobility or energy efficiency amelioration will indirectly improve the citizen welfare.

4. Conclusions

Along our research we found that there are no specific instruments or tools for assessing Smart City strategies regarding new and evolving technologies in a simple way. The methodology presented attempts to provide an easy-to-use tool to bring urban developers and policy-makers closer to technology and give them an instrument that they can use to compare different alternatives in order to choose the best option. The use of this technology analysis matrix (TAM) could be an efficient method for evaluating the implementation of new technologies and systems in Smart Cities, as well as consider their interrelationship with other systems in a SC strategy. The assessment of systems and their impact on all issues that affect people urban life would arise as a helpful tool for urban planners and city managers, to be able to determine the most efficient and useful technologies that are truly important for building city intelligence. The TAM would also contribute to detecting what technologies would be unnecessary in Smart City policies, thus providing the city with important savings on investment in nonessential projects. Finally, it may also be useful to evaluate the investments in complementary technologies, taking into account a well-balanced SC strategy.

5. Materials and Methods

The methodology followed for this evaluation, and all data processing, was carried out during the corresponding thesis research work of the first author of this manuscript. It was presented in several publications [17,19,20] and conferences [18,27,28]. We are now testing it in several scenarios in order to improve their performance with the main goal of being useful for urban managers, designers, urban planners, and policy-makers. At the same time, we are working with different cities for the development of the Smart City matrix (SCM).