Business Models of Public Smart Services for Sustainable Development

Abstract

The smart city concept has entered the public debate over the last decade as a concept for the development of urban space for the efficiency, improvement and availability of public and private services and sustainability. The Business Models Canvas is most often used in the literature for the creation of business models of smart services. On the basis of the above, we investigated whether the Business Models Canvas is the most used tool for creating business models for public smart services in Slovakia and whether cities and municipalities need to evaluate their models for the provision of public smart services. However, there is no commonly used methodology for evaluating smart city business models to help both practitioners and researchers choose the best option. The goal of the research is to create a tool for evaluating business models of public smart services in smart cities. The base method used was the Delphi method, based on the previous primary (content) analysis process of the Business Model Canvas best practices. In total, 709 towns and villages participated in the primary research. Subsequently, the obtained data were evaluated and used for further research using the Delphi method, in which 28 experts participated. The research was carried out between 2020 and 2023 in Slovakia. Primary research confirmed that the Business Models Canvas is the most used tool for creating business models for public smart services in Slovakia and cities and municipalities need to evaluate their models for the provision of public smart services. Areas and basic building blocks were also identified for the design of the evaluation methodology of business models for public smart services. The proposal of the methodology for evaluating business smodels for public smart services in Slovakia was implemented using the Delphi method with the cooperation of 28 experts. Based on the results of the Delphi method, a methodological procedure for evaluating business models for public smart services was established. The methodology proposed in the paper is a simple, organized, flexible and transparent system that facilitates the work of evaluators of business models of public smart services and marketing.

1. Introduction

The most important subject in a smart city should be citizens. To implement a smart city initiative is not only to achieve technological success, but also to use technology to create public values and marketing [1,2,3,4]. Business models represent the planning and direction of business and marketing activities. They require connecting smart projects with a specific initiative, such as providing high-quality electronic services, achieving results that citizens consider desirable, and increasing trust in public institutions [3,4,5,6]. Social, technological, and economic changes, the demand of citizens for the modernization of services, the new development of ICT and the economic situation that requires more efficient public administration enabled local governments to adopt ICT for the provision of public services [7,8,9]. This represents a revival of the concept of smart city. A deeper analysis of specific services provided through smart technologies is needed. [7,10,11,12].

The cities need new business models because their role in the increasingly complex ecosystem of public services’ value creation is currently changing [13,14,15]. The increased complexity of the challenges facing cities and the plurality of stakeholders means that governments are resorting to new models of collaboration and co-production when designing and delivering services [4,15,16,17,18]. Municipal governments and providers of local services are therefore no longer the only parties responsible for delivering value to their residents but also become one of the several actors responsible for these key factors [15,18,19,20]. For municipal governments, this means that they operate in a service system where value creation depends on the involvement of a wide range of stakeholders [21,22]. In this system of services, the role of the city government is increasingly to guide and supervise the provision of services, and not just to create and provide services to passive consumers [5,6,22,23,24,25].

In the process of development towards smart cities, local governments are moving from traditional business models that are managed by the local government and its budget is formed by a third party (usually the taxpayer paying taxes) to models that strengthen the complex relationships between stakeholders and different financial systems [22,25,26,27,28]. It is possible to use business models such as direct public provision, public–private partnerships, public–private–academic partnerships, public–private–academic–civic partnerships, and urban/data communities, which differ in scope and participation of the parties involved [29,30,31]. The involvement of all interested parties creates the so-called Penta-helix model, which is increasingly used in smart cities [32,33,34,35]. The Penta Helix concept is an evolution of the Triple Helix concept. The innovative Penta-Helix model involves government, academia/research, industry, community and media as the five key stakeholder groups that need to cooperate to achieve this national goal [36,37,38].

These business models can support the continuous development of technologies and services in cities [14,39]. By using these key factors, it is possible to find several new opportunities that can create added economic and social values and significantly increase performance and competitive advantages for cities [40,41,42,43,44,45]. Business models help city governments to actively assess the net balance of their activities in terms of economic viability, environmental sustainability and social inclusion and acceptability, and to make changes if the results are not compatible with the city’s strategic goals [25,46,47,48,49]. The introduction of public smart services and their financial viability requires a proper understanding of the real business opportunities that can be created [48,50,51,52,53].

The scientific works and surveys provide a range of tools that help organizations to correctly formulate business models [30,54]. Business Model Canvas became an interesting tool for creating business models among authors [30,55]. This tool has also undergone various modifications to adapt to current trends [39,54,56]. For example, the Business Model Canvas For Mission-Driven Organizations was created, which is intended for organizations whose goal is not to make a profit [43,57,58,59]. Another modification is the Triple Layered Business Model Canvas, which, in addition to the economic layer, also contains a social and environmental layer [47,57,58,60]. Likewise, the Social Business Model Canvas complements the classic Business Model Canvas with two more building blocks focused on social–environmental impacts [38,51,52,56]. With the development of smart and sustainable cities, the idea of their application to create business models of public smart services began to spread in scientific circles [51,52,61,62]. However, through the performed analysis, we concluded that this approach is not correct, which was also confirmed by authors such as P. Giourka et al., K. Timeus et al. and N. Walravens and P. Ballon [30,39,51,52,57,62]. Therefore, they decided to create new tools for the creation of business models that would directly correspond to the criteria of business modeling of public services in smart cities [57,60,63]. These tools provide a holistic and integrated view of the business model in smart city conditions and direct it to sustainable value creation [17,60]. Their main purpose is to show how the city creates and maintains value for its citizens. The values have to be created by social well-being, environmental sustainability, and economic prosperity [60,61,63,64].

The authors P. Giourka et al., who proposed the Smart City Business Model Canvas, and the authors K. Timeus et al., who created the CMC, were again based on the original Business Model Canvas. Authors N. Walravens and P. Ballon came up with a completely new tool, which was called FPBM [35,42,56,58,60]. In this case, we see an advantage precisely with the link to the Business Model Canvas, which is already proven and recommended by the scientific community. From the mutual comparison of these tools, we concluded that the Smart City Business Model Canvas is a suitable and comprehensive tool for creating business models of public smart services [32,40,41,42,56,60,62,65]. This extended version of the Business Model Canvas also considers the important network characteristics of smart city business models. It adds building blocks that systematically assess the social and ecological impacts of the business model [40,41,42,62,66]. At the same time, several building blocks were adapted to include relevant aspects of smart city solutions in opposition to other standard business models.

Technological development and implementation of business models for the provision of smart services can help cities operate more efficiently and improve services for citizens and businesses. While the definition of a smart city and business models are still evolving, it is clear that smart cities use information and communication technologies to improve service levels, citizen well-being, sustainability and economic development. The growing development of technology can make cities more innovative and efficient in terms of development in the coming decades. The following

Table 1. Specific examples in the Smartcity concept.

Specific Examples of the Smart City Concept	Characteristics	Authors
Cities with more effective decision-making based on big data	Advances in big data and connected devices have given cities access to previously unavailable information. It enables them to access and analyze vast amounts of information and easily derive meaningful and actionable insights, especially in high-risk environments. In particular, big data and the Internet of Things offer endless possibilities to enable stronger decision-making, which improves the lives of residents by reducing costs and improving services.	[67,68,69]
Greater involvement of citizens and the government	Together with a bottom-up planning approach, these smart technologies help increase citizen engagement and trust in local government officials	[70]
Safer communities	Various security devices can also provide complete monitoring of the health and safety of users or residents of cities and towns.	[71,72]
Reduced environmental footprint	Various security devices can provide data to monitor air quality, ecological footprint, environment and identify the causes of pollution.	[63,73]
Improvement of transport	Connected transport systems are meant to improve efficiency across the city.	[74]
Greater digital equality	Smart city technology can create a fairer environment for citizens.	[75]
New opportunities for economic development	By providing an open data platform with access to city information, businesses can make informed decisions by analyzing data from integrated smart city technologies.	[75]
Effective public services	Smart technology networks enable two-way communication, help identify errors and increase efficiency.	[76,77]
Improvement of infrastructure	When correlated with advanced design, Building Information Modeling, and city business models, smart technology can provide cities with predictive analytics to identify areas for development and better use and manage resources from the perspective of citizens.	[78]
Increased workforce engagement	Implementing smart technologies helps reduce manual tasks.	[79]

shows specific examples of the Smartcity concept, at least in tabular form indicating the possible development of scenarios—change of models.

However, it is essential that the Smart City Business Model Canvas is not used by individual actors in the ecosystem solitary [43,54,62]. solutions for smart cities in co-operation with directly involved actors. In this iterative process, the various partners should first map their network and then articulate exactly how they create value for the other actors in the network and the end users of the integrated solution [35,42,57,58].

The tools for creating business models of public smart services have a dual purpose. They can be used as a tool for city governments to express their role in the smart city public service system and use it as a starting point for creating new services. In this sense, it is a descriptive tool [59,80,81]. Over time, the city’s business model can be used to analyze the development of various elements that are important for the development of a smart economy. This perspective recognizes that cities are a dynamic ecosystem whose key actors, political priorities and communities are constantly changing [30,31,57].

Business model innovation is considered a complex process that is generally characterized by some uncertainty regarding its decision-making and expected results, especially in the early stages of business model innovation [24,28,81]. To reduce this complexity and uncertainty, in addition to trial-and-error learning practices, the tools and techniques can be used by organisations aimed at analyzing and evaluating business models [28,57]. In response to this and to support decision-making, the research was devoted to the development of qualitative and quantitative tools and techniques for evaluating business models that generate an overview of the expected performance of business models [35,62,82].

Specifically, at least two tools were created to evaluate the business models of public smart services. The first is an evaluation tool by the authors N. Walravens and P. Ballon, which is based on the Framework of the Public Business Model. The second tool was designed by the authors R. Díaz-Díaz et al. under the name Business Model Evaluation Tool for Smart Cities. These tools in smart cities make it possible to compare different business models [30,35,40,41,42,57,62]. For example, it is possible to compare the business model of a public smart service to a public service model provided by traditional systems, as well as between different business models that could be potentially implemented using smart technologies. The goal is to assess the suitability of replacing the traditional business model of public services with a model of smart technologies [30,35,40,41,57]. However, these tools were not adopted in practice, and no scientific articles were concerned with these specific cases. These tools are critically evaluated because of the difficulty of their use [30,35,40,41,62].

A useful contribution could be the creation of an analytical and evaluation tool that would be comprehensible and easy to use, and at the same time is based on a tool for the creation of business models, which would be directly intended for public smart services. Therefore, the goal of the research is to create a tool for evaluating business models of public smart services in smart cities—proposal of the assessment methodology.

2. Materials and Methods

A content analysis of the available literature and professional articles on the topic of business models in smart cities identified a research gap was carried out, namely the absence of an evaluation tool for business models of public smart services. The main goal of our work was the creation of an evaluation tool for business models of public smart services in smart cities.

The Delphi method was used to create a tool for evaluating business models of public smart services. The use of this method primarily involves the participation of experts from practice, who express their suggestions, attitudes, and other comments in several rounds [83].

The experts who were approached for potential cooperation were selected based on the information obtained from them, their work, or their field of activity. They primarily represent experts from practice who deal with the issue of smart cities, especially the creation or evaluation of business models/plans/strategies/projects of public smart services. Important experts from three universities were involved in the research, the Ministry of the SR MIRRI (department of smart cities), smart service providers in Slovakia and the Czech Republic, innovation agencies that implement and ensure the proper operation of the provided smart services, and representatives of six smart cities who are responsible for creating/ evaluation of business models. A total amount of 28 experts participated in this research. The communication and cooperation took place via a form created in the Google Forms environment. The form was primarily used for the collection of proposed parameters for the assessment tool of business models of public smart services. The pretest carried out on a sample of five respondents helped to reveal incomprehensible and unclear parts of the form, which were subsequently removed.

The use of the Delphi method required the completion of three phases, namely the preparatory, implementation and final phase (Figure 1 [83]).

The use of the Delphi method required the completion of three phases, namely the preparatory, implementation and final phase [83].

The preparatory phase consisted mainly of identifying the research topic, analyzing the current state, finding out the need to solve the given issue, selecting and addressing experts and creating a tool through which the experts were to express their opinions and propose evaluation parameters.

The implementation phase initially represented the selection of building blocks for which evaluation parameters will be created. Out of the 2004 Slovak cities and municipalities approached, 709 respondents successfully filled out the questionnaire. The return rate of the questionnaire was at the level of 35.4%. Through the calculation of the selection sample, it was found that out of the total number of 2890 cities and towns of the Slovak Republic, with 95% reliability, it was necessary to obtain answers from at least 340 respondents. The necessary sample of 340 respondents was filled and at the same time, it was possible to fulfill the condition that at least 340 respondents provide at least one public smart service. The introductory part of the questionnaire consisted of identification questions. Respondents had at the beginning of the questionnaire, a portfolio of public smart services is also available, so that they have an idea of what kind of services they are. The core of the questionnaire consisted of questions focused on the types of smart services provided, the types of tools for creating a business model of public smart services used, the need for evaluating the business model of public smart services, the basic blocks of the business model of public smart services, etc. As part of the primary research, based on the literature review, the basic assumptions and research questions were investigated:

• The most frequently used tool for creating a business model of public smart services is the Business Model Canvas.
• Cities and municipalities need to evaluate the business models of public intelligent services.

And established hypothesis:

H1. 

The need to evaluate the business models of public intelligence services increases the use of the business model of public intelligence services, the Business Model Canvas.

As part of the research, research objectives and hypotheses were determined, which were tested using the Student’s test. The Student’s t-test is the most frequently used parametric test for testing hypotheses. This test evaluates the statistical significance of the tested difference between the mean values of the investigated parameters. The mutual relation among independent variables and individual responses was analysed by the correlation coefficient. It is used to measure the strength of a relationship between two variables. We interpret the values of the correlation coefficient according to Chráska [84]. He describes their dependence as follows: values from 0.9 to 1 (−0.9 to −1) are considered to be very highly dependent, so there is a very strong interdependence among the variables. Values from 0.7 to 0.9 (−0.7 to −0.9) are highly dependent, from 0.4 to 0.7 (−0.7 to 0.4) are moderately dependent, and from 0.2–0.4 (−0.4 to −0.2) are with low dependence. Values from the interval 0–0.2 (−0.2–0.0) are considered to be weak (without dependence). The closer the correlation coefficient is to 0, the weaker the relationship between the studied variables, or non-existent. Conversely, the closer it is to 1 or −1, the stronger the relationship between the variables [84].

In addition to the above, other data were also collected. Respondents were also asked to mark all the parameters and building blocks that they included in their business model of public smart services.

These data provided basic information for the further process of the Delphi method, where experts were involved and based on the results of primary research, they evaluated the individual identified building blocks and parameters. The involved experts should then, within the framework of cooperation, propose individual parameters that would sufficiently and comprehensively evaluate these building blocks. A form was sent to the experts with the request that they create criteria or parameters that, in their opinion and experience, would help to evaluate the created and used business models of public smart services. After the first round, all proposals were collected and incorporated into the initial proposal. Subsequently, the draft was sent to the experts so that they could comment and add comments or other proposals. If all respondents agreed, it would be possible to consider the evaluation tool as final. There was no consensus in the first round, so this procedure was repeated in four rounds. In the fourth round, the respondents agreed on the final form of the evaluation tool, together with the method of evaluation of individual parameters. After successful cooperation, a final tool for evaluating business models of public smart services was designed, which consists of 191 parameters (or a further 379 subparameters).

The final phase was represented by the achievement of a common consensus of all involved experts. The final draft of the assessment tool was created after completing four rounds, when the experts had no further comments and suggestions.

The resulting proposal was processed in the form of a table. The first column represents the created parameter, and the second column represents the evaluation method on the corresponding scoring scale. This form was chosen for its simplicity and clarity. Not all parameters are scored on the same scale. The point scale was also assessed by 28 experts.

3. Results

The primary research was conducted in 709 cities and municipalities in the Slovak Republic. On the basis of the primary research the research questions and hypotheses were established and further analyzed, see

Table 2. Statistical evaluation of primary (content) research.

Research Questions		Number of Positively Evaluated Answers	Correlation Coefficient
1	The most frequently used tool for creating a business model of public smart services is Business Model CANVAS	98	−0.8845
2	Cities and municipalities need to evaluate the created business models of public smart services	125	−0.8375
Hypothesis H1
The need to evaluate the business models of public intelligence services increases the use of the business model of public intelligence services—the Business Model Canvas
F-test		0.8511
F critical value		0.4160
Assessment of the normality of variables by F-test		The hypothesis of equality of variances is rejected
Hypothesis testing t-Test for difference of variances		The test for unequal variances is valid
Area minus		−1.9672
Area plus		1.9672
t Stat		−1.5562
Hypothesis testing t-test for equality of variances		The hypothesis of equality of variances is accepted
Pearson correlation coefficient		0.8375

, and the most frequently fulfilled areas and building blocks of business models of smart services provided in cities and municipalities in Slovakia were identified.

Research question 1 was confirmed, as up to 98 respondents identified the Business Canvas model as the most frequently used tool for creating a business model of public smart services in Slovakia (compared to Lean Canvas models 30 responses, BMCMDO—1 response, SBMC—34 responses, other frameworks 52 answers, the other respondents do not use any framework for the creation and evaluation of business models of public smart services). The most common reasons for negative evaluation in BMC, Lean Canvas, BMCMDO, TLBMC and SBMC include the insufficient number or focus of building blocks and the focus of the tool primarily on businesses (building blocks were not adapted to the conditions of the city/municipality). The strength of the relationship (correlation coefficient) is −0.88451, which is a strongly negative correlation. This means that the sum of the squares of the distances of the measured points from the regression line is the smallest possible and there is a strong dependence. Based on the data mentioned above, we accepted the research question Number 1.

The research question Number 2 was also accepted, because the cities and municipalities have a need to evaluate business models of public smart services in Slovakia (125 respondents answered yes, only 44 respondents answered no, and the other respondents took a neutral position. Neutral position—can assume that due to not using any framework for creating business models of public intelligent services). The strength of the relationship (correlation) is −0.83750 strong negative correlation). Based on the above, we accept research question 2, because the sum of the squares of the distances of the measured points from the regression line is the smallest distance and a strong negative dependence is proven here.

Based on the statistical evaluation of the established hypothesis—H1—there is a statistical dependence between the need for evaluation and the use of business models of public services—H1—The need for evaluation of business models of public intelligence services increases the use of the business model of public intelligence services—Business Model Canvas—this hypothesis is accepted. Although the hypothesis of equality of variance was not confirmed, the test of inequality of variances is valid and the evaluation t-test confirmed the dependence between the studied variables, as the t-Stat value (−1.5562) is located between the evaluation regions, which means that the t-Stat test statistic lies in the region of acceptance of the null hypothesis, which is the interval −1.9672 to 1.9672. The strength of the relationship between the investigated variables is strongly positive (0.8375).

On the basis of the above-mentioned data, we can conclude that the need to evaluate the business model of public services has a strong positive impact on the creation of business models in cities and towns. The respondents involved in the research further marked all areas and building blocks, which they included in their business model of public smart services. It was clear from the results that the most frequently filled areas, resp. building blocks that are part of the classic Canvas business model. On the contrary, building blocks such as data, deployment channels and environmental and social impacts, which should definitely be part of the business model of public smart services, were part of these business models to a much lesser extent. Most of the respondents said that they lacked the building blocks that would define the overall goal and would complement the building blocks with the environmental area, social area and sustainability. When marking the “other answer” option, the respondents most often commented, as in the previous case, that until the goal is set, it is not possible to create a business model correctly, but if cities/municipalities choose and fill in at least the necessary building blocks, it is a suitable auxiliary tool.

Based on the data from the primary survey, building blocks and their parameters were identified, which were subsequently used as input data for the implementation of the Delphi method for co-exploration with experts. In this part, the final draft of the assessment tool for business models of public smart services is presented and described in more detail, the parameters of which were designed primarily by experts dealing with the issue being addressed (Delphi method). The proposed business model of smart city services consists of 4 parts and 15 building blocks, see

Table 3. Evaluating business models of public smart services in smart cities.

Blocks	Number of Evaluation Parameters/Sub-Parameters	Description	Parts	Appendix A
Setting the mission (goal)	12 parameters/ 13 sub-parameters	Represents a clear definition of the goal public smart service.	Value proposition	Table A1
Value proposition	13 parameters 22 sub-parameters	Describes the specific needs of network recipients.		Table A2
Data	15 parameters 38 sub-parameters	Deals with the analysis of what data is necessary for the provision of the service.		Table A3
Network beneficiaries	5 parameters 14 sub-parameters	Defines who will directly benefit from the value proposition.	Providing value	Table A4
Deployment channels	14 parameters 35 sub-parameters	Solves the technological basis needed for the provision of public smart services.		Table A5
Key actors	9 parameters 26 sub-parameters	Identification of stakeholders who offer the best opportunities to access multiple resources necessary for the proper provision of public smart services.	Creating value	Table A6
Key activities	14 parameters 30 sub-parameters	Describes all activities that need to be implemented for the proper provision, operation, and maintenance of a public smart service.		Table A7
Relations with actors	9 parameters 16 sub-parameters	Describes the links between key actors for the provision of public smart services.		Table A8
Offer of key actors	15 parameters 22 sub-parameters	Describes what each actor can offer in the framework of joint creation.		Table A9
Key actor co-creation operation	13 parameters 25 sub-parameters	Describes what types of operations each key actor can potentially provide.		Table A10
Key resources and infrastructure	16 parameters 28 sub-parameters	Describes what resources are necessary for the proper provision of a public smart service.		Table A11
Budget costs	15 parameters 25 sub-parameters	The budget cost building block mainly refers to the economic costs that will be incurred during the implementation of public smart services.	Triple bottom line	Table A12
Revenue stream	13 parameters 27 sub-parameters	Describes all revenue streams associated with public smart services, including pricing strategies, user fees, revenue models, advertising, sponsorship, or other revenue generation mechanisms.		Table A13
Environmental impacts	14 parameters 29 sub-parameters	Mainly focused on comparing how the planned strategy will generate more environmental benefits than negative impacts.		Table A14
Social impacts	14 parameters 29 sub-parameters	It describes the impacts, both positive and negative, that a public smart service can have on residents and communities.		Table A15

, which supplement the appendices characterizing the individual parameters shown in Appendix A 

Table A1. Parameters of the assessment tool of the business model of public smart services for the building block—determining the mission (goal).

Parameters	Evaluation
1. Was a mission (goal) established in the BM, which outlines the overarching purpose, vision and direction of the service? 1	−1/+1
2. Is the stated mission (goal) consistent with the overall goals and priorities of the organization or entity responsible for providing the service, such as a city government or public agency? 3	−2/−1/0/+1/+2
3. Is the established mission (goal) specific, measurable, achievable, relevant, and time-bound and does it provide a clear framework to guide the development, implementation, and evaluation of the service? 3	−2/−1/0/+1/+2
4. To what extent does the stated mission (goal) express the needs and expectations of the service’s network recipients, as well as the broader social or community goals to which the service wants to contribute, as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
5. Does the stated mission (goal) provide a compelling and motivating vision that inspires and engages stakeholders, including staff, partners, users, and the wider community, towards achieving the service’s goals? 3	−2/−1/0/+1/+2
6. Is the established mission (goal) flexible and adaptable, allowing adjustments or fine-tuning over time based on changing needs, priorities or external factors while maintaining the overall strategic direction of the service? 3	−2/−1/0/+1/+2
7. To what extent was the set mission (goal) achieved? 4	−3/−2/−1/0/+1
8. Does the achievement of the stated mission (goal) of BM contribute to a sustainable, open and accessible city/municipality? 3	−2/−1/0/+1/+2
9. To what extent does the achievement of the stated mission (goal) of BM contribute to a city/municipality in which the knowledge-based economy and productive innovations prevail? 3	−2/−1/0/+1/+2
10. To what extent does achieving the set mission (goal) of BM contribute to a creative city/municipality? 3	−2/−1/0/+1/+2
11. To what extent does the achievement of the stated mission (goal) of BM contribute to a civically engaged and socially cohesive city/municipality? 3	−2/−1/0/+1/+2
12a. Were the challenges, obstacles or gaps that may arise in the fulfillment of the set mission (goal) identified in the BM, and ways of their appropriate solution or overcoming proposed? 1	−1/+1
12b. Are there (have arisen) any challenges, obstacles, or gaps in the alignment of the mission (goal) and operational activities or results, for which no suitable solution has been proposed and the provided service cannot (couldn’t) overcome them? 3	−2/−1/0/+1/+2

Note: 1, 2, 3, 4: The index indicate the used method evaluating individual parameters, Table 4.

,

Table A2. Parameters of the assessment tool of the business model of public smart services for the building block—value proposition.

Parameters	Evaluation
1a. Has the value proposition that the service can provide been identified in BM? 1	−1/+1
1b. Is there any evidence or data, such as case studies, successful applications, or testimonials from satisfied users, to support the selection of this offer? 1	−1/+1
2a. Were requests for the service ascertained directly from the recipients? 1	−1/+1
2b. Were these requirements included and considered in BM? 1	−1/+1
2c. To what extent does the service provided meet the needs of the beneficiaries, as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
3a. Were the requirements of the city/municipality for the service identified, included, and taken into account in the BM? 1	−1/+1
3b. To what extent does the provided service meet the requirements of the city/municipality? 4	−3/−2/−1/0/+1
4a. Was it ascertained directly from the recipients what kind of profit they are interested in from the delivered value? 1	−1/+1
4b. Was this profit included and considered in BM? 1	−1/+1
4c. To what extent does the service provided deliver the desired profit as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
5a. Was the recipient directly asked what needs/problems the delivered value can fulfill/solve? 1	−1/+1
5b. Were these needs/problems included and considered in the BM? 1	−1/+1
5c. To what extent did the provided service fulfill/solve all identified needs/problems as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
6. Does the service’s value proposition differ from competing solutions or alternatives in terms of its features, functionality, performance, cost-effectiveness, or other key attributes? 6	−2/−1/0/+1/+2
7. By how much % is the service provided less/more used than current alternatives? 7	−4/−3/−2/−1/0/+1/+2/+3/+4
8. How many % of network recipients are interested in the provided service? 8	−2/−1/+1/+2
9. Does the service improve the quality of life of citizens? 3	−2/−1/0/+1/+2
10. Does the service effectively use its value proposition to attract and retain users, generate revenue, secure funding, or achieve other goals? 3	−2/−1/0/+1/+2
11a. Were identified (evaluation) threshold values in BM, or KPI for the service provided? 1	−1/+1
11b. If so, to what extent were the (evaluation) threshold worthy, or KPI achieved during retrospective evaluation? 4	−3/−2/−1/0/+1
12. Have opportunities or strategies been identified in the BM that the service can explore to improve its value proposition, such as introducing new features, improving the user experience, expanding the user base or exploring new markets? 1	−1/+1
13. Is the value proposition continuously adapted and refined based on changing user needs, market dynamics or technological advances to remain relevant and competitive in the rapidly evolving smart city environment? 3	−2/−1/0/+1/+2

,

Table A3. Parameters of the evaluation tool of the business model of public smart services for the building block—data.

Parameters	Evaluation
1a. Was the data necessary for the provision of the service identified in BM? 1	−1/+1
1b. Were additional data needed in the implementation phase that were not identified in BM? 2	−1/+1
1c. What % of the necessary data was (is) available during the implementation and provision of the service? 4	−3/−2/−1/0/+1
2a. Has the data to be collected through the provided service been identified in BM? 1	−1/+1
2c. What % of the necessary data is collected? 4	−3/−2/−1/0/+1
2d. What % of collected data is necessary? 4	−3/−2/−1/0/+1
2e. Do the obtained data have a future processor (so that they are not useless)? 3	−2/−1/0/+1/+2
2 f. Is the quality of the collected data sufficient (for example for further analysis)? 3	−2/−1/0/+1/+2
3. Has (Is) a robust data acquisition and integration strategy in place that ensures that relevant data from a variety of sources, both internal and external, can be accessed and used to support its operations, decision making and value creation? 1	−1/+1
4. Has (Is) a clear data governance framework in place that outlines policies, procedures, and practices for managing data throughout its lifecycle, from collection to storage, processing, analysis and sharing? 1	−1/+1
5a. Has the purpose of using the obtained data been determined in BM (to generate actionable knowledge, provide information for decision-making, optimize operations, and create value for its users, stakeholders and the wider community)? 1	−1/+1
5b. What % of the obtained data is used for the specified purpose? 4	−3/−2/−1/0/+1
6a. Was it determined in BM how and where the obtained data will be stored? 1	−1/+1
6b. In the implementation phase, it was necessary to use a different method and a different location of data storage, which was not specified in BM? 2	−1/+1
6c. What % of data is stored in the most appropriate way and in the most appropriate place (no more appropriate alternative is available)? 4	−3/−2/−1/0/+1
7a. Was it determined in the BM how the obtained data will be further processed? 1	−1/+1
7b. In the implementation phase, was it necessary to choose another/another method of processing that was not mentioned in BM? 2	−1/+1
7c. What % of data is processed in the most suitable way (no more suitable alternative is available)? 4	−3/−2/−1/0/+1
8a. Have appropriate security, data protection and data breach response measures, including access control, encryption, data backup, disaster recovery and incident response plans, been chosen at BM to mitigate the risks associated with data breaches or other security incidents? 1	−1/+1
8b. In the implementation phase, it was necessary to use other methods of security and data protection that were not chosen in BM? 2	−1/+1
8c. What % of data is secured in the most suitable way (no more suitable alternative is available)? 4	−3/−2/−1/0/+1
9a. Was it determined in the BM how and to whom the data will be made available? 1	−1/+1
9b. Was it necessary to change the method of making data available in the implementation phase? 2	−1/+1
9c. What % of data is made available in the most appropriate way and only to relevant persons (no more appropriate alternative is available)? 4	−3/−2/−1/0/+1
9d. Has data been stolen or misused? 2	−1/+1
10a. Was it specified in BM how the data should be anonymized? 5	−1/0/+1
10b. Was it necessary to change the method of data anonymization in the implementation phase? 6	−1/0/+1
10c. What % of data is anonymized in the most suitable way (no more suitable alternative is available)? 4	−3/−2/−1/0/+1
11a. Have mechanisms been identified in BM to ensure the ethical use of data, including addressing issues such as bias, fairness, transparency, and accountability? 1	−1/+1
11b. Was it necessary to change the mechanisms for ensuring the ethical use of data in the implementation phase? 2	−1/+1
11c. To what extent are ethical principles observed in procedures related to data? 4	−3/−2/−1/0/+1
12a. Has the BM established a method for sharing data and collaborating with stakeholders such as other public agencies, private companies, academic institutions, or community organizations? 1	−1/+1
12b. Was it necessary to change the method of data sharing and cooperation with other interested parties in the implementation phase? 2	−1/+1
12c. What % of data is shared in a cooperative and responsible approach? 4	−3/−2/−1/0/+1
13a. Were identified (evaluation) threshold values in BM, or KPI for data? 1	−1/+1
13b. If so, to what extent were the (evaluation) threshold worthy, or KPI achieved during retrospective evaluation? 4	−3/−2/−1/0/+1
14. Were cross-sectional data also used in the processing of BM, if available? 5	−1/0/+1
15. To what extent does the service comply with relevant data management, privacy, security and ethics laws, regulations and standards and demonstrate a commitment to responsible data practices? 4	−3/−2/−1/0/+1

,

Table A4. Parameters of the evaluation tool of the business model of public smart services for the building block—network beneficiaries.

Parameters	Evaluation
1a. Were the network recipients of the service identified in the BM, considering their demographics, characteristics, needs and preferences? 1	−1/+1
1b. Were network recipients changed in the implementation phase, compared to those identified in BM? 2	−1/+1
1c. For what % of the required beneficiaries is the service provided? 2	−3/−2/−1/0/+1
1d. Were network recipients appropriately segmented, if segmentation was necessary? 3	−2/1/0/+1/+2
1e. To what extent have the offer, features and communication been appropriately tailored to the needs, preferences, behavior of network recipient segments as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
1f. Would it be more appropriate to provide the service to other/additional network recipients? 2	−1/+1
2a. Has a feedback method/mechanism or user engagement strategy been established in the BM to continuously collect and incorporate user feedback into the design, development, and improvement processes of the service? 1	−1/+1
2b. Was it necessary to change/supplement the method/mechanism of feedback or the strategy of user involvement in the implementation phase, compared to those set out in the BM? 2	−1/+1
2c. Is the satisfaction, engagement and feedback of network beneficiaries regularly evaluated and assessed through surveys, interviews, focus groups or other means? 3	−2/1/0/+1/+2
2d. Is the feedback obtained used to increase the value proposition for network recipients? 3	−2/1/0/+1/+2
3. What is the overall satisfaction level, adoption rate or user perception of the service in your target group as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
4. What % of network recipients benefit from the service as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
5a. Were possible barriers, challenges, or obstacles (language barriers, digital literacy, accessibility) identified in the BM, which may arise during interactions of the service with its target group, and ways of their appropriate solution or overcoming proposed? 1	−1/0/+1
5b. Are there (have there been) any barriers, challenges, or obstacles in the interactions of the service with its target group, for which a suitable solution has not been proposed and the provided service cannot (couldn’t) overcome them? 2	−2/1/0/+1/+2

,

Table A5. Parameters of the assessment tool of the business model of public smart services for the building block—deployment channels.

Parameters	Evaluation
1a. Have the necessary deployment channels been identified in BM? 1	−1/+1
1b. It was necessary to use other deployment channels in the implementation phase that were not identified in BM? 2	−1/+1
1c. What % of the necessary deployment channels were (are) used? 4	−3/−2/−1/0/+1
1d. What % of deployment channels used were (are) necessary? 4	−3/−2/−1/0/+1
1e. Were the deployment channels correctly chosen (or would it be better to replace some)? 3	−2/1/0/+1/+2
2a. Were the beneficiaries’ preferences, possibilities and needs for service accessibility ascertained directly? 1	−1/+1
2b. Have these preferences, behaviors and accessibility needs been included and considered in BM? 1	−1/+1
2c. How well do these channels align with the preferences, behaviors and accessibility needs of network recipients as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
3a. Has the method/strategy of addressing and involving the intended beneficiaries of the service been determined in the BM? 1	−1/+1
3b. In the implementation phase, was it necessary to choose a different method/strategy of addressing and involving the intended recipients of the service, compared to those set out in the BM? 2	−1/+1
3b. Has the reach, adoption and use of the service been optimized through reaching and engaging the intended recipients of the service? 3	−2/1/0/+1/+2
4a. Has a multi-channel approach to deployment been designed in BM that considers the diverse needs, preferences, or limitations of network recipients? 1	−1/+1
4b. Was it necessary to change/use a different approach to deployment in the implementation phase, compared to the one specified in BM? 2	−1/+1
4c. Is the service provided through multiple channels if necessary? 5	−1/0/+1
4d. To what extent does the service tailor its offerings, features, or functions appropriately for different deployment channels to ensure consistent user experience and satisfaction, as evidenced by user feedback, market research, or satisfaction metrics? 4	−3/−2/−1/0/+1
5a. Has a method/mechanism been established in BM to collect feedback, insights, or analysis from various deployment channels such as user feedback, usage data or performance metrics? 1	−1/+1
5b. In the implementation phase, was it necessary to change/supplement the method/mechanism of feedback, overviews or analyses from various channels of deployment, compared to those stipulated in the BM? 2	−1/+1
5c. Performance, efficiency, etc. are regularly assessed. deployment channels based on feedback, reports, analyses? 3	−2/1/0/+1/+2
5d. Is the feedback obtained used to improve decision-making processes and to ensure that channels operate in accordance with the evolving needs, preferences, and expectations of network recipients? 3	−2/1/0/+1/+2
6. What % of network recipients access the service through the chosen deployment channels? 4	−3/−2/−1/0/+1
7. What is the level of user satisfaction, adoption, or engagement with the service across various deployment channels as evidenced by user feedback, market research, or satisfaction metrics? 4	−3/−2/−1/0/+1
8a. Have measures been established in the BM to ensure the security, privacy, and integrity of the offering across different deployment channels, including encryption, access control, authentication, and other best practices? 1	−1/+1
8b. In the implementation phase, was it necessary to change the method of securing, protecting privacy and integrity of the offer in individual channels of deployment, compared to those stipulated in BM? 2	−1/+1
8c. To what extent is the security, privacy and integrity of the offering ensured across different deployment channels? 4	−3/−2/−1/0/+1
9. Were the deployment channels used the most cost-effective option (no better alternative available)? 3	−2/1/0/+1/+2
10a. Has the BM established a way to maintain, monitor and support the offering across various deployment channels, including resolving issues such as software updates, hardware maintenance, user support or service interruptions? 1	−1/+1
10b. In the implementation phase, was it necessary to change the way of maintenance, monitoring and support of the offer across different deployment channels, compared to those stipulated in BM? 2	−1/+1
10c. To what extent is service continuity, reliability and availability ensured through deployment channels? 4	−3/−2/−1/0/+1
11a. Has the BM established a way to manage the integration and interoperability of the offering across different deployment channels, such as ensuring seamless data exchange, user authentication or service continuity? 1	−1/+1
11b. Was it necessary to change/use a different way of managing the integration and interoperability of the offer across different deployment channels in the implementation phase, compared to the one set in the BM? 2	−1/+1
11c. To what extent is seamless data exchange, user authentication and service continuity ensured? 4	−3/−2/−1/0/+1
12a. Have challenges, obstacles or problems that may arise in relation to deployment channels been identified in the BM and ways to appropriately address or overcome them proposed? 1	−1/+1
12b. Are there (have) been any challenges, obstacles or issues related to deployment channels for which a suitable solution has not been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1
13. To what extent does the service comply with relevant laws, regulations and standards relating to security, data protection and integrity of the offer across deployment channels? 4	−3/−2/−1/0/+1
14. To what extent do deployment channels contribute to achieving goals such as improving service delivery, user experience or zaint engagement heresed parties? 4	−3/−2/−1/0/+1

,

Table A6. Parameters of the assessment tool of the business model of public smart services for the building block—key actors.

Parameters	Evaluation
1a. Have key actors been identified in BM? 1	−1/+1
1b. Was it necessary to involve other/additional actors in the implementation phase who were not identified in the BM? 2	−1/+1
1c. What % of necessary actors are involved in providing the service? 4	−3/−2/−1/0/+1
1d. What % of involved actors is necessary? 4	−3/−2/−1/0/+1
1e. How many % of actors are correctly chosen (or would it be better to replace some)? 4	−3/−2/−1/0/+1
2. What share of actors brought the expected benefit/result? 4	−3/−2/−1/0/+1
3a. Has the BM established a way to identify, engage and collaborate with key stakeholders, including establishing clear lines of communication, defining roles and responsibilities, and managing expectations to ensure effective coordination, alignment and mutual understanding of service objectives and benefits? 1	−1/+1
3b. Was it necessary to change the method of identification, involvement, and cooperation with key actors in the implementation phase, compared to the one stipulated in the BM? 2	−1/+1
3c. Is effective coordination, alignment and mutual understanding of the goals and benefits of the service with key actors ensured? 3	−2/1/0/+1/+2
4a. Has the BM established a method/mechanism for building and maintaining positive relationships with key actors, such as partnerships, agreements, or governance structures? 1	−1/+1
4b. In the implementation phase, was it necessary to change/use a different method/mechanism for building and maintaining positive relations with key actors, compared to the one stipulated in the BM? 2	−1/+1
4c. Are relationships, dynamics or effectiveness of key actors regularly evaluated? 3	−2/1/0/+1/+2
4d. Are the results used to inform strategy, action or policies related to stakeholder engagement, collaboration, or management to ensure that key stakeholders are willing to work together on an ongoing basis in line with the aims and objectives of the service? 3	−2/1/0/+1/+2
5a. Was the method of involving key actors in decision-making processes related to the design, development, implementation, or evaluation of the service determined in the BM? 1	−1/+1
5b. Was it necessary to change/use a different method of involving key actors in decision-making processes in the implementation phase, compared to the one stipulated in the BM? 2	−1/+1
5c. To what extent are diverse perspectives, needs or interests of actors considered and integrated into the service? 4	−3/−2/−1/0/+1
6a. Has a method/mechanism been established in the BM for collecting feedback on satisfaction, engagement, or trust among key actors in relation to the service to increase their involvement, support and commitment to the provision of the service? 1	−1/+1
6b. In the implementation phase, was it necessary to change/add to the method/mechanism of obtaining feedback, compared to the one stipulated in the BM? 2	−1/+1
6c. Is the satisfaction, engagement, trust and perception of actors regularly assessed, based on feedback? 3	−2/1/0/+1/+2
6d. Is the feedback obtained used to improve decision-making processes and to ensure that the involvement, support, and commitment of key actors in the delivery of the service is increased? 3	−2/1/0/+1/+2
7. What is the level of satisfaction, engagement, or trust among key stakeholders in relation to the service, as evidenced by stakeholder feedback or satisfaction metrics? 4	−3/−2/−1/0/+1
8a. Has the BM established a way to manage any conflicts of interest, differing expectations or challenges that may arise between key actors? 1	−1/+1
8b. In the implementation phase, was it necessary to change/use a different method of resolving conflicts, different expectations, and challenges, compared to the ones stipulated in the BM? 2	−1/+1
8c. Does it facilitate constructive dialogues, negotiations, or solutions to ensure smooth functioning and satisfaction of stakeholders? 3	−2/1/0/+1/+2
9a. Have risks, dependencies or uncertainties associated with key factors that may arise in connection with changes in policies, regulations, financing, or market conditions been identified in the BM, and ways of appropriately addressing or overcoming them suggested? 1	−1/+1
9b. Are there (have arisen) any risks, dependencies, or uncertainties with the provision of permanent support from actors for which a suitable solution has not been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1

,

Table A7. Parameters of the evaluation tool of the business model of public smart services for the building block—key activities.

Parameters	Evaluation
1a. Were the necessary key activities identified in BM? 1	−1/+1
1b. Has continuous technological progress, best practices or innovative approaches been considered when planning the necessary activities? 1	−1/+1
1c. Are key activities flexible and adaptable, allowing adjustments or fine-tuning to improve service delivery, scalability or sustainability over time based on changing needs, priorities or external factors while maintaining the overall strategic direction of the service? 3	−2/1/0/+1/+2
1d. Are key activities adapting to the changing needs, expectations or demands of users, stakeholders or market dynamics as evidenced by user feedback, market research or satisfaction metrics? 3	−2/1/0/+1/+2
1e. Was it necessary to carry out other/additional activities in the implementation phase that were not identified in the BM? 2	−1/+1
1f. What % of the necessary activities are carried out? 4	−3/−2/−1/0/+1
1g. What % of the performed activities is necessary? 4	−3/−2/−1/0/+1
2. What proportion of the activities brought the expected benefit/result? 4	−3/−2/−1/0/+1
3. Are key activities effective in terms of resource utilization, process optimization and service quality? 3	−2/1/0/+1/+2
4. Do the key activities contribute to achieving the set goals, impacts, benefits, solving challenges or supporting innovation and cooperation? 3	−2/1/0/+1/+2
5a. Were the tasks, responsibilities and communication channels defined in the BM, which should ensure the smooth execution of activities? 1	−1/+1
5b. In the implementation phase, was it necessary to modify/supplement the tasks, responsibilities, and communication channels for carrying out activities, compared to those stipulated in the BM? 2	−1/+1
5c. To what extent is the execution of activities smooth in terms of defined tasks, responsibilities and chosen communication channels? 4	−3/−2/−1/0/+1
6. What proportion of activities are carried out in accordance with the planned and set conditions (adherence to the set time schedule, delivery of the set result, observance of the set costs, etc.)? 4	−3/−2/−1/0/+1
7. Did some activities have a negative impact on the result (delay, delivery of less benefit, increase in costs, etc.)? 2	−1/+1
8a. Did any of the activities have negative externalities? 2	−1/+1
8b. If so, were they identified in the BM as potentially possible and thus a solution was proposed for them? 1	−1/+1
8c. If so, were they compensated in any way? 1	−1/+1
9a. Has BM established a way to ensure user satisfaction and involvement in key activities such as user onboarding, training, or support? 1	−1/+1
9b. Was it necessary to change/use a different method of ensuring user satisfaction and involvement in key activities in the implementation phase, compared to the one specified in the BM? 2	−1/+1
9c. Is there regular and active involvement of network recipients in key activities? 3	−2/1/0/+1/+2
10. What is the level of user satisfaction, engagement or involvement with key activities as evidenced by stakeholder feedback or satisfaction metrics? 4	−3/−2/−1/0/+1
11a. Has the BM established a method/mechanism for obtaining feedback, insights or input from users and key actors to inform improvements, innovations, or modification of activities? 1	−1/+1
11b. Was it necessary to change/use a different method/mechanism for obtaining feedback, insights or input from users and key actors in the implementation phase, compared to the one established in the BM? 2	−1/+1
11c. Is the satisfaction or perception of users and actors regularly assessed, based on feedback? 3	−2/1/0/+1/+2
12. What is the level of satisfaction or perception among users and key actors in relation to key activities as evidenced by feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
13a. Have key performance indicators (KPIs) or metrics used to assess the performance, effectiveness or efficiency of key activities been established in the BM to ensure they remain relevant, effective, and efficient in delivering the intended value proposition and achieving the desired service outcomes? 1	−1/+1
13b. If so, to what extent were the (evaluation) threshold worthy, or KPI achieved during retrospective evaluation? 4	−3/−2/−1/0/+1
14a. Have the risks and challenges that may arise in relation to key activities, such as data privacy, security, interoperability, or regulatory compliance, been identified in the BM and ways to appropriately address or overcome them proposed? 1	−1/+1
14b. Are there (Have there been) any risks and challenges in connection with key activities for which a suitable solution has not been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1

,

Table A8. Parameters of the evaluation tool of the business model of public smart services for the building block—relations with actors.

Parameters	Evaluation
1a. Were the necessary relationships with actors identified in the business model? 1	−1/+1
1b. Was it necessary in the implementation phase to modify or supplement relationships that were not identified in BM? 2	−1/+1
2. Are the relationships between the actor’s collaborative, cooperative and mutually beneficial (use of resources, expertise and skills) in achieving the desired results? 3	−2/1/0/+1/+2
3. What proportion of established relationships brought the expected benefit/result? 4	−3/−2/−1/0/+1
4a. Has a method/mechanism been established in BM for collecting feedback regarding satisfaction, engagement of actors within the framework of mutual relations and cooperation? 1	−1/+1
4b. Was it necessary to change/use a different method/mechanism for gathering feedback from key actors in the implementation phase? 2	−1/+1
4c. Is the satisfaction, engagement and perception of actors regularly assessed, based on feedback? 3	−2/1/0/+1/+2
4d. Is the feedback obtained used to improve decision-making processes and to ensure that improvement, development, innovation support or co-creation opportunities occur? 3	−2/1/0/+1/+2
5. How satisfied are stakeholders with the level of engagement, responsiveness and support they receive in addressing their needs, concerns and expectations as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
6. Are actors involved in decision-making processes, designing, and planning in shaping the direction and priorities of services? 3	−2/1/0/+1/+2
7. Do relationships with stakeholders affect the overall performance, sustainability, and success of the service in achieving its intended goals and creating value for the community as a whole? 3	−2/1/0/+1/+2
8a. Has the BM established a way of managing relations with actors in terms of building trust and maintaining transparency to ensure a supportive and collaborative environment? 1	−1/+1
8b. Was it necessary to change/use a different method of managing relations with actors in the implementation phase, compared to the one stipulated in the BM? 2	−1/+1
8c. Does effective and timely management of relations with actors take place? 3	−2/1/0/+1/+2
9a. Were the risks and barriers that may arise in relations with actors, such as conflicting interests, power dynamics or differences in values and goals, identified in the BM, and ways of their appropriate solution or overcoming proposed? 1	−1/+1
9b. Are there (have) there been any risks and barriers in relations with actors for which a suitable solution has not been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1

,

Table A9. Parameters of the evaluation tool of the business model of public smart services for the building block—offer of key actors.

Parameters	Evaluation
1a. Was the supply of key actors identified in BM? 1	−1/+1
1b. Was the offer of individual key actors correctly identified (did they really have the offer that was set in the BM)? 3	−2/1/0/+1/+2
2. Does the key players’ offering support the sustainability or long-term viability of the service, for example through partnerships, revenue-sharing models or other collaborative arrangements? 3	−2/1/0/+1/+2
3. Does the actors’ offer match the needs, requirements and expectations of the beneficiaries, stakeholders, or the service, as evidenced by user feedback, market research or satisfaction metrics? 3	−2/1/0/+1/+2
4. Is the actors’ offer comprehensive, innovative, or differentiated in terms of features, functionalities, capabilities or added value? 3	−2/1/0/+1/+2
5. Is the key players’ offering reliable, scalable, and sustainable in terms of technical performance, data quality, service availability and support, as evidenced by various metrics, analytics, etc.? 3	−2/1/0/+1/+2
6. Is the offer of key players cost-effective, transparent, and competitive in terms of prices, licensing or other business terms? 3	−2/1/0/+1/+2
7. Does the key players’ offering provide better value for money compared to the benefits or outcomes of alternative solutions and services? 1	−1/+1
8a. Was it determined in the BM how the offer of key actors will be used? 1	−1/+1
8b. In the implementation phase, was it necessary to change/use a different way of using the offer of key actors, compared to the one stipulated in the BM? 2	−1/+1
8c. To what extent are the actual results of using the key actors’ offer the same as planned? 4	−3/−2/−1/0/+1
9a. Has a feedback method/mechanism, user involvement strategy or collaborative development been established in the BM so that the offer of key actors leads to continuous improvement, development, innovation support or co-creation opportunities? 1	−1/+1
9b. Was it necessary to change/use a different method of feedback or user involvement in the implementation phase, compared to the one specified in the BM? 2	−1/+1
9c. Is the feedback of actors and network beneficiaries regularly evaluated and assessed through surveys, interviews, etc.? 3	−2/1/0/+1/+2
9d. Is the feedback obtained used to improve decision-making processes and to ensure that the service evolves, grows, and adapts to changing needs and opportunities? 3	−2/1/0/+1/+2
10. To what extent does the key players’ offering meet expectations and ensure service satisfaction as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
11. Is the offer of key actors flexible and adaptable to meet the unique needs or preferences of the service and its users? 3	−2/1/0/+1/+2
12. Are the key players offering flexibility, adaptability for modifications, integrations, or extensions to meet changing requirements or future developments? 3	−2/1/0/+1/+2
13a. Have the risks, challenges and limitations that may arise in connection with the offer of key actors, such as data privacy, security, interoperability, or compliance with regulations, been identified in the BM, and ways to appropriately address or overcome them proposed? 1	−1/+1
13b. Are there (have) been any risks, challenges, and limitations in relation to the offer of key actors for which no suitable solution has been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1
14. Does the offer of key actors contribute to achieving the overall goals, impacts and benefits of the service? 3	−2/1/0/+1/+2
15. Is the offer of key actors in line with the strategic goals, mission, and values of the service? 3	−2/1/0/+1/+2

,

Table A10. Parameters of the evaluation tool of the business model of public smart services for the building block—co-creation operations of key actors.

Parameters	Evaluation
1a. Have operations been identified in BM that key actors can provide? 1	−1/+1
1b. Was it necessary in the implementation phase to add an actor who would implement the necessary operations? 2	−1/+1
1c. Do key actors contribute expertise or resources to service co-creation operations? 3	−2/1/0/+1/+2
2. Are key actors involved in service co-creation operations such as joint planning, design, development, implementation, or evaluation of the service? 3	−2/1/0/+1/+2
3. Are co-creation operations inclusive, diverse, or representative in terms of involving key actors from different stakeholder groups such as citizens, businesses, academia, government, or civil society? 3	−2/1/0/+1/+2
4. To what extent are co-creation operations transparent, communicative, or participatory in providing timely, accurate or accessible information to key actors as evidenced by key actor feedback? 4	−3/−2/−1/0/+1
5a. Was the method of measurement, monitoring and evaluation of the achieved results of co-creation established in the BM? 1	−1/+1
5b. In the implementation phase, was it necessary to change/use a different established method of measuring, monitoring, and evaluating the achieved results of co-creation, compared to the one established in the BM? 2	−1/+1
5c. Are the results of co-creation regularly measured, monitored, and evaluated? 3	−2/1/0/+1/+2
5d. Are the obtained results used to improve decision-making processes and to ensure the correct implementation of changes? 3	−2/1/0/+1/+2
6a. Have incentives, motivations or rewards been identified in the BM to be used to encourage key actors in terms of participation, contribution, or commitment to co-creation? 1	−1/+1
6b. Was it necessary to change/use other incentives, motivations, or rewards for key actors in the implementation phase, compared to those set out in the BM? 2	−1/+1
6c. Are key actors actively involved in co-creation operations, and their sufficient contribution and commitment to co-creation can be monitored? 3	−2/1/0/+1/+2
7. Did the implemented co-creation operations help to identify aspects that could not be identified otherwise? 5	−1/0/+1
8a. Have collaborative mechanisms and tools or processes been established in the BM to facilitate co-creation operations between key actors, such as co-design workshops, innovation labs, hackathons, or stakeholder engagement initiatives? 1	−1/+1
8b. In the implementation phase, was it necessary to change/use other cooperation mechanisms and tools or processes to facilitate co-creation operations, compared to those stipulated in the BM? 2	−1/+1
8c. Are various workshops, hackathons and other stakeholder engagement initiatives held regularly? 3	−2/1/0/+1/+2
9a. Has a method/mechanism been established in the BM to collect, analyze, or incorporate feedback from key actors to continuously improve, develop and support co-creation operations? 1	−1/+1
9b. Was it necessary to change/use a different method/mechanism for collecting, analyzing, or incorporating feedback from key actors in the implementation phase, compared to the one set out in the BM? 2	−1/+1
9c. Is the feedback of key actors regularly evaluated and assessed through surveys, interviews, etc.? 3	−2/1/0/+1/+2
9d. Is the obtained feedback used to improve management, coordination or facilitate co-creation operations? 3	−2/1/0/+1/+2
10. Are co-creation operations flexible and adaptable, allowing for modification, development or fine-tuning over time-based on changing needs, technologies or stakeholder dynamics while maintaining the overall strategic direction of the service? 3	−2/1/0/+1/+2
11. Do co-creation operations contribute to building long-term relationships, partnerships or networks that can increase the sustainability, efficiency, or scalability of the service? 3	−2/1/0/+1/+2
12. Do co-creation operations lead to tangible results such as improved service quality, increased user satisfaction, efficiency, or innovation? 3	−2/1/0/+1/+2
13. Are the co-creation operations carried out in accordance with the overall goals, objectives and vision of the service? 3	−2/1/0/+1/+2

,

Table A11. Parameters of the assessment tool of the business model of public smart services for the building block—key resources and infrastructure.

Parameters	Evaluation
1a. Were the key resources and infrastructure necessary for the implementation and provision of the service identified in BM? 1	−1/+1
1b. Were other key resources and infrastructure needed in the implementation phase that were not identified in BM? 2	−1/+1
1c. What % of the necessary resources and infrastructure were (are) available? 4	−3/−2/−1/0/+1
1d. If all the necessary resources and infrastructure were not available, was the reason due to a factor that could not be controlled (unexpected inflation, lengthy VO, insolvency of the supplier, etc.)? 8	−1/0
1e. If it was a factor that cannot be influenced, was it identified as potentially possible in the BM and was a method of eliminating negative impacts proposed? 1	−1/+1
2. Are key resources and infrastructure reliable, secure, and resilient in terms of minimizing downtime, protecting against cyber security threats, ensuring data integrity, or maintaining service continuity? 3	−2/1/0/+1/+2
3a. Was the method of use of key resources and infrastructure determined in BM? 1	−1/+1
3b. Was it necessary to change/use a different way of using key resources and infrastructure in the implementation phase? 2	−1/+1
3c. What % of key resources and infrastructure are used in the most appropriate way (no more appropriate alternative available)? 4	−3/−2/−1/0/+1
4. Were (are) key resources and infrastructure fully secured during the entire implementation period? 3	−2/1/0/+1/+2
5. Were (are) resources and infrastructure provided by actors who were identified as providers of these resources and infrastructure? 3	−2/1/0/+1/+2
6. To what extent are key resources and infrastructure cost-effective or optimized in terms of resource utilization, operating costs, or total costs of ownership? 4	−3/−2/−1/0/+1
7a. Has the way of integration, interoperability and compatibility of key resources and infrastructure with other systems, technologies or stakeholders in the service ecosystem been established in the BM? 1	−1/+1
7b. Was it necessary to change/use a different method of integration, interoperability and compatibility of key resources and infrastructure in the implementation phase, compared to the one specified in the BM? 2	−1/+1
7c. Is seamless data exchange, integration or collaboration between different components or actors ensured and facilitated? 3	−2/1/0/+1/+2
8. Are key resources and infrastructure sustainable, environmentally sound, or socially responsible in terms of minimizing negative impacts on the environment, society, and public health? 3	−2/1/0/+1/+2
9a. Has the BM established a way of managing, monitoring, and optimizing key resources and infrastructure in terms of performance metrics, analytics or feedback mechanisms? 1	−1/+1
9b. Was it necessary to change/use a different method of management, monitoring and optimization of key resources and infrastructure in the implementation phase, compared to the one specified in the BM? 2	−2/1/0/+1/+2
9c. Are key resources and infrastructure regularly monitored, measured, analyzed and is feedback regularly obtained? 3	−2/1/0/+1/+2
9d. Does the chosen method of management, monitoring and optimization contribute to continuous improvement, innovation, or optimization of the service? 3	−2/1/0/+1/+2
10. To what extent do key resources and infrastructure meet the availability, quality, expectations or requirements of the service and its users as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
11. Are the key resources and infrastructure flexible and adaptable, scalable, updatable, and capable of being modified in terms of the need to adapt to changes in service demand, user requirements or technological advances? 3	−2/1/0/+1/+2
12a. Were possible barriers, limitations or risks related to resources and infrastructures, such as unexpected inflation, lengthy VO, supplier insolvency, etc. identified in the BM, and ways to appropriately address or overcome them proposed? 1	−1/+1
12b. Are there (have) there been any barriers, restrictions, or risks in connection with resources and infrastructures for which a suitable solution to eliminate negative impacts has not been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1
13. Are key resources and infrastructure compliant with regulations and standards, certified or regulated to meet legal, regulatory or industry standards such as privacy, security, or interoperability requirements? 3	−2/1/0/+1/+2
14. Do key resources and infrastructure support the growth, expansion, or evolution of the service over time? 3	−2/1/0/+1/+2
15. Do key resources and infrastructure contribute to the reliability, efficiency, or effectiveness of the service? 3	−2/1/0/+1/+2
16. Do key resources and infrastructure contribute to the financial sustainability or viability of the service? 3	−2/1/0/+1/+2

,

Table A12. Parameters of the assessment tool of the business model of public smart services for the building block—budget costs.

Parameters	Evaluation
1. Were the budget costs identified in the BM, which needed to be spent in the process of providing the service? 1	−1/+1
2. Are the budgeted costs amenable to possible adjustment or revision to optimize the affordability and inclusiveness of the service while maintaining financial viability? 3	−2/1/0/+1/+2
3. Are budgeted costs effectively adjusted and revised to ensure cost competitiveness, profitability, or financial resilience? 3	−2/1/0/+1/+2
4. Do budgeted costs consider potential changes in technology, regulatory requirements or other external factors that may affect the cost structure of services? 3	−2/1/0/+1/+2
5. Are the costs associated with the new service higher, the same, or lower than the current alternative? 9	−1/0/+1
6. Was there an opportunity to save or reduce budget costs and was it used? 1	−1/+1
7. To what extent have the budgeted costs of the service been correctly estimated, projected or planned, including cost forecasting, budgeting, or financial modelling, as evidenced by thorough analysis and studies? 4	−3/−2/−1/0/+1
8a. Has the method of monitoring, analysis and reporting of budget costs, including the processes of cost control, cost reduction and cost optimization, been established in BM? 1	−1/+1
8b. Was it necessary to change/use a different method of monitoring, analysis and reporting of budget costs in the implementation phase, compared to the one set in the BM? 2	−1/+1
8c. Are the budget costs of the service regularly monitored, analyzed, and evaluated? 3	−2/1/0/+1/+2
8d. To what extent is the service being operated within budget constraints and financial targets? 4	−3/−2/−1/0/+1
9a. Was the method of management, optimization, and diversification of the budget costs of the service determined in the BM to reduce risks, ensure cost efficiency or financial sustainability of the service? 1	−1/+1
9b. In the implementation phase, was it necessary to change/use a different method of managing, optimizing, and diversifying the budget costs of the service, compared to those specified in the BM? 2	−1/+1
9c. Are the budget costs of the service effectively managed, optimized and diversified? 3	−2/1/0/+1/+2
9d. Compared to the service provided by the traditional system, are the sources of costs less, more or equally diversified? 9	−1/0/+1
10a. Have operational controls, audits or reporting mechanisms been planned in BM to ensure that the service is operated in a transparent, compliant, and financially responsible manner? 1	−1/+1
10b. Are there regular or planned operational controls, audits or reporting mechanisms? 3	−2/1/0/+1/+2
10c. Are the services operated in a transparent, compliant, and financially responsible manner? 3	−2/1/0/+1/+2
11a. Has the BM established a way to project, plan or optimize the budgeted costs of the service to support future growth or replication efforts? 1	−1/+1
11b. Was it necessary to change/use a different method of projecting, planning, and optimizing the budget costs of the service in the implementation phase, compared to the one specified in the BM? 2	−1/+1
11c. Do the budgeted costs of the service consider the potential scalability, expansion, or replication of services to other areas or regions? 3	−2/1/0/+1/+2
12. Do the budgeted costs of the service reflect market dynamics, competition or trends in the relevant industry or market? 3	−2/1/0/+1/+2
13. Are cost management efforts integrated into the broader strategy, value proposition or social impact of services? 1	−1/+1
14. Are the budgeted costs of the service in line with the overall strategy, value proposition or financial sustainability plan of the service? 3	−2/1/0/+1/+2
15. Are the budgeted costs of the service in compliance with relevant laws, regulations or guidelines related to labor, procurement, or other operational aspects, as evidenced by operational controls, audits or reporting mechanisms? 3	−2/1/0/+1/+2

,

Table A13. Parameters of the evaluation tool of the business model of public smart services for the building block—income streams.

Parameters	Evaluation
1. Were the individual income streams from the service identified in BM? 1	−1/+1
2. Are pricing strategies, user fees and other revenue mechanisms amenable to possible adjustment or revision to maximize revenue generation while ensuring affordability and inclusiveness of the service? 3	−2/1/0/+1/+2
3. Are revenue streams effectively adjusted, expanded, or diversified to take advantage of emerging opportunities and respond to changing market dynamics or mitigate potential risks and disruptions? 3	−2/1/0/+1/+2
4. Do service revenue streams consider potential changes in technology, regulatory requirements, or other external factors that may affect the service’s revenue-generating potential? 3	−2/1/0/+1/+2
5. Is the generated revenue associated with the new service higher, the same, or lower than the current alternative? 9	−1/0/+1
6a. Were the recipients directly asked how much and in what way they are willing to pay for the service (for example, subscription, transactional, etc.)? 1	−1/+1
6b. Were these findings included and considered in BM? 1	−1/+1
6c. To what extent does the service provided meet the price requirements of the recipients as evidenced by user feedback, market research or satisfaction metrics? 4	−3/−2/−1/0/+1
6d. To what extent are the service’s revenue streams aligned with the needs, preferences, or willingness to pay off target users or customers, as evidenced by user feedback, market research, or satisfaction metrics? 4	−3/−2/−1/0/+1
6e. Was it necessary to change the price additionally for there to be interest (or higher interest) in the service? 2	−1/+1
7a. Has the method of implementation, monitoring and evaluation of revenue streams from the service, including processes for revenue forecasting, tracking, and reporting, been established in BM? 1	−1/+1
7b. In the implementation phase, was it necessary to change/use a different method of implementation, monitoring and evaluation of revenue streams from the service, compared to those set out in the BM? 2	−1/+1
7c. Are revenue streams from the service regularly monitored and evaluated? 3	−2/1/0/+1/+2
7d. To what extent does the service generate sufficient revenue to cover costs, achieve profitability, or achieve financial sustainability? 4	−3/−2/−1/0/+1
8a. Has the BM established a way of managing, optimizing and diversifying income streams to reduce risks or increase financial resilience? 1	−1/+1
8b. Was it necessary to change/use a different method of management, optimization, and diversification of income streams in the implementation phase, compared to the one stipulated in the BM? 2	−1/+1
8c. Are the service’s revenue streams diversified, including the number of revenue sources, their relative contribution to total revenue, and their potential risks or dependencies? 3	−2/1/0/+1/+2
8d. Compared to the service provided by the traditional system, are the sources of income less, more, or equally diversified? 9	−1/0/+1
9a. Have financial controls, audits or reporting mechanisms been planned in the BM to ensure that the service is operated in a transparent, compliant, and financially responsible manner? 1	−1/+1
9b. Are there regular or planned financial controls, audits or reporting mechanisms? 3	−2/1/0/+1/+2
9c. Is the service operated in a transparent, compliant, and financially responsible manner? 3	−2/1/0/+1/+2
10a. Has the BM established a method for designing, planning, and optimizing revenue streams from the service to support future growth or replication efforts? 1	−1/+1
10b. In the implementation phase, was it necessary to change/use a different method of projecting, planning, and optimizing revenue streams from the service, compared to the one specified in the BM? 2	−1/+1
10c. Are the revenue streams from the service commensurate with the potential scalability, expansion, or replication of the service to other areas or regions? 3	−2/1/0/+1/+2
11. Do service revenue streams reflect market demand, competition or industry or market trends? 3	−2/1/0/+1/+2
12. Are the revenue streams of the service in line with the overall strategy, value proposition or financial sustainability plan of the service? 3	−2/1/0/+1/+2
13. Are the service revenue streams in compliance with applicable laws, regulations or guidelines regarding pricing, invoicing, taxation, or other financial aspects? 3	−2/1/0/+1/+2

,

Table A14. Parameters of the assessment tool of the business model of public smart services for the building block—environmental impacts.

Parameters	Evaluation
1a. Were positive environmental impacts of the service identified in BM? 1	−1/+1
1b. Are the actual positive environmental impacts associated with the service smaller, larger or the same as planned? 9	−1/0/+1
2a. Is there an alternative service that would bring more positive environmental impacts? 2	−1/+1
2b. Are the actual positive environmental impacts of the service smaller, larger, or the same as the current alternative? 9	−1/0/+1
3a. Were the negative environmental impacts of the service identified in BM? 1	−1/+1
3b. Was it determined in the BM how these negative impacts will be eliminated as much as possible? 1	−1/+1
3c. Has there been a planned reduction of negative environmental impact? 3	−2/−1/0/+1/+2
4a. Is there an alternative service that would bring less negative impacts? 2	−1/+1
4b. Are the actual negative environmental impacts of the service less, greater, or the same as the current alternative? 10	−1/0/+1
5. Does the service contribute to the preservation, protection, and restoration of the environment, including initiatives such as promoting sustainable transport, reducing greenhouse gas emissions, conserving natural resources, or increasing biodiversity? 3	−2/−1/0/+1/+2
6. Does the service promote environmental awareness, education or engagement among users, stakeholders, or the wider community? 3	−2/−1/0/+1/+2
7a. Has the BM established a way to assess, manage and optimize environmental impacts arising during the life cycle of the service, such as material sourcing, production, use and end-of-life management, to minimize environmental damage and promote sustainability? 1	−1/+1
7b. Was it necessary to change/use a different method of assessment, management, and optimization of environmental impacts in the implementation phase, compared to the one stipulated in the BM? 2	−1/+1
7c. Are there actually minimization of negative environmental impacts and promotion of sustainability, as evidenced by various analyses, metrics, etc.? 3	−2/−1/0/+1/+2
8a. Has the BM established a way to integrate environmental aspects into the design, development, or operation of the service, including elements such as energy-efficient technologies, waste reduction strategies or water conservation measures? 1	−1/+1
8b. Was it necessary to change/use a different way of integrating environmental aspects in the implementation phase, compared to the one stipulated in BM? 2	−1/+1
8c. Are environmental aspects effectively integrated into the design, development, or operation of the service? 3	−2/−1/0/+1/+2
9a. Has the BM established a way of working with relevant environmental organisations, experts, or stakeholders to ensure that environmental impacts are properly assessed, resolved, or mitigated? 1	−1/+1
9b. Was it necessary to change/use a different method of cooperation in the implementation phase, compared to the one stipulated in BM? 2	−1/+1
9c. Are the outputs of this cooperation integrated into the overall BM or service sustainability plan? 3	−2/−1/0/+1/+2
10a. Has the BM established a way to support circular economy principles, such as recycling, re-use, or waste reduction? 1	−1/+1
10b. Was it necessary to change/use a different way of supporting the principles of the circular economy in the implementation phase, compared to the one stipulated in the BM? 2	−1/+1
10c. Are these principles effectively integrated into the value proposition of the service, supply chain management or other operational processes of the service? 3	−2/−1/0/+1/+2
11a. Measurable criteria, (evaluative) threshold values, or KPI for achieved environmental positive/negative impacts? 1	−1/+1
11b. If so, to what extent were the measurable criteria, (evaluative) threshold values, or KPI achieved during retrospective evaluation? 4	−3/−2/−1/0/+1
12a. Have potential environmental risks, vulnerabilities, or uncertainties, including those related to climate change, natural disasters, or other environmental factors, been taken into account in the BM and ways of appropriately addressing or overcoming them proposed? 1	−1/+1
12b. Are there (have) been any risks, vulnerabilities or uncertainties related to environmental impacts for which a suitable solution has not been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1
13. Was (Is) the service implemented and operated in accordance with relevant environmental regulations, standards, certifications, or best practices? 3	−2/1/0/+1/+2
14. Is the service consistent with the wider environmental objectives, policies, or strategies of local, regional or national governments? 3	−2/−1/0/+1/+2

and

Table A15. Parameters of the evaluation tool of the business model of public smart services for the building block—social impacts.

Parameters	Evaluation
1a. Were the positive social impacts of the service identified in BM? 1	−1/+1
1b. Are the actual positive social impacts associated with the service smaller, larger or the same as planned? 9	−1/0/+1
2a. Is there an alternative service that would bring more positive social impacts? 2	−1/+1
2b. Are the actual positive social impacts of the service smaller, larger, or the same as the current alternative? 9	−1/0/+1
3a. Were the negative social impacts of the service identified in BM? 1	−1/+1
3b. Was it determined in the BM how these negative impacts will be eliminated as much as possible? 1	−1/+1
3c. Has there been a planned reduction in the negative social impact? 3	−2/−1/0/+1/+2
4a. Is there an alternative service that would bring less negative impacts? 2	−1/+1
4b. Are the actual negative social impacts of the service smaller, larger, or the same as the current alternative? 10	−1/0/+1
5. Does the service promote social inclusion, diversity, or equality, including measures to reduce the digital divide, ensure equal access to services or address potential bias or discrimination in the provision of services? 3	−2/−1/0/+1/+2
6. Does the service promote digital literacy, skills development or capacity building among users, stakeholders, or the wider community? 3	−2/−1/0/+1/+2
7a. Has the BM established a way of assessing, monitoring, reporting, or mitigating social impacts to ensure positive social outcomes and mitigate any negative social impacts? 1	−1/+1
7b. Was it necessary to change/use a different method of assessment, monitoring, reporting, or mitigating social impacts in the implementation phase, compared to the one specified in the BM? 2	−1/+1
7c. Is there actually regular assessment, monitoring, reporting or mitigation of social impacts? 3	−2/−1/0/+1/+2
8a. Has the BM established a way to integrate social aspects into the design, development, or operation of the service, including elements such as accessibility features for people with disabilities, measures to promote social justice or strategies to promote community involvement? 1	−1/+1
8b. Was it necessary to change/use a different way of integrating social aspects in the implementation phase, compared to the one stipulated in BM? 2	−1/+1
8c. Are social aspects effectively integrated into the design, development, and operation of the service? 3	−2/−1/0/+1/+2
9a. Has the BM established a way of working with relevant social organisations, experts, or stakeholders to ensure that social impacts are properly assessed, addressed, or maximised? 1	−1/+1
9b. Was it necessary to change/use a different method of cooperation in the implementation phase, compared to the one stipulated in BM? 2	−1/+1
9c. Are the outputs of this cooperation integrated into the overall BM or social impact plan of the services? 3	−2/−1/0/+1/+2
10a. Has the BM established a way to integrate social aspects into the user experience, user interface or service user engagement strategies? 1	−1/+1
10b. Was it necessary to change/use a different way of integrating social aspects in the implementation phase, compared to the one stipulated in BM? 2	−1/+1
10c. Are social aspects effectively integrated into the user experience, user interface or user engagement strategies of the service? 3	−2/−1/0/+1/+2
11a. Measurable criteria, (evaluative) threshold values, or KPI for achieved social positive/negative impacts? 1	−1/+1
11b. If so, to what extent were the measurable criteria, (evaluative) threshold values, or KPI achieved during retrospective evaluation? 4	−3/−2/−1/0/+1
12a. Have potential social risks, vulnerabilities, or uncertainties, including those related to privacy, security, ethics, or social justice, been considered in the BM? 1	−1/+1
12b. Are there (have been) any risks, vulnerabilities or uncertainties related to social impacts for which no suitable solution has been proposed and the service provided cannot (couldn’t) overcome them? 2	−1/+1
13. Was (Is) the service implemented and operated in accordance with relevant social regulations, standards, or best practices? 3	−2/−1/0/+1/+2
14. Is the service consistent with wider social goals, policies, or strategies of local, regional or national governments? 3	−2/−1/0/+1/+2

and Table 3 characterizing the evaluations of the individual parameters. Appendix A Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9, Table A10, Table A11, Table A12, Table A13, Table A14 and Table A15 describe the parameters and sub-parameters of individual building blocks for evaluating the business model of smart city services.

Table 4. Method of evaluation of individual parameters.

Index	Evaluation	Description
1	−1/+1	(−1 if the answer is no; +1 if the answer is yes)
2	−1/+1	(−1 if the answer is yes; +1 if the answer is no)
3	−2/−1/0/+1/+2	(−2 no; −1 rather no; 0 medium value/can’t judge; +1 rather yes; +2 yes)
4	−3/−2/−1/0/+1	(−3 for/at less than 25%; −2 for/at 25% to 49%; −1 for/at 50% to 74%; 0 for/at 75% to 89%; +1 for/at 90% up to 100%)
5	−1/0/+1	(−1 if the answer is no; 0 if the question is not relevant (for example, the service is not an alternative to the current service or there was no need to perform the given action); +1 if the answer is yes)
6	−1/0/+1	(−1 if the answer is yes; 0 if the question is not relevant (for example, the service is not an alternative to the current service or there was no need to perform the given action); +1 if the answer is no)
7	−4/−3/−2/−1/0/ +1/+2/+3/+4	(−4 if the service is used 75% to 100% less than the current alternative; −3 for 50% to 74%; −2 for 25% to 49%; −1 for less than 25%; 0 if the service has no current alternative; +1 if it is used 1% to 24% more; +2 for 25% to 49%; +3 for 50% to 74%; +4 for 75% to 100%)
8	−1/0	(−1 if the answer is no; 0 if the answer is yes)
9	−1/0/+1	(−1 if the answer is lower/less; 0 if the answer is the same; +1 if the answer is higher/more)
10	−1/0/+1	(−1 if the answer is greater; 0 if the answer is the same; +1 if the answer is less)

characterizes the individual assessment methods used for individual parameters and sub-parameters.

The obtained evaluation parameters of each block of the business model are described in Appendix A Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9, Table A10, Table A11, Table A12, Table A13, Table A14 and Table A15. Each appendix represents the evaluation of individual building blocks belonging to one of the four parts—value proposition, value provision, value creation and triple bottom line. At the same time, the evaluation of the business model is conceived from two points of view: whether the business model was correctly created (it contained all the necessary elements that were not changed during the implementation phase), and whether it was possible to achieve the set goals and requirements of the effective provision of public smart services.

The building blocks of the value proposition belong to the first part of the business model of public smart services. In this part, the fulfillment of the set mission (goal), the success of the delivery of the required value proposition and the provision of the delivery and collection of the necessary data are assessed through the evaluation tool.

The building block setting the mission (goal) represents a clear definition of the goal, which outlines the overarching purpose, vision, and direction of the public smart service. For this building block, 12 evaluation parameters were created (13 together with sub-parameters), which are captured in Table A1.

The value proposition building block describes how the specific needs of network recipients will be met and what value in the form of a public smart service will be delivered to them. For this building block, 13 evaluation parameters (22 together with sub-parameters) were created, which can be seen in Table A2.

The data building block mainly deals with the analysis of what data is necessary for the provision of the service, how these data will be handled, what data will need to be collected and who will work with these data, or have access to them, etc. For this building block, 15 parameters (38 including sub-parameters) were created, which can be seen in Table A3.

The second part of the business model represents a grouping of building blocks aimed at providing value. The evaluation tool for this part primarily monitors whether the network beneficiaries are correctly segmented, to what extent they are involved in the decision-making process, how satisfied they are with the fulfillment of their requirements and needs through the value proposition, and whether all deployment channels are necessary for effective service provision. It represents the field of marketing focused on communication with users within the used distribution channels and in changing the philosophy and vision of smart city services.

The network beneficiaries building block clearly defines who will directly benefit from the value proposition. It is about determining the target group for which specific needs or issues are proposed to be solved through a public smart service. For this building block, five parameters were created (14 together with sub-parameters), which are contained in Table A4.

The deployment channels building block solves what technological basis is needed for the provision of public smart services, how the services will be provided (web portals, mobile applications, kiosks, physical installations or other means), whether it is possible to use already implemented portals, applications, etc., within which additional services would be provided and, last but not least, it describes the methods/measures of communication and distribution of values created by various actors to network beneficiaries, support mechanisms for these beneficiaries, etc. For this building block, 14 parameters were created (35 together with sub-parameters), which are contained in Table A5.

The third part of the business model includes the building blocks that are dedicated to creating value. The created parameters for this part mainly assess the correct selection and involvement of key actors, the provision of necessary activities, the creation and maintenance of relationships between individual actors and access to the necessary resources and infrastructure and their effective use for delivering the required value proposition.

The building block of key actors refers primarily to the precise identification of stakeholders who offer the best opportunities to access multiple resources necessary for the proper provision of public smart services, such as government agencies, private companies, non-profit organizations, citizens, or other relevant entities. For the analysis of this building block, nine parameters (26 together with sub-parameters) were created, which are listed in Table A6.

The key activities building block describes all activities that need to be implemented for the proper provision, operation, and maintenance of a public smart service, such as data collection and analysis, service deployment, user support, marketing or performance monitoring. These activities are ensured by individual actors who were identified in the previous building block. For the evaluation of this block, 14 evaluation parameters (30 together with sub-parameters) were created, which are shown in Table A7.

The building block relations with actors refers primarily to the key relationships of all key actors in the network and at the same time to their ties to the provided public smart services. It is about defining the relationships that need to be created to facilitate and streamline the co-creation practices of the actors involved in the ecosystem of the service provided, and the resulting mutual relationships that need to be organized so that value is co-created. These are, for example, supplier, distribution, marketing or implementation relationships. Nine evaluation parameters (16 together with sub-parameters) were designed for this building block, which are shown in Table A8.

The building block offer of key actors is mainly focused on the description of what each actor can offer in the framework of joint creation, i.e., technology, research and development, intellectual property rights, development of new products, processes and services, citizen involvement, etc. For the evaluation of this block, 15 evaluation parameters (22 together with sub-parameters) were created, which are shown in Table A9.

The key actor co-creation operation building block describes what types of operations each key actor can potentially provide to contribute to the creation and provision of a public smart service and maximize its impact. They enable cities/municipalities to systematically evaluate opportunities and weaknesses or gaps that need to be addressed for the successful implementation of an integrated smart city solution. Thirteen evaluation parameters were chosen for this building block (25 together with sub-parameters), which are captured in Table A10.

The building block key resources and infrastructure describes what resources are necessary for the proper provision of a public smart service. These are mainly financial, physical, mental, and human, but also infrastructural resources, such as data centers, cloud infrastructure, communication networks, hardware, software, or other technological assets. In the business model, however, it is necessary to focus on those that are necessary for the provision of the service and that can differentiate a particular city/municipality from others. For this building block, 16 evaluation parameters (28 together with sub-parameters) were created, which are shown in Table A11.

The last part of the business model represents the grouping of triple bottom-line building blocks. The assessment tool in this part mainly monitors the effective use of budget costs, their amount and diversification, the appropriateness of the chosen pricing strategies, the amount and diversification of income streams and what positive and negative environmental and social impacts result from the service provided and whether the service can bring more positive effects than the current one alternative.

The budget costs building block mainly refers to the economic costs that will be incurred during the implementation of public smart services. These costs include all costs from investments in infrastructure projects to soft incentives for network beneficiaries or users. In this block, it is also necessary to consider the possible reduction in costs through economies of scale. Fifteen evaluation parameters were selected for this building block (25 together with sub-parameters), which are shown in Table A12.

The revenue stream building block describes all revenue streams associated with public smart services, including pricing strategies, user fees, revenue models, advertising, sponsorship, or other revenue generation mechanisms. Cities and towns have a range of revenue streams to evaluate, including taxes, user fees and external grants. Alternative sources of income can come from the creation of entirely new economic segments in these sectors. After analyzing the cost structure, this building block will show whether the chosen business model is economically viable or whether other sources of financing will need to be considered. For this building block, 13 evaluation parameters were created (27 together with sub-parameters), which are captured in Table A13.

The building block environmental impacts is mainly focused on comparing how the planned strategy will generate more environmental benefits than negative impacts. Even if the goal of a smart city is to achieve benefits for the environment, it is important to consider what its negative impacts may be. Environmental impacts can be monitored using various indicators that should be included in the business model. Subsequently, the city/municipality can receive a clearer picture of where it should direct its attention and resources. For this building block, 14 evaluation parameters were created (29 together with sub-parameters), which are captured in Table A14.

The social impact building block is mainly focused on social sustainability. It describes the effects, both positive and negative, that a public smart service can have on residents and communities. These are all aspects that directly result from the interventions of smart cities but may also include indirect benefits. As in the previous building block, the social impacts should also be measurable using selected indicators that will enable the city/municipality to evaluate over time whether the services provided have a positive or negative impact on the social well-being of the company. For this building block, 15 evaluation parameters were created (29 together with sub-parameters), which are captured in Table A15.

In addition to the parameters, a method of their evaluation was also proposed. The obtained evaluation parameters are captured in Table 4. Each row represents the evaluation of individual building blocks belonging to one of the four parts—value proposition, value provision, value creation and triple bottom line. In addition to the parameters, a method of their evaluation was also proposed, see Table 4.

A separate rating scale was created for each parameter (Table 4). The rating scale/parameters/values in Table 3 are based on primary research carried out in municipalities and cities in Slovakia and were subsequently adjusted by experts based on their professional knowledge of cattle and agreed within the implemented Delphi method. The used method for evaluating individual parameters and sub-parameters of given building blocks is indicated by an index within the proposed evaluation of building blocks, Appendix A Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9, Table A10, Table A11, Table A12, Table A13, Table A14 and Table A15. The evaluation logic is the same for all parameters. Achieving negative values means insufficient inclusion of the evaluated element in the business model or failure to meet the set goals related to the evaluated element. Conversely, positive values can be achieved if the evaluated elements were correctly included in the business model, or if the set requirements and goals were met. Therefore, the goal should be to achieve the greatest possible number of points (sum of positive values).

4. Discussion

Based on the above findings, we can state that the most frequently used tool for creating a business model for public smart services is the Business Model CANVAS, even in the conditions of Slovakia. This result identifies with other research, such as Timeus et al. [40] or Poeppelbuss, J., Durst, C. [85] and others. In municipalities and cities in Slovakia, business models of public smart services are not given enough attention. Research in this area is more focused on specific areas of companies, as Stalmašeková et al.—The impact of using the digital environment in transport [86], Paculík et al.—Low level of digitization and innovations in business models of companies in Slovakia [87], Loučanová et al.—Open business model of eco-innovation for sustainability development: Implications for the open-innovation dynamics of Slovakia [34], Bertot et al.—Universal and contextualized public services: Digital public service innovation framework, Kollarova et al.—Conceptual model of key aspects of security and privacy protection in a smart city in Slovakia etc.

The stated results point to the fact that the need to evaluate the business model of public services has a strong positive impact on the creation of business models in cities and towns. Therefore, as part of the research, we subsequently proposed a methodology for evaluating the business models of public smart services using the Delphi method. The need for this methodology arises from the examples shown in Table 1, which point to the trend of developing smart city services, which represent an increase in efficiency, added value and a number of other benefits, as stated by the authors [67,68,69,70,71,72,73,74,75,76,77,78,79].

Smart cities are characterized by a high level of complexity, social diversity and contextual uncertainty. As various other studies have shown [88,89,90,91,92], it is difficult to capture this complexity in assessments. However, they agree on future perspectives and new approaches to understanding and evaluating the business models of public smart services and at the same time illustrate their own problems with evaluation (specific to the country, technology and other parameters). Based on our analysis, we will outline the method of evaluating the business models of public smart services in the conditions of Slovakia, based on the needs of municipalities and cities, and created on the basis of the Delphi method in cooperation with experts in the field. The assessment represents an innovative, open assessment methodology that addresses cities and municipalities in Slovakia and has the potential to lead to more inclusive smart city management processes. The presented research results generate rich, context-specific data and simplify complex phenomena. The application of the proposed assessment can help to work more efficiently with smart city data and use them for the development and e-government of cities.

The presented draft of the evaluation tool should serve primarily for creators, respectively, evaluators of business models of public smart services, who would use the tool to evaluate whether they have fulfilled all the necessary building blocks, considered all aspects and conditions of a smart city, and directed the provision of services to the goals defined by the smart city concept itself. If there will be several business models created with the same logic (the building blocks of the Smart City Business Model Canvas will be included), it will be possible to compare them, evaluate them and make quick strategic decisions, which would gradually lead to the fulfillment of the original goal.

Among the main advantages of the created tool, we would include especially the process of creating the proposal itself, in which 28 experts from practice were involved and who, after several consultations and the incorporation of their comments and suggestions, consider the tool to be sufficiently suitable for use for the specified purpose. This tool should be intended for them, so we see a great benefit that they were involved in the design and that they see the point in its creation and continued use. On the other hand, there is room to question whether the involvement of 28 experts is sufficient. Until the tool is tested in practice, it is not possible to assess its main shortcomings, or limitations, and therefore not even determine whether the number of experts involved was sufficient. However, it is worth considering continuing this research and involving other experts to modify the final tool to best suit the needs and requirements of its users.

After evaluating the final form of the tool, we believe that the goal of creating a simple, clear, and comprehensible evaluation tool for all interested parties, including citizens, was fulfilled. This claim is mainly based on the assumption that the assessment tool is based on the Business Model Canvas, which is itself characterized as a very easy-to-understand and use tool. At the same time, the parameters are also formulated in the form of control questions (which has also been proven with the evaluation tool Business Model Evaluation Tool for Smart Cities by the authors Díaz-Díaz, Muñoz and Pérez-González [51,52] and not complicated indicators (as in the case of FPBM by N. Walravens P. Ballon [35], who was criticized by the scientific community for this) [30,35,40,41,48,51,52]. The evaluation of individual parameters represents numerical scoring on the relevant scales, which also ensures a clear, unambiguous, and transparent evaluation.

The tool includes a wide range of parameters, or evaluation questions (191 in total). That is why we consider a large scope of these parameters to be a benefit, through which a detailed analysis and evaluation of the monitored areas, goals and requirements set in the business model, but also by the smart city concept itself, can be ensured. When using the tool, the challenge will be to limit the subjective attitude of the evaluator. Predicting the future performance of business models requires a subjective component, and therefore the evaluation tool leads to debatable decisions. Nevertheless, it facilitates decision-making, increases transparency and accountability, leads to fairer decisions, and strengthens citizens’ trust. When assessing the adequacy of the business model to the strategic plan of the city or municipality, it is a subjective assessment, and in addition, the other parameters of the assessment tool are also subjective to a certain extent.

In fact, the cost structure may be objective in the case of fixed-price public service concessions, but in other cases, the value of this parameter is based on forecasts, and therefore we may encounter a subjective factor when using the evaluation tool. The evaluation of parameters from the building blocks of income streams and network beneficiaries also depends on forecasts, which can be successful, as happens in all businesses [42,48,51,52]. As for the parameters of social and environmental costs and benefits, as well as the value proposition, their subjective nature results from the personal opinions of each member of the evaluation committee. Despite the existence of a certain degree of subjectivity in the evaluation of business models using the evaluation tool, its use should remain useful for those facing the choice of a business model, as it facilitates analysis and decision-making considering different perspectives and increases accountability and transparency [30,35,48,51,52].

As part of the effectiveness assessment, the impact on stakeholders should be investigated, i.e., what is the impact of the service on various stakeholders, including users, the public sector, businesses and the community; achieving goals, whether the business model effectively achieves the set goals of the service, such as solving social challenges, improving the provision of public services or increasing the quality of life; user satisfaction, i.e., whether users are satisfied with the service provided and consider it of value; social inclusion, i.e., whether the business model supports social inclusion by ensuring fair access and service benefits for all segments of the population, including vulnerable or marginalized groups, and public value, i.e., whether the business model creates public value by addressing societal needs and providing benefits to the wider public beyond individual users [30,35,48,51,52].

From the point of view of efficiency, the evaluation parameters should determine the use of resources, whether the business model for providing the service uses effective resources, including financial, human and technological; process efficiency, marketing, i.e., whether the service’s operational processes and workflows are efficient, improved and optimized to deliver the service on time; performance metrics, whether there are well-defined performance metrics to measure the effectiveness of the service, such as service delivery time, response time or resource utilization rate and innovation potential, i.e., whether the business model supports innovation, marketing and continuous improvement in the provision of public smart services by encouraging experimentation, learning and feedback bonds [22,48,51,52,54,64,80].

When evaluating the effectiveness, the time to market should be taken into account, i.e., whether the business model is able to provide a public smart service on time, taking into account the dynamic nature of technological progress and the needs of citizens, or users; agility, whether the business model has the flexibility and adaptability to be able to respond in a timely and efficient manner to the changing demands of the market or users; scalability, i.e., whether the business model is designed to scale the service in light of potential increases in demand or expansion into new areas or user segments; stakeholder engagement, whether the business model effectively involves stakeholders including users, the public sector, businesses and the community in the design, implementation and operation of the service and collaboration, marketing, i.e., whether the business model facilitates collaboration and partnerships between different stakeholders, such as public–private partnerships, with the aim of using collective resources, expertise and skills in providing a public smart service [30,31,35,44,53].

It is also important to note that the evaluation parameters of business models of public smart services may differ depending on the specific context, the nature of the service and the evaluation objectives. The proposed evaluation parameters serve as a starting point and can be adapted or extended as needed to meet specific evaluation requirements.

Based on the above, the proposed assessment tool for business models for smart services is in accordance with the goals of sustainable development (United Nations 17 Sustainability Development Goals—17 SDGs). It primarily fulfills goal 11 Sustainable cities and communities, but also partially contributes to the other goals, because business models help city governments to actively assess the net balance of their activities in terms of economic viability, environmental sustainability and social inclusion and acceptability, and to make changes if the results are not in line with the city’s strategic goals [25,46,47,48,49,93]. These tools direct it to sustainable value creation [17,60]. Their main purpose is to show how the city creates and maintains value for its citizens. Value must be created by social well-being, environmental sustainability, and economic prosperity [60,61,63,93].

5. Conclusions

The mentioned research is unique in Slovak conditions and represents an innovative tool for evaluating business models for the provision of smart services, as the research confirmed that the need to evaluate business models of public services has a strong positive impact on the creation of business models in cities and municipalities.

The created proposal should serve primarily for creators, or evaluators of business models of public smart services. The tool provides a simple and clear list of the most important areas that should be included in the business model. The tool also includes a proposal for the evaluation of individual parameters, which enables a mutual comparison of several business models. The results of the business model evaluation using this tool also represent an understandable and transparent indicator of success for all interested parties. The simplicity of the tool also contributes to quick and simple understanding by citizens as target users of public smart services and brings these services closer to citizens through marketing tools so that their efficiency increases. The limitation s of this study is the implementation in the conditions of the Slovak Republic, but it is to be implemented it also in other countries. The limitations of this study are in the limitations of the methods used, in which it is difficult to assess the level of expertise of individual survey participants, it has a high degree of sensitivity of the results to ambiguously formulated questions, and limited survey time.

The evaluation tool was designed specifically for smart cities, but it could be adapted to evaluate business models in other areas, such as public administration at any level or even for-profit organizations that feel responsible for the development of the communities in which they operate. Therefore, we consider the testing of the revised proposal in other areas, etc., as a future direction of research. Further research should be aimed at verifying the proposed methodology for evaluating Business Models for Innovation of Public Smart Services and evaluating its effectiveness in cities and municipalities.