Assessment of the Croatian Open Data Portal Using User-Oriented Metrics

Abstract

Open data portals are web services that serve as a central access point for all government-published open data and can exist at local, regional, national, and international levels. They are an important element of most open data initiatives that have enabled a large amount of government data to be widely available. However, data quantity and quality are not the only aspects that should be considered when publishing data. To improve the reusability of data and to achieve greater impact and benefits from open data, it is important to consider user-oriented aspects of the portal management, discovery, and use of data (e.g., organizing the portal in a user-centric way, providing accurate metadata, using a standardized and open data format, etc.). In this paper, we adopted the metrics proposed by the European Commission to assess compliance of the Croatian Open Data Portal with 10 user-oriented principles that open data portals should implement in terms of sustainability and added value. While the results show the government’s efforts in publishing data, some aspects such as better collaboration with data providers and other data portals, offering different visualization tools, etc. need to be improved to achieve active use and impact.

1. Introduction

Public organizations produce and commission large amounts of data and information [1], which, when made available to citizens for further reuse is referred to as Open Government Data (OGD). According to the Open Definition by the Open Knowledge Foundation, open data are data that anyone can freely access, use, modify, and share for any purpose. Publishing open data can bring enormous benefits to society, e.g., increased transparency, increased public engagement, increased collaboration, economic growth, and easier discovery of data [2]. Therefore, the requirements for Open Government Data are widely recognized by countries that want to promote transparent, accountable, and collaborative governance [3]. Open data can facilitate the innovation of new digital products, create new jobs, increase value creation, and contribute to a better and more prosperous society [4].

1.1. European Legislation on Open Government Data

In the European Union (EU), the public sector is one of the most data-intensive sectors [5] and is responsible for managing and publishing a large amount of data. Because of that, the European Commission has played a leading role in setting the open data agenda. Through the so-called PSI Directive of 2003 [6], the European Commission provided a legal framework for reusing public sector information. However, only after two revisions of the PSI Directive, the term ‘open data’ was introduced in the Directive on open data and the re-use of public sector information [7]. The Open Data Directive, as well as other directives published by the European Commission, consists of numbered recitals beginning with the word “Whereas” that set out the reasons for the content of the enacting terms (i.e., articles) of a legislative act [8]. According to Recital (16) of the 2019 Open Data Directive [9], Member States should be encouraged to promote the creation of data that are open by design and default and at the same time ensure the protection of personal data. Another—more recent—key pillar of the European data strategy is the Data Governance Act [10], whose main goal is to increase trust in data sharing and availability, to overcome technical obstacles of data reuse, and will be applicable from September 2023, following a 15-month grace period [11]. Recital (26) of the Data Governance Act highlights the need for a single access point for information sharing, such as open data portals.

Open data portals are web services that serve as a central access point for all government-published open data and can exist at local, regional, national, and international levels. An example of a national open data portal is the Croatian Open Data Portal. Croatia joined the European Union in July 2013, and as a Member State, it is obliged to align its national legislation with the constantly evolving EU acquis [12]. For this reason, open data and the reuse of public sector information have been regulated in Croatia since 2013 when the Act on the Right of Access to Information was adopted. The Act enacted the PSI Directive (Directive on the re-use of public sector information), and soon after, in March 2015, the national Croatian Open Data Portal was launched [13].

In addition, data.europa.eu, formerly known as the European Data Portal, is an example of an international data portal. This portal was established in December 2012 and serves as a single access point for all data published by administrations in European countries, EU institutions, agencies, bodies, and other non-EU countries [14]. This is accomplished by harvesting the metadata from open data portals and geodata portals [15]. As of February 7, 2023, data.europa.eu has more than 1,600,000 datasets from 36 countries available on the portal through 176 data catalogs. In a one-year period, from 1 January 2022 to 1 January 2023, the total number of open datasets increased by almost 200,000, with Germany, the Czech Republic, and France being the top three countries in data publishing [16]. Three of the catalogs made available on the portal are from the Republic of Croatia: The City of Zagreb Open data portal catalog (local example), the National Spatial Data Infrastructure Geoportal of Croatia catalog, and finally the Open Data Portal of Croatia catalog (the latter two are examples of national open data portals). Although the publication of open data is not limited to governments, most of the data made available on the open data portals is published by public institutions [17].

1.2. Assessing the User Experience of Open Data Portals

Open data portals usually use some open data platforms solution. According to Bogdanoivć et al. [18], “in most cases, Open Data Portals are built on open-source platforms, such as Comprehensive Knowledge Archive Network (CKAN) or Drupal Knowledge Archive Network (DKAN), while others use proprietary software, such as Socrata or Opendatasoft.” Data platforms simplify the publishing and management of open data on the web through existing guidelines [19] that make it easier for data providers to publish and for users to find data. CKAN powers hundreds of data portals worldwide, from national and regional government organizations throughout the European Union, North and South America, Asia, and Oceania [20]. The previously mentioned data.europa.eu and Croatian Open Data portal both use the CKAN platform, which enables the application programming interface (API) by default [17]. This is one of the reasons for its popularity since it allows working with data without having to download it first.

Thanks to these open data platforms and open data initiatives, a large amount of high-quality government data are now available on open data portals. However, the quantity and quality of the data are not the only aspects that should be considered when publishing. The data provided on the portal should be made available in a way that facilitates its reuse and the creation of new value, because only publishing data that are not reused does not bring any benefit. One of the main goals in achieving greater open data reuse should be making it easier for users to navigate open data portals and provide them with a better user experience (UX). User experience can be described as any interaction the user has with a product or service, and the design of a high-quality UX takes into account various elements to provide a simple, efficient, reliable, and enjoyable experience for the user [21]. User experience plays an important role in the success or failure of products and services, so the main goal is to create a product that is highly usable and fulfills user needs [22].

This research was conducted as part of the Horizon 2020 Twinning Open Data Operational (TODO) project whose main goal is to leverage the interdisciplinary scientific excellence and innovation capacity of the University of Zagreb, Croatia, in the field of open data to boost the supply and use of open government data in Croatia and beyond. The main motivation for this research was to inform the government and portal owners about the overall quality of the portal, its usability, user experience, and the state of open data awareness in Croatia. Since users and user experience should be the focus of all Open Data portals, our goal was to assess the extent to which the Croatian Open Data portal is user-oriented and to make recommendations that can lead to greater reuse and awareness of open data and its benefits.

The main objectives of this paper are: (i) to provide an overview of some existing open data and open data portal assessment frameworks; (ii) to select the appropriate metrics and assess the extent to which the Croatian Open Data Portal is user-oriented; (iii) to assess how user-oriented the Croatian Open Data portal is compared to other European open data portals; (iv) to identify the features of the portal that need improvement; and (v) to make recommendations for achieving greater open data reuse.

This paper is structured as follows (with corresponding objectives): Section 2 summarizes various benefits and limitations of existing open data and open data portal frameworks (i); Section 3 presents the methodology and metrics used for the assessment of the Croatian Open Data portal (ii); Section 4 presents the results of the assessment of the Croatian open data portal based on user-oriented metrics (iii) and a comparison with different European open data portals (iv); and Section 5 provides our conclusion and recommendation that can lead to improvement of the portal (v).

2. Backgrounds of Open Data Assessment Frameworks

This section provides an overview of existing open data and open data portal assessment frameworks that have been developed in recent years. To evaluate how user-oriented the Croatian Open Data portal is, the most appropriate assessment framework with corresponding metrics must be selected. Assessment frameworks can be used for monitoring the status of open data in a single or multiple countries [23] and therefore track governments’ progress in opening data. They also serve as a tool for improving open data governance [24]. The next subsection provides an overview of six relevant open data assessment frameworks, the focus of their assessment, but also their limitations, geographic scope, and the timing of their last assessment.

2.1. Open Data Assessment Framework

The Open Data Barometer project, developed by the World Wide Web Foundation, is a global measure that aims to uncover the true impact of open data initiatives around the world based on accountability, innovation, and social impact [25]. There have been four editions so far, measuring open data implementation in more than 100 governments, with the last one dating back to 2016. In 2018, the Leaders Edition was published. The report analyzed the readiness, implementation, and emerging impact of 30 advanced open data governments only. This edition underwent a methodological revision compared to the 2016 edition. The 30 governments that were analyzed had either adopted the Open Data Charter principles, meaning data should be open by default, timely and comprehensive, accessible and usable, comparable and interoperable, for improved governance and citizen engagement, for inclusive development and innovation [26] or the equivalent G20 Anti-Corruption Open Data principles [27]. The methodology was last updated for the 2018 assessment.

The Global Open Data Index, run by the Open Knowledge Foundation, measures the openness of government data according to the Open Definition [28]. This is an independent assessment from a civic perspective that focuses only on the data publication, while data quality, use, and impact created from the data are not covered. This project is no longer active, but the methodology behind this assessment is available on their website.

The Open Data Inventory (ODIN) framework was developed by the Open Data Watch, and it assesses the coverage and openness of official statistics to identify gaps, promote open data policies, and encourage better collaboration between data providers and users [29]. Coverage refers to the availability of important statistical indicators in 22 categories of social, economic, and environmental statistics and is assessed on five elements (e.g., availability of data in the last five years, availability of data at the first administrative geographic level), while openness assesses compliance with international standards of openness also using five elements (e.g., availability of data in machine-readable formats, availability of download options). The last report was published in 2020, while the methodology was last updated in 2022, with an outgoing evaluation currently in progress.

The Open Data Maturity framework conducted by the European Data Portal is also worth mentioning. This framework provides insights into the state of open data among the EU Member States. This study assesses the level of maturity using four dimensions: Policy, Portal, Impact, and Quality, and it divides countries into four different maturity groups: trend-setters, fast-trackers, followers, and beginners [30]. The first report was published in 2015 with new reports published annually. The last 8th Open Data Maturity Report was published in December 2022, and all reports can be downloaded directly or accessed via a dashboard from the data.europa.eu. Since the first report of 2015, the methodology has been updated, the first major update was in 2018 and the last major update was in 2022. The 2018 report introduced four assessment dimensions for the first time: Policy, Impact, Portal, and Quality, which have since been maintained to provide consistency in evaluation. In the 2022 revision, the assessment is still made based on these four dimensions, but the methodology has been changed. According to the 2022 Open Data Maturity Report [31], the Policy, Portal, and Quality dimensions have not been significantly restructured, but the Impact dimension has undergone a comprehensive update. This year’s revision of the methodology allows for changes due to the new Open Data Directive and the implementation of the newly defined high-value datasets as well as a better distinction between data reuse and the impact created from data reuse.

The 2022 Open Data Maturity Report showed that EU27 countries have an average open data maturity score of 79%. This score has decreased by 2% from the last report in 2021. For two dimensions, Policy and Impact, the score has decreased compared to last year’s report, while the ‘Portal’ and ‘Quality’ dimensions have remained the same. The country that received the highest open data maturity score is France, putting it at the top of the trend-setter’s group, with an impressive 97% score. Other top-performing countries making it in the trend-setter group are Ukraine, Ireland, Poland, Cyprus, Estonia, Spain, and Italy. When it comes to open data maturity in Croatia, the results indicate that Croatia is below the EU27 average of 79%, with an overall score of 72%. Based on these results, Croatia is placed in the Followers group, taking 22nd place in this assessment. After ranking 18th for the past two years, these results appear to indicate that other countries are doing a better job of implementing their policies and achieving a greater reuse of open data. The greatest weakness seems to be in creating the impact of open data, because for this dimension, Croatia reached only 53%, while the EU27 average is at 71%. The lower impact score for Croatia, especially compared to last year’s score of 85%, may be due to the revision of methodology on how open data reuse is being measured and how the impact of data reuse is created. On the other hand, the highest-scoring dimension is the Portal dimension with an overall score of 81%, where metrics portal features scored 90%, portal sustainability scored 86%, portal usage scored 72% and data provision scored 66%.

The Global Data Barometer project, which aims to measure the state of open data for solving specific societal issues, emerged in 2019 from the Open Data Barometer project. This is a recent study, issued in 2022, that assesses the state of data in 109 countries and is based around four pillars: data governance, capability, availability, and use and impact of data for public good [32]. Governance looks into the open data polices; capability looks into capabilities of governments, civil society, and the private sector to collect, manage, share, and use data; the data availability is all about assessing a wide range of public datasets; while for use and impact, pillar use cases were identified. Unfortunately, only a small number of use and impact indicators were included in this edition; therefore, this pilar had limited weight (only 4%) in the final score [33].

2.2. Open Data Portal Assessment Framework

When it comes to assessing the quality of open data portals, according to Máchová et al. [34], the literature suggests that there are two main approaches. The first approach aims to evaluate the quality of metadata provided on the portals and the datasets themselves, while the authors supporting the second approach emphasize the importance of portal features, feedback, and users requesting specific datasets, because there seems to be a lack of engagement from stakeholders forming an active open data ecosystem.

Pollock [35] was the first to propose the concept of the open data ecosystem in 2011, where a “one-way street” model of data processing should be replaced by an ecosystem approach that consist of different life cycles, and they advocated for sharing data back to publishers and between intermediaries. So, for an open data ecosystem to come to life, there needs to be a constant collaboration between all stakeholders (public sector, citizens and non-governmental organizations, private sector) involved in the process. In this way, the efficiency and transparency of the public sectors can be achieved, as well as the creation of new innovative services and economic growth. Van Loenen et al. [36] argue that the existing open data ecosystem is exclusive; it is government-driven and consists mainly of open government data. There is a need to include users in a more sustainable open data ecosystem, and this can be achieved by focusing on improving the quality of open data portals and encouraging citizens to use the data. The release of data and the design of open data portals should therefore be carefully managed and planned, taking into account the needs of citizens and data professionals.

Recognizing the need to engage users in a more sustainable open data ecosystem, the European Commission published an analytical report in 2020, “The Future of Open Data Portals”, outlining ten ways in which open data portals must evolve to be sustainable and to create value from data reuse [37]. According to [37], the portal should be organized for use, promote use, and be discoverable; in addition, metadata should be published, the use of standards should be promoted, co-located documentation must be provided. It is also necessary to link the data, be measurable, co-locate tools and be accessible. This report was soon followed by another European Commission study “Sustainability of Open (Data) Portal Infrastructures: Open Data Portal Assessment using User-oriented Metrics” [38], which provided exact metrics and methods for assessing these ten user-oriented principles. In [38], the conformity of ten European Open Data portals from different Open Data maturity levels with these user-oriented sustainability principles is also assessed.

2.3. Summary of Open Data Assessment Frameworks

This subsection outlines some notable examples of open data assessment frameworks that can provide an insight into the state of open data around the world, while the next subsection focuses on the open data portal assessment frameworks.

As seen in

Table 1. Summary of open data and open data portal assessment frameworks provided in Section 2, what they measure, their limitations, geographic coverage, last edition and if repeated.

Open Data Assessment Framework	Measure	Limitation	Geographic Coverage	Repeated	Last Edition
Open Data Barometer	Readiness		115		2016 (4th)
	Implementation	Data provider’s perspectives only	30	Yes	2018 *
	Emerging impact		(Global)
Global Open Data Index	Openness	It covers only data publication, and the project is no longer active	94 (Global)	Yes	2016/2017 (4th)
Open Data Inventory	Data coverage	Data publication only	187	Yes	2020 (Ongoing assessment)
	Openness		(Global)
Open Data Maturity report	Policy Impact Portal Quality	Oriented on the policy implementation	35 (Europe)	Yes	2022
Global Data Barometer	Governance Capability Availability Use and Impact	Data provider side	109 (Global)	First edition	2022
Open Data Portal Assessment using User-Oriented Metrics	Sustainability of open data portal	Only one assessment conducted	European Union (10 open data portals)	First edition	2020

* The Leader edition (the report analyzed the readiness, implementation, and emerging impact of 30 advanced open data governments only).

, these frameworks measure different aspects of open data (e.g., readiness, implementation, openness, impact) and have various limitations (e.g., methodology not being regularly updated, project no longer being active, focusing only on the data provider side). Some of these assessment frameworks focus more on the implementation of open data policies and the publication of data rather than on understanding users and their needs. It is not beneficial for anyone to focus on publishing a large number of datasets if those datasets are not used. This is supported by the fact that the European Commission has defined high-value datasets (HVD), of which reuse can have major benefits for the economy and society [9]. Since the user, user experience, and data reuse should be the focus of a sustainable open data portal, we decided to use the metrics proposed by the European Commission in [38] to assess the compliance of the Croatian Open Data portal with 10 user-oriented principles.

Other assessment frameworks discussed in this section do not cover both the user and provider side, and their methodology is often out of date. Metrics used for the assessment should be up to date and meet the ever-changing technologies and users’ needs. The chosen metrics are therefore the most appropriate for assessing how user-oriented the Croatian Open Data portal is. We believe that our assessment will also allow us to identify which metrics may need to be corrected, suggest how they can be improved, and encourage other portal owners to conduct the same assessment. The assessment will show which portal features need to be improved to create a more user-oriented portal that promotes data reuse and increases the impact of open data.

3. Open Data Portals Assessment Methodology Using User-Oriented Metrics

This research adopted the same metrics and methods proposed in [38] by the European Commission. We applied these metrics to assess the conformity of the Croatian Open Data Portal with ten user-oriented principles that open data portals should implement in terms of sustainability and added value, as presented in [37]. The Croatian Open Data Portal was established in 2015 and represents a central node for collecting, categorizing, and distributing public sector open data in Croatia [39].

In [38], the conformance of ten European open data portals (from different open data maturity stages) with ten user-oriented principles was assessed. We compared these assessment results with our Croatian Open Data Portal assessment results. To make this assessment as objective as possible and the results as reliable as possible, the assessment was carried out independently and separately by two authors of the paper and by a researcher from the Faculty of Geodesy, University of Zagreb. The authors of this paper conducted the assessment in November 2022, while the researcher carried out the assessment at the beginning of February 2023.

All assessment results were then compared to determine if there were any differences between them. There were some minor differences in the scoring results, because the descriptions for assigning points for scoring some metrics were not clear enough. For this reason, some points had to be awarded based on the assessor’s (expert doing the assessment) opinion. In addition, the comparison revealed that there were no changes to the Croatian Open Data Portal features that would have affected the assessment result in the three-month period. The only difference in portal features was detected when evaluating the metrics of the principle “Be discoverable”. For this specific metric (in a previous assessment, the portal provided statistics to see if permanent, discoverable URI/URLs were used for each dataset), we decided to use the last assessment result. There were some minor differences in the scoring results, so the average score of all three assessments was calculated for the remaining nine principles.

This section briefly describes each of the ten user-oriented principles with their corresponding metrics adopted from [38].

3.1. Organize for Use

The first principle from the [37] states that open data portals should be “Organized for use”, which is not always the case because user behavior and user experience are often not analyzed. User experience focuses on having a deep understanding of users: their needs, values, and abilities but also their limitations [40]. Therefore, it should be analyzed to provide users with the experience they need. The following metrics from [38] were used for the assessment of the first principle. These metrics take into consideration the user’s needs and give them the ability to quickly identify whether the dataset they are currently examining is useful to them and, if possible, provide more suitable recommendations. In this way, it is not necessary to download and delete datasets if they do not meet the user’s needs. According to [38] (p. 11), the “Organize for use” metric consist of awarding one point for each of the following five elements of the portal:

• Each dataset is accompanied by a comprehensive descriptive record.
• An extract of the data can be previewed.
• The portal provides recommendations for related datasets.
• The portal enables users to review/rate the datasets.
• Keywords from datasets are linked to other published datasets.

3.2. Promote Use

The second user-oriented principle from [37], “Promote use”, discusses promoting the use of the datasets through sharing of knowledge. According to [38] (p. 13), one point is awarded to the portal for each of the following:

• The portal is connected to social media to create a social distribution channel for open data.
• The portal provides users with online support for feedback, to request/suggest the publication of new datasets, and when problems arise during use.
• The portal provides a way for users to keep informed of updates to the data (e.g., news feed).
• Datasets are accompanied by links or resources that provide user guidance and support.
• Examples of reuse (fictitious or real) are provided.

3.3. Be Discoverable

The third principle “Be discoverable” is all about simplicity and efficiency in finding desired datasets. Alexopoulos et al. [41] believe that some open data portals have simple search functions and data filtering features that make it difficult to obtain relevant data. For assessing the “Be discoverable” principle, we used metrics from [38] (p. 13), originally proposed by Walker et al. [42], where points for any datasets are awarded as follows:

• The publisher/owner of the data has an open data portal.
• The publisher/owner of that portal publishes an updated, searchable list of datasets.
• The publisher/owner of that portal publishes an updated, searchable list of datasets with synonyms.
• The publisher/owner of that portal publishes a list of datasets that are known to exist but are not currently available.

3.4. Publish Metadata

The fourth principle, “Publish metadata” from [37], emphasizes the importance of publishing good-quality metadata. Metadata are structured information that make it easier to retrieve, use or manage an information resource [43]. According to Zuiderwijk et al. [44], metadata can improve the discoverability of data, so it is important to provide accurate and complete metadata. The 5-level Maturity Schema for Metadata Management [45] based on Tim Berners-Lee’s Open Data schema [46] was adapted by the European Commission and used to assess this principle. The description and direction for assessing each level on the 5-level Maturity Schema is straightforward and easy to understand. According to [38] (p. 15), the portal receives points based on the place it occupies on the following 5-level Maturity Schema for metadata management [45]:

• Metadata Ignorance;
• Scattered or Closed Metadata;
• Open Metadata for Humans;
• Open Reusable Metadata;
• Linked Open Metadata.

3.5. Promote Standards

“Promoting standards” is the next assessed principle. Standards make it easier for people and organizations to publish, access, share and use better-quality data [47]. According to [38] (p. 16), one points is awarded for each of the following items:

• Permanent, discoverable URI/URLs are used for each dataset (e.g., URI/URLs can be used as universal, unique identifiers by appending a serial number or other internal naming system to a domain).
• The portal uses versioning of datasets (to maintain the history of a dataset).
• Dates are available in a standard format (facilitates the automated exploitation of date-type data and their conversion according to specific needs or constraints).
• Metadata associated with each dataset are available in a standard format to enable automated metadata retrieval and the import of metadata from other data catalogues.
• The metadata catalogue can be retrieved using a standard protocol (e.g., automatic retrieval of the metadata catalogue using RDF or HTTP GET).

3.6. Co-Locate Documentation

“Co-locating documentation” is also required to promote the sustainability of portals. In [38] (p. 17), the metric adopted by Walker et al. [42] was used for assessing this principle. Therefore, we have also adopted the same metric where points are awarded based on portal ranking on the following scale:

• Supporting documentation does not exist.
• Supporting documentation exists but as a document which has to be found separately from the data.
• Supporting documentation is found at the same time as the data (e.g., the link to the document is next to the link to the data in the search).
• Supporting documentation can be immediately accessed from within the dataset, but it is not context sensitive (e.g., a link to the documentation or text contained within the dataset).
• Supporting documentation can be immediately accessed from within the dataset, and it is context sensitive so that users can immediately access information about a specific item of concern (e.g., a link to a specific point in the documentation or the text contained within the dataset).

3.7. Link Data

The next user-oriented sustainability principle is “Link data”. According to [38] (p. 18), the metrics for this assessment have been taken from the Tim Bernes-Lee 5-star deployment scheme for open data, which was proposed in 2010. This is a well-known measure of open data, and points for this principle are awarded based on ranking on the following scale [46]:

• Data are available on the web (in whatever format);
• Data are available in a machine-readable (structured) format;
• Data are available in a non-proprietary open format;
• Use of RDF standards (URIs are used to denote content);
• Linked RDF (data are linked to other data to provide context).

3.8. Be Measurable

The eighth user-oriented principle “Be measurable” is all about providing users and publishers with metrics so both groups can have insights into data that are needed the most and the quality of provided data. According to [38] (p. 20), points for the “Be measurable” principle are given based on the portal’s ranking at the following scale:

• Portal has no analytics;
• Portal has site analytics;
• Portal has use analytics;
• Portal has impact analytics.

3.9. Co-Locate Tools

“Co-location of tools” is another principle proposed that should be implemented to achieve open data portal sustainability. It is important for users to visualize and interact with the data but also to have the ability to discuss with other users that share the same interest. According to [38] (p. 22), the following scale for portal assessment was used (with increasing value):

• The portal does not provide visualization or collaboration tools for users to engage with the datasets;
• The portal provides visualization tools to enable users to engage with the datasets;
• The portal provides visualization and collaboration tools to enable users to participate in the governance of the portal (e.g., dataset rating), but the engagement with other users is limited or mediated by the administrator;
• The portal provides visualization and collaboration tools to enable users to collaborate innovatively with other users.

3.10. Be Accessible

The final sustainability principle from [37] is “Be accessible”. This user-oriented principle is focused on data being available to a wide range of users. This metric does not use a numeric scale but rather a “never, sometimes, always” scale. According to [38] (p. 24), the metrics for the “Be Accessible” principle are following:

• The portal uses human and machine-readable and non-proprietary formats.
• The portal provides different types of formats for the same dataset.
• The mechanisms for accessing and interacting with datasets are documented.
• Multilingual support is available on the portal.
• The portal supports the visually and hearing impaired.

4. Results and Discussion

This section presents the results of Croatian Open Data Portal assessment using user-oriented metrics as well as the discussion of the used metrics. In

Table 2. Ten user-oriented sustainability principles with metrics for assessing each principle and points awarded in the assessment of the Croatian Open Data Portal.

Principle	Metrics/Criteria	Points
1. Organize for Use	Each dataset is accompanied by a comprehensive descriptive record	1
	An extract of the data can be previewed	1
	The portal provides recommendations for related datasets
	The portal enables users to review/rate the datasets	1
	Keywords from datasets are linked to other published datasets	1
2. Promote Use	The portal is connected with social media	1
	The portal provides users with online support for feedback to request/suggest the publication of new datasets	1
	The portal provides a way for users to keep informed of updates to the data	1
	Datasets are accompanied by links or resources that provide user guidance and support	1
	Examples of reuse (fictitious or real) are provided	1
3. Be Discoverable	The publisher/owner of the data has an open data portal
	The publisher/owner of that portal publishes an updated, searchable list of datasets	2
	The publisher/owner of that portal publishes an updated, searchable list of datasets with synonyms
	The publisher/owner of that portal publishes a list of datasets that are known to exist but are not currently available
4. Publish Metadata	Metadata ignorance
	Scattered or closed metadata
	Open metadata for humans
	Open reusable metadata	4
	Linked open metadata
5. Promote Standards	Permanent, discoverable URI/URLs are used for each dataset	1
	The portal uses versioning of datasets
	Dates are available in a standard format	1
	Metadata associated with each dataset are available ina standard format	1
	The metadata catalogue can be retrieved using a standard protocol	1
6. Co-Locate Documentation	Supporting documentation does not exist
	Supporting documentation exists but as a document which has to be found separately from the data	2
	Supporting documentation is found at the same time as the data
	Supporting documentation can be immediately accessed from within the dataset, but it is not context sensitive
	Supporting documentation can be immediately accessed from within the dataset, and it is context sensitive so that users can immediately access information about a specific item of concern
7. Link Data	Data are available on the Web
	Data are available in a machine-readable format
	Data are available in a non-proprietary format	3
	Use of RDF standards
	Linked RDF
8. Be Measurable	Portal has no site analytics
	Portal has site analytics
	Portal has use analytics	3
	Portal has impact analytics
9. Co-Locate Tools	The portal does not provide visualization or collaboration tools for users to engage with the datasets
	The portal provides visualization tools to enable users to engage with the datasets
	The portal provides visualization and collaboration tools to enable users to participate in the governance of the portal (e.g., dataset rating), but the engagement with other users is limited or mediated by the administrator	3
	The portal provides visualization and collaboration tools to enable users to collaborate innovatively with other users
10. Be Accessible	The portal uses human and machine-readable and non-proprietary formats	Most of the time
	The portal provides different types of formats for the same dataset	Most of the time
	The mechanisms for accessing and interacting with datasets are documented	Most of the time
	Multilingual support is available on the portal	Sometimes
	The portal supports the visually and hearing impaired	Never

, a summary of all 10 user-oriented principles with the corresponding metrics/criteria for awarding points was given. The table also includes points awarded to the Croatian Open Data Portal for each metric/criteria of the assessed principle.

For the first principle “Organize for use”, 20 different datasets from the Croatian Open Data portal were selected for the assessment. When selecting these, we decided to use datasets published by different data providers, choosing different thematic categories (i.e., not all datasets are in the thematic category “economy and finance” or “government and public sector”). Ten different data providers from the portal were selected, with each provider represented by two datasets. All these datasets were published recently: in the second half of 2022. A list of the selected datasets is provided in Appendix A.

4.1. Results of the Croatian Open Data Portal Assessment Using User-Oriented Metrics and Discussion

For the first principle, the Croatian Open Data Portal received four out of five points, because the portal fails to provide users with recommendations for related datasets. Most of the datasets evaluated from the portal are accompanied by some kind of a descriptive record, but for a large percentage of datasets, this is just a repetition of the dataset title. Since in [38] it is not specified what a descriptive record should contain, here, even the mere repetition of the title is considered as the presence of a description. Because of that, and since all datasets had some kind of description, they received one point. The portal provides a possibility to link datasets by specific keywords, but not all datasets fulfill this requirement. However, this is not the portal’s fault, since the feature is present, but rather that of the data publisher who does not define the specific keywords. Therefore, one point was awarded to the portal because of the existing capability, even though not all datasets met the required condition.

For the second principle, the Croatian Open Data Portal received the maximum five points rating. Assessment of this user-oriented principle based on the metrics from [38] was clear for most points. However, the metric that should be explained better is “datasets are accompanied by links or resources that provide user guidance and support”. It should be explained in more detail what kind of user guidance or support is accepted. For this assessment, we interpreted this to mean that datasets must include additional information about using a specific dataset, to have a link to the data publisher website or contact information to contact the data publisher. It is important to note that some of the Croatian Open Data Portal features (commenting, requesting new data, and following specific datasets) are only available to registered users. The portal also provides six examples of data reuse, but they are out of date, and new data reuses are not published.

For the third principle, the portal received two out of four points. The Croatian Open Data Portal publishes an updated, searchable list of datasets, but it does not use synonyms while searching, and there is no list of datasets that exist but are currently available. For this principle, we believe that the metrics should be reformulated or explained better. From the metric description, it is interpreted that all data publishers should have an open data portal. We believe that it is not necessary for all public organizations and agencies responsible for publishing data to have their own open data portals, as this would cause even more confusion between users and more time spent searching for datasets. This metric will provide realistic results if points are awarded for each portal that needs to be evaluated rather than for each dataset on the portal. After multiple analyses and reviews of their own evaluation of 10 European Open Data Portals based on this metric, we concluded that the authors of the report intended this metric to be used to evaluate every Open Data Portal and not each dataset on the portal. However, this cannot be concluded from metric description and should, therefore, be reformulated.

For the fourth principle, the portal received four points. Metadata from the Croatian Open Data Portal is centrally managed, in a machine-readable format, and an API is provided for computers to access. It belongs to the category “Open, reusable metadata”, which is ranked 4th in the 5-Level Maturity Schema for Metadata Management. To receive the fifth star on the proposed schema, the use of RDF and URIs should be enabled.

For the fifth principle, promoting standards, 20 previously selected datasets were used to assess the first three metrics, while the answer for the last two metrics was found in the user instructions of the Croatan Open Data Portal. The portal scored four out of five points, since there is no old versioning of datasets available. When first assessed in November 2022, the metadata quality report available on the portal showed that only 79% of all datasets available had a discoverable URL. Therefore, in this first assessment, the portal did not receive a point for this metric, since only 79% of datasets met the desired condition. When the assessment was conducted for the second time in February 2023, this statistic was no longer included in the metadata quality report. Therefore, the availability of permanent URI /URLs was tested for only 20 selected datasets, and only these results were used in the final assessment. Since all 20 datasets had discoverable URI/URLs, one point was awarded to the portal.

The portal received only two out of five points for the “Co-locating documentation” principle, because after assessing the 20 previously selected datasets from the Croatian Open Data Portal, the supporting documents could not be found directly on the portal. This documentation is generally available but is found separately from the data. The metrics for this principle are well described; however, it is difficult to score the portal against these metrics in general. The score largely depends on the specific dataset and data publisher being assessed. This means that the same Open Data Portal may receive different scores depending on the number of datasets and data publishers assessed. For this reason, we believe that metrics should clearly state the number of datasets assessed or indicate that all data made available on the portal should have supporting documentation.

When applying the 5-star scheme for the linked open data principle, it was found that datasets on the Croatian Open Data Portal are usually available for download in a non-proprietary format. That is why the portal received three points for the “Link data” principle.

The eighth principle “Be measurable” was a bit more difficult to assess, as there are no clear instructions on which analytics belong in which category. For this principle, the portal scored three out of four points, because we could not find any metrics that could measure impact analytics.

By using the metrics for assessing the principle of “Co-locate tools”, the Croatian Open Data Portal received three out of four points. The portal provides visualization and collaboration tools (e.g., datasets can be previewed and if the data contain spatial components, it can be displayed on a map), but users are not able to collaborate innovatively with other users.

For the last, tenth, principle “Be accessible” in [38], we suggested using the “never, sometimes, most of the time, always” scale. We believe that the use of a numerical scale for this principle would be more appropriate to be consistent with the scales used previously. Since all nine principles are evaluated based on a numerical scale, it is logical to do the same for the last principle as well. The “never, sometimes, most of the time, always” scale has four possible values that can be translated to digits between 0 and 1. We decided to translate these values on the following numerical scale (never = 0; sometimes = 0.33; most of the time = 0.66; always = 1). This means that if the value “never” was appointed to the metric/criteria of the principle, it is now translated into 0 points on the new proposed scale. Based on this new proposed numeric scale, the Croatian Open Data Portal scored 2.31 points. The results of the assessment showed that data provided on the portal are generally in a machine-readable and non-proprietary format, that there are different types of formats for the same datasets, English language is available in addition to Croatian for some portal features, and that there are some documented mechanisms for interacting with the datasets. What seems to be lacking completely from the portal is support for the visually and hearing impaired.

Table 3. Results of the Croatian Open Data Portal assessment using user-oriented metrics (the table contains the points received for each principle as well as the highest possible score that each principle can achieve).

User-Oriented Principle	Points Awarded in the Assessment	Maximum Possible Points
1. Organize for Use	4	5
2. Promote Use	5	5
3. Be Discoverable	2	4
4. Publish Metadata	4	5
5. Promote Standards	4	5
6. Co-Locate Documentation	2	5
7. Link Data	3	5
8. Be Measurable	3	4
9. Co-Locate Tools	3	4
10. Be Accessible	2.31 1	5 1

¹ The results for this principle were translated to the newly proposed numerical scale.

provides a summary of the results of the Croatian Open Data Portal assessment using user-oriented metrics.

4.2. Comparison with the European Open Data Portals

In addition to providing metrics for assessing ten user-oriented principles, the 2020 report [38] also provides an assessment of ten European open data portals using the proposed metrics: the Cyprus National Data Portal Data.gov.cy [48], Finnish National Open Data Portal Avoindata.fi [49], Belgian National Open Data Portal Data.gov Belgium [50], Slovak Republic National Data Portal Data.gov.sk [51], Portuguese National Data Portal Dados.gov Portugal [52], Icelandic Open Data Portal Island.is [53], EU Open Data Portal data.europa.eu [14], London Datastore data.lonodn.gov.uk [54], National Geoportal of the Grand-Duchy of Luxembourg Geoportail.lu [55], and Open Data Portal Trento dati.trentino.it [56]. Unfortunately, the Croatian Open Data Portal was not among the assessed open data portals from the report, which means that we could not compare our results (from 2023) with the 2020 results, as this is the first time that these metrics are used on the Croatian Open Data Portal.

In this section, we compare the results of the Croatian Open Data Portal assessment shown in Section 4.1 with the assessment results of those ten European open data portals from the 2020 report [38], since the same metrics were used. The comparison was made not to rank the portals with each other but to examine the positive practices, features and functions that have already been implemented by some portals and to determine the aspects that the Croatian Open Data Portal can improve. We want to emphasize that the original report, which assessed ten European open data portals using user-oriented metrics, was published in 2020, while our final assessment of the Croatian Open Data Portal, using the same metrics, was made at the beginning of 2023. For this reason, it is possible that these previously assessed open data portals would receive a different score if evaluated today. Because of this three-year gap, we decided (in March 2023) to do a quick scan of the previously assessed European open data portals from the report to see if there have been any major changes since the report was published. We have not re-evaluated all the principles for each of the portals but rather only some of them to obtain a general overview of the situation and to see if some improvements have been made.

After summarizing our Croatian Open Data Portal assessment results, an average score for all ten European open data portals from the 2020 report [38] was calculated to make a comparison between them (see

Table 4. Comparison of the 2023 Croatian Open Data Portal assessment score and average score of ten European open data portals from [38] using user-oriented metrics.

User-Oriented Principle	Croatian Open Data Portal Score	Average Score of the European Open Data Portals Assessed in 2020
1. Organize for Use	4	2.9
2. Promote Use	5	4.3
3. Be Discoverable	2	2
4. Publish Metadata	4	3.4 (3.8) 1
5. Promote Standards	4	2.7 (3.0) 1
6. Co-Locate Documentation	2	2.4
7. Link Data	3	3.4
8. Be Measurable	3	1.7
9. Co-Locate Tools	3	2.5
10. Be Accessible	2.31 2	2.52 2

¹ The results in parentheses take into account the change in score that our 2023 assessment has discovered. ² The results for this principle were translated to the newly proposed numerical scale.

and Figure 1).

For the first principle, “Organize for use”, the Croatian Open Data Portal received four points, which turned out to be the highest score in this category and brought the portal to the top of the list together with the London Datastore data.lonodn.gov.uk [54] and the Slovak Republic National Data Portal Data.gov.sk [51]. The assessed European open data portals from the report [38] have an average rating of 2.9, from which we can conclude that organizing portals to meet the needs of users is something that should be given more attention in general. Our quick assessment of the London Datastore data.lonodn.gov.uk [54], Slovak Republic National Data Portal Data.gov.sk [51](which scored highest for this principle) and Belgian National Open Data Portal Data.gov Belgium [50] (which scored lowest on the list with one point) completed in March 2023 was conducted to see if some changes were made. The assessment showed that no changes in points occurred; London Datastore data.lonodn.gov.uk [54] still has four points (rating the datasets is not available), as does the Slovak Republic National Data Portal Data.gov.sk [51](there is no recommendation for related datasets), while the Belgian National Open Data Portal Data.gov Belgium [50] has only one point (datasets are accompanied by a comprehensive descriptive record).

For the “Promote the use” principle, all European open data portals scored highly in the 2020 report [38], with five of them receiving the highest score, which is the same as the Croatian Open Data Portal. The average score for this principle is 4.3, while the lowest rated portals from the 2020 report (with three points) are the Belgian National Open Data Portal Data.gov Belgium [50] and Icelandic Open Data Portal Island.is [53]. For this reason, we decided to conduct the assessment again for those two portals. The Belgian National Open Data Portal Data.gov Belgium [50] received three points in our 2023 assessment (we could not find any link to social media on the portal, and the portal does not provide a way for users to check for data updates). Evaluating the Icelandic Open Data Portal Island.is [53] was a bit more difficult, as the portal is only available in Icelandic. However, our assessment showed that the portal again receives three points (we could not find any way of online support and examples of data reuse), as it did three years ago.

The third principle “Be discoverable” turned out to be one of the lowest-rated principles. In our assessment, the Croatian Open Data Portal received two out of four points. All the European open data portals from the 2020 report also received two points. We decided to re-evaluate the EU Open Data Portal data.europa.eu [14], but no changes in the score occurred. In the search bar on the portal, we used the word “water” and narrowed the search to show only results from the health category and from the data.gov.uk catalogue. A total of nine datasets were found, all of which contained the word water in either the title or description of the dataset. This assessment found that the EU Open Data Portal data.europa.eu [14] publishes an updated, searchable list of records, but it does not use synonyms. Since data should be easy for users to find, this generally low score indicates that all open data portals should improve their search engines and provide users with a list of datasets that have not yet been published and their expected publication dates.

The Croatian Open Data Portal scored four points for the “Publish metadata” principle. In the 2020 assessment, only two portals (EU Open Data Portal data.europa.eu [14] and Cyprus National Data Portal Data.gov.cy [48]) received a maximum of five points, while four of them (Portuguese National Data Portal Dados.gov Portugal [52], London Datastore data.lonodn.gov.uk [54], National Geoportal of the Grand-Duchy of Luxembourg Geoportail.lu [55], and Belgian National Open Data Portal Data.gov Belgium [50]) received only two points. We re-assessed this principle for the following open data portals: Belgian National Open Data Portal Data.gov Belgium [50], Portuguese National Data Portal Dados.gov Portugal [52], Slovak Republic National Data Portal Data.gov.sk [51] and Icelandic Open Data Portal Island.is [53]. In our assessment, the Slovak Republic National Data Portal Data.gov.sk [51] and Icelandic Open Data Portal Island.is [53] have scored again four points (both use the CKAN platform, which enables the application programming interface (API) by default), while there was a change in score of the Belgian National Open Data Portal Data.gov Belgium [50] and Portuguese National Data Portal Dados.gov Portugal [52]. Because these portals now provide an API for computers to access, query, and reuse the available metadata and data catalogues, they also scored four points in our evaluation, which is progress from three years ago.

Assessment for the “Promote standards” principle showed that the Croatian Open Data Portal is at the top of the list with three more European portals (EU Open Data Portal data.europa.eu [14], Cyprus National Data Portal Data.gov.cy [48] and Slovak Republic National Data Portal Data.gov.sk [51]) scoring four points, while three of them received only one point. The average score for the evaluated European open data portals in 2020 was 2.7, showing that portals should do more to comply with this principle. We re-applied metrics for this principle on the lowest scoring data portals from the report: the London Datastore and Dados.gov Portugal [52]. Our evaluation showed that the London Datastore data.lonodn.gov.uk [54] still receives only one point (permanent, discoverable URI/URLs are used for each dataset), while the Portuguese National Data Portal Dados.gov Portugal [52] has again showed improvement and now scores four points (the fifth point is missing because the portal does not use old versioning of datasets).

The “Co-locate documentation” principle turned out to be one of the lowest-rated principles in general. The Croatian Open Data Portal is below the European open data portals’ average of 2.4, with only 2 points received. The EU Open Data Portal data.europa.eu [14] is one of the highest rated portals in the report (with three points), so we decided to assess it again to see if any positive changes have been made. It turned out that this grading largely depends on the data publisher. For example, the dataset “Change of immigration status permits by age, sex and citizenship” [57] provided by Eurostat has supporting documentation that is found at the same time as the dataset. On the other hand, on the same portal, the dataset “Charging station Garenne Colombe” [58] provided by IZIVIA has supporting documentation that needs to be found separately from the data. Since there is no information that all datasets need to meet the condition of providing supporting documentation, we decided to give the portal three points.

For the “Link data” principle, the Croatian Open Data Portal scored three points and is below the European open data portals’ average of 3.4. Four of the European open data portals assessed in the report are positioned at a four-star level using RDF standards, and none is at a five-star level, while the rest of them, similarly to the Croatian Open Data portal, are at the three-star level.

The assessment for the eighth principle “Be measurable” showed that this is one of the best areas of the Croatian Open Data portal, while other European open data portals from the 2020 report are not doing so well. The National Geoportal of the Grand-Duchy of Luxembourg Geoportail.lu [55] and Portuguese National Data Portal Dados.gov Portugal [52] are leaders in this category with three points, while five portals are awarded only one point. We decided to check the situation with two of these lowest scored portals: the London Datastore data.lonodn.gov.uk [54] and Belgian National Open Data Portal Data.gov Belgium [50]. Our assessment showed that these two open data portals again scored only one point because they do not have site analytics available.

For the next principle, “Co-locate tools”, the National Geoportal of the Grand-Duchy of Luxembourg Geoportail.lu [55] and Portuguese National Data Portal Dados.gov Portugal [52] are at the top of the list with four points, while the lowest rated portals from the 2020 report are the Belgian National Open Data Portal Data.gov Belgium [50] and Icelandic Open Data Portal Island.is [53]. The Croatian Open Data Portal scored three points in our 2023 assessment, which is better than the European average of 2.5. We re-assessed the lowest rated European open data portal for this principle; the Belgian National Open Data Portal Data.gov Belgium [50] and Icelandic Open Data Portal Island.is [53]. Again, our assessment showed that there were no changes in the 2023 assessment. The portals received only one point because we could not find any visualization or collaboration tools available for users.

The last principle that was compared is “Be accessible”. This comparison was much easier to make after we translated the score from a “never, sometimes, most of the times, always” scale to our new numerical scale. The results show that the Croatian Open Data portal scored 2.31 and is a little below the EU average of 2.52 points. The portal is relatively accessible in terms of data formats and variety, but it does not support the visually and hearing impaired, which is a situation similar to those of the other evaluated portals.

Even though the assessment of the Croatian Open Data Portal was conducted in 2023, while the assessment of ten European open data portals from [38] was conducted in 2020, our quick scan of these portals in 2023 showed that there have been no significant changes in their ratings (Table 4). To determine the exact average of all ten European open data portals in 2023, each principle must be re-evaluated for each portal. However, as our assessment did not reveal any significant changes, the 2020 results are still valid. Our comparison showed that the Croatian Open Data Portal performs well in most of the areas assessed and is below the European average in only two categories (“Co-locate documentation” and “Link data”). This gives a good insight into the Croatian government’s efforts to develop a high-quality open data portal, which is in close proximity to EU trendsetters in the field of open data. The comparison also highlighted some portal features and practices (e.g., using various site analytics, collaborative tools, providing accurate metadata, supporting the visually and hearing impaired) that can be implemented to encourage citizens to reuse data, making the portal sustainable.

5. Conclusions and Recommendations

This article presents an assessment of the Croatian Open Data Portal using metrics developed by the European Commission in [38]. Our overview of existing open data and open data portal frameworks has shown that these metrics are best suited to assess how user-oriented and user friendly the portal is. By conducting our assessment, we were able to compare the results of the Croatian Open Data Portal with ten previously assessed European open data portals from the [38] report. The results of our assessment were based on evaluating a certain number of datasets; therefore, the results for some principles may vary if others were selected. In addition, the compared assessment results of the Croatian Open Data Portal are from 2023, while the assessment results of ten European open data portals are from 2020. However, our quick review of these European open data portals in 2023 showed that no major changes have occurred, so the comparison results are valid.

This assessment helped us to identify the features of the Croatian Open Data Portal that need improvement as well as those that have not yet been implemented. The Croatian Open Data Portal scored the highest for the principle “Organize for use”, followed by “Promote use”, “Publish metadata” and “Promote standards”. Even though “Publish metadata” was rated highly with four points because the metadata catalog is centrally managed and accessible via standard protocol, this assessment showed that the quality of the metadata on the portal is not at a satisfactory level. Very often, basic metadata are missing such as publisher name, contact, release date, update date, and license. Providing better metadata quality would significantly contribute to the reuse of open data. The assessment has also shown that the Croatian Open Data Portal needs to improve the principles of “Be discoverable” and “Co-locating documentation”. One of the main problems with the portal is that users spend too much time trying to find the datasets they want. In addition to the poor quality of the metadata, simple search mechanisms and the lack of a publication plan also contribute to slower data retrieval. For better discoverability, it would be best if the portal included a list of the datasets that are about to be published with the exact dates. In this way, the user will not waste time searching for data that may not even exist. In addition, portal owners should work on implementing the function of providing recommendations for related datasets, which for example is made possible by the London Datastore data.lonodn.gov.uk [54]. This enables the users to find the relevant dataset more quickly. Supporting documentation is also very important, as it contains additional information about the dataset and should be provided directly on the portal for each dataset. It is important to provide users with all the information available. For the Croatian Open Data Portal to be sustainable, it should work on encouraging citizens to use the data.

While these metrics helped us identify the portal’s weaknesses and understand what needs to be improved, their application was not always straightforward and often required technical knowledge. The possible limitation of this study can be found in the fact that assessment was conducted by a limited number of field experts. Future efforts should consider a larger group of assessors with diverse backgrounds, participating in the assessment process. However, this research has helped us identify weaknesses in the existing assessment framework, based on which we make the following suggestions for improvement.

Although the metrics proposed by the European Commission in [38] are generally well structured and combine all relevant metrics from the literature that focus on users and improving user experience and data reuse, we identified certain problems during our assessment. Two main problems were identified when applying the proposed metrics in our Croatian Open Data Portal assessment. The first problem is the lack of sufficient description and guidelines for assessing some criteria of the principle, while the second problem is that some metrics do not have a simple yes or no answer (i.e., 0 and 1 point value) when trying to make an assessment. In addition, this framework requires an assessment not only of the portal but also of the data available on the portal. However, nowhere in [38] is it specified how representative the data sample should be.

For example, the metric “Datasets are accompanied by a descriptive record” for the “Promote use” principle does not specify what that description should include. Therefore, it depends largely on the person who makes the assessment. More objective results would be achieved by specifying that the dataset description should include information that is not simply taken from the title but also provides additional information that helps the user select the correct dataset. Another example is the metric “The mechanisms for accessing and interacting with datasets are documented” for the “Be Accessible” principle. It is not specified what type of mechanisms (document, video, or similar) and whether that mechanism should be made available to data publishers, data users, or both. These examples show us that if the description for assessing some of the principles is not improved, the final score will depend on the person conducting the assessment.

This research has also shown that for the evaluation of certain criteria of the principles, the simple use of 0 and 1 values is not applicable. This problem was noted in [38] for assessing the tenth principle, where a new “never, sometimes, most of the times, always” scale was proposed. However, even though the same problem exists in other principles, this was not detected in [38].

For example, the metric “permanent, discoverable URI/URLs are used for each dataset”, used for the assessment of the “Promote standards” principle, or metric “Keywords from datasets are linked to other published datasets” used for the assessment of the “Organize for use” principle, can clearly have values that are between 0 and 1. The number of datasets used in the assessment also plays a major role. It is not the same whether 5 or 50 datasets are being evaluated. Current metrics for the “Co-locate documentation” principle are awarding points to the portals based on the existence of supporting dataset documentation, meaning that here, the score also depends on the dataset that is being evaluated. This assessment framework should clearly specify the number of datasets that needs to be evaluated and how to assign the maximum score if only some datasets meet this condition. In this article, we have proposed using the new numerical scale, which is based on the “never, sometimes, most of the time, always” scale. We translated these descriptive values in the following numerical scale for awarding points (never = 0; sometimes = 0.33; most of the time = 0.66; always = 1). Our suggestion is to apply this scale to all the principles that can reach values between 0 and 1.

We would also suggest that the assessment of the principle “Publish metadata” should consider the quality of the metadata provided on the portal, as the results based on the current metrics are misleading and hide the real problem of missing metadata.

This assessment was only the first step in our research: to create awareness of the performance and usability of the Croatian Open Data Portal and to identify existing features and practices that should be adopted by portal owners. We will share the results of our findings directly with portal owners so that they can use these assessment results and focus on improving the areas with the lowest scores. The sooner the lowest scoring areas are improved, the sooner the benefits of open data can be realized.

Although the Croatian Open Data Portal received good grades and at first the situation seems to be favorable for the user, the reuse of data is still lacking. This good rating is due to the fact that metrics are focused on assessing how user-oriented the portal is, but they do not consider the quality and relevance of the data on the portal. However, if the data published on the portal are not of high quality and value to users, it is of no use to anyone, no matter how advanced the portal is. Mutual communication and collaboration between all stakeholders forming an open data ecosystem (portal owners, data providers and users) is crucial.

Since publishing data are only useful if being reused, it is important to provide both a user-oriented portal and high-quality and valuable data. These conclusions are in accordance with the new EU Open Data Directive [9] and Data Governance Act [10]. Our future work should therefore focus on providing a new open data assessment framework that combines user needs for a well-organized and sustainable open data portal with the need for high-quality and high-value data.

Therefore, future research should focus on engaging users and encouraging them to collaborate with portal owners and find a way to build trust in data sharing and step from open data portals to an open data ecosystem.