Beyond Open Data Hackathons: Exploring Digital Innovation Success

Abstract

Previous researchers have examined the motivations of developers to participate in hackathons events and the challenges of open data hackathons, but limited studies have focused on the preparation and evaluation of these contests. Thus, the purpose of this paper is to examine factors that lead to the effective implementation and success of open data hackathons and innovation contests. Six case studies of open data hackathons and innovation contests held between 2014 and 2018 in Thessaloniki were studied in order to identify the factors leading to the success of hackathon contests using criteria from the existing literature. The results show that the most significant factors were clear problem definition, mentors’ participation to the contest, level of support to participants by mentors in order to launch their applications to the market, jury members’ knowledge and experience, the entry requirements of the competition, and the participation of companies, data providers, and academics. Furthermore, organizers should take team members’ competences and skills, as well as the support of post-launch activities for applications, into consideration. This paper can be of interest to organizers of hackathon events because they could be knowledgeable about the factors that should take into consideration for the successful implementation of these events.

1. Introduction

Open data is a significant topic both for academics as well as practitioners. Open data are considered to be free, accessible, and reusable without restrictions. Open data have two main motivations. The first one is economic motivation, as the amount of open data held by government or companies have economic benefits for developers, citizens, and other companies in private sector. Governments and companies provide updated data to nascent entrepreneurs in order to develop applications and create new services and improve the effectiveness and efficiency of the existing ones. The second one is social motivation, as governments provide data in order to reform bureaucracy, as well as encourage transparency, accountability, responsiveness, and participation in government [1,2,3,4,5,6,7,8]. Moreover, the accessibility of the information to a wider public, as well as the permission to induce innovative use and application, are some more advantages. Sieber and Johnson (2015) [9] determined benefits for citizens. These benefits enhance transparency, accountability, and civil participation. It is stated that open data support accountability, increasing both trust and citizens’ satisfaction as they can think about alternatives and make better decisions [10,11,12,13,14,15].

Except for the value that open data provide to public administration, it also increases the value for companies and nascent entrepreneurs. Recent studies highlight the benefits for entrepreneurs that use open data. Startups use open data in order to create new business models, as well as increase profitability and competitiveness. Moreover, open data contributes to the development of new products and services, which increase the innovation and firm profitability [13,16,17,18,19,20,21]. Thus, open data initiative must be part of a whole ecosystem in order to achieve these motivations, improve productivity of existing companies or government institutions, and enable the creation of innovative products and services through the use of enabled information technology (IT) platforms [17,20]. An important opportunity for public and private entities to collaborate in order to establish a win-win situation for them is their participation in open data hackathons.

Open data hackathons or innovation contests are events where individuals from different fields cooperate to develop applications that will offer value to citizens establishing a win-win situation for all involved bodies. Unfortunately, many applications that have been developed in hackathons are abandoned, leading them to only provide open data access, which is not enough to increase economic value [22,23,24]. Despite the significance of hackathons, previous researchers have not widely studied the motivations of developers and the challenges of open data hackathons, and limited studies have focused on the preparation and evaluation of these contests [25,26,27,28,29]. Furthermore, these studies have only focused on the implementation of these events, ignoring the factors that contribute to their success. Thus, the purpose of this paper is to identify factors leading to the effective implementation and success of open data hackathons and innovation contests. Six case studies of open data hackathons and innovation contests held between 2014 and 2018 in Thessaloniki were examined. Based on the existing literature regarding the implementation and evaluation of innovation contests and hackathons, six criteria were used to examine the success of hackathon contest. Thereafter, an analysis was conducted in order to identify how each contest responded to the identified criteria.

This paper is organized in five sections. The next section presents theory related to open data challenges and the motivations in hosting open data hackathons and innovation contests. Section 3 describes the methodology of this paper. Section 4 presents the results of the analysis of each hackathon. Finally, Section 4 provides conclusion and suggestions for future research.

2. Literature Review

Hackathons are internal or external events with different purposes or execution processes with common characteristics and structure where small groups cooperate to produce software prototypes in a limited amount of time. The main theme of hackathons is technology, and participants are motivated to take part in these events in order to meet new people and experiment with technologies. Internal hackathons are organized by companies that aim to support participants to generate new ideas in software engineering. Furthermore, participants have the opportunity to learn more about new technologies. These hackathons are innovative-oriented because they focus on the generation of ideas. Although hackathons are usually technologically oriented, this does not exclude participants’ development of prototypes based on hardware. There are also other categories of hackathons depending on application type, application programing interface (API) or language used, or demographic characteristics of participants. Another category of hackathons is civic or socially oriented hackathons that aim to improve public services for citizens. Civic hackathons are time-limited events where workers from the public sector and companies, as well as citizens, collaborate in order to develop software applications that improve citizens’ life [26,30,31].

Hackathons provide many opportunities to organizations because they can test their new products and services, as well as generate new ideas. As the technology which is considered to be provided to developers is considered to be a strong motivation to participate because they will learn something new and useful, participants are energized to develop new applications [27]. Previous researchers have presented many models for hosting open data hackathons or digital innovation contests trying to meet participants’ expectations. In the first phase, organizers have to define the goals and objectives of the hackathon. Then, they have to announce the event through social media, emails, and posters, and send invitations to encourage participation. The third phase includes activities that are related to the preparation of the event, such as the technical infrastructure, APIs, software libraries, Wi-Fi access, physical space, networking facilities, small breakout conference rooms, and logistics of the event. As all hackathons have a reward structure and a set of criteria, these have to be determined in this phase. Last but not least, the jury committee that will evaluate the applications, as well as the sponsors that will fund winners, have to be invited.

The applications which are developed during hackathons remain as property of the developers. They have the opportunity to sell them to the general public once the competition is over. In Helsinki, there are sales channels of mobile applications and online markets (e.g., iTunes and Ovi Store) which support the developers of mobile applications to sell their developed services without significant investments in distribution and permit for practically unlimited upscaling of the usage of the applications [32]. Specifically, Juell-Skielse et al. [18] conducted a survey among participants of an open data hackathon, and they concluded that despite the fact that more than 80% of the teams planned to expand their service further, only one-third had achieved the development after the contest. This could be explained due to the limited support to developers by organizers after the contest. Lee et al. [33] claimed that when organizers in open data hackathons are knowledgeable about developers’ motivations, they could involve entrepreneurs and venture capitalists on the panels of judges. Moreover, they could organize competitions and closing ceremonies that involved potential funders. Thus, developers will have many opportunities to expand their applications because they could discuss about their applications, present them in real-time, and obtain funding.

The existing motivations for developers who participate in digital innovation contests are the fairness of the judgment system, training, collaboration, and new knowledge [27]. Also, fun, enjoyment, intellectual challenge, status and reputation, user need, professional and personal identity, autonomy, learning and skills development, money, extrinsic reciprocity, signaling, and career concerns can be factors that motivate developers to take part in the innovation contests [18] and persuade them to explore ideas that involve high market technical uncertainties [19,26,34].

However, the actors who participate in open data hackathons or digital innovation contests face many challenges which incommode the use of open data. These challenges are related to the lack of local data and utility of open data, lack of technical readiness to use data sources due to complex data format or interfaces, unclear licensing of open data, technical obstacles concerning data publishing platforms, and the capability of users, application developers, and businesses to realize the opportunities that stem from open data in order to use data to develop new services. Other challenges refer to legal issues, such as data which are not ceremoniously open by decision of owner, weakness to derive appropriate data sources for application purposes, guarantee of quality and credibility of data, and a lack in availability of regional data sources for creating applications for local services [2,35,36,37]. Education, experience from users, citizens, and enterprises, as well as maintenance from government, are necessary in order to use open data [17]. Moreover, Juell-Skielse et al. (2014) [18] indicate that the most significant barriers which face organizers and developers during hackathons are the lack of time or money, the lack of marketing competence, the lack of partner co-operation for technical development, the lack of external funding, the lack of information, difficulties establishing licenses for API’s and other services, the lack of technical competence and innovation experience, the high market competition, obstacles in existing service-dependent platforms, and difficulties of finding competent team members.

3. Methodology

As the aim of this paper is to identify the factors that affect the success of open data hackathons or digital innovation contests, success was defined as the accomplishment of an aim or purpose. Organizers are the ones who identify the main purpose of each hackathon and it should be aligned with participants’ expectations. As the research question is “What are the factors that increase the success of an open data hackathon or an innovation contest?”, this paper focuses only on the success resulted from the actual contests and not on the measurement of success after the contests. The case selection aimed at finding cases that vary contextually and involve groups that represent different roles in Thessaloniki’s ecosystem and have different skills in order to compare the implementation process of each hackathon and how specific factors affect their success.

Six case studies of open data hackathons and innovation contests held between 2014 and 2018 in Thessaloniki were examined. The actual environments of six hackathons have been analyzed by both collecting data from the site of these events and conducting informal interviews and discussions with mentors or project managers of these contests and participants [28,38]. Interviews allow the understanding of hackathons and the factors and activities taking place within such environments. Also, the identification of the six factors leading to success was derived from the data gathered from the interviews. The questions were related with six factors that affect the success of hackathons or innovation contests based on the existing literature [26,27,28]. These factors refer to clear problem definition, winners’ rewards, team members’ competences and skills, mentors’ participation to the contest, level of support to participants by mentors in order to launch their applications to the market, jury members’ knowledge and experience, and the entry requirements of the competition.

The events that were examined lasted 1–3 days and organizers were responsible for the planning of the contest (e.g., timing, physical location, technical infrastructure, and logistics) since the day of the actual execution. Then, participants developed their applications, which were evaluated by the jury committee. The purpose of hackathons and innovation contests in Thessaloniki was to inform and encourage developers and citizens to use open data and applications, which were developed based on data in order to increase the benefits both for participants and citizens.

4. Results

The following studied contests were “Hackathess,” “Apps4Thessaloniki,” and “Apps4thessaloniki tourism edition,” which all organized by the Municipality of Thessaloniki in cooperation with the Open Knowledge Foundation Greece, and the Urban and Regional Innovation Research unit. In addition, “Afixis Hackathon” was analyzed, which was hosted by the NGO Afixis. Finally, universities in Thessaloniki hosted two hackathons: “Let’s have a Hackathon!” was organized by the Aristotle University of Thessaloniki, and “Datathon 2018” was hosted by University of Macedonia. In each hackathon, 11–25 teams consisting of 2–4 developers/students participated and developed their projects. Specifically, developers who participated at “Hackathess,” “Apps4Thessaloniki,” “Apps4thessaloniki tourism edition,” and “Afixis Hackathon” came from Greece, the United States, Australia, South Africa, Germany, Cyprus, and Ireland.

Table 1. Overview of open data hackathons and digital innovation contests.

Hackathon	Organizers	Purpose	Expectations
Hackathess	Municipality of Thessaloniki	The development of new applications in order to improve citizens’ life, based on open data	Organizers expect to improve the quality of citizens’ life through the use of developed applications
Apps4 Thessaloniki	Municipality of Thessaloniki	The creation of web and mobile applications that improve different functions of the Municipality and the city	Organizers expect to improve the quality of citizens’ life through the use of developed applications
Apps4 thessaloniki tourism edition	Municipality of Thessaloniki	New opportunities for tourists; New experiences for tourists; Benefits for tourists	Organizers expect to improve tourists’ experiences through the use of developed applications
Afixis Hackathon	Afxis	Creation of innovative educational programs for students	Organizers expect to develop innovative educational programs
Let’s have a Hackathon!	Aristotle University of Thessaloniki	Development of digital capabilities	Organizers expect to develop participants digital capabilities
Datathon 2018	University of Macedonia	Development of ideas and applications to improve daily problems	Organizers expect to develop participants capabilities in order to conduct statistical research using open data

presents an overview of these events.

Each open data hackathon or innovation contest had a purpose. The task was given to participants and they had to develop an application that would meet the purpose of the event (cases 3, 4, 5, 6). In these cases, organizers identified a certain problem within the area of the theme, therefore directing participants to develop an application for that specific problem. In cases 1, 2, and 3, organizers created a platform where citizens could be registered and submit their ideas, which could inspire teams to develop new applications. In this platform, participants were encouraged to share their ideas. This helped participants to develop applications based on citizens’ needs and problems and they created solutions for the improvement of transportation, social services, economic conditions, and government. On the other hand, participants developed an application for the municipality using only open data sources provided by the municipality (cases 1, 2). In these cases, organizers selected topics within the theme (such as smart city, crowdsourcing, and public data) and participants could freely develop any application that would be included in these topics. The availability of open data can in some cases force participants to develop applications based on them. For example, in cases 1, 2, and 6, the use of open data from the local municipality was mandatory, and as a result, participants were required to use them in order to develop solutions.

Participants completed their registration to each hackathon or innovation contest, then created their teams in order to collaborate and develop applications. All open data hackathons and digital innovation contests made use of presentations in order to guide participants toward successful solutions. Specifically, in case 4, mentors and professionals helped participants to develop their applications. After many hours of coding, the developers presented their applications to juries in order to evaluate them. The solution design was not a viable product, but a prototype which included only a minimum set of key features, and it had to provide value to its potential customers and allowed acceptance tests. A final pitch was conducted in front of a jury committee in order to select the most innovative prototypes which should be further developed in order to become a viable) product for customers. In cases 1, 2, and 3, experts, academics, potential customers, mentors, investors, senior managers of local companies, and members of the Municipality of Thessaloniki participated in the jury committee. In cases 4 and 5, academics, companies, and professionals participated in the jury committee, while in case 6, only academics participated.

Table 2. The analysis of the six success factors for open data hackathons and digital innovation contests.

Hackathon	Clear Problem Definition	Winners’ Rewards	Team’s Skills	Mentors’ Participation to the Contest and Support	Jury Members’ Knowledge and Experience	Entry Requirements of the Competition
Hackathess	Well defined	Prizes for winners	Enough diversified competences	Satisfactory communication with participants	Strong jury	Participating teams had to use open data sources provided by the local government and develop applications based on citizens’ ideas
Apps4 Thessaloniki	Well defined	Prizes for winners	Enough diversified competences	Satisfactory communication with participants	Strong jury	Participating teams had to use open data sources provided by the local government
Apps4 thessaloniki tourism edition	Well defined	Prizes for winners	Enough diversified competences	Satisfactory communication with participants	Strong jury	Participating teams had to use open data sources provided by the local government
Afixis Hackathon	Satisfactory	Small venture capital	Well diversified competences	Mentors in place	Strong jury	Team size
Let’s have a Hackathon!	Satisfactory	Voucher Certifications	Well diversified competences	Mentors in place	Strong jury	Team size
Datathon 2018	Satisfactory	Certifications	Enough diversified competences	Satisfactory communication with participants	Satisfactory jury	Participating teams had to use open data sources

presents how each event responded to the six identified factors.

The expectations both of participants and organizers are reflected through these factors. As previously mentioned, organizers identified the theme of each event, and it was communicated to the participants through the event website, presentations that were held, and mentors in place. Thus, participants had the ideal solution in mind. The type of reward may act as a motivation for participants in order to participate to the events. Participants with different skills and competencies cooperated in order to develop digital solutions. A jury committee including academics, professionals, experts, mentors, and members of the local community organizers was able to award the most appropriate solution developed and set any entry requirements for the event.

Although new applications were created, not all cases are believed to have been successful, as the organizing committee did not support the winners to expand their applications and create startups. Each factor, as well as how the examined events responded, are presented down below. Cases 1, 2, and 3 had defined an area per se and participants had to develop solutions based on this area using open data provided by the local community. In cases 5 and 6, the purpose was not so specific, and participants could develop any solution they would think could meet this purpose. Concerning winners’ rewards, all events had some sort of an award for the team with the winning solution which was not the same in all cases. In cases 1, 2, and 3, organizers gave prizes to winners in order to motivate them to launch their applications to the market. Specifically, in case 3, the organizers of the event provided winners with an opportunity to participate at a competition for startups at Tel Aviv. Additionally, in case 4, winners gained a small venture capital in order to expand their solutions and commercialize them. In contrast, hackathons organized by Greek Universities did not provide money to winners. An explanation could be that participants did not develop final applications, but only prototypes. In these cases, it is more difficult for winners to expand their prototypes and launch them to the market. Starting with a concept or prototype means additional software development activities will be necessary in order to develop a solution with a minimum set of features and provide it to customers.

Regarding participants’ skills, developers in all cases had the right competences that were needed for the creation of solutions. Developers with different skills and experiences cooperated, created a team, and competed with other teams in order to generate new ideas, create prototypes, and develop new solutions. It was a big challenge for them to create a team which would work together in such a short timeframe and obtain useful results. All cases had mentors, and practitioners had the opportunity to communicate with them. In cases 4 and 5, mentors played a significant role because they supported developers with the creation of their applications. In other cases, participants could communicate with mentors, but this communication was often short and rarely informative enough. Jury members had expertise to identify the solution needed. In cases 1, 2, and 3, the jury committee included academics, professionals, experts, mentors, and members of the local community. Organizers were more able to award the most appropriate solution developed and set any entry requirements for the event. In cases 1, 2, 3, and 5, the jury committee evaluated winners using criteria such as the quality of the applications, the usability of solutions, the technology that was used, and the extent to which participants’ ideas were innovative. Finally, the effects of entry requirements on the outcome of the contest differ. In cases 1, 2, 3, and 6, participants required the appropriate skills to use open data sources in order to develop applications. In cases 4 and 5, participants required a variety of skills in order to create digital solutions. Participants could not submit their ideas before each contest, so they could not work on their prototypes before the start of the contest.

There are some differences in the planning process among hackathons. Hackathons in cases 1 and 2 were civic and socially oriented. The third case included a single-application hackathon, whereas the last case was an innovation-jam hackathon. Specifically, in case 4, mentors and professionals helped participants develop their applications. Mentors could participate in all hackathons that were examined in order to increase the effective development of applications. Specifically, in cases where participants had to use open data in order to create applications, mentors could help them with the use of datasets. Finally, in the jury committee, different types of judges could participate in all cases in order to holistically evaluate winners’ solutions. Other differences concern the background of participants and the team size. In cases 1, 2, 3, and 6, developers had to use open data, so they required knowledge regarding their use. Furthermore, when participants created prototypes (e.g., in cases 1, 2, 3, 4, 5, and 6), they had to combine a set of skills, which consisted of not only technical skills, but also skills related to product development, marketing, getting funding, etc.

5. Conclusions

The results show that all factors are crucial for the success of open data hackathons or digital innovation contests, but certain factors play different roles. All factors have, in fact, a relation to one another. The identification of the purpose of each contest is a crucial factor because it helps participants to have a guide in order to develop their applications. This factor can affect others, so the clear definition and communication of the purpose of each event can be a win-win situation both for organizers and participants. It is important to identify clear goals and prepare a professional hosting of the event as well as of follow-up activities in order to develop and launch a viable product for the customers. Sponsorship, such as small venture capitals, is required in order to find the right people to be involved in the hackathon and support follow-up activities such as developing, launching, and marketing an innovative product or service [2,3]. Participants should have well-diversified competences in order to cooperate and create prototypes based on the purpose of each contest. Technical knowledge, exclusively, is not a sufficient factor that leads to the success of a hackathon. Each group has to efficiently cooperate and create different types of skills in order to generate an idea, expand it to a prototype, develop an application, and launch it to the market. Another crucial factor that interacts with the previous one is the involvement of mentors during the hackathon in order to help practitioners to create their digital solutions. Mentors could be academics, professionals, experts, or members of the public community. The cooperation among universities, research institutes, and other public organizations is necessary in order to publish usable data and support participants to develop their applications. Furthermore, businesses and consultants should be involved in innovation contests in order to support participants to expand their ideas in services which meet citizens’ needs and gain revenues from them. Both mentors and members of the jury committee should help winners to expand their applications and launch them to the market after the event. In contrast, awards are not a crucial factor which significantly affect the success of a contest. Participants need venture capitals in order to launch applications to the market. Hackathon awards are not sufficient.

The contribution of this paper refers to the factors that influence the planning process and the success of hackathons. This paper can be of interest to organizers of hackathon contests because they could be aware of the factors that affect the effective implementation of these contests, as well as their success. As previously mentioned, these factors are not well-defined in each contest, but each one of them plays an important role in the outcome of hackathons, and each factor is related to the other. Practitioners who focus on the improvement of these factors can overcome organizational challenges and facilitate the innovation process in hackathons’ implementation. Furthermore, the results of this paper shares practical experience with academics and researchers by providing new insights regarding to the preparation, the implementation and the evaluation of contests. It is clear that there is not a single way of how to organize a hackathon or an innovation contest, but the whole set-up must be defined specifically for the needs of each public institution and should even be adapted for each specific use case.

This paper presents an analysis of six cases held in Thessaloniki. A more in-depth analysis of the innovation-driven hackathon pattern and experiences of its application would help practitioners to examine how it could be applied to each hackathon, as hackathons differ in purpose, preparation, execution, and follow-up activities. Furthermore, similar cases from other cities or countries could be examined in order to develop a holistic planning process for organizing hackathons and generalize the findings of this paper. Also, future researchers could measure the success of each hackathon both during the event, as well as after the contest. The existing literature does not provide results regarding the satisfaction of organizers and participants after the end of the contest. The evaluation of developers’ or organizers’ satisfaction could help decision makers to identify the strong and weak points of the events and take specific actions to improve them [39,40,41,42]. It is important to identify the criteria that can measure the success of a hackathon. This could help organizers to plan and implement each event more effectively in a strategic manner.