Fostering Continuous Value Proposition Innovation through Freelancer Involvement in Software Startups: Insights from Multiple Case Studies

Abstract

[Context] The software startups could continuously innovate business model value proposition by involving freelancers as a source of innovative ideas (that enhance customer perceived value) and as experts for implementing the innovative ideas (by undertaking software engineering tasks). Startups employ one of three strategies for associating with freelancers i.e., task based (association ends with completion of the outsourced task), panel based (outsourcing task to a panel of freelancers associated with startup), or hybrid. Uncertainties, terminology issues, high technical debt, lack of documentation, lack of systematic decision making processes, lack of resources, lack of brand values, need for the continuous involvement of the freelancer to incorporate continuous validated learnings, merging freelancer perspectives, and deciding the level of their involvement in individual requirement engineering (or value proposition innovation) activities, are the main inhibitors for associations with freelancers. The availability of good freelancers and their long term and continuous commitments are necessary requirements for value proposition innovation. The theory about freelancer association with the software startups is extended by studying the real practices of startups, which successfully involved freelancers for value proposition innovation by capturing innovative ideas and acquiring the freelancer’s skills to implement those ideas. [Objectives] The objective of this paper is to explain the strategies adopted by the software startups to foster value proposition innovation by continuously involving the freelancers and the way they overcome the challenges arising because of the associations. The findings are driven by the study of real practices of startups that proved to be successful in the market by involving freelancers and continuous innovations leading to increased market shares. [Method] This paper performs empirical studies through case studies of three software startups located in Italy, France, and India, which are at the verge of being transforming into big companies, with large market share. The current practices highlighting the successful way of executing freelancing association strategies for value proposition innovation and the way to overcome the arising challenges are reported. The findings are also compared with those of two young startups based in Switzerland and India, to bring useful lessons for the young startups. The case study results are validated by employees from the studied startups (both those who participated in data collection and those who did not). [Results] The results indicate that freelancer involvement during value proposition activities, which is the core business operation, is beneficial to the both freelancers and the startups. Startup teams could reduce the development costs, shorten time to market, and increase customer satisfaction (by providing features addressing real market needs) by correctly involving the freelancers uniformly across value proposition activities. The startups could manage innovation with small teams (compared to human resources in companies) if done jointly with the freelancers, that helps the team members to learn new skills, upgrade their skills, and learn new perspectives about their markets. Business impacts due to freelancer involvement are stronger if the level of freelancer involvement across various value proposition activities is higher compared to their involvement across few activities only. The studied startups are not completely dependent on the freelancers but the freelancer’s perspectives and skills are valued as a rich source of market success. Freelancer involvement is taken as an opportunity to gain access to global market perspectives, which otherwise would be effortful for in-house teams to collect. In addition, they resolve technical debt, are a source of upgrading skills for undertaking future innovation, and help reaching customers for marketing (promoting product and gaining access to the feedbacks). Overall, the value proposition innovation in the studied startups have different levels of involvement of the freelancers but these startups have reported positive impacts on the business in terms of development cost reductions, shorten time to market, and high customer satisfaction (measured on early attainment of product/market fit and fast growth thereafter). The case study results are validated by the startup employees (member checking). The responses collected are analysed using box plots, which shows a higher level of result agreements among the employees.

1. Introduction

Freelancers are self-employed persons that do not have full time employee relationships with the companies. They temporarily work for a short duration to successfully execute some outsourced tasks. Freelancers could also be involved in the execution of outsourced tasks on a continuous basis, with freelancers continuously innovating their work products by better utilizing their experiences. Freelancers bridge the lack of in-house expertise in the company for the completion of tasks, by providing their services in exchange for monetary rewards. Freelancers are not employees of the companies and, hence, not covered under the employee benefits like tax benefits, insurance cover, social security cover, allowances etc. The companies hire freelancers for individual tasks by selecting them from their professional networks, referrals, and freelancing platforms like freelancer.com (www.freelancer.com), Upwork (www.upwork.com), Guru (www.guru.com) etc.

The different platforms have different ways of associating companies with the freelancers. However, in general, each platform allows a company to upload details of the task to be outsourced (like price, duration, quality expectations, milestones etc.), freelancers to register their interest for the outsourced task, support hiring process in form of access to reputation metrics and conducting interviews, providing mechanisms to support communication between the company and freelancers, payment release, and sharing of the completed work documents. Freelancers through their expertise and skills help make business processes agile as they give them the ability to quickly respond to market changes as reported in [1]. The long term association of freelancers with the companies helps the latter to develop the firm-specific human capital, which brings competitive advantage to the company [2].

Startups could associate with the freelancers by motivating them to be the part of the freelancer panel maintained by them (panel based association), associate just for the execution of the outsourced task (task based freelancing), or use an optimal combination of the two types (hybrid association) [3]. Uncertainties, terminology issues, high technical debt, lack of documentation, lack of systematic decision making processes, lack of resources, and lack of brand values are the main inhibitors for associations with freelancers [3]. To involve freelancers for the value proposition innovation as a source of creative ideas and as experts to implement the high priority ideas, continuous involvement of the freelancers to incorporate continuous validated learnings and merging freelancer perspectives with the startup team learning, further complicates the freelancer involvement [4].

A startup is a novel organization that offers innovative products to the market and searches for repeatable and scalable business models. Often, startup teams have little understanding of the markets which require continuous involvement of the customers (and users) in the software development process. This helps the startup team to continuously perform experimentations to validate their assumptions (also called hypothesis) about business model canvas (like value propositions, customer segments, customer relationships etc.) to continuously improve their understanding of the market (and, hence, the business model). This is equivalent to saying that startups use a lean approach to continuously perform experimentation with customers using the build, measure, and learn (B-M-L) cycle which allows them continuously to improve the underlying hypotheses because of the validated learning brought by the co-creation process with the customers. As the experimentation progresses and new learning enhances the market understanding, the level of uncertainty decreases and the business model evolves to a more accurate version, finally resulting in a model that is repeatable and scalable. Once this happens, the maturity of the startup team with the market enhances, which increases confidence in assumptions about business model aspects and, hence, reduces the uncertainties.

However, after finding a scalable and repeatable business model, startups must maintain competitive advantages in the highly competitive markets by innovating the ways to create, deliver, and capture value. For competitive advantages to be maintained, the startups must continuously innovate, i.e., change the way they work in a manner that generates more value for the customers (or users). Continuous innovation requires ideas from different stakeholders and accordingly it could be closed or open innovation. If the ideas are collected from within the organisation then the innovation is closed innovation and if they are collected from outside the organisation boundaries (i.e., from external stakeholders, especially customers, users, investors etc.) then it is called open innovation. Involving freelancers for value proposition innovation is a form of open innovation.

Innovation, thus, requires a continuous flow of ideas to the startup team responsible for innovation, continuous evaluation of ideas, and implementation of those ideas that enhances the conceived customer value of the product (benefits minus cost). Business model innovation requires creative ideas about a different business model canvas, which affects the customer value directly or indirectly. Value proposition innovation deals with creative ideas about value proposition canvas, i.e., what value the software is providing to the users that will force them to buy the product or service. Moreover, the innovation must be continuous and sustainable.

Value proposition specifies the value that software is promising to deliver to the customers by addressing their problems in an efficient manner (i.e., addressing customer needs by addressing their pain points and enhancing the benefits). For example, software startups could get ideas about value proposition innovation in the form of new functionality to be implemented (value proposition canvas), ideas about how to lower the development costs, or ideas about new programming technology, which improves performance of the software, which directly affects the customers’ perceived value.

Freelancers could be interested in suggesting innovative ideas and providing their expertise in implementing those ideas by undertaking corresponding outsourced software engineering tasks for which startups do not have in-house expertise. Freelancers’ involvement in suggesting creative ideas helps to provide the startup team with different perspectives about the problem domain that serves as a magnifying lens to the underlying problem domain. The startup team could get new directions with the freelancer perspectives and could incorporate the features that match the market needs in a unique way. This paper divides the innovation management into three categories including:

• Invention of ideas related to value proposition. This includes exploring problem domain, identifying new ideas about solutions addressing identified problems, and validating these solutions using prototypes. The ideas in form of the software features could be further refined, prioritized, and documented. This activity is analogous to the requirement engineering activity of the software engineering.
• Implementation of the innovative value proposition ideas. This includes the software engineering activities like designing, coding (both minimum viable product (MVP) and software version), testing, and release to the markets.
• Supporting activities: These include the activities that are neither idea inventing nor innovation activities. These activities support the activities that convert the ideas into working software solutions. For instance, requirement documentation is a supporting activity.

Often, large companies undertake innovation by using intrapreneurship, bootlegging, R&D, internal venture, subsidiaries, joint ventures, venture capital, spin-offs, and crowdsourcing [5]. Startups are characterized by being innovative, having lack of resources, having more uncertainties, working under time-pressure, having small teams, being highly reactive, and rapidly evolving [6]. Software startups have challenges which are unique to them. This includes little or no operating history, limited resources, multiple influences from stakeholders, dynamic technologies, and markets [7]. Moreover, they have a less experienced team, high degrees of uncertainties, and tight market release deadlines. They are not self-sustained, highly innovative, highly reactive to changes, and face rapidly scaling requirements [8]. High level of uncertainties and lack of resources do not make the innovation initiatives like intrapreneurship, bootlegging, internal ventures, and spin-offs feasible in startup contexts. External crowdsourcing (involving crowds of customers and freelancers) could be employed by startups but these will be light weight, flexible customized processes within startups contexts.

The knowledge transfer of innovation practices from large companies to startups is yet to be investigated by researchers. Freelancers could be a good source of not only getting innovative ideas related to creating, delivering, and capturing values but also using their skills to implement these ideas within limited time and cost. The objective of startups is to turn into large organizations in the future, so the innovation practices must evolve into effective processes that could best take advantage of freelancing support for fostering innovations.

Authors in [3] reported through an exploratory study the various freelancing models executed by software startups to involve freelancers, challenges, and issues with such associations and quantitative business benefits brought to the startups. The exploratory study was conducted because literature lacks reported studies about freelancer’s involvement in startups. The freelancer involvement in value proposition activities in two young startups along with their associated challenges are reported in [4]. The practices are yet to mature as startups are yet to mature to turn into big companies and face innovation challenges to keep competitive advantages. The study [4] highlighted that the freelancer’s involvement from the beginning of the startups could help innovate value propositions by providing different perspectives (which are very effortful to collect otherwise) by virtue of their expertise and interactions with customers as startup team members. The freelancer involvement in requirement engineering activities at varying levels depends on startup contexts and is highly influenced by the difficulty to select them optimally, ensuring long term association, building trust, mechanisms to integrate their perspectives, establishing communication, negotiations, and strategic pricings.

This paper performs the exploratory case study to explore how the studied startups successfully involved freelancers for value proposition innovation activities and how they were able to turn the challenges into business opportunities. The same startups were interviewed in a case study reported in [3] but the previous study differs in terms of objectives and validation of their freelancer association practices.

The objective of the previous study reported in [3] was to explore the way the startups involve freelancers, the challenges faced by the startups in involving freelancers, and business impacts of such associations. The focus was on a holistic view of freelancer association rather than specifically on individual aspects of the innovations. The studied startups have successfully involved freelancers for the value proposition innovation, which have helped them to successfully innovate their offered products to continuously capture the changing market needs, compared to the time when data was collected for the previous study. The results are based on data provided by the startups that have attained growth in the markets and are on the verge of turning into big companies. The focus of the case study reported in this paper is to explore the freelancer supported value proposition innovation.

The freelancer association strategies in the previous case study were subjected to continuous experimentation for improvement as the startups were in the process of fostering market growth as they were in the mid of the growth phase of the startup life cycle. At the time this case study was conducted, the startups were able to continuously evolve their freelancer association practices over the time (relative to the time when data for the previous study was collected and companies were struggling to acquire success) by a continuous process of implementing, measuring results, learning from results, and incorporating validated learning to further improve practices. These practices are now more matured and better validated in a real context than those reported earlier. These practices were in the stage of being continuously evolved and their market results were initial during the previous study. The considered startups have involved freelancers for value proposition innovation, an activity that is usually not preferred for outsourcing, as it is the core business operation deciding the future of the startups in the market.

The results of the study are also compared with results reported in [4] about the freelancer involvement in value proposition innovation in the two young startups based in Switzerland and India, to bring useful lessons for the young startups. The sample of startups is limited because the number of startups with large involvement of freelancers in their software development process and innovation practices related to the value proposition is limited.

This paper is structured as follows; Section 2 presents the related work of freelancer involvement in the software startups. The case study protocol that forms the basis of the case study reported in this paper is given in Section 3. The case research design is presented in Section 4 and the cases studied are described in Section 5. The results obtained after the analysis of the collected data are given in Section 6, followed by the overall discussion about freelancer involvement for value proposition innovation in Section 7. The data reported by the case study is validated with the startup employees using member checking as given in Section 8, followed by the concluding of the paper and discussing the future work in Section 9.

2. Literature Survey

2.1. Case Study Guidelines

Authors in [9] presented guidelines to conduct and report case studies in software engineering. The case study research includes five steps, i.e., case study design, data collection procedures, collecting evidence, analysis of collected data, and reporting. Case study design results in the specification of the case study protocol. The protocol contains the various case study elements as planned for the case study like case selection procedures, data collection procedures, analysis procedures, validity, and ethical considerations etc. The data collection methods are executed to collect data from various data sources, which are analysed using various analysis techniques leading to the generation and/or testing of a hypothesis. The data is usually qualitative but small quantitative data could also be collected, which is subjected to descriptive statistical analysis. The results of the case study are then reported to their intended audience using suitable reporting formats. The case study is executed considering validity and ethical issues. The objective is to achieve the research objectives in the most trustworthy manner, satisfying the ethical issues involved in the research. Various checklists associated with the individual case study steps are provided to assist the researcher to conduct the case study in an effective and efficient manner.

2.2. Freelancers in Software Startups

This section highlights freelancer associations with software startups to identify models, issues, challenges, and benefits associated with their involvement with the software startups.

Authors in [3] conducted an exploratory case study involving three software startups located in Italy, France, and India, followed by a survey of 54 freelancers. The 54 freelancers include 21 who are working with the startups and 33 who prefer to not work with startups but had some associations with them in the past. This helps to get perspectives of startups and freelancers (including negative cases, i.e., those who do not prefer to work with startups). Data collection is done using interviews and observations of the cases, the responses of which were analysed using a grounded theory approach based on series of theory building and testing iteratively. The objective of the exploratory case study was to identify the way the startups involve freelancers in software development tasks, challenges that both parties face, and the impacts of such associations on startups. The overall objective was to identify the freelancer and startup collaboration model (or framework) that could provide further directions for research work to boost the associations to create synergies between them.

The results indicated that the freelancer association strategy (or association model) is task based (ending with the completion of tasks to be outsourced), panel based (outsourcing task to the panel of freelancers associated with startup without any employment benefits), or hybrid. The strategies are quite flexible and vary with the maturity of the software startups. The outsourcing is usually competition based and otherwise done on an individual basis due to the urgency of task completion or resource limitations. The previous relations with startups as former freelancers or based on personal relations with startup employees (or founders) is the main basis of individual selections (although there are negotiations related to price and deadlines). Uncertainties, terminology issues (i.e., different software engineering related terminologies used by outsourcers and freelancers), high technical debt, lack of documentation, lack of systematic decision making processes, lack of resources, and lack of brand values are the main inhibitors for associations with freelancers. The availability of good freelancers and long term associations positively impacts startups.

2.3. Freelancers in the Context of Software Development

This section highlights the studies that focuses on the freelancer involvement in software development to highlight the state of art of freelancer supported software engineering.

Authors in [10] conducted a systematic mapping study based on the guidelines, as given in [11], with the objective of providing the state of art in the area of freelancer supported software development and further to identify the focus on software startups. Four bibliographic databases, i.e., IEEE Xplore, ScienceDirect, ACM, and Springerlink were triggered against the string (Freelancer*) AND (“Software Engineering” OR “Software Development”) using the timescale of 2015 to 2020 (both ends included). The major inclusion criteria were the studies based on freelancers (both crowdsourced and not crowdsourced) used in software engineering activities. The result yielded 17 studies, which were subjected to forward snowballing by extending the search over their Google citations, resulting in 19 more studies, leading to 36 total studies subjected for the further analysis.

The results indicated that the studies reported the use of freelancers for generic software development (78%) rather than for individual development activities. The reported challenges include collaboration and coordination (33%), developer recommendation (or selection) (19%), team formation (14%), task recommendation (allocation) (14%), task decomposition (11%), privacy and security (confidentiality) (11%), budget estimation (8%), recognition (8%), trust issues (8%), market dynamics (6%), intellectual property issues (6%), participation of crowd workers (6%), and capacity utilization (3%). These challenges are highly interdependent, and each challenge impacts all other challenges. Most of the studies are validation approaches (72%) followed by empirical studies (25%), and the solutions approaches based on toy (or hypothetical) examples (3%). Recent focus of research (total 7 studies in 2019) is on generic software development handling the collaboration and coordination (3 studies out of 7), developer recommendation (2 studies out of 7), and task recommendation (2 studies out of 7).

2.4. Freelancers for Fostering Value Proposition Innovation

The authors in [4] conducted a case study with two startups located in Switzerland and India to explore the freelancer involvement in the requirement engineering activity. The result indicates that the studied startups involved freelancers for value proposition innovation by utilising freelancers in exploring global customer segments and problem domains, and utilising their expertise in implementing the software engineering activities. Freelancers helped these startups to avoid making mistakes based on wrong perspectives about the problem domain and also to validate the startup’s assumptions about the business model.

The freelancer involvement is inhibited by factors related to freelancer associations as identified in [3], especially those that impact value proposition innovation. This includes the need to optimally establish the freelancer involvement from the beginning of the startups with a promise for long term benefits in exchange for their trustworthy and accurate perspectives. However, freelancers could help the startups to explore the problem domain by representing the samples of globally distributed customer segments as sources of problem domain related information. In addition, customers could become startup team representatives to establish direct interactions with global customer segments.

In general, literature provides evidences (although limited) about the freelancer’s involvement in the business operations by the software startups. However, literature have limited ability to support the startup community through rich empirical evidences as this research area is still in an infancy state, although, it is gaining attention of the researchers these days. Literature does provide the challenges in associating with the freelancers but the challenges specific to the individual software engineering phases is missing from the literature. The solutions addressing these challenges are also too limited to be of any use for the startup community. The freelancer involvement for the value proposition innovation (or requirement engineering) is yet to be studied and reported in the literature. Startups do want to conquer the global markets but limited resources make it hard for them to achieve these objectives optimally.

3. Case Study Protocol

The research paper employs a case study to explore the way the software startups include freelancers for software development activities. The exploratory nature of the case study is necessitated because the literature lacks knowledge about associations between freelancers and companies, implications of such associations and the opportunities to improve their product offering overcoming resource limitations. The case study follows the guidelines as proposed in [9]. The case study protocol uses the template as proposed in [12] and is given below:

Background	The paper tries to reach the following objectives: (a) To identify the freelancer involvement in value proposition innovation in the startups (Obj1). (b) To identify the practices of the startups to overcome the issues and challenges in freelancer supported value proposition innovation (Obj2). (c) To study the impact of freelancer involvement in the value proposition innovation on the customer′s perceived value (Obj3).
	The paper tries to find answers to the following research questions: RQ. 1  How do the software startups managed to execute successfully the freelancer association model in practice? RQ. 2  How do the software startups successfully managed to overcome the issues and challenges in establishing freelancer associations? RQ. 3  What impact do the associations have on the customer’s perceived value as reported in real?
	The research questions were formulated using the PICO (Population, Intervention, Comparison and Outcomes) framework as suggested in [13]. Population: Software startups that are successful in associating with freelancers. Intervention: Involvement of freelancers. Comparison: (not considered to better explore the problem domain). Outcomes: Impact on customer perceived value. The relation between the research objectives and the research questions is shown in Figure 1.
Design	Propositions/hypothesis The case study design is an embedded multiple case study. This means that there are multiple cases that are studied by analysing multiple units of analysis within each case. This research has following design elements: Context: Value proposition innovation at startups. Case: Software projects and individual freelancers. Units of analysis: Startup practices focused on freelancer associations and associated business impacts.
Selection	The cases are selected on the basis of their ability to fulfil the purpose of the research study. Thus, the cases were selected using a purposive sampling technique. Compared to the previous study, two new cases are added and all three cases that participated previously are interviewed again.
Proceduresand Roles	All the researchers were involved in all procedures related to the design and execution of the case study protocol.
Data Collection	Data collection is done using interviews, observations, and the analysis of archival records (meeting notes, online shared project related documents, daily meetings reports). Semi-structured interviews were conducted with 18 participants across three startups. Interview sessions were conducted in an elaborated and interactive manner. Startup founders, co-founders, chief executive officers (CEO), chief technology officers (CTO), and senior software engineers were the data sources. Data triangulation was achieved by collecting data from multiple sources, thereby ensuring the validity of results. Result validity is further enhanced by performing member checking with the startup employees. The interview was recorded and transcribed. The field notes were elaborated at the end of the interview session by series of discussions with fellow researchers and participants. The field notes were prepared during interview sessions and were elaborated after the sessions.
Analysis	The grounded theory approach was used to analyse the qualitative data, which was collected through data collection. The recordings of the interview session were transcribed, coded, codes merged to generate answers to the formulated research questions by iteratively generating and testing hypotheses [14].
Plan Validity	The research focused on treating validity issues with care. The objective was to enhance trustworthiness of the case study. The four threats to validity as proposed in [9] were handled in following manner: Construct Validity: Multiple data sources were used, and study protocol was discussed between researchers and evaluated by the experts. The transcripts were shown to participants to ensure the common understanding of information between researchers and participants. Further, member checking was used to ensure that the case study results are valid, and no new perspectives are left unexplored during the interviews. Internal Validity: Internal validity is associated with the case studies that involve the examination of the casual relationships. As the research is exploratory, internal validity was not the threat to the study. External Validity: The study results apply to software startups, but they similarly hold for any software company. The reason is that non-startups are better placed than startups in terms of their resource capabilities and, hence, they could better use freelancing opportunities. To ensure this, the founders of three startups (who worked at senior positions in software companies) were asked to share their feedback about the applicability of the research results for the big companies. Their feedback supported the applicability. The feedback helped to ensure external validity. Also, the study results will be quite useful and applicable for different startups having different contexts (for instance, internal and external startups). Reliability: The use of data triangulation (use of multiple data sources and instruments), investigator triangulation (use of multiple researchers during research), theory triangulation (use of multiple perspectives to build theory, i.e., Founders, co-founders, chief executive officers (CEO), chief technology officers (CTO)), and senior software engineer’s), and methodological triangulation (comparative data analysis of qualitative data collected from multiple sources and multiple instruments) helped to ensure that the results of the study are independent of the specific researchers. In other words, if another researcher will conduct the same case study by following the case study protocol strictly as mentioned in this paper, then results will be the same.
Study Limitations	This case study could be extended in the future to include more insights brought by more successful startups that involve freelancers for value proposition innovation and those new entrants that emerge with more promising practices. Due to a limited number of startups that publicly announce their dependence on freelancers (may be due to funding issues), the limitation could be these unknown startups may provide some more insights about their practices. However, these unknown startups could be almost negligible as the outsourcing of the core business activity of value proposition innovation is very unusual for the majority of startups.
Reporting	The case study is reported for the startup founders, startup software engineers and project management team to learn lessons from practices of successful startups to get benefitted from involvement of the freelancers to continuously innovate, implement innovative ideas and enhance customer perceived value. This report is also very meaningful to the researchers to conduct more research to provide the startup community the effective solutions to involve freelancers in value proposition innovations.

4. Research Design

The case study research design consists of four steps as mentioned below:

(a)

Pre-study: In this step, an online workshop on “innovation, freelancing, crowdsourcing, and software development” was organized with the employees of the three startups (41 employees of all ranks, including founder). The objective was to make them aware of the objectives of the research study, to take their consent for voluntary participation, to help resolve the terminological differences between researchers and startup team, to arouse participation interest among employees, and finally to establish prior relations with them that could help to establish a friendly atmosphere during interviews.

In this workshop, they were made aware of the terminologies like panel based freelancing model, task based freelancing, hybrid freelancing, crowdsourced models, non-crowdsourced models, software engineering terminologies, freelancers, and gig economy were discussed with short examples. The discussion with them and a short presentation with their representatives (management and technical representatives) helped researchers and startup employees to resolve terminology differences, to better understand the concepts, and to refine the interview guide to be used during data collection. As they have participated in previous case studies, it was very easy to have a common understanding of the terminologies and research objectives.

After the online lecture and discussions, out of 41 employees, 24 were interested to participate in the study. Startup founders of three startups sampled the representatives from their employee list (amongst 24 employees) based on their experience, designations, and ongoing work commitments leading to 18 members to be interviewed. All 41 employees agreed to participate in the result assessment phase to ensure higher accuracy of the results.

This involves founders, co-founders, chief executive officers (CEO), chief technology officers (CTO), and senior software engineers as the representatives of the cases to be studied. Compared to the previous study conducted with the three startups, these startups now have dedicated officers for taking strategic decisions (CEO), and technology related decisions (CTO). Rationale behind involving the CEO was to get information related to the freelancer association policies from management perspectives while involving CTO help to get information from a technical perspective (software engineering related). The role of CEO and CTO was only not found in three grown startups at the time they participated in the previous case study. The role of CEO was undertaken by the founder and that of CTO by a senior software engineer in the studied startups in the previous case study.

(b)

Data Collection: The different methods for data collection as used in this case study include interviews, observations, and analysis of archival records. In the previous study [3], the startups had not maintained any documentation records, which made analysis quite impossible. However, as these startups grew and the number of employees increased, they started circulating documentations in the form of meeting records, daily meeting status on Trello, and shared documents on Google Drive. However, to meet the objectives of the study, they were insufficient to provide another perspective to the information as acquired through interviews.

The manner in which freelancers were involved for generating innovative ideas, the manner they were analysed, evaluated and decided for implementation, and the manner in which freelancers were selected for undertaking technical implementation of the selected ideas (for instance, software engineering activity) was also observed by the first author of this paper for startups A, B, and C.

The semi-structured interviews were conducted very flexibly using an interview guide. The interview guide had background questions and research specific questions, which were asked in zigzag order as per the directions, set by the ongoing interviews. The questions of the interview guide were open questions. Founders, co-founders, chief executive officers (CEO), chief technology officers (CTO), and senior software engineer’s involvement helped to get data from multiple perspectives to reach data triangulation, ensuring research validity by reducing bias. Participants were informed with respect to the issues related to informed consent, review board approval, confidentiality, handling of sensitive results, inducements, feedback as mentioned in [9] in the pre-study phase.

One researcher performed the interviews while another one prepared field notes. The interview sessions were voice recorded, which helped to prepare the transcript of the conversation and elaborate the field notes. Interviews were held with subsequent feedback. Stated in other words, the same representative of the startup was interviewed again to resolve doubts, get more insights, or confirm the interpretation of the researcher. Adding new representatives helped to get more information from another perspective, which helped to gain additional knowledge.

The field notes were discussed between researchers at the same time they were made and discussed with the startup representative, in order to enhance the quality of the notes and ensure its validity. This ensured that researchers and representatives agreed regarding their interpretations and disagreements were resolved by involving a third researcher and more representatives.

(c)

Data Analysis: The grounded theory approach was used to analyse the qualitative data, which was collected. The recordings of the interview session were transcribed, coded, and codes were merged to generate answers to the formulated research questions. This also helped to generate propositions/theory that could benefit other startups with the results. The process of generating hypotheses and their testing continued to be conducted in an iterative fashion as mentioned in [14].

(d)

Result Assessment: The results of this study were sent back not only to the participants of the case study (18 employees), but also to their randomly sampled colleagues who participated in the pre-study phase (and not in data collection) of the research (18 randomly selected employees out of 23 employees). The result assessment includes two groups i.e., G1 (18 employees who participated) and G2 (18 randomly selected employees who did not participate in case study except pre-study phase). The survey’s open questions were emailed to groups G1 and G2. Due to active participation of the startup founders, the researchers were able to get responses from all employees, i.e., all 36 employees participated in the result assessment. Involving group G1 helped to validate the results and ensure that it exactly matched the knowledge that they wished to share. Involving group G2 ensured to validate that no new perspectives were missed and, hence, the results were complete. The responses from both groups were analysed individually and jointly using box plots. Analysis results indicate that the results are accurate, credible, and are able to fully represent their experiences. Results assessment is explained in detail in Section 7. The questionnaire for the result assessment is given in Appendix A.

5. Details of Studied Units

The studied startups include three startups that participated in the previous study [3]. The three startups are based in Italy, India, and France. Compared to the time when data was collected for the previous study, these startups have successfully grown and have successfully turned freelancer opportunities into success results. This case study reports the matured freelancer based strategies of these startups that have worked well for them, i.e., they are tested in real practice. This will help to gain better insight into the journey that these startups undertook to reach successful freelancer strategies. The experiences of the startups in this journey will provide lessons for young startups to better convert freelancer opportunities into business success.

The startup life cycle is divided into three phases, i.e., startup, stabilization, and growth [15]. The considered startups experienced massive growth and, hence, are at the verge of becoming larger companies. The role of founders, co-founders, chief executive officers (CEO), chief technology officers (CTO), and senior software engineers are found in these startups (CEO and CTO are new roles added to their team to manage growth). The characteristics of the startups that participated in the research study as reported in this paper (referred under “Reported Study” column) are given in

Table 1. Startup categories.

S. No.	Start-up Name	Country	Previous Study [3]		Reported Study
			Number of Employees	#Employees Interviewed (Including Founder)	Number of Employees	#Employees Interviewed (Including Founder)	Software Market Category
1.	A	Italy	8	3	10	06	Social sector
2.	B	India	7	2	16	06	Financial markets
3.	C	France	12	2	15	06	Educational software

. To maintain the anonymity of the participating startups, their legal names are replaced with the letters A, B and C.

6. Result Analysis

The study of three software startups through a series of interviews and observations is highlighted for each research question.

RQ. 1 

How do the software startups managed to execute successfully the freelancer association model in practice?

This research question is answered from three perspectives, i.e., sources of freelancer’s information, value proposition innovation activities outsourced to freelancers, and strategies to continuously involve freelancers. The responses given by the studied startups from each perspective is discussed below.

(a)

Sources of Freelancers

The startups identified various sources of getting access to available freelancers for further analysis and selection. As these startups have attained growth in the markets and are about to turn into larger companies, the various sources of freelancer’s information that could be of great help to any startup stage, are given in

Table 2. Sources of freelancers.

S. No.	Freelancer Association Strategy	Sources of Freelancer related Information
1.	Panel Based Approach	Non-crowdsourced approaches Employee referrals, college interns, professional networks, freelancing platforms, freelancer referrals, social networking sites like Facebook and LinkedIn. Crowdsourced approaches Freelancer panel (managed by startups) and freelancing platforms (managed by third parties) (only during early startup lifecycle stages).
2.	Task Based Approach	Non crowdsourced approaches Employee referrals, college interns, professional networks, freelancing platforms, freelancer referrals, social networking sites like Facebook and LinkedIn. Crowdsourced approaches Freelancing platforms (managed by third parties like guru.com, freelancer.com etc.).
3.	Hybrid Approach	Crowdsourced approaches Freelancing platforms (managed by third parties) and freelancer panels (managed by startups).

.

In non-crowdsourcing based approaches, the panel starts from few “known freelancers” and is continuously growing into a big pool of diverse freelancers. The freelancer’s information is accessed from the referrals of the employees and the referrals of the freelancers (who are currently working or have worked with the startups before). The professional network of the founder could also be a useful asset for the information. The profiles of individual freelancers on the freelancing platform could be another source of the information. Social networks like Facebook, LinkedIn etc., are also useful sources of accessing the freelancer profiles.

In crowdsourcing based approaches, the freelancing platforms that are managed by third parties like guru.com, upwork.com, freelancer.com etc., could be an important source of freelancers for startups executing the hybrid and task based approaches (for entire lifecycle stages) and during early life cycle stages for the startups executing the panel based approaches. Freelancer databases, i.e., panels of freelancers managed by the startups executing panel-based approaches, are used and preferred in later life cycle stages.

As per one of the study participants of Startup A, “The level of uncertainties is higher initially with fast failing and fast learning, with the objective to release quickly to the market. The objective is not only to bridge the skill gaps (as the need for particular skills evolves due to pivots) for particular tasks but to establish long term relations with freelancers, to be able to get their continuous support for managing the rework arising because of the pivots”.

As per another study participants of Startup B, “Our effort is not only to select the cost effective freelancer from multiple sources with the job description as the only basis, but the effort is to intrude into his professional network to quickly manage the changes as made during life cycle stages especially initially”. The views of the participants highlight that a large number of sources of freelancer information helps to identify suitable freelancers for the current business needs. However, as there are a large number of pivots (for instance, value proposition hypothesis modifications), efforts must be made to select someone who could provide support for managing these changes on a continuous basis. This support is offered through continuous rework and/or execution of new tasks (that arises as a result of the changes with new skills requirements) by providing access to the network of suitable freelancers with required skills. The overall objective is to realize a short time to market by achieving the following:

• Minimisation of time to locate suitable freelancers as per job description skill requirements.
• Quick execution of the outsourced tasks (rework) to synchronise with the continuous changes brought by the validated learning of market facts as a result of experimentations.

(b)

Value proposition innovation activities outsourced to freelancers

The value proposition defines the benefits that are to be offered to customers by solving their problems. Identifying the value proposition requires continuous interactions with the customers to identify their needs. The solutions addressing the identified needs are delivered to the customer and their feedback is used to drive the further evolutions. The interactions with customers are a continuous process of identifying their needs, validating the learning, and evolving the future solutions driven by the improved level of learning.

This process of identifying and delivering the value proposition to the customers involves the following activities:

• Problem validation or exploration: This includes exploring the problem domain to validate if the initial ideas of the startup are based on the problem observed in the market. As a result of continuous interaction with customers, the startup team will gain a better understanding of the customer problems and this will lead to a better refinement of the abstract ideas.
• Problem/solution fit: The abstract ideas are mapped to suitable solutions, i.e., products and services (and features) that are expected to provide benefits to the customers. There could be many solutions to address the problem as identified in the previous stage and selecting the best one will be an opportunity to harness. Continuous interaction with the customers by showing them the features and observing how they are interacting with the prototypes (if used) will help to understand whether the solution is the right way of addressing the problem. This stage results in modifications of the solutions (detailed ideas), which ensures that problem/solution fit is obtained. Paper prototypes (e.g., mockups) or working prototypes could be used in this stage.
• Product/market fit: In this stage, the software requirements are identified and prioritised from the solutions identified in the previous stage. The Minimal Viable Product (MVP) is created (using software engineering activities like design, coding, testing, release), which is offered to early adopters and continuously evolved based on their feedback and the feedback of other new customers.

The value proposition identification is similar to the requirement engineering activity of software engineering (ignoring the issues with the use of terminologies) [4]. The value proposition is delivered to the customers in the form of a software solution to fulfil the needs of the customers. The value is delivered through distribution channels (like app stores or distributors in case of software DVDs). The software solution is built by following the software engineering process model activities, which include design, coding, testing, and release, which could be performed in numerous ways, represented by models like agile, waterfall etc.

The building of the software solution (i.e., mapping the problem domain into working solution) is the implementation of the innovation, i.e., innovative ideas about value proposition. The value proposition is innovated by continuously identifying new pain points of the users and implementing them as the new software version with enhanced features.

Due to the terminological differences between requirements and the customer needs (the former is more technical and solution domain oriented, while the latter is more nontechnical and problem domain oriented), the engineering activities associated with the value proposition innovation is divided into two categories, i.e., pre-product/market fit and post-product/market fit. The startup life cycle is divided into these two categories on the basis of the product/market fit. This categorisation is used to highlight the key points associated with the customer ability to express problems or requirements (or features).

• Initial startup cycles (pre product/market fit)

• Customer inputs gathering: This activity aims to gather customer needs (pain points and gains). It helps to explore the problem domain.
• Prototype Generation: This activity generates prototypes, which could be paper prototypes like drawings or working prototypes like software code.
• Requirement validation: This helps to ensure the requirement is the right solution for the customer problem (problem/solution fit). This is achieved by using prototypes as the means of communication with customers. For instance, if a working prototype is used then customers could be shown how the idea will solve their problem. The startup teams could also observe the interaction between customer and the working prototype if customers are allowed to access it.
• Requirement Prioritization: The validated requirements are prioritised by the startup team to select the high priority features that could be released to the market. The learning that the team gets as a result of customer interactions supports this decision making process.
• Requirement Documentation: The requirements may be documented. This serves as a blueprint for the software engineers to start with the software design.
• Minimal Viable Product generation (MVP) (Implementation/Coding): The software product with a minimal number of features that are just required to satisfy early adopters is called MVP. The MVP helps the startup to test the actual release in the market and gather feedback to drive further improvement of the product.
• Product validation: The MVP is released to the market and the customer reaction to the launch is witnessed carefully. The sales of the product are an indication of how well the product satisfies its customer segments. The feedback of customers is collected, analysed, and implemented (repetition of previous stages of value proposition innovation). Once enough sales are reported, the product/market fit is obtained.

• Later Startup cycles (post product/market fit)

• Requirement gathering: This activity aims to gather their needs (pain points and gains). Some of the customers specify their needs in terms of software requirements/product features.
• Prototype generation.
• Requirement validation (using prototype).
• Requirement prioritization.
• Requirement documentation.
• Increment implementation (coding).
• Product validation.

The other activities, such as software design, software testing etc., will also be executed in the process of converting the prioritised requirements into working software solutions. The number of such activities depends on the software development life cycle model followed by the software startup.

The value proposition innovation activities consider following important aspects:

• Software features are considered similar to the software requirements in this paper.
• Prototypes are different from MVPs. Prototypes are the system models that are created to foster learning but are not released to the markets like throwaway working software, drawings etc. MVP is also a system model created to gather learning about the product but is released to the actual market and, hence, learning is brought from its actual release to the market.
• Initially, the customers talk in terms of their pains and the gains they expect. Being nontechnical in nature and less familiar with the product, they do not speak in terms of “software requirements” (or software features). They speak by highlighting their problems and it is the responsibility of the startup team to map them to the suitable product features/requirements. Thus, initial startup phases involve a better understanding of their problem domains (in case product/market fit is not attained), which is termed as “customer input gathering”. During later startup phases (after product/market fit), the customers are marginally experienced with the product and some of the customers start talking about their needs in terms of features. Hence, it is termed as “requirement gathering”. This differentiation is made to show the difference in the way the customer specifies their needs, i.e., as problems (during initial startup phases) or as features (during later startup phases). However, some customers may talk in terms of the solutions (or features) during initial startup phases but quantitatively they represent negligible samples of the customer population.
• MVP keeps on changing until the sales grow, i.e., product/market fit is obtained. Here, the business model is considered scalable and repeatable.
• The product is continuously evolved (even after product/market fit) because of the changing customer needs and changes in the environment.
• Thus, the value proposition of the software is always innovated to capture the market and maintain competitive advantages. The innovation could be incremental, component, architectural, or disruptive depending on the idea that provides value to the customers. The ideas for innovating the value proposition could come from employees, customers, vendors, distributors etc. However, the innovations (except disruptive) brought by inputs from the customers are considered in this paper. The reason is that the disruptive innovation is very rare and the studied startups (who are at the verge of becoming companies) are young enough to have encountered disruptive ideas.

These activities are grouped into two categories, i.e., those generating innovation ideas regarding value proposition and those implementing the innovation ideas.

Table 3. Categorisation of software engineering activities.

S. No.	Initial Startup Cycles (Pre Product/Market Fit)		Later Startup Cycles (Post Product/Market Fit)		Supporting Activities
	Invention of Ideas	Innovation Implementation	Invention of Ideas	Innovation Implementation
1.	Customer inputs gathering	Minimal Viable Product generation (MVP)	Requirement gathering.	Increment Implementation (Coding).	Requirement prioritization.
2.	Prototype generation	Any other software engineering activity like designing, testing etc.	Prototype generation	Any other software engineering activity like designing, testing etc.	Requirement Documentation.
3.	Requirement validation		Requirement validation
4.	Product validation		Product validation.

highlights this categorisation.

The classification of the activities among those generating the innovation ideas and those implementing the innovation is done on the rationale that requirement engineering activities will collect, analyse, and prioritise the value proposition ideas rather than implementing them into working code. For instance, prototypes help the requirement team to gain deeper insight into the solutions and to refine them on the basis of customer inputs (or their interactions with the prototypes). So, they are giving more insights about the value propositions rather than implementing them.

Requirement prioritization leads to ranking the requirements without inventing new ideas or their implementation, so it is categorised as a supporting activity. Requirement documentation is also categorised as a supporting activity, as it is neither an invention nor an implementation activity, since its objective is to document the software requirements so that this document can drive the work of software designers. The categorization of some of the software engineering activities as idea invention of ideas, innovation implementation or supporting activities, strongly depends on the objectives reached after their successful execution. For instance, if requirement prioritization is performed with customers, it could lead to a generation of creative ideas and it could be categorised as invention activity. However, in a startup scenario, requirement prioritization is usually done by the startup team informally without the customer involvement, so this paper categorises it as a supporting activity.

The product validation results in rich feedback for fostering future software developments, it is the main source for getting innovative ideas. It is classified in the invention of ideas category. In general, most of the requirement engineering activities that are performed with the customers are categorised as an invention of ideas activity, while other software engineering activities that implement the invented ideas are categorised as implementation activities. The activities belonging to none of these categories are called supporting activities.

(c)

Strategies to continuously involve freelancers

The studied startups involve freelancers across different value proposition innovation activities.

Startup A involves freelancers for problem exploration (customer needs or requirement gathering), prioritization of the problems, and documentation. Freelancers are also involved in the Minimal Viable Product development and evolution of software increment (implementation phase of the innovative ideas).

Startup B involves freelancers for problem exploration (customer needs or requirement gathering) in the form of expert consulting and prototype development (both paper prototype and working prototype). The working prototypes are transformed into actual software code by an in-house team, which results in reducing efforts associated with coding the features.

Startup C employs freelancers for prototype development for validation of the innovative ideas. The value proposition innovation implementation is outsourced to the freelancers and includes model designing (UML) and coding related tasks. In coding related tasks, the freelancers are assigned only those tasks for which in-house resources are not available and/or coding tasks that are very effortful to execute in-house (and comparably less value generating) and continuous refactoring of the code (to improve the software performance).

For refactoring of the code, the panel of freelancers (usually college interns) are assigned the challenge to improve the code fragments. The changes suggested by the panel are selected for final integration and as a replacement of the actual code after a quality check by the in-house team. However, the involvement of freelancers during value proposition innovation is minimal in startup C.

The way the freelancers are involved for value proposition innovation by each startup is discussed below:

Startup A

The value proposition innovation activities are ongoing continuous activities throughout the startup life cycle. Thus, the freelancers are involved from the beginning of the startup phase, with a long term focus. Startup A uses a panel based freelancer association strategy, i.e., the startup prefers to maintain an expert database of freelancers, which could be offered the opportunity to undertake the outsourced tasks rather than searching for them on third party freelancing platforms.

The startup uses the scheme called “percentage share” scheme to motivate the freelancers to join the panel and contribute continuously in various activities. Because they actively participate in problem exploration (and requirement elicitation) activities, the freelancers have a clear understanding of the problem domain (or market needs). Their perspectives are utilised in prioritizing the identified problems (i.e., needs of the market) so that the highest priority problems can be addressed first, considering uncertainties and resource limitations. This lowers the burden of the requirement analysts to map the problem domain understanding into a solution domain (i.e., requirements) due to the ranking of the problems. The task of the requirement team to select fewer requirements, which will be the part of the MVP, becomes less effortful and more accurate.

The rewarding of the freelancer is justified, as startups need to reward those perspectives that bring value to the firm and the customer by unearthing the customer problems. As per one of the startup A participants, “Rewarding is a great way to keep freelancers motivated for continuous involvement to bring only the perspectives about the customer problems addressing which could generate value to the customers. Such perspectives help to avoid failures which otherwise could have been much costly for the startups”.

Freelancers are also involved in the development and continuous evolution of the MVP by identifying new needs and changes requested by the customers (learning). The task is benefitted by the freelancers’ rich understanding of the problem domain (because of their frequent interactions with customers) and solution domain (because of their technical expertise, which makes it easy for them to understand the mapping of problems into requirements). The value proposition activities are not done entirely by the freelancers but the in-house software team jointly works with them. For instance, the MVP is developed by a freelancer together with an in-house developer. This team working is beneficial, as the in-house developer focuses on improving coding practices to address technical debt and gain experience, which could be helpful as the software complexity will grow in the future. The freelancer panel grew in size with the growth of the startup and their continuous support helped them to continuously innovate the value proposition.

Freelancers are also involved in the documentation activity that is usually conducted informally by startups. The requirements selected for the implementation, i.e., release set of the MVP or the next software version, are documented by the freelancers in the database maintained by the startups. This task requires a good understanding of the solution domain and technical expertise. The in-house developer and freelancer jointly work with documentation as well. The freelancers also interact with customers and help the startup to grow their customer base. The interaction not only results in sales, but also the exchange of the feedback that drives the software evolution. The freelancers were invited to participate in the innovation process through non-crowdsourcing based strategies during the initial startup life cycle stages, but they turned into crowdsourced based strategies as the panel size grew bigger with more outsourcing work to offer to the freelancers.

The process model associated with the freelancer involvement in each value proposition innovation activity is briefly explained below:

(a)

Problem Exploration (customer needs gathering and requirement gathering): The “percentage share” scheme works as per the logic formulated as Algorithm 1 (identified as Percentage-share). The pseudocode representation of the Algorithm 1 is given below.

Algorithm 1. Percentage-share.

Pseudocode Let P: Set of user segment problems, U: List of potential users of the software solution and F: Set of freelancers with the cardinality of M. Lp is the list of the common problems identified by the set of freelancers (F). Startup selects the freelancers that are local to the global segments that startups are targeting. For each geographical customer segment (G) (a) Each freelancer submits the list of problems and list of potential users. (b) Lists of potential users are merged and stored as potential users list for future communication. (c) List of problems submitted by all freelancers are analysed and following actions are taken: i. Common problems are identified and set aside (List Lp). ii. Other problems are handed to other freelancers to validate them through interactions with the customers. The problems on top of the ranks of all freelancers and those agreeing with the startup perspectives, are also merged with the list Lp. iii. The problems that startup teams believe to be of high importance but have different rankings in the freelancer’s perspectives, are validated by the team leading to merging them with the list Lp or rejecting them. The list of potential users can also be contacted by the team. iv. The list Lp is subjected to prioritization using the Problem Prioritization algorithm. (d) Each problem in the list Lp (and associated software requirements, when mapped) is linked with the IDs of the freelancers that suggested the corresponding problems. Software solution is released to the promising segments with the features that address the problems in the list Lp. The software solution addressing the problems in the list Lp, if successful in the market, the percentage share of sales is distributed among the freelancers linked to the problems addressed by the solution. End of Algorithm 1.

(b)

Problem Prioritization: The number of problems (even after validation by the startup team) is large enough and could be considered for providing software solutions in a current planned market release. Only a few highest priority problems are considered, while others are put in the backlog for consideration later on. With the growth of the user base, the number of problems also grows considerably in size. The reason for gathering problems rather than focusing on features (or requirements) is that users specify their pain points (rather than expressing requirements, which require solution domain knowledge). In case they specify the features, the studied startups prefer to identify their problem first to better predict the suitable solution for the underlying problem. Freelancers’ perspectives on the problems ranking is merged with the in-house team perspectives to generate final rankings. Algorithm 2 (identified as Problem-prioritization) represents the problem prioritization process employed by startup A.

Algorithm 2. Problem-prioritization.

Pseudocode This algorithm ranks the problems of the users of the selected user segment such that highest priority represents market opportunity. Data Items P: Set of user segment problems with cardinality N. F: Set of freelancers with the cardinality M. ST: Set of in-house startup team with the cardinality S. Share: Set of values with each value representing the cumulative share earned by the particular freelancer. Priority: set of the priority values associated with individual problems. Size of this set equals the size of the set P. Algorithm steps 1.  For i = 1 to M Share_norm[i] = Normalise (Share[i]). End of loop. [This normalises the share values in the range of 0 to 100 by dividing by a large number say 10,000 and assigns it to a new array called Share_norm]. 2.  Assign each problem in set P to each freelancer in the set F. 3.  Each freelancer rates the problem using the scale 1 to 5, with 5 signifying highest priority. 4.  For i = 1 to N For j = 1 to M Priority[i] = ∑ (Share_norm[j] * rating[j])/j. End of inner loop End of outer loop. 5.  Take the pair of two values from the set P and the set Priority. Assign the pair to the set ST for validation. The startup team could agree to the ranking or could modify the ranking as per their perspectives. 6.  Update the set Priority as per the outcome of the set 5. 7.  End of Algorithm 2.

Algorithm 2 first ranks the problem on the basis of the weighted average of the freelancer perspectives about the problem priority. The weights are driven by the monetary share values earned by the freelancers. The rationale is that the freelancer with a higher share of sales reward means higher activity, contribution, and higher agreements between the customer related perspectives and the final outcome (i.e., sales). The ranked list is then displayed for validation to the startup team members, who could either accept the ranking or modify it. The team has a final decision about the ranking. Once they finalise their decision, the final ranks are updated in the system. The team will work on the highest priority problem(s) only due to scarcity of the resources.

(c)

Minimal Viable Product development (and evolution of software increment): The in-house development team maps the high priority problems into solutions, validate the solution using prototypes with the sample of users (existing, referrals, and those drawn from the list shared by the freelancers in the problem exploration activity) and prioritise the number of features. After completing these activities, the developer is selected from an in-house team to work with the freelancers drawn from the panel, to implement the MVP or higher software version. The involvement of freelancer and developer in the same team promotes joint learning, developing a shared understanding of coding standards and quality levels, and keep the team motivated for further innovations. The pricing is made on a per module basis.

(d)

Documentation: The requirement related documents are continuously being maintained by the freelancers. The college interns (part of the panel) are usually given such tasks, which are priced lower than other activities. The quality of documentation is reviewed by the startup team.

Startup B

Startup B employs a task based freelancing strategy. This means that the task is outsourced to the freelancers when the need arises with no focus on continuous involvement of the freelancers. Startup B involves freelancers for problem exploration (and requirement gathering) and prototype development activities of value proposition innovation. The startup used to perform these activities with freelancers using a crowdsourcing based strategy except in the few cases (for instance, strict deadlines) where freelancers in the professional network of the startup team or those referred by the other freelancers (non-crowdsourcing based) were also used. The process model associated with the freelancer involvement in each value proposition innovation activity is briefly explained below:

(a) 

Problem exploration (customer needs gathering and requirement gathering):

The face-to-face interactions with the customers are dealt with by the startup team only. The perspectives of the expert freelancers are also considered by the startup team in form of continuous expert consultancy. In order to associate with the experts of suitable product markets in a particular geographical area, the call for consultancy requirements is made on the freelancing platforms and the prices are specified usually on an hourly or per advice basis. However, the startup mentions in the open call that the consultancy (or expert advice) is required on a continuous basis but will be priced on an hourly basis or per advice session.

(b) 

Requirement Validation (through prototype development):

This activity involves developing paper prototypes (for instance, multimedia documents, drawings etc.) and working prototypes (for instance, software code). The prototype is shown to users to see if the solution is the right match with the identified problem. By using prototypes the team could enlarge their understanding about solutions that could effectively work in the markets to address the identified customer problems. The paper prototype is designed by the freelancers. Freelancers are selected for this task through competitive crowdsourcing using third party freelancing platforms as well as competitions/hackathons organised among college students. These prototypes are validated by the users and are very helpful as the blueprints of the software design to the in-house developers.

Working prototypes are also assigned to the freelancers through crowdsourcing on third party freelancing sites. One prototype per one requirement is designed to felicitate easy integration in the software code and to reduce the development time. The prototype is evolved into high quality code and integrated with the software solution by the in-house developer after the successful validation with the user. This reduces the effort to code and release the next higher software version.

Startup C

Startup C employs a hybrid freelancing strategy. The freelancers have minimal involvement with the value proposition innovation activities related to generating the creative ideas, which is limited to the prototype development activity. The innovative ideas implementation activities supported by the freelancer involvement include model designing and coding related tasks.

The startup uses two models of freelancer association, i.e., panel based freelancing and task based freelancing (crowdsourced version) in parallel. For instance, coding tasks are outsourced to both panels and freelancing sites. If the selection is not done from the panel, freelancers are selected from the pool of freelancers on the freelancing sites. Code refactoring is usually done by college interns that are panel members and usually cost less for this effortful task. This is explained below:

(a) 

Prototype development: The startup involves a hybrid strategy based on panel members and an external crowd of freelancers, to invent the design of the prototypes, which could help to validate the requirements. This startup involves the panel freelancers also, so that their continuous support ensures continuous updating to the prototypes as new learning is integrated in the innovation process. In many instances, the freelancer that provided the startup with the prototype solution undertakes the prototype development of the task that is an evolved version of the older task. Thus, they continuously modify the prototypes into better versions.

(b) 

Model Designing: The startup designs the UML diagrams to graphically present the aspects related to the system. The support of the freelancers is taken for designing and evolving the UML design documents. These design documents reduce the efforts of the in-house team, and they could focus on interacting with the customers, coding, and testing activities.

(c) 

Coding tasks (including MVP): In case the startup does not have in-house resources for coding a particular module or the module is too effortful to be implemented (and not generating much value), the conditional panel based freelancing is used. Efforts are made to outsource it at the lowest price to the panel. In case a larger number of modules are to be outsourced, the startup executes an open call for participation through crowdsourcing platforms along with the conditional panel based association.

Refactoring of the code is done on a frequent basis with the involvement of the members of the panel of freelancers. The outsourcing call sets the lowest price for such tasks, so usually the college interns agree to undertake these tasks in exchange for a small token of money and internship experience. Their outcomes are reviewed by the in-house team and by the crowd of the other panel members (usually college interns). In case of a positive outcome (or after suggested modification), their code replaces the original low quality product.

A summary of freelancer involvement for value proposition innovation in three studied startups is given in

Table 4. Summary of freelancer involvement.

Startup	Activities	Category	Involved Perspectives	Pricing	Level of Dependency on Freelancers
			(F: Freelancer & S: Startup Team)		Individual Activity	Overall Value Proposition Innovation
A	Problem Exploration	Invention of ideas	F & S	Percentage share of sales.	Maximum	Invention: Maximum Implementation: Maximum
	Problem prioritization		F & S		Maximum
	MVP	Implementation of ideas	F & S	Pricing per module to be coded.	Maximum
	Documentation	Supporting activity	F	Prices set per hour/per task.	Maximum
B	Problem Exploration	Invention of ideas	F & S	Prices set per hour or advice session.	Low	Invention: Medium Implementation: None
	Requirement Validation (Prototype Development)	Invention of ideas	F	Per work output.	Maximum
C	Requirement Validation (Prototype Development)	Invention of ideas	F	Per work output.	Maximum	Invention: Low Implementation: Medium.
	Model Designing	Implementation of ideas	F	Per work output.	Maximum
	Coding	Implementation of ideas	F	Per module to be coded or refactored.	Medium (Refactoring + few coding tasks only)

.

The reasons for the dependency of the startups on the freelancers for undertaking set of value proposition activities are discussed below:

(a)

Startup A: The startup employs the freelancer’s active involvement in almost every value proposition innovation activity (both invention of value proposition ideas and its implementation). This is the reason for the level of involvement specified at a rating of “maximum”. However, the freelancer’s perspective is always merged with the learning gained by the startup team. The reason is to lower the in-house efforts and enhance the success chances by getting multiple perspectives about the customers. Freelancers are also employed in one of the supporting activities, i.e., requirement documentation.

(b)

Startup B: Startup B involves the freelancer in two main activities of the identification of the innovative value proposition. Freelancers are not used for the implementation of the innovation (for instance MVP or coding). The overall level of involvement for invention is “medium” and for implementation is specified at a rating of “none”.

(c)

Startup C: Startup C involves freelancers minimally for the identification of the innovative value proposition. However, the freelancers are involved in implementation related activities like designing and coding. Only few coding tasks of new features are outsourced to the freelancers and the remaining are undertaken in-house only. The reason for continuous outsourcing of the refactoring tasks to the freelancers is to make code better and avoid any technical debts, also to provide value to the customers. Due to less involvement of the freelancer in invention activities, the ratings of involvement are “low” and during implementation it is “medium”.

RQ. 2 

How do the software startups successfully managed to overcome the issues and challenges in establishing freelancer associations?

The major challenges in establishing associations with freelancers for generic activities (software development or business operations like marketing) and for value proposition innovation (or evolving requirement engineering) is shown in

Table 5. Freelancer association challenges.

S. No.	Type of Activity	Reference	Challenges
1.	Generic Activity	[3]	Task description issues which include price setting, deadline estimation issues, ambiguously defined tasks, poor software artefact support to enhance understanding etc. (Task Description) Selection of freelancer from the pool of diverse options. (Selection) Coordination, collaboration and communication. (3C) Probabilistic nature of outsourcing needs. (Probabilistic Arrival) Demotivation to existing employees. (Employee Demotivation) Evaluation of the freelancer work output. (Evaluation) Trust issues (for both startups and freelancers). (Trust Issues) High technical debt that limits future outsourcing of tasks. (Technical Debt)
2.	Value Proposition Innovation.	[4]	All issues as mentioned under generic activity. Additional challenges (specific to value proposition innovation) include: Ensuring optimal long term association for continuous rework arising because of continuous learning (low cost, high quality involvement of freelancers for continuous services). (Continuous Rework) Merging freelancer perspectives with the validated learning that startup teams have (collected through direct interactions with the customers). (Perspective Merging) Difficulty in negotiations due to startup reputation issues. (Negotiations)
		As per interviewed startups (A, B and C)	Additional challenges (specific to value proposition innovation) include: Decision about outsourcing value proposition innovation activities as opposed to undertaking them in-house, which activities to outsource and so on. Value proposition activities are core business activities. Outsourcing decision must resolve trade-offs between current needs of the startups and long term business impacts of the outsourcing. (Outsourcing Decision) Deciding the level of reliance on the learning shared by the freelancers (or level of participation of the freelancer). (Dependency on Freelancers) Decision about outsourcing learning as opposed to opportunity to attain the learning as per market facts by the startup team in-house. (Learning Issues)

.

The challenges in involving freelancers during software development become more complicated for the startup teams if freelancers are also involved in value proposition innovation activities (due to the challenges specific to the freelancer supported value proposition innovation/requirement engineering). The freelancers could be involved to identify the innovative ideas about the value proposition and also for their implementation through software engineering activities. The existing startups handle the freelancer association challenges as mentioned in

Table 6. Strategies to overcome freelancer association challenges.

Startup	Challenge	Method to Overcome Challenges
A	Task Description	Technical expertise of the freelancers makes it possible to specify the features to be implemented in the MVP. Standard template to report the customer needs, makes it possible to overcome the task description issues. Same value proposition activities are executed continuously with the same templates, so the understanding of the outsourced tasks is always higher.
	Selection	Panel is created by starting with the freelancers in professional relationships with the startup team. As the freelancers start earning money, others are motivated to participate. Same activity is performed by multiple freelancers. The redundancy helps to overcome the wrong selection decisions.
	3C	Replication of the problem exploration by multiple freelancers, use of standard templates and weekly meetings, helped to achieve 3C.
	Probabilistic Arrival	This issue was not reported. This is because the startup executed the same value proposition activities repeatedly.
	Employee Demotivation	Joint work of freelancer and in-house teams helped to motivate both parties. Moreover, the freelancer perspectives are always validated by the perspectives of the in-house team, so the in-house team always remains motivated.
	Evaluation	The work is evaluated by the validation by the in-house team. Live evaluation is done by measuring the product sales on the basis of how the freelancers are awarded a percentage share of the sales.
	Trust Issues	The reputation of the founder helped to establish trust of freelancers in the startup. Due to the involvement of a large number of freelancers and involvement of the in-house team, the startups’ product offering is less vulnerable to few non-trustworthy freelancers.
	Technical Debt	Coding performed jointly by freelancer and the in-house developer. Documentation is always kept updated by outsourcing documentation tasks to the freelancers.
	Continuous Rework	Monetary reward in terms of percentage share and price for MVP development helps to involve freelancers continuously. Moreover, an in-house team is involved, so they gain sufficient experience that makes them self-reliant in undertaking the value proposition innovation tasks.
	Perspective merging	The freelancer’s perspectives are merged with in-house team perspectives. In case of contradictory perspectives, the other freelancer perspectives are also considered. The in-house team also does the research with the market to bring fresh perspectives to accept, modify, or reject the conflicting freelancer perspectives.
	Negotiations	Percentage share helps to motivate freelancers to be the part of the panel. If freelancer perspectives appear to be true, then only they are to be offered percentage share of the sales. It is a win-win situation for both parties. As the size of the panel grows, crowdsourcing makes the negotiations better.
	Outsourcing Decision	Same activities are outsourced, and this integrates the freelancers in the value proposition innovation process model in the startups.
	Dependency on Freelancer	Freelancers’ perspectives are always validated with the startup team perspectives. The dependency on freelancers is maximum, but the startup team also performs the same tasks themselves.
	Learning Issues	The learning in the startup is not completely outsourced to the freelancers. Freelancers’ learning provides different perspectives to the startup team learning. The team also learns by interacting with the customers and also uses the freelancer perspectives to create a holistic view of the system.
B	Task Description	Task related to consultancy and prototype development is easiest to design. The market prices are used as a benchmark for establishing the prices.
	Selection	Freelancers are selected from the crowdsourced platform on the basis of their ratings, sample of work available and short interviews. It is also possible to establish competitive crowdsourcing which gives a wide array of options to the startups to select and reward the best prototype. The working prototype is associated with the single software requirement, so the complexity of the code is small, which allows it to be outsourced through competitions. Otherwise, an in-house team will evolve the working prototype into actual software code, which means that bad selection of the freelancer does not affect the code quality.
	3C	The tasks are less effortful and less time consuming. Less coordination is required, and the freelancer submits the output after completion of the work on freelancing sites.
	Probabilistic Arrival	The consultancy advice and prototype development are outsourced continuously, which means that they do not have probabilistic arrival.
	Employee Demotivation	Freelancer work is just a small supporting work, which is evolved into final high quality code. Employees are provided help by freelancer involvement, which is not any cause of demotivation.
	Evaluation	The advice can be evaluated by comparing the actual outcomes with the advice. Prototypes can be evaluated by the in-house team on the basis of their quality, in order to support evolution into final code and ability to support customer interactions.
	Trust Issues	Freelancing websites provide data that drives the establishment of trust factors. The previous ratings of the startups help to foster freelancer trust into the startups.
	Technical Debt	The prototype is evolved into final code by the in-house team. Due to freelancer involvement, the effort with coding is much reduced. The in-house team can ensure high quality to avoid technical debt.
	Continuous Rework	The value proposition innovation activities are executed mainly by an in-house team. They are responsible for continuous rework. For prototype development, the continuous involvement of freelancers is an effortless task.
	Perspective merging	There is no need to merge the perspectives for prototype development. The consultant perspective could be taken as one direction, which must be carefully researched before being traversed.
	Negotiations	Crowdsourcing freelancing sites helps to get different rate quotes for the task to be outsourced. The lowest bid could be negotiated with.
	Outsourcing Decision	Same activities are outsourced, and this integrates the freelancers in the value proposition innovation process model in the startups.
	Dependency on Freelancer	Startup is less dependent on the freelancer as major activities are executed by an in-house team.
	Learning Issues	Major activities are taken in-house so the startup team experiences maximum learning. Learning is not outsourced to the freelancers.
C	Task Description	Prototype development, model design, and refactoring of the code are usually outsourced repeatedly. Low prices are set among the panel members as there is a high likelihood to find cost effective solutions in the panel only. The bids obtained from the freelancing websites are used in case of failed search in the panel.
	Selection	Outsourced tasks are less effortful and less challenging. Refactoring results are checked by the in-house team before being converted into final code. Freelancers are selected on the basis of the crowdsourcing call among the panel members and freelancing websites. The large size of the pool helps to get a wide range of the prices and the lowest price is selected.
	3C	The tasks are less effortful and less time consuming. Less coordination is required, and the freelancers submit the output after completion of the work on freelancing sites.
	Probabilistic Arrival	The tasks are outsourced continuously, which means that they do not have probabilistic arrival.
	Employee Demotivation	Employees validate the refactoring code generated by the freelancers. Prototypes are also validated by the in-house team. The support is the help provided to make future work of the in-house team less effortful. Employees are not demotivated by freelancer involvement.
	Evaluation	Prototypes can be evaluated by the in-house team on the basis of their quality to support customer interactions. Refactored code is evaluated on the basis of use of coding standards.
	Trust Issues	Freelancing websites provide data that drives the establishment of trust factors. The previous ratings of the startups help to foster freelancer trust regarding the startups. Panel members are always having higher trust levels regarding the startups.
	Technical Debt	Refactoring helps to streamline the future developments and reduction of technical debt. Designing UML models helps to document the system related information that promotes system understanding and removes debts.
	Continuous Rework	The value proposition innovation activities are executed mainly by an inhouse team. They are responsible for continuous rework. For prototype development, the continuous involvement of freelancers is an effortless task.
	Perspective merging	There is no need to merge the perspectives for prototype development. The consultant perspective could be taken as one direction, which must be carefully researched before being traversed.
	Negotiations	Crowdsourcing freelancing sites help to get different rate quotes for the tasks to be outsourced. The use of both crowdsourcing platforms and the panels helps to control the prices. The lowest bid could be negotiated with.
	Outsourcing Decision	Same activities are outsourced, and this integrates the freelancers in the value proposition innovation process model in the startups.
	Dependency on Freelancer	Startup is less dependent on freelancers as major activities are executed by an in-house team.
	Learning Issues	Major activities are taken in-house so the startup team experiences maximum learning. Learning is not outsourced to the freelancers.

.

RQ. 3 

What impact do the associations have on the customer’s perceived value as reported in real?

The association with the freelancers helped the startups to identify innovative value propositions continuously and implement innovative ideas to bring value to the customers. Involving the freelancers, the startups were able to undertake value proposition innovation at less cost and in less time.

In case of startup A, a maximum use of the freelancers resulted in a smaller number of in-house teams, resulting in the reduction of the costs. The benefit of cost reduction was transferred to the customer as the reduction in the prices. An increased use of the freelancers helped them to enlarge the search over the problem domain to provide features that address high priority problems. This enhanced customer perceived value as benefits increased and prices were reduced due to the reduction in the development costs.

In startup B, the use of consultants helped them to reduce the costs with continuous experimentations with the customer segments as expert advice provided necessary guidelines for problem exploration with fewer mistakes. The prototype development at low costs helped to quickly validate requirements and convert working prototypes into working code in less time and effort. This reduced costs and time to market, thereby increasing customer perceived values.

In startup C, the prototype development at low cost helped to quickly validate the requirements with the customers. UML models helped to capture the system related information, which reduced technical debt. Refactoring improved the code performance and reduced technical debts. The validation with customers helped to enhance product success chances and refactoring helped to improve product performance, which jointly enhanced customer perceived value.

As per the estimate of the startups, they are able to achieve good market shares in less time only because they captured customer problems, provided efficient solutions to address the problems, and continuously innovated the value proposition with the help of freelancer involvement. Freelancers helped them to save costs with exploring problem domains by face-to-face continuous interactions with the customers, implementation related activities at a low cost and in less time. The cost and time saved by freelancer involvement is approximately mentioned by the startups, which is shown in the

Table 7. Impact of freelancer association on customer perceived value.

Startup	Activity	Cost Saving	Time to Market Reduction	Time to Product/Market Fit
A	Invention of ideas	€ 1500	6 months	6 Months
	Implementation of Ideas	€ 10,000
B	Invention of ideas	₹ 320,000	4 months	6 Months
	Implementation of Ideas	-------
C	Invention of ideas	€ 2500	3 months	8 Months
	Implementation of Ideas	€ 1000 (low saving because few coding tasks are outsourced)

.

Cost saving is calculated on the basis of the expenditure on the activity if the in-house team has to solely execute the activity. The effort with the activity, per day salary of the employee, and outsourcing task price are considered to calculate cost saving.

Time to market reduction is calculated on the basis of the average reduction of the software delivery time with respect to the time that would have been incurred if the activity had been executed in-house only. It is the average time saved for all market releases.

Table 7 shows that startup A is able to have cost savings and time reduction to the market. Reduction in the cost benefit could be transferred as the reduction in the prices, which increases perceived customer value. Startup B is also able to reduce cost and time to market because of avoiding too many failures during identifying a value proposition’s fit with the market by freelancer involvement. Although the level of freelancer involvement was not high during the implementation phase, startup C was not able to reduce costs like the startups A and B managed because it had little involvement of the freelancers in ideas invention activity of innovation and had average involvement in ideas invention activity (i.e. outsourced few coding tasks infrequently to the freelancers which resulted in small cost reductions). Outsourcing of the refactoring will provide benefits in the future, the measurement of which will take some time. In the near future, the savings for startups would grow as the complexity of the system will grow and refactoring benefits will become evident.

All the studied startups reported that freelancer support was meaningful to accurately understand the problem domain and efficiently execute implementation tasks. They attained the product/market fit in an average of 6.7 months and then grew quickly in the market.

One important observation is that the savings in the development cost and time to market does not depend on the level of freelancer involvement across a few activities, but it depends on the level of involvement uniformly across various value proposition innovation activities.

7. Discussion

The startups can involve freelancers in different value proposition activities. The involvement could also signify different levels of dependencies on them for future innovations. This means that startups could either completely depend on the fresh perspectives brought by the freelancers and their skills for implementation of the innovative ideas, or they could take their inputs as mere opinions for validating the startup team assumptions about the markets. The overall objective is to convert their participation into an opportunity of success, growth, and learning using limited resources, managing market uncertainties and meeting tight release deadlines. The partial dependency on the freelancers means that the startup team executes all value proposition activities in-house, with the involvement of the freelancers. This helps the startup team to avoid outsourcing the value proposition activity which is the source of learning to the startup team and a way to establish professional relations with the potential customers. The different perspectives brought by the startup team and the freelancers, provide the startup team with a holistic view of the solution that has the capability to satisfy the market needs in an efficient manner. The diversity in the perspectives is a major success factor for fostering innovation in startups. The freelancer perspectives act as a guiding road for the startup team that could help them avoid unnecessary failures and prevent wasting the resources on the experimentations that could otherwise be avoided with market facts represented by the freelancer perspectives. For instance, in a previous study [4], the Swiss startup avoided the mistake of considering the wrong customer segments because of the freelancer perspectives, which served as an initial red flag. The startup team could merge their learning with the perspectives brought by the freelancers to follow the unified perspective that is supported by strong evidence of representing the market.

The startups serve one or more customer segments with a niche offering for segments that may be globally distributed across the globe. With app stores and other digital platforms as distribution channels for the software industry, it is possible to cater to the needs of the big customer segments across global markets. The identification of the repeatable and scalable business models requires the startup teams to establish direct communication with customers to better understand their problems, which is very effortful with global segments. Freelancers can use their understanding of the segments local to them, their experience in the similar industry and their interactions with the customers as startup team virtual members. This may help the startup team to collect the problems of global segments in an efficient manner. These perspectives could also be modified by the learning brought by the startup team driven by their interaction with global customer segments.

The implementation of the innovative ideas is no longer challenged by the lack of skills in-house as the freelancers could undertake the execution of the software engineering activities for implementing the innovative ideas, bringing the innovative features early to the market. The joint implementation of the innovative features by freelancers and in-house developers establishes trust, learning, and quality (by following coding practices). The software team does not compromise the learning acquisition in the process of innovation, and they keep on building skills that could help them to innovate faster in future. The involvement of freelancers for undertaking the activities like documentation, refactoring and so on, helps to support the future innovations by improving software performance and the ability of the code to be modified in the future.

The joint involvement of freelancers and the startup team for inventing and implementing innovative ideas for value proposition innovation, helps the startup to grow fast in the market (for instance, higher sales leading to capturing large market shares), reduce development costs, and reduce time to market. The startups should not be completely dependent on freelancers but should involve them as team members, motivating them to bring fresh perspectives, using such perspectives for validation and rewarding the promising ones driven by real market success. The reward mechanism should motivate the freelancers for their continuous involvement during the innovation activities, which require a lot of rework due to continuous learning brought by continuous experimentations with the market. The panel based freelancing model seems to provide the startups with the advantage of the long term associations with continuous involvement as the panel strategy will have more trust between freelancers and startup teams [3].

The uniform involvement of the freelancers in all value proposition activities (both inventing ideas and implementing ideas) is having higher business impacts compared to nonuniform involvement of the freelancers across few activities. From the studies of startups, the business impacts are higher if freelancers are involved in inventing ideas compared to just implementing ideas. The business impacts grow with a stronger involvement of the freelancers across inventing and implementing activities. This means that the business impacts do not depend on the level of freelancer involvement across a few activities, but it depends on the value proposition innovation activities, which involved the freelancers, and on how uniformly the freelancers were involved across different activities. For instance, startup B has more freelancer involvement in ideas invention activities compared to startup C, which has more freelancers involved in the implementation phase. Yet, startup B is able to have better business impacts compared to startup C. If startup B involves them during the implementation phase and startup C involves them more during invention phase, they will have better business impacts.

The challenges associated with freelancer association and the ones associated with the involvement in value proposition innovation activities are successfully managed by the studied startups. Providing the monetary rewards based on actual market performance (like percentage share in case of startup A), continuous involvement in similar value proposition activities, virtual team establishment with in-house team involvement along with freelancers, valuing freelancers perspectives, involving freelancers for documentation and refactoring related activities, to name a few, could help to overcome various hurdles in freelancer involvement. Panel based freelancing provides better support for promoting trust between freelancer and startup teams that help to involve them continuously for innovating the product to grow quickly in the markets. Task based and hybrid freelancing was also successfully executed by startups B and C. However, to foster continuous support of freelancers during inventing ideas phase, efforts must be made to optimise the freelancer selection from the pool and ensure their long support to take advantage of their continued learning with the problem domain.

Further, the studied startups are not completely dependent on the freelancers but their perspectives are valued as a rich source of success. The situation is similar to the two startups studied in the previous study [4], where freelancers were involved in requirement engineering activities but only to get opinions for decision making rather than completely trusting their perspectives. The startups are able to outsource the core business operation of value proposition innovation by considering freelancer perspectives as one of the inputs along with in-house perspectives for decision making. Freelancer involvement is taken as an opportunity to resolve technical debt challenges and as a source of upgrading skills for undertaking future innovation implementation tasks effectively. The freelancer supported value proposition practices adopted by the three successful startups is a good hope for resource limited startups who wish to gain quick success in the global markets. These successful freelancer supported value proposition practices hold rich value for the startup community especially when the literature has limited ability to support the startups by providing them with the rich empirical studies, evaluated solutions and guidance that could be adopted by the startups as per the unique working contexts [16].

8. Result Assessment

To ensure the validity and correctness of the case study results, the results were shared with the two groups of the startup employees, i.e., G1 (18 employees who participated in data collection) and G2 (18 randomly selected employees who did not participate in case study except the pre-study phase). The results were shared by sending them the two artefacts, i.e., (a) results obtained (without disclosing startup details) and the context of the startups, (b) questionnaire with a list of 7 closed questions and 1 open question. The scale of 1 (strongly disagree) to 5 (strongly agree) is used for rating purposes by the participants.

The startups have their own limitations, uncertainties, markets, working environment, learnings, working practices, and so on, which makes it hard to formulate the questionnaire in a manner that will allow other startups to assess the results brought after studying all startups. This is because the startups have their own practices specific to their contexts that vary from other startups. However, all startups share one thing, i.e., the understanding of the limitations that startups have and the need to foster continuous innovation and utilizing external expertise under tight limitations.

This makes it possible for the studied startups not only to assess their reported data but also to analyse how best the practices of the other startups could be applied in reality in the context of other startups. Both groups G1 and G2 were given access to the questionnaire and a time limit of one week to complete the process. Their quantitative responses were subjected to analysis using boxplots for each question for each group.

The responses obtained from the two groups, for each question, are summarized in

Table 8. Descriptive statistics for Group (G1) responses.

Question Number	Min	First Quartile (Q1)	Median	Third Quartile Q3	Max
Q1	1	3	3	4	5
Q2	3	4	4	5	5
Q3	1	3	3	4	5
Q4	2	4	4	5	5
Q5	2	4	4	5	5
Q6	2	3	4	5	5
Q7	3	4	4	5	5

and

Table 9. Descriptive statistics for Group (G2) responses.

Question Number	Min	First Quartile (Q1)	Median	Third Quartile Q3	Max
Q1	1	2	3	3	4
Q2	3	3	4	5	5
Q3	1	2	3	3	5
Q4	2	4	4	5	5
Q5	3	4	4	5	5
Q6	3	4	4	5	5
Q7	3	4	4.5	5	5

, which are graphically represented using boxplots in Figure 2 and Figure 3.

The boxplot for the responses obtained from the groups G1 and G2 are shown in Figure 2 and Figure 3. The boxplots show that there is a high level of agreement among the participants (Group G1) about the results. Further, the participants who did not participate in data case study (Group G2) also agreed to the results. The boxplots show that the responses are scattered across the data range, but most of the responses belong to the range of 3 to 5 ratings. The reason for the ratings in the scale of 1 to 2 is because the startups have different working contexts, which makes it hard for a few individual participants to accurately rate other startup practices from their applicability point of view from innovation and implementation perspectives.

The boxplot for the responses obtained from group G2 (Figure 3) show that except for question 1 and 3, most of the data belong to the range of 3 to 5 ratings. For the questions 1 and 3 (Figure 3), the response ratings are distributed in the range of 2 to 4 (compared to corresponding ratings of group 1, which are in the range of 3 to 5 (Figure 1)).

The last question, i.e., Q8, which was an open question, was analysed for the possible explanation for such variation between ratings of two groups for Q1 and Q3. This difference seems to be arising because of uncertainties in the mind of startup employees about the applicability of this strategy in all startup working practices. This is because it is very unusual for any employer to think about commitment from freelancers to undertake nonperiodic tasks, by maintaining long term voluntary relations with them, which helps startups to take their services any time (even ignore them if better freelancers are available) and for freelancers it is just a “free” relationship. One of the participants from group G1 responded that, “Panels of freelancers is a very innovative concept and we never thought about such associations. Before reading this panel based strategy of one of the startups, it was hard even to imagine that freelancers would like to have such voluntary association with us. But I am not very sure how much generalizable this association practice is for all startups”. One of the participants from group G2 responded for the hybrid freelancing based on panels that, “I am quite happy to see that the startup has very well addressed resource limitations by maintaining a panel of freelancers. It is really good to see that the panel is maintained well by continuously involving freelancers for giving innovation ideas as well as addressing expertise gaps in-house. However, I am just unsure about how well we will be able to replicate (if not innovate) this panel type freelancing in our startups”. However, the slightly better ratings for these questions for group G1 arises because the same startups have participated in the case study before.

The higher agreement among those rating in the two groups for questions (other than Q1 and Q3) is due to their higher level of understanding of the task based freelancing, which motivated participants to get involved in reading other startup practices with an objective to innovate their work practices. The high confidence resulted in higher ratings and minimal differences between pairs of responses, which increased the correlation. One of the from group G2 responded, “We all faced almost similar challenges which kept us at risk for a long time. The startup practices shared by researchers are very much interesting to handle diverse challenges and I personally feel that these could be combined and further innovated to suit the unique demands of startups”.

The results are very high for Q7, because limitations in the context of startups, which further complicate freelancer associations, are the main pain point of all startup team members. The practices of all startups to handle such challenges had developed deep interest and motivation inside the group members to learn from other practices and think about its adaptability in their own contexts.

Overall, the participants of group G1 and group G2 seem to be in agreement over the results reported by this case study. The responses from the two groups helped not only to ensure that the data is collected and analysed correctly by the researchers, without introducing elements of subjective judgement, but also to ensure that the responses from the samples (group G1) correctly covers the perspectives of other employees (group G2), i.e., no new perspectives emerge if new samples are interviewed.

9. Conclusions and Future Work

The finding of the paper reveals that the startups could successfully innovate their value propositions to maintain competitive advantages by involving freelancers as a source of the creative ideas and as skill resources for implementing the ideas. Involvement in value proposition requires continuous commitment from the freelancers with the agility to continuously improve the learning resulting into market successes. This requires the startups to build trust relations with them and motivate them to contribute continuously. Three freelancing association models, i.e., panel based, task based, and hybrid, may have different impacts on freelancer associations, but the startups employing them have reported positive impacts on their businesses by involving freelancers.

The startups may not like to completely depend on the freelancer support but to complement their own knowledge with the perspectives brought by the freelancers. The freelancers’ support during implementation may help startups to optimally implement innovative ideas, support skill upgrade of the in-house team, promote team working, promote trust, and ensure that high coding standards are followed. Business impacts due to freelancer involvement does not depend on the level of freelancer involvement across a few activities, but it depends on the level of involvement uniformly across various value proposition innovation activities. Freelancer involvement will help startups to avoid costly failures and reach growth quickly overcoming the resource limitations. The studied startups were able to innovate successfully, taking support of the freelancers even though they managed to have this with the small team size (which increased marginally compared to the high growth, relative to the time the previous study was conducted (as reported in [3])). Freelancer involvement reduced the efforts of the in-house team and involvement of the in-house team in all innovation activities helped them learn new things, which kept them motivated.

In future, it is expected that more startups will foster innovation by involving freelancers and will report their experiences and market successes, which could be helpful to the startup community. More evidence in the future will help to extend the theory as reported in this paper. Enriching literature with more studies will be suitable for the research community to address some of the pain points in freelancer associations, as to be reported in future studies, and to the startup community to receive lessons driven by empirical evidence leading to their adoption in their working context.