Design for Motivation: Evaluation of a Design Tool

Abstract

Design for motivation constitutes a design practice that focuses on the activation of human motives to perform an action. There is an increasing need to design motivational and engaging mechanisms for voluntary systems, such as innovation platforms, where user participation is a key target. When designing for motivation, a challenge of the early design phases is the selection of appropriate design tool and strategy. The current work presents a design tool, namely DEMO (DEsign for MOtivation), and evaluates its design process. The tool provides multidisciplinary teams with a user-centred, structured method to ideate and ultimately develop a consistent design plan to engage the users of innovation platforms. The evaluation study analysed the tool’s contribution to the design of motivational innovation platforms, utilising three data collection methods: a protocol analysis, interviews and questionnaires. The results discuss the experiences of 32 users with the development of motivation concepts, the group and the user activities, as well as their creativity aspects. Structured processes and the use of artefacts were found to be productive practices in the early design phases. The results also highlight the importance of multidisciplinary and user-centred teams that can enhance collaboration and communication during the design processes.

1. Introduction

Design embraces many ways to affect the use of interactive systems and to motivate end-users to perform an action or adopt a specific behaviour. Strategies of design that lead to user engagement may differ depending on the use context, the nature of the targeted interactive systems, the use purpose and the utilised extrinsic and/or intrinsic motivators (e.g., [1,2,3]). As the need for creating motivational and engaging systems increases, especially for voluntary systems (e.g., [4,5,6]), it is important to efficiently adopt and employ techniques from related fields, such as persuasive design, gamification, game design, motivational design, and instructional design [7,8,9,10,11]. Design for motivation constitutes a “design practice focused on the activation of human motives, with short or long-term effects, to perform an action” [12] and originates from the recent increasing interest in the development of systems that target behavioural change, persuasion and engagement.

When designing for motivation, a challenge during the early design phases is the selection of tools. A number of studies on design tools have focused on the presentation of these very tools (e.g., [13,14,15,16]); however, there is no dedicated process for selecting the appropriate tools, no clear description of the design process and no specific instructions regarding how to apply the tools [12]. Designing for motivation could benefit from a design tool that operates at a higher level than current tools, thus constituting a design meta-tool which utilises the related theories to produce a detailed design plan for the early stages. Furthermore, employing design processes with stages, provides guidance related to the content of communication as well as a specific focus for the decisions at each stage, leading to design solutions [17].

The current work presents and evaluates a design process with a design tool, namely DEMO (DEsign for MOtivation: www.designformotivation.com). DEMO is a design tool used during the early design stages that targets the development of design plans for motivating and engaging users of interactive systems.

The meta-tool qualities of DEMO provide multidisciplinary teams with a user-centred, structured method for ideation related to motivational design. The tool has previously been evaluated in a usability study [12], leading to the tool’s current second version.

The tool was primarily developed according to the identified design directions [12,14,18,19,20,21,22] for specific interactive systems, namely innovation platforms (e.g., [3,5,23]). Innovation platforms are a representative example of voluntary interactive systems that urge user motivation. When developing innovation platforms, it is important to determine which factors trigger user participation [5]. Previous research on innovation platforms and other similar research communities have identified a number of participation motives [1,2,3,4,5,6,23,24,25]; however, the design processes involved in developing motivational innovation platforms are still unclear [1,23,24].

An evaluation study was organised to examine the design process in the case of innovation platforms and to better understand how the DEMO tool supports the design process of developing motivational platforms.

The contribution of the paper is threefold. First, the paper describes the design process in detail, as well as how the tool is utilised during the process. Second, the combination of methods used to study and analyse design processes in this context to provide rich results are discussed. Third, the presentation of results can be useful in other contexts and tools when designing motivational systems.

Design Challenges and Directions

The systematic development of motivational systems is limited by a number of challenges [12,18,19]. Four main challenges have been discussed in the literature of gamification, game design and persuasive design for existing design approaches. First, the existing tools refer to designers, such as game designers and experience designers, as their target group that utilises these tools [19]; however, the suitability of designers in developing motivational systems has been questioned due to the complexity of the design task, as it requires expertise from diverse areas and knowledge of various aspects of technology [20,21]. Second, the theoretical foundations of motivation have been interpreted and applied in different ways in the literature [14,18]. The majority of applied research lacks a theoretical connection and practical use for the existing design approaches in the systems’ designs. Moreover, the dialogue with end-users should influence the design process from the beginning. Communication between the developing team and the end-users and their involvement are central to the design process [22,26]. Finally, the design of motivational systems is an activity that designers need to acknowledge, explore and understand as they design things [22]. The challenge in this case is to create dialogue within the design team that differs from a traditional communication process and to use artefacts in a material form, which could influence ways of thinking and acting when designing for motivation [22,26].

To delineate the complexity of design for motivation, key design directions have been suggested. Given the intricacy of the design problem, it has been suggested to focus on optimising design as a team-based activity [17,20,22,27]. A multidisciplinary team of professionals, such as game developers, experience designers, and psychologists, could discuss how design, use and objects are related to create motivational systems [17,22]. Discussions based on artefacts is suggested as a means of dialogue generation for multidisciplinary teams, which surpasses traditional communication processes [22,28,29,30]. Furthermore, the use of a user-centred approach ensures that users’ needs, motivations, and goals are considered during the early design phases [17,18,22]. A user-centred design could be used to define the area in terms of methods and approaches rather than as a specific type of design [22]. Finally, the motivational design process could benefit from structured approaches with steps to guide the design team to a solid outcome [17,27,31]. Existing motivation concepts, theories, instruments, design methods, game elements and other existing tools should be available to the target group in order to make informed decisions to achieve a common outcome [27,31].

The DEMO tool was developed based on the directions discussed, addressing an artefact-based, user-centred, structured design approach for multidisciplinary teams. A specific description of DEMO is presented in the following section.

2. DEMO: A Tool to Design for Motivation

The concept of developing a tangible tool emerged while working with industrial partners, who wanted to increase user participation in innovation communities. Therefore, DEMO (DEsign for MOtivation) was introduced as a design tool used during the early development stages of innovation platforms, ultimately targeting an increase in user motivation and participation in such platforms [12]. The tool was built according to the design directions (Section 1), while it is theoretically grounded on motivation theories and incorporates design concepts for motivation through a stepwise process. The novelty of the DEMO tool and its process lies mainly in three aspects. First, the tool is considered a ‘meta-tool’, as it offers an overview of the existing tools, methods and related theories that lead to efficient communication during early-phase, limited-time sessions. Similar tools were not previously available. Second, the tool has been evaluated before its use during projects. The first version of DEMO was evaluated by experts for its usability performance [12,18]. This paper discusses the second evaluation of motivation- related tools based on empirical studies. The existing tools have been applied without previous evaluation on their usability. Third, the design of the tool incorporates the design directions that are discussed in the literature. The characteristics of artefact-based, user-centred, structured and multidisciplinary teams are embedded in DEMO’s design. In previous approaches, only the characteristics of user-centred and structured processes were usually discussed.

The DEMO tool consists of three main parts: the DEMO cards, the DEMO board and the team roles, which are utilised during a semi-structured, workshop-based design process. The board and the cards are utilised as artefacts that serve as partial or complete representations of the process that is being produced [32]. In other words, DEMO offers a bottom-up approach to the design task, focusing on developing the activities and the process for a ‘motivation plan’, i.e., an early-phase plan of a design task.

2.1. DEMO Cards

The DEMO cards are designed to inspire group discussions and to suggest specific activities related to design, management and research. Collaborative card-based techniques and game-like approaches already have a long tradition of use in participatory design, serving as communication tools between members of the design team and users to initiate design discussions and to accelerate the refinement and iteration of ideas [33,34,35,36]. The cards are inspired by previous card-based design approaches [37,38] and incorporate terms from related theories.

In DEMO, four card categories, namely people, methods and tools, resources and expectations, aim to further clarify the activities for each step. The cards display an icon with a term from motivation (motivation theories, gamification, game elements), design (interaction design, user experience), management (needs, material) and research activities (research methods, tools). Figure 1 presents examples of the cards.

• ‘People’ cards: Specify who (e.g., industrial partners, users or administration) should be involved in a particular step.
• ‘Method and tools’ cards: Specify which methods and tools (e.g., personas, qualitative data or surveys) should be used in a step.
• ‘Resources’ cards: Specify which resources (e.g., the platform, policies or points) are needed to complete a step.
• ‘Expectations’ cards: Specify what the expectations (e.g., feedback, learning or new product/service) are for each step.

2.2. DEMO Board

The activities on the cards are structured on a table board. The DEMO board is used as a design and information space that supports the structure of the design process and the collaborative reflection through the steps. In design literature, empirical studies have showed that applying methods based on structured methods produces the most useful outcomes [39,40]. The board and the cards assist the team members in navigating through the process of developing a motivation plan and help them to include all information essential to the group discussion. The DEMO board triggers the identification of important aspects of the design process through four phases with titles and suggested questions underneath that the team can use as discussion-starting material. The phases include:

• Objective: The team describes and clarifies the objective of designing to motivate the user, i.e., increase participation or quality of participation.
• User: The team defines the characteristics of the target user and the subject of the design process and utilises the innovation platform.
• Experience: The team discusses and specifies the targeted user experience that should result from the innovation platform to motivate the user and reach the objective.
• Motivation: The team chooses the motivational mechanisms that will be employed by the innovation platform to motivate the user.

2.3. Team Roles

The target group of DEMO is a multidisciplinary team that collaborates in a participatory workshop. The team may consist of designers, non-designers, end-users and other experts, where each participant has a ‘role’ according to his/her background. The roles have two primary purposes within the design process: (a) they support the participants’ engagement in a ‘gamified’ role-playing process, and (b) they ensure the inclusion of several perspectives during the workshop [12]. The team roles are inspired by previous work on design tools [41], design thinking research [42], participatory design approaches [43], and game-like approaches [44]. The roles are described as follows:

• Manager: This is a management-oriented role that focuses on how to organise and deliver an optimal solution to a problem. Managers, such as project and/or innovation managers, are characterised by a high level of practice and they are comfortable working with a wide variety of different tasks [12]. The project manager can also be the workshop facilitator and may apply the DEMO tool.
• Designer: This is a design-oriented role that focuses on design-related activities (sketches, drawing, implementation) and contributes to the designs artefacts throughout the process. Designers, who may be interaction designers, service designers or interface designers, must be able to communicate and define concepts for the other roles [12].
• Developer: This is a technical role that focuses on how to develop the suggested solutions. Developers, who may be computer scientists, or software engineers, contribute to the feasibility of the suggested solutions/designs [12].
• Psychologist: This is a role that requires experience and/or a background in psychology, motivation, and human factors. The psychologist contributes to the analysis through discussions regarding motivation and human factors as well as suggestions regarding how to address the platform’s users more effectively.
• End-user: For a user-centred process, this role is necessary to gain user insight. The end-user has experience or is familiar with the problem and contributes to the applicability of suggestions, designs and solutions from a consumer/customer perspective.

2.4. Design Process

The design process consists of pre-workshop preparation and the workshop (Figure 2). The preparation for the workshop refers to the selection of participants and their familiarisation with the DEMO tool, the processes of the workshop and the design task. It is not mandatory for participants to have any experience in workshop participation or in using design tools. Pilot sessions can also be arranged to familiarise participants with the tool.

The workshop begins with a short introduction to the design task, the roles and the tool, thus ensuring a shared understanding of the process and its desired outcome. The DEMO tool suggests a step-wise process that works iteratively between these two stages. The duration of the introduction should be limited to ten minutes. The workshop consists of two stages: the conceptualisation and the implementation stages.

Conceptualisation: The goal of this step is idea generation and concept building on how to solve the design task. The DEMO tool suggests a step-wise process that works iteratively. There are no set rules for turn-taking, and cards may be combined in the way the participants deem productive; however, it is useful to begin by clarifying the objective and the user, as they are described in the board. Managers control the design process and the discussion flow, following the board’s suggestions and asking questions. All participants contribute to idea generation, problem finding and/or the evaluation of ideas by suggesting and combining the cards on the board and through argumentation related to their choices. The psychologist and the end-user play central roles. This stage lasts approximately one hour. At the end, the participants summarise the concluding concepts, reflect on the resulting design concepts and take a short break. This stage ensures a common understanding of the motivation concepts, rather than an evaluation of concepts in terms of whether they are appropriate or realistic.

Implementation: The goal here is to determine how the previously suggested concepts could be implemented. Again, this is an iterative process that uses the same phases of DEMO. At the implementation stage, the experience and motives are the central discussion topics; therefore, the developer and the designer play central roles. All participants contribute to idea generation, problem finding and/or the evaluation of ideas. Other techniques, such as drawings and sketches, are employed. This stage lasts approximately one hour. At the end, the participants summarise how they will implement the motivation concepts into designs and reflect on the resulting designs. This stage ensures a common understanding of the connections between conceptualisation and implementation, as well as the in-between activities. After the workshop, a plan for future activities is developed and the workshop is repeated, if necessary.

An evaluation study was conducted to examine the tool’s features and performance in design processes, which is described in the following section.

3. Evaluation of the Design Process with DEMO

The above design process was applied and evaluated, as part of an ongoing research project. The objective of this study was to provide an in-depth understanding of the design process of motivational innovation platforms, while the process was supported by the DEMO tool. The research question was: How do the features of the tool (artefact-based, multidisciplinary, structured, user-centred) support the design process of developing motivational innovation platforms?

The research question was examined through an exploratory case study that is generally used to gain an in-depth understanding of the concerned phenomena in a real-life setting [45]. The case study was conducted over the span of three months (January–March 2017).

3.1. Participants and Study Setup

Participants were recruited during January and February 2017, and they were assigned to one workshop. In total, one pilot study and six workshops took place during this time in a Scandinavian research institution. Each workshop consisted of 4–5 participants with a background in one of the following disciplines: design, computer science, psychology or project management. The selection and delegation of the participants in workshops were based on their backgrounds and expertise. The team composition were reflected newly grouped teams of people with multidisciplinary backgrounds, though it is beyond the scope of this paper to make a definitive causal connection between group members and familiarity levels within the group; however, it is of interest to investigate the effect of the members’ backgrounds on their contribution to the design process. During the workshops, the researcher of this study participated as an observer and performed appropriate interventions (through discussions) when requested or considered necessary. Participants signed an informed consent form and were informed of the data collection process in advance. The total duration of each workshop, including pre- and post-sessions, was approximately three hours.

3.2. Measures and Study Procedure

The study examined participants’ practices and perceptions regarding the use of the DEMO tool during the design process, their motivation practices and creativity. Several data collection methods were utilised: (a) before the workshop: a pre-questionnaire, (b) during the workshop: observations with field notes and video-recordings and (c) after the workshop: a post-questionnaire and audio-recorded interviews.

Before the workshop, participants completed a questionnaire using both a Likert scale and open questions. Questions were asked regarding basic demographic information, such as age, gender, and field of expertise, as well as prior experience in design tools, workshops, and creativity capacity. In addition, participants were asked to describe any motivation techniques and rewarding mechanisms they utilised in the past. The purpose was to identify general concepts related to how users think and act in response to motivation. Questions about creativity concerned the self-rated creativity capacity, which was adapted from an existing questionnaire on creativity styles [46].

The workshop began with a short self-introduction and introduction to the DEMO tool, and an online manual of the tool had been sent to participants beforehand. The design task and the participants’ goal was to develop a motivational plan for an innovation platform that would increase voluntary user participation. During a two-hour workshop, the participants worked on the design task while they interacted with the tool (Figure 3). The workshop included two phases of teamwork: concept development and the implementation phase, where concepts were transformed into designs and sketches of the platform. Figure 4 shows a screenshot of a workshop with a completed motivation plan and other generated materials.

During the post-workshop session, the participants completed a questionnaire about their experience with the DEMO tool, collaboration with other people and satisfaction with the final design. Both a Likert scale and open questions were applied. In addition, all participants were invited to individual interviews. The semi-structured interviews focused on their experience with the DEMO tool, satisfaction with the final design, and group work experience. Selected questions were repeated during the interview to evaluate or complement their replies from the pre- and post-questionnaires. The study was approved by the Norwegian Centre for Research Data. All data were anonymised, and personal identifiable information was removed from the study results.

3.3. Data Analysis

To enhance the rigour and analysis of the empirical data, additional verification methods were employed, such as data triangulation [45,47]. The data from the workshops (video recordings and other generated materials) were used as a primary data source, while both the questionnaire and interview materials were used as secondary sources to compare and complement the data. The method followed for each dataset is described in the next sections.

3.3.1. Workshop Analysis

An existing protocol analysis method [48] was employed to examine 14 h of video recordings of the workshops and to examine the participants’ behaviours and performance in detail. The design process was captured and represented as a sequence of events in time. These events were organised into creative episodes, which include the segmented verbal utterances that contain a series of transactions among participants about a theme with the aim to move the discussion forward and to achieve the goal [48]. Episodes were qualified as ‘framing-moving-reflection’ cycles. ‘Framing’ refers to an action in which a participant frames a (sub) problem or (partial) solution as a context for further activities (problem finding), while ‘moving’ is considered a problem-solving exercise and refers to actions such as generating ideas, making an inventory, sorting information, combining ideas or comparing concepts. ‘Reflecting’ refers to actions that lead to a complete reconsideration of the design task and involves the critical thinking of the team and their actions.

Table A1. Examples of “framing-moving-reflecting” activities during the workshops.

Episodic Activity	Code	Description	Utterances
Framing	FRA	Identify a design problem	“From a design stand-point when we are talking about motives and experiences, you mentioned that the culture of the workplace maybe impacted on we should create this platform, while if there is a lot of competition style in the workplace, then we should build on that. Or if there is other aspect that influences the culture, we should focus on the platform as well” (Designer).
Framing with cards	FRA-C	Identify a design problem based on cards	“We have two kinds of different aims in a way, these (cards) are more like friendly themes, against competition, not like “who is the first?”. So, we have to decide if we want to go towards competition or something that is more collaborative” (Manager).
Moving	MOV	Proposed explanation of components for the design task	“If the work environment is not on competition, and count on collaboration then we can reward the group instead of a person” (End-user).
Moving with cards	MOV-C	Proposed explanation of components for the design task, based on cards	“I was thinking about these two cards: ‘users’ and ‘registered users’, because the idea is to get more people to use it. So, if we want to improve it, we need experiences from the users of the platform. So, it should be, maybe one of our stakeholders I guess” (Designer).
Reflecting	REF	Evaluate or judge the explanation in moving regarding components of the design task	“I actually liked the idea you proposed, considering that you use your real name. But what we could do if people are afraid of posting ideas because they don’t want their manager to pick them up is to give them an option” (Psychologist).
Reflecting with cards	REF-C	Evaluate or judge the explanation in moving regarding components of the design task, based on cards	“If the goal is to produce high quality ideas, we need to give them some sort of reward for it. We can give them rewards and feedback but that’s going to be enough” (End-user comments on the ‘rewards’ and ‘feedback’ cards).

(Appendix A) presents examples of the ‘framing-moving-reflecting’ coding for the workshops. The creative episodes were used to capture two instances: user activities with the DEMO tool and without it. Additional information used to analyse the video recordings was included, such as the time and duration of episodes, the participants involved in each episode, discussion topics and details of episodes. Other data from the workshop, such as field notes and generated materials, were merged to understand the context of the design process. Initially, the analysis was conducted by two researchers to discuss the data, gather feedback and make changes, if necessary. Later, one researcher continued the retrospective protocol coding. JMP software was used for data visualisation and the analysis of the behavioural patterns during the design process.

3.3.2. Interview and Questionnaire Analysis

The interviews were audio-recorded and transcribed verbatim. In total, 10 h and 45 min of interviews were transcribed. Data were coded using the software NVivo 10 to better organise, browse and manage the data. The grounded theory was used for the qualitative analysis of the data [49]. Using open coding, basic codes and then themes were identified, by grouping previous codes and classifying them under higher-order headings. For the final step of the content analysis, a general description of the research topic was formulated by generating categories (and sub categories), and selective coding was integrated to represent the central phenomenon of the data. Furthermore, two questionnaires were used to assess the participants’ backgrounds, creativity and experiences with DEMO. Using a qualitative analysis method, the questionnaires were analysed into codes and themes. Data were coded using the software NVivo 10, while the SPSS Statistics 24 was used for the statistical analysis.

4. Results

The study generated rich results for the research question. First, the participant demographics are reported. Then, the results are organised into four sub-sections: (a) user experience with the DEMO tool, (b) group activities and communication, (c) motivation concepts and thematic activities, and (d) creativity and user activities across disciplines.

4.1. Demographics

A total of 32 participants, 21 males and 11 females, voluntarily participated in this study. All workshops consisted of a minimum of four and a maximum of five participants, with a mean age ranging from 25–43 years old. The participants were coded according to their roles, i.e., manager, designer, developer, psychologist and end-user. All participants were invited to adopt the roles in the workshops according to their professions, apart from end-users, who were invited as end-users. Specifically, designers had expertise in service design, interaction design and user experience design, while developers had expertise in computer science, software engineering and information technology development. Furthermore, managers had expertise in research and marketing, service science, software and data management. Psychologists’ backgrounds focused on psychology, work and organisational psychology, motivation and learning, while end-users had expertise in interaction design, human-computer interaction (HCI), and service design. The expertise level was self-reported. The mean years of expertise was quite high, especially for managers (M = 11 years), developers (M = 10 years) and designers (M = 6 years).

The participants also reported previous experience with design activities and design tools. Nineteen participants (60% of the total) had previous experience in design activities during workshops, in which almost half of them (48%) participated in workshops more than six times, while the remainder participated in up to five similar workshops (26% 1–2 times, 26% 3–5 times). The participants mentioned that the purpose of these workshops was to define a problem, to design and create solutions, user testing and evaluation, to gain insights or to engage participants.

4.2. User Experience with the DEMO Tool

In the post-questionnaires, participants reported how they utilised the tool during the workshop, while in interviews, the participants confirmed and expanded their answers with specific comments about DEMO’s components. The overall user experience with the tool underscored the usefulness and fun as the most important characteristics of the DEMO tool according to the post-questionnaires. The importance of fun and usefulness was confirmed during the interviews, in which participants provided 61% positive comments and 24% negative comments, as well as 15% suggestion comments for the tool’s future improvements. Positive comments referred mainly to the general experience with the tool and were less specific regarding the components of DEMO. According to the participants, the general experience with the tool was fun and game-like (22%), structured (16%), inspirational (14%), easy to use and a beneficial approach (10%). The fun and game-like aspects were explained in terms of the group-based, board-game like and time-limited activities. The structure of the tool was highlighted as a positive feature in terms of the process, design task, project and groups’ backgrounds. Inspirational aspects of DEMO mainly referred to the role of the cards, whereas the beneficial aspects of the tool were described in terms of the process or the components of the tool.

“I think it’s fun to use; it becomes like a game and that probably shaped the discussion and focus towards the idea”.

(End-user)

“The tool was helpful, partly for organising the work—Okay you should look at the objective, you should look at the users, motivation and so on. And this set up with these steps also worked as a checklist and inspiration. Also the cards sometimes work when things don’t run so smoothly. We picked up a card and looked at it and continued working”.

(Developer)

The important factors of user experience with DEMO were prioritised differently across disciplines. For example, designers considered the tool primarily fun to use and useful, while developers favoured the fun and simplictic aspects when using DEMO. End-users considered the tool primarily fun and satisfactory to use. Managers and psychologists considered the tool primarily fun to use and useful. Furthermore, some negative experiences with the tool were reported. The DEMO was argued to require time to learn (24%), while the learning threshold was perceived differently across disciplines. Also, in general, the process of using DEMO was found to be confusing (22%) due to the lack of strong guidance, and also it was also incicated that it limits participants’ creativity (6%) due to the previous use of open-ended practices in work.

“I took some time to understand how to use the board, how to use the cards, when, if there was an order. I thought the way to use it was quite random, but later, I understood it”.

(Psychologist)

“When we had the cards, we didn’t try to think. On the other hand, when we ran out of ideas, we could use the cards for inspiration. It is both positive and negative. Maybe it would have been easier to be creative if we didn’t have the cards”.

(Manager)

Lastly, the interviewees suggested design and instructional improvements for DEMO regarding the design of components, such as the improvements on the content and physical size, and the structure of the design process, such as including training time, navigation through the process, justification of the participants’ roles and suggestions for application domains. Figure 5 shows a screenshot from a workshop with generated materials.

4.3. Group Activities and Communication Using the DEMO Tool

The groups worked intensively during the workshops on the design task for approximately two hours to create a motivation plan for innovation platforms. To better examine the group activities, the design process was segmented into episodes that were qualified as ‘framing-moving-reflection’ cycles [48]. The problem finding (Framing-F), idea generation (Moving-M) and idea evaluation (Reflecting-R) episodes were mapped for the workshops, with fluctuating numbers of episodes across time. Also, the episodic activities were represented under two conditions, with cards and without cards. For the purpose of this study, each workshop was coded as W1–W7, including the pilot workshop that fulfilled the aim and standards of the study. This analysis produced a total of 213 episodes for the 14 h of data (Figure 6). The lengths of the episodes varied from a total of 1 min to 11 min. The workshop activities were divided into conceptualisation and implementation episodes. On average, 20 episodes of conceptualisation and 10 episodes of implementation activities represented a typical workshop.

Then, the workshops were examined based on the use of DEMO. Figure 7 presents the overall episodic activities of the workshops. The mean value for the episodic activity (number of episodes) was mapped for both conditions on the timeline: with and without cards. The time corresponds to the actual time of the workshops that took place from 12.00 PM–14.00 PM. Some interesting patterns emerged. The episodic activity was quite balanced in the workshops. The problem finding (framing) and the idea generation (moving) activities presented higher mean values when the team used cards. The evaluation of the suggested ideas (reflecting) had higher values in general for the same condition, but not in all workshops separately. Problem finding (with cards) always began more intensively in the first phase of conceptualisation and faded out in the late phase. The idea generation and evaluation (with cards) usually began later in the design process, with fluctuating values. It was observed that problem finding presents a complementary effect between the two conditions, where the activity with cards seemed to increase when the activity without cards decreases.

According to the secondary data sources, DEMO was useful during concept formulation, to inspire creativity, to negotiate the meanings or the concepts and to better describe the process, from conception to implementation. The use of the cards is primarily connected to the generation of ideas, similar to brainstorming processes, which promote group discussions on the topic and involve everyone in the group. The cards were seen as reminders of the existing literature, meaning the motivation theories, gamification and other design and research methods. Similarly, the board helps to create discussion and generate ideas for relevant topics on motivation, which lead to defining and clarifying the meanings and concepts, understanding the relations among themes and documenting and providing an overview of the discussions. Furthermore, the group communication was associated with the tool’s characteristics. Overall, the group work was evaluated as good and constructive collaboration with friendly people and useful discussions. Specifically, the participants built on each other’s ideas and obtained feedback from different disciplines. The majority of participants (91%) reported that the DEMO tool supported the group communication and indicated that the tool acted as an intermediary or boundary object:

“It supported the communication with the tool, yes. I think just as a sort of, like I said, not just communication with the tool, but sort of just creating a space for noting ideas and having them, being present in some categories—that kind of is useful to getting a high-level overview of what is going on in this meeting. Rather than having everything completely invisible. There is something that is visible on the table”.

(Designer)

“We had great conversations, no interruptions; the participants had diverse backgrounds. So, people approached the project from different angles, having different understandings. I liked listening to other points of view, such as the psychologist’s perspective”.

(End-user)

4.4. Motivation Concepts and Thematic Activity

In the pre-questionnaire, participants briefly described some examples of how they motivated other people to perform an activity, mainly from their work environments. Participants mentioned the steps of motivation processes, such as explaining the benefits, revealing opportunities, providing relevant examples and peer discussions. Specifically, 81% of participants had previously used non-monetary rewards, such as learning, while 19% of them used monetary rewards.

“As a part of several projects at the university, I have needed help from users to evaluate products or concepts in development. We have briefly explained the goal of the project and why we have needed external persons as part of the evaluation. Their motivation has usually been helping a friend out and getting a glimpse of what interaction design is about”.

(Developer)

In general, all workshops suggested a combination of intrinsic and extrinsic motivations, which are implemented in various ways in innovation platforms. The conceptualisation of motivation began at the beginning of the workshops. The discussion on motivation was a continuous negotiation of the meanings that capture motivation in relation to the board’s phases and cards, such as the user type, community type, interaction type, host environment of the community, community’s target and users’ target. For example, the selection of user types was also connected with motives, such as the ‘socialiser’ user type (W1), and influenced the description of the experience (fun and engaging platform). The motives were then discussed accordingly. During the workshops, the presence of end-users assisted the team in defining the users’ needs and describing the motivation concepts. Any concept that was related to psychological, cognitive, social or emotional needs was discussed in relation to the components of DEMO, the cards and the board and how that could support the design task and users’ motivation in the innovation platform. Examples of motivation concepts that were suggested for implementation include entertainment and fun, community support, social interaction and personalisation, positive reinforcement, creativity, peer recognition and feedback from experts. From an implementation perspective, motivation concepts were addressed as interface elements, a series of online user activities or long-term strategies.

“We got a bit too stuck on the anonymity as a motive and ended up with a small check box, but I think that it reflected what we had in mind in the first part”.

(End-user) (suggestion for an interface implementation)

“We used achievements, to get some kind of reward for participating in a platform that has one main motive. And the other was the personalisation of the platform. We wanted people to get attached to their ideas, the platform and using it. So, we suggested a personal space where they could comment, look at other people’s work and start to make it a little more personal and meaningful”.

(Designer) (suggestion of motivation concepts as series of online activities)

“The short-term motive is peer recognition and experts’ reviews. The long-term motive is the possibility to work on and develop your idea within a real project with funds, and a team”.

(Manager) (suggestion for motivation concepts as part of the long-term strategies)

Furthermore, an analysis of the discussion topics during the workshops was conducted to examine the content of discussions. The analysis of all 213 episodes showed that 10 discussion topics occurred in the workshops. Specifically, the four phases of the board (objective, user, experience, motives) were complemented by additional topics, such as user-objective, user-experience, motives-objective and experience-motives among other topics. Two main discussion topics that were more intense and therefore are considered the most critical in this study are the discussions on the ‘objective’ and the ‘experience-motives’. Examples of cross thematic topics in the workshops, between the ‘experience’ and ‘motives’ phases, refer to the gamification of user interface elements with a user profile ranking, the visibility of achievements and locations in the user interface, rewards based on a future commitment and receiving funding to develop an idea. During all workshops, considerable time was spent discussing one of these two topics; for example, W1 focused more on the objective-motives, while W4 focused more on discussing the experience-motives. One exception is W5, where the topics were discussed equally. In total 145 episodes involved the cross-thematic topic of experience-motives. This illustrates that the transformations of motivation concepts to an online activity stimulates discussions.

At the end of the workshops, all participants evaluated the outcomes and the suggested motivation plans, both in questionnaires and interviews. The majority of participants (84%) was satisfied with the concluded motivation plans, and many commented on their potential improvement (42%); however, some participants (16%) were not satisfied with the outcomes due to the perceived mismatch between the conceptualisation and implementation phases, a lack of innovative suggestions or disagreement on the selected motives.

4.5. Creativity and User Activity across Disciplines

The pre-questionnaire measured the self-rated creativity capacity of the participants, according to which designers and end-users considered themselves more creative than other participants. Designers and end-users reported that they are involved in creative types of work on a regular basis, while end-users’ backgrounds also involved design activities. Developers considered themselves creative and regularly involved in creative work. Interestingly, managers also considered themselves creative with regular involvement in such activities, while it was expected that psychologists would rate lowest in their creativity capacity.

During the workshops, the episodic activities across five disciplines of our study were mapped to examine the level of participation and contributions during framing-moving-reflecting activities (Figure 8 and Figure 9). Overall, when using cards, the end-users spent more time on problem finding (framing) compared to other disciplines, while designers spent more time on idea generation (moving) and psychologists on the evaluation of ideas (reflecting). Furthermore, in the second condition without cards, designers spent more time on the framing activity compared to other disciplines, while developers spent more time on the moving activity and psychologists on reflecting, without cards. Nevertheless, all disciplines spent a considerable amount of time on framing-moving-reflecting activities, although the focus was on different activities.

“It was hard work individually. The other people had many good ideas and worked faster than I do. I followed them. I am not a creative person when I work alone; I need a team to be creative. Very good to be a part of the team”.

(Psychologist)

All participants argued that they were more creative while working in a group setting with the tool rather than working alone with the tool. In the interviews, it was confirmed that all participants liked the group work with the tool, while two participants preferred a combination of individual and group work. Negative remarks on group communication involved the lack of familiarity with the group members, perceived lack of contribution (e.g., psychologists) in the discussions compared to other participants (e.g., managers) or perceived lack of a significant role in the design process. The roles in the group work were important for group communication, while some roles, such as the role of manager, were perceived as more significant than others. Therefore, participants’ suggestions referred to the equal involvement of the group members, training time for managers or facilitators and a division of the design task to subtasks. In addition, the use of motivation techniques during the design process varied across disciplines. For designers, developers and psychologists their main technique was to modify and combine existing ideas. Managers mostly combined existing ideas and worked on evaluating ideas, while to a larger extent, end-users combined ideas. Finally, the perceived inspiration source was measured. The perceived inspiration source was considered by designers to be associated with the cards, while other participants argued that the board inspired them. Other inspirational sources were mentioned in addition to the tool, such as social media, personal experience with similar projects, psychology theories and group work.

5. Discussion

The results indicate that the design process of creating motivational innovation platforms is supported by the features of the DEMO tool. The use of DEMO was considered to inspire creativity, to assist in concept formulation, to negotiate the various meanings and to assist in the process description from conceptualisation to implementation. Figure 10 shows the implementation sketches of motivation concepts from two workshops. Hereafter, the main findings are discussed according to the results.

5.1. The Use of Artefacts Inspires Creativity in the Design Process for Motivation

The use of artefacts during the design process, such as the cards and the board in this case, is a common practice to scaffold ideation and concept development in the early phases of design [37,50]. First, the use of artefacts created a game-like activity that triggered creativity among multidisciplinary teams. The game-like activity was the strongest factor in the participants’ experience with DEMO that maintained their interest and focus on the design task. In particular, it was found that when working with a complex topic, such as motivation, a game-like activity provides structure for concept building. Usefulness was also considered an important factor for utilising the tool, and this was also reflected in the workshops’ episodic activities.

“It was a lot of fun. I learnt a lot from not using it, so that when we didn’t use it, it was more messy. It was less structured at least because it will be messy in a way”.

(Manager)

Second, the components of DEMO inspired the design process in different ways. The cards helped initiate a group discussion on a topic and involved everyone in the group. The majority of episodes were triggered by cards, while cards were mainly used for problem finding and idea generation. The evaluation of ideas with cards showed mixed results among the workshops, as it was dependent on qualifications, such as critical thinking, personal experience and the backgrounds of participants. Consequently, participants elaborated on the evaluation of ideas according to their professional backgrounds and experiences.

The use of the cards was primarily connected with the generation of ideas. The cards seemed to function when creativity and idea generation decreased, while the introduction of cards generated new activities. The cards were viewed as reminders of the existing literature of motivation theories, examples and related work, which are essential when working with the complex topic of creating motivational platforms. Similarly, the board focused the discussion and ideas on relevant topics of motivation, such as the objective to design for motivation and the targeted user, leading to the definition and clarification of motivation concepts as well as to understanding the relations among themes, to document and obtain an overview of the discussions.

5.2. Multidisciplinary Groups Negotiate the Meanings of Motivation Using Artefacts

The results confirmed the value of using artefacts for argumentation and dialogue generation in multidisciplinary teams in this field, as previous studies suggested, e.g., [22,37]. The detailed analysis of all 213 episodes (Figure 6) revealed that the discussion topics of the workshops covered a cross thematic area, usually involving two main topics, such as ‘user-objective’ and ‘experience-motives’; however, one discussion topic was considered the most critical for this study: the negotiation of experience-motives. In total 145 episodes included the cross-thematic topic of experience-motives. One explanation is the lack of shared conceptual models and the lack of standardised structures or processes for motivation concepts.

The roles of the artefacts, cards and board, were perceived as similar to the boundary objects, in which standardisation is a key component. The negotiation of meanings for experience-motives created a common understanding of the content of the motivation plan at the end, satisfying the informational requirements of all group participants.

“I think having something to work with functions as a boundary object between us; it’s a good starting point instead of starting from nowhere. Also, it helped us to get to know each other. We as a group had to negotiate the meanings of the board, and that’s actually to start using the board”.

(End-user)

“It’s always good to use tangible elements in a discussion […] If the discussion froze, it’s important to have cards. When you need them, they can be really valuable, and the cards are like boundary objects”.

(Designer)

The role of the artefacts as boundary objects indicates why the level of participation and contribution during the design process was significantly higher when using cards, compared to not using cards. Using the artefacts, team members contributed in negotiations in different ways. For example, managers and psychologists contributed more in evaluation of ideas using cards, while the contribution from designers, developers and end-users was concurrent in problem finding and idea generation using cards.

Given the fact that designers are not necessarily trained to design motivational systems, other professionals are also involved in this design process [20,21,22]. Therefore, the design process would benefit from alternative types of interactions that create a dialogue among professionals relevant to design for motivation.

5.3. Structured Tools Support the Creation of a Shared Vocabulary to Design for Motivation

When designing for motivation, the development of shared mental models is critical. During workshops, psychologists analysed motivation concepts from a theoretical viewpoint, offering related theories and explaining relevant study results, while end-users were mainly concerned with their own needs and desires from the innovation platform. All groups spent considerable time during the workshops in clarifying the objective from different perspectives.

“For me, with psychology as a background, my main concern was ‘why are they supposed to do this?’ […] I think we locked down on the reward quite early, and I don’t know if that was the most advantageous […] But many studies show that if you feel like you have control and are part of the decision in your everyday work life, you have greater motivation”.

(Psychologist)

The differences across disciplines in describing approaches to motivation were also evident from the different techniques used to work with ideas during the workshops. The technique of modifying and combining existing ideas was used by designers, developers and psychologists, while end-users mainly combined ideas. Managers mostly combined and evaluated existing ideas to move the process forward. The use of a structured tool with explicit phases prompted the participants to define and build their meanings for the motivation plan. The study revealed that each participant had his/her own definitions; therefore, the creation of a shared vocabulary was a necessity for building a motivation plan. According to participants, DEMO had a dual role in this: it created a shared vocabulary through the cards and organised the process through the board.

“It created a shared vocabulary first of all. Also, it allowed us to focus on certain areas—‘Okay we start from here.. and then we discuss this..’. As I said, motives and experience are interdependent and objectives and users are clear. If I had to summarise, it created a structure and provided a shared vocabulary”.

(Manager)

In this study, the structured process during the early design phases was appreciated by the majority of the participants, while it was found that structure is needed to begin an unknown process and to establish the basic elements of a motivation plan. Thus, during the implementation phase the participants used more intuitive and less structured ways to design their solution.

5.4. User-Centred and Multidisciplinary Groups Are More Creative with Structured Tools for Motivation

The workshops generated high numbers of episodes in a short time period with the use of the tool, while high numbers of episodic activities during workshops suggest that the process and the tool support a high activity pace (Figure 6). The user-centred approach contributed to efficient group activity. The group discussions began with end-users’ needs and experiences with innovation platforms, especially during the conceptualisation phase. Overall, the end-users spent more time on problem finding compared to other disciplines. The end-user helped the group frame the problem and the objective quickly, while they were active participants throughout the design process and supported idea generation as well.

“Basically, we used the board the entire time, trying to separate thoughts and ideas […] We went a bit back and forth. So, I think it helps shape how you present ideas and how you discuss them, and it also helps you to focus on different aspects”.

(End-user)

Other disciplines assisted in the discussion in different areas. For example, designers spent more time on idea generation (moving) and psychologists on the evaluation of ideas (reflecting). The presence of different disciplines and backgrounds had a positive effect in regards on creativity. Two types of creative behaviour were observed in our study: (a) structuralists who follow an orderly method with steps and (b) inspirationalists who favour familiar structures to be creative [51]. The structured process of the board and the structure of the tool were appreciated by all participants, supporting the group creativity and inspiration when needed. During the workshops, managers, psychologists, developers and some end-users behaved like structuralists, while designers and end-users with a design background displayed the behaviour of inspirationalists. Psychologists and managers required sources of insights to be inspired and generate ideas, and they used the board more often than other disciplines. On the contrary, designers used fewer sources of insights, such as the cards, for problem finding compared to other disciplines (e.g., end-users, psychologists). This can be explained by the dominance of end-users on problem finding; however, designers provided support for end-users’ problem-finding efforts through questions. In the beginning of the workshops, designers were focused more on active listening, while in later phases, they were focused on drawing and sketching during group discussions. Therefore, their episodic activities focused mainly on idea generation and idea evaluation, when using cards. At the end of the workshops, all disciplines provided feedback on the suggested designs, while designers, developers and managers played a more active role in determining the implementation details. This is indicative of the different needs during the structured processes based on the differences in professional background and work practices.

5.5. Study Limitations and Future Work

One limitation of evaluating DEMO is related to the group dynamics during the workshops. Apart from the negotiation processes among the participants, other interactional processes occurred, such as persuading and educating others. This was perceived by some participants as a negative factor regarding their active participation in discussions. On the other hand, politeness among participants was perceived as a non-productive factor for the delivery of high-quality outcomes, leading to the ‘groupthink trap’ [52].

Another limitation for group creativity involves the dominant viewpoints that were likely adopted by the group without the critical examination of other alternatives. Therefore, if there is no facilitator, the role of the manager is of great significance to balance the group dynamics, to achieve the best possible outcome. Although all managers had extensive experience in managing group dynamics, a new design process often requires more training time. Regarding the roles within the multidisciplinary teams, they were assigned based on participants’ background, but during the design process the participants were not restricted to their roles and participated more openly in the discussions.

From a methodological point of view, a limited number of relevant studies on design tools, was found to justify the results. Given the lack of similar studies and evaluations of frameworks, part of the results are interpreted based on design literature.

In future work, it is suggested that DEMO could be applied in other domains and contexts, such as education and learning, health and wellness, online communities and social networks that lack of user motivation. In these cases, DEMO should be tailored to a specific context to be efficient, with an embedded domain-specific vocabulary and theories on the cards. In any application context, a longitudinal study of the design process using DEMO should be organised to examine the evolvement of the design process throughout a project based on user motivation. In addition, the exploration of other artefacts and materials in relation with the application domain may be relevant.

6. Conclusions

This study examined the design process of developing motivational innovation platforms and analysed the contribution of the tool’s features to this process. Structured and artefact-based design tools, i.e., DEMO, show promising design directions in the field to organise a design process. Also, multidisciplinary and user-centred processes are essential to developing well-defined motivation concepts.

The use of artefacts inspires participants’ creativity and helps multidisciplinary groups to negotiate the meanings of motivation, while structured tools stimulate creativity and support the creation of a shared vocabulary for multidisciplinary groups when designing for motivation. The findings also demonstrate that the full creative picture of designing for motivation emerges when episodic activities are enriched by the activities of the multidisciplinary group using the tool. Furthermore, the evaluation study showed that DEMO encourages creativity in conceptualising motivation and suggesting implementation techniques in a time-efficient workshop. By analysing the design process using various methods, the sequences of episodic activities and events were captured, as well as the actual use of the tool and participants’ contributions. Most importantly, the study indicates ‘how’ the participants negotiated motivation concepts, ‘which’ topics these negotiations involved most and ‘who’ among the team was responsible for each activity during the design process.

The DEMO tool presents a novel approach to designing motivational interactive systems. Design for motivation, as a design practice, embeds theories, concepts and practices from related fields and makes available a wide-spectrum of the existing tools to the users. The findings of the current study highlight several implications for designers, practitioners and researchers who work with motivation issues in interactive systems. First, the DEMO tool can be applied and tested in other contexts as well with appropriate modifications of the terms on the cards, if needed, according to the context-specific literature. Examples of use contexts could be any interactive system or community, such as social media communities and online communities of practice. Second, the methodological approach based on episodic activities provided an insightful approach and highlighted some of the more salient design factors regarding the use of the tool in practice and it can be employed by similar studies. Lastly, the workshops’ procedure, including the conceptual and implementation phases, provided an efficient way to quickly extract design concepts during the early design phases, revealing the consensus of the development team.