**5. Evaluation**

The evaluation was conducted through a series of interviews with specialists in nutrition and physical activity from three different countries: Spain (one male and two females), Mexico (two males and one female) and Costa Rica (one male and two females). *Sensors* **2018**, *18*, 271 11 of 21 **5. Evaluation**

For each interview, a physical prototype and a storyboard were presented to the participant and a series of questions were asked. In each session, the same interview protocol was used. The protocol consisted of three main parts: participants' information, prototype usefulness and prototype usability/ergonomics. The questions in the protocol are detailed in Appendix D. The evaluation was conducted through a series of interviews with specialists in nutrition and physical activity from three different countries: Spain (one male and two females), Mexico (two males and one female) and Costa Rica (one male and two females). For each interview, a physical prototype and a storyboard were presented to the participant and

The goal with this evaluation was to assess the system's perceived usefulness and usability. Moreover, the thoughts of the specialists were gathered and added to the prototypes. a series of questions were asked. In each session, the same interview protocol was used. The protocol consisted of three main parts: participants' information, prototype usefulness and prototype usability/ergonomics. The questions in the protocol are detailed in Appendix D.

The system was presented using the physical device and storyboards [31]. Using storyboards, we presented the main use cases for the prototypes and their benefits. Participants were presented with two storyboards, one for healthy eating behaviors and the other for sportswear reminders. The goal with this evaluation was to assess the system's perceived usefulness and usability. Moreover, the thoughts of the specialists were gathered and added to the prototypes. The system was presented using the physical device and storyboards [31]. Using storyboards,

The developed interview protocol and questionnaire was designed using two main references: The first set of questions (focused on the prototype) were based on different scales [32] with the intention of gather the perspective of the participants on the usefulness of the system. The second part was based on the standard ISO 9241-210:2010 Ergonomics of human-system interaction [33]. we presented the main use cases for the prototypes and their benefits. Participants were presented with two storyboards, one for healthy eating behaviors and the other for sportswear reminders. The developed interview protocol and questionnaire was designed using two main references: The first set of questions (focused on the prototype) were based on different scales [32] with the

To provide a better understanding we will describe our interpretation of the ISO 9241-210:2010 terminology. By effectiveness we mean the accuracy and completeness with which users achieve specified goals. By efficiency we mean resources expended in relation to the accuracy and completeness with which users achieve goals. By satisfaction we mean freedom from discomfort and positive attitudes towards the use of the product. intention of gather the perspective of the participants on the usefulness of the system. The second part was based on the standard ISO 9241-210:2010 Ergonomics of human-system interaction [33]. To provide a better understanding we will describe our interpretation of the ISO 9241-210:2010 terminology. By effectiveness we mean the accuracy and completeness with which users achieve specified goals. By efficiency we mean resources expended in relation to the accuracy and completeness with which users achieve goals. By satisfaction we mean freedom from discomfort and

Conciseness, consistency and self-descriptiveness deal with the amount of information required to deliver the message and the way in which this data is displayed every time. The characteristics of presented information (i.e., detectability, legibility, discriminability, clarity and comprehensibility) are related to freedom from distraction and interpretability. positive attitudes towards the use of the product. Conciseness, consistency and self-descriptiveness deal with the amount of information required to deliver the message and the way in which this data is displayed every time. The characteristics of presented information (i.e., detectability, legibility, discriminability, clarity and comprehensibility) are related to freedom from distraction and interpretability.

Finally, suitable for individualization means that, users can modify interaction and presentation of information to suit their individual needs and suitable for learning means that the system conforms to user's expectations and it prevents or tolerates errors. Finally, suitable for individualization means that, users can modify interaction and presentation of information to suit their individual needs and suitable for learning means that the system conforms to user's expectations and it prevents or tolerates errors.

#### **6. Results and Discussion 6. Results and Discussion**

In this section, we present the results of the interviews. The evaluation protocol was explained in the Evaluation Section. All results are positive. However, some of the evaluators provided feedback and improvement opportunities that we will also discuss. We asked the evaluators to use three words to describe the system. Figure 4 shows the results of this exercise. The most common word used to describe the system was useful (5 out of 9) followed by simple (4 out of 9). In this section, we present the results of the interviews. The evaluation protocol was explained in the Evaluation Section. All results are positive. However, some of the evaluators provided feedback and improvement opportunities that we will also discuss. We asked the evaluators to use three words to describe the system. Figure 4 shows the results of this exercise. The most common word used to describe the system was useful (5 out of 9) followed by simple (4 out of 9).

**Figure 4. Figure 4.** Word cloud of main characterizers used by evaluators. Word cloud of main characterizers used by evaluators.

Some words caught the interviewer's attention. For instance, when a reviewer used the word "accessible", we followed up to determine that she was referring to the possibility of many people to reach the information on the device. This includes everyone in the household and since the system uses graphical representations it is also accessible for children that do not know how to read or write. *Sensors* **2018**, *18*, 271 12 of 21

One of the interviewees also used the word "basic" we followed up to determine that she saw potential to add other functionalities to the device. Therefore, the version presented was too basic. Another word was "organized" delving in this we determined that the characterization was for the users and not the system. Therefore, organized people are potential users of the system. Some words caught the interviewer's attention. For instance, when a reviewer used the word "accessible", we followed up to determine that she was referring to the possibility of many people to reach the information on the device. This includes everyone in the household and since the system uses graphical representations it is also accessible for children that do not know how to read or write.

Two questions were asked to the evaluators to assess their willingness to either use the system or to recommend it to their friends, family or patients. Figure 5 shows the answers to this question. It is interesting to observe that nine out of nine evaluators (specialists in nutrition and physical activity) are somewhat interested in using the system. However, eight out of nine evaluators consider that they would recommend the system. To further explain this, we must note that usually specialists in nutrition and physical activity have a healthy lifestyle, therefore they might not see fit to use the system. However, during the interviews most of the evaluators stated that remembering healthy snacks was one of the main problems of their patients. One of the interviewees also used the word "basic" we followed up to determine that she saw potential to add other functionalities to the device. Therefore, the version presented was too basic. Another word was "organized" delving in this we determined that the characterization was for the users and not the system. Therefore, organized people are potential users of the system. Two questions were asked to the evaluators to assess their willingness to either use the system or to recommend it to their friends, family or patients. Figure 5 shows the answers to this question. It is interesting to observe that nine out of nine evaluators (specialists in nutrition and physical activity) are somewhat interested in using the system. However, eight out of nine evaluators consider that they would recommend the system. To further explain this, we must note that usually specialists

Some of the evaluators stated that they would not use the system as it is. But they would reconsider if it included some of their feedback (especially screen size and the amount of information that can be delivered through the device). The evaluation had two main focuses, the first one was to assess the evaluator's opinion on the system, the second one was focused on the look and feel of the device (human factors and ergonomics perspective). We asked a question regarding the evaluators' satisfaction with the look and feel of the devices. All the answers were positive (5 somewhat satisfied, 3 very satisfied and 1 extremely satisfied). Figure 6 shows the responses to this question. in nutrition and physical activity have a healthy lifestyle, therefore they might not see fit to use the system. However, during the interviews most of the evaluators stated that remembering healthy snacks was one of the main problems of their patients. Some of the evaluators stated that they would not use the system as it is. But they would reconsider if it included some of their feedback (especially screen size and the amount of information that can be delivered through the device). The evaluation had two main focuses, the first one was to assess the evaluator's opinion on the system, the second one was focused on the look and feel of the device (human factors and ergonomics perspective). We asked a question regarding the evaluators'

Most of the improvement opportunities mentioned by the evaluators include: enlarging the device screen and making it in color, changing the shape of the case, adding sound to the notifications, and adding text to the notification and not only images. This were the main concerns regarding the device. satisfaction with the look and feel of the devices. All the answers were positive (5 somewhat satisfied, 3 very satisfied and 1 extremely satisfied). Figure 6 shows the responses to this question. Most of the improvement opportunities mentioned by the evaluators include: enlarging the device screen and making it in color, changing the shape of the case, adding sound to the notifications, and adding text to the notification and not only images. This were the main concerns regarding the

One of the evaluators recommended to add text, this text would be used to provide information about the ingredients of the snack. In Costa Rica, the two of the evaluators asked to change the shape of the snack reminder case to an apple as it is traditional in Costa Rica to have an apple shaped magnet fridge. device. One of the evaluators recommended to add text, this text would be used to provide information about the ingredients of the snack. In Costa Rica, the two of the evaluators asked to change the shape of the snack reminder case to an apple as it is traditional in Costa Rica to have an apple shaped magnet fridge.

**Figure 5. Figure 5.** Likelihood to use the system or recommend the system. Likelihood to use the system or recommend the system.

Another important observation was that some of the evaluators compared the system with mobile applications with similar functionalities. Other evaluator requested for the system to sync with mobile applications. Another important observation was that some of the evaluators compared the system with mobile applications with similar functionalities. Other evaluator requested for the system to sync with mobile applications.

*Sensors* **2018**, *18*, 271 13 of 21

Delving on the device functionalities, most of the comments were focused on the snacking reminder device. The first recommendation was to add reminders not only to people to carry their snacks but also to prepare them in advance. This would be helpful to assure that the snacks are ready to go when they are required. One evaluator recommended to use only one device with more information (i.e., combining both devices). Delving on the device functionalities, most of the comments were focused on the snacking reminder device. The first recommendation was to add reminders not only to people to carry their snacks but also to prepare them in advance. This would be helpful to assure that the snacks are ready to go when they are required. One evaluator recommended to use only one device with more information (i.e., combining both devices).

The last part of the evaluation focused on the ergonomics of human-system interaction. In this section, 13 characteristics of the system were assessed by the evaluators. Figure 7 shows the results of this evaluation segmented by country. Table 4 shows the five-number summary and a reliability analysis of the data depicted in Figure 7. The last part of the evaluation focused on the ergonomics of human-system interaction. In this section, 13 characteristics of the system were assessed by the evaluators. Figure 7 shows the results of this evaluation segmented by country. Table 4 shows the five-number summary and a reliability analysis of the data depicted in Figure 7.

**Figure 6.** Evaluator's satisfaction with the look and feel of the system. Scale (0, 3, 6, 0) represents the number of participants in each category. **Figure 6.** Evaluator's satisfaction with the look and feel of the system. Scale (0, 3, 6, 0) represents the number of participants in each category.

An interesting finding of this evaluation is that both Mexico and Costa Rica on average consider the system good (4.3 out of 5 points). However, in Spain, the opinions were less positive (3.9 out of 5 points). We do believe that being Latin-American countries, Costa Rica and Mexico share opinions on how the system could be used. An interesting finding of this evaluation is that both Mexico and Costa Rica on average consider the system good (4.3 out of 5 points). However, in Spain, the opinions were less positive (3.9 out of 5 points). We do believe that being Latin-American countries, Costa Rica and Mexico share opinions on how the system could be used.

In general, the best ranked categories were system's comprehensibility, the perceived effectiveness and clarity (4.5 out of 5 points). The worst ranked categories are the system's suitability for learning and it discriminability (3.7 out of 5 points). In general, the best ranked categories were system's comprehensibility, the perceived effectiveness and clarity (4.5 out of 5 points). The worst ranked categories are the system's suitability for learning and it discriminability (3.7 out of 5 points).

with mobile applications.

information (i.e., combining both devices).

analysis of the data depicted in Figure 7.

number of participants in each category.

on how the system could be used.

Another important observation was that some of the evaluators compared the system with mobile applications with similar functionalities. Other evaluator requested for the system to sync

Delving on the device functionalities, most of the comments were focused on the snacking reminder device. The first recommendation was to add reminders not only to people to carry their snacks but also to prepare them in advance. This would be helpful to assure that the snacks are ready to go when they are required. One evaluator recommended to use only one device with more

The last part of the evaluation focused on the ergonomics of human-system interaction. In this section, 13 characteristics of the system were assessed by the evaluators. Figure 7 shows the results of this evaluation segmented by country. Table 4 shows the five-number summary and a reliability

**Figure 6.** Evaluator's satisfaction with the look and feel of the system. Scale (0, 3, 6, 0) represents the

An interesting finding of this evaluation is that both Mexico and Costa Rica on average consider the system good (4.3 out of 5 points). However, in Spain, the opinions were less positive (3.9 out of 5 points). We do believe that being Latin-American countries, Costa Rica and Mexico share opinions

In general, the best ranked categories were system's comprehensibility, the perceived

**Figure 7. Figure 7.** Evaluation of system characteristics based on the standard ISO 9241-210:2010. Evaluation of system characteristics based on the standard ISO 9241-210:2010.

**Table 4.** Minimum, maximum, quartiles and reliability analysis for Likert scales depicted in Figure 7. All items Cronbach Alpha: 0.8307, Std. Alpha: 0.8431. Reliability analysis calculated using [34].


A system is suitable for learning when it supports and guides the user in learning to use the system. In our case since the notification and device are supposed to share little information the suitability for learning suffers. To be able to learn to use the system a simple guide should be provided. The discriminability problem is mostly related to the device screen size and resolution. The implemented prototype used a 0.96-inch monochrome OLED graphic display; to improve this, a bigger screen should be used. Moreover, images that avoid ambiguity could be used. Enlarging the screen would also allow for text to be shown, improving discriminability of the notifications.

The final improvement opportunities for the system are focused on the notifications. One evaluator suggested adding personalization based on the user's data and preferences (already considered in the system). However, she added the possibility to add diseases and allergies into the notifications. Another evaluator suggested the necessity of creating profiles (also considered in the system), but he requested to be able to change between profiles using the device.

In general, one recommendation is to add more nutrition facts of the snacks. For instance, alimentary groups, portion sizes and water consumption.

One confronted opinion between evaluators was if the personalization should be performed by the user or if the definition of snacks should be personalized by the nutritionist. One of the evaluators suggested that she would like to be able to specifically set the food characteristics for a given day in their patient's device. Another evaluator suggested doing a consumption analysis of the user. However, this would require sensing capabilities that the system does not provide as it is designed for notifications.

The last opinion on the system is that two evaluators would prefer this to be a mobile application rather than a physical device. The main reason provided is that most people already have a mobile device. During the design phase of this system, this discussion was carried.

One of the main arguments to avoid push notifications on smart devices is that the indiscriminate use of these notifications is causing people to stop paying attention to them. Social media notifications, dedicated apps (e.g., Netflix, Yelp, and Amazon), email, and other mobile phone notifications are just some examples of this overwhelming amount of notifications.

Finally, all evaluators highlighted the positive effect of using these devices in day to day to prevent overweight and obesity if activities and tasks associated with notifications are carried out properly.
