Development and Evaluation of an mHealth App That Promotes Access to 3D Printable Assistive Devices
Abstract
:1. Introduction
2. Materials and Methods
2.1. Pre-App Development: Establishing the Devices through Experiential Learning
- Tags: Categorization of each design based on the World Health Organization’s International Classification of Functioning, Disability, and Health (ICF) categories [27] to ensure the reach of 3DAdapt would be accepted by international stakeholders and to communicate with all healthcare professionals and collaborative stakeholders within a common language. Each device was tagged into the three major ICF categories: body function, activities and participation, and environmental factors.
- Purpose: A short summary (1–2 sentences) of the AT design including its purpose and any special caveats (such as additional supplies or tools needed to assemble).
- Simple Name: A design name using the common practice language of a rehabilitation professional and what a PWD may understand to be helpful for them. For example, the original title of “Fork and spoon support for person with disabilities” (https://www.myminifactory.com/object/3d-print-fork-and-spoon-support-for-person-with-disabilities-5480, accessed on 30 April 2024) was renamed to “Universal cuff for utensils”. The original device name was still provided for search purposes.
- Printsets: To overcome the need to determine which of the necessary STL file(s), and potentially multiple STL files, to download and use, all necessary STL files to make up the entirety of a single AT device were uploaded as a new term titled “printset”, which is the collection of all necessary STL files, pre-scaled, in a “one-click” download.
2.2. Design and Development of the Mobile App
2.2.1. Design of Mobile App
2.2.2. Design of the Server
- Availability: If the server experiences unforeseen problems, requests can be rerouted to alternative instances without noticeable interruption to the end user.
- Reliability: An instance of the server will give consistent responses regardless of how internet traffic is routed or loads are balanced across multiple servers.
- Scalability: The resources utilized by the server can be automatically scaled based on the demand for the service, reducing the cost to keep the service running by only having the resources needed at any given moment in use.
- Basic Information Collections:
- users—private information about users (emails, permissions, subscriptions, etc.)
- profiles—public information about users (bio, URLs to photos, etc.)
- designs, designs.files, and designs.printsets—information about the designs
- makes—the information about the design reviews (internally referred to as makes)
- collections and collections.designs—the information about the collections
- designs.comments and makes.comments—the comments posted on designs and reviews
- tags and categories—the available tags and categories used by the designs
- licenses—the available licenses for designs and their details
- Relational Collections that associate between other collections:
- likes—which designs, reviews, and collections are liked by which users
- watches—which designs, reviews, and collections are watched by which users
- followers—which users and following which other users
- Collections that store temporary or cached information:
- designs.files.customizations and designs.printsets.customizations—information about recently customized (or currently generating) models
- importers.cache—temporary information for importing designs from other sites
- email_verifications, password_reset, and token_blocklist—collections used to assist in verifying users, resetting passwords, and logging users out securely
- email_queue and updates—collections used to send users notifications of updates that they have requested
2.2.3. Deployment of the Server
2.3. Beta Testing and App Evaluation
Expert Content Analysis
- Participants and App orientation
- Data Collection
- Data Analysis
3. Results of Expert App Review
3.1. Results of Qualitative Content Analysis
3.1.1. Theme 1: Utility and Relevance of 3DAdapt
- Acceptability
- Demand
- Implementation
- Practicality
- Adaptation
- Integration
3.1.2. Theme 2: Actionable Steps for Modification
3.1.3. Theme 3: Areas for Improvement in the Future
3.2. App Modifications Based on the Content Analysis
4. Discussion and Conclusions
4.1. Overview
4.2. Implications
4.3. Strengths, Limitations, and Future Directions
4.4. Principal Findings
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Appendix A. Interview Guide for the Experts
- Interview Protocol
- Permission: First of all, would you mind if we video record this zoom call because that would make it easier for me to summarize the results afterward?
- Questions for Experts
- Could you talk about your experience as an OT? What type of practice areas have you worked in? Could you describe your experiences and expertise with the application of 3D printing tech with AT provision?
- Do you have experience with evaluating health apps like this one? And if so, could you tell us something about what kind of rehabilitation apps you have evaluated?
- Quality of information
- What do/did you like about this app?
- Prompt: Does the app fulfill your needs? Why or why not? Do/did you enjoy sessions with your health app? Is/Was working with the app satisfying? Is the app worth recommending to others?
- How easy is/was using the app?
- Prompt: What makes the app information clear and understandable? How do/did you find the font size and representations?
- Who do you think could benefit from this app?
- Is app content correct, well-written, and relevant to the goal/topic of the app?
- Have you sometimes not known (did you sometimes not know) what to do next with this app?
- Prompt: Are there any parts that you do not use because they are complicated? What app features do/did you find intuitive? Do you sometimes wonder if you are using the app the right way? Who do you turn to help using the app?
- What medical or technical jargon have you seen in this app that a potential app user may not understand?
- Are you satisfied with the time taken to perform tasks on this app?
- Prompt: How about time to display graphs? Time to synchronize information?
- Acceptability/Benefit to users
- Could you discuss to what extent this app would be judged as suitable or attractive among the intended recipients of the app?
- Could you think about what benefits this app is going to provide the families, caregivers, and professionals working with users needing assistive technology? For example,
- Awareness: This app is likely to increase awareness of the importance of addressing…
- Knowledge: This app is likely to increase knowledge/understanding of…
- Attitudes: This app is likely to change attitudes toward improving…
- Intention to change: This app is likely to increase intentions/motivations to address…
- Help-seeking: Use of this app is likely to encourage further help-seeking for [insert target health behavior]…
- Behavior change: Use of this app is likely to increase/decrease…
- As you navigate the search-and-find features, are the devices you find practical for the end user?
- Demand
- Have any users expressed interest in ways of searching and finding 3D devices and needing a method that is more user-friendly?
- Prompt/probe: How much demand for this AT app is likely to exist among users who would benefit from it?
- Effective implementation
- Could you discuss the extent, likelihood, and manner in which we will be able to make this app available for users as planned and proposed? What barriers do you foresee?
- Prompt: What existing barriers do you foresee to implementation? [I.e., sots, access, user needs and preferences, and availability of similar apps.] What recommendations do you have on how to overcome them?
- How much and what type of resources will be needed to implement it?
- What factors can you foresee affecting the ease or difficulty of implementing the app?
- Based on your experiences, how can we, as rehabilitation professionals, help the users of this app better?
- Prompt: In your opinion, what is the best way to collaborate with users to train about this app? How can we spread awareness about this app?
- Practicality
- To what extent can this app be introduced among intended users using existing resources and current rehabilitation circumstances without outside support?
- What are your thoughts on the privacy protection concerns of using this app?
- Efficiency, speed, or quality of implementation
- Could you think about any positive/negative effects on the intended users?
- Ability of participants to use this app?
- Integration
- Could this app be integrated into the workday? Are there barriers?
- Research
- Clinical applicability
- Classroom instruction
- Thinking of your existing organizational setting where you provide 3D printing, how easy or difficult would it be to integrate this app into your available services to clients? Probe: What is the fit with your organization’s current infrastructure? What are your thoughts on the sustainability of this app for the future of how you design or deliver 3D-printed devices as part of your services or research? Do you foresee any costs (monetary or other perceived barrier) associated with integrating this app into your organization’s offerings?
- Adaptation
- Could you think about some changes/modifications we need to make to accommodate different users’ environmental and social contexts?
- Prompt: What about possible modifications considering potential users’ varying disabilities and accessing the information?
- Two Aspects: (1) Thoughts on the adaptive devices that are searchable and included within 3DAdapt? Does this consider adaptive devices for multiple different disabilities? (2) Thoughts on the user experience of the app itself? Does it lend itself to be accessible to users with various levels of abilities?
- Prompt: What makes the app information clear and understandable? How do/did you find the font size and representations?
- What customization features would you like to see in this health app?
- Wrap-up questions
- Is there anything I did not ask that we should consider for refining the app further?
- Thank you again for taking the time to meet with us. Please rest assured that your anonymity will be protected and that your name will not be mentioned in any of the reports or presentations.
References
- Dawson, R.M.; Felder, T.M.; Donevant, S.B.; McDonnell, K.K.; Card, E.B., III; King, C.C.; Heiney, S.P. Identifying the strengths and limitations of existing mobile application evaluation tools. Nurs. Inq. 2020, 27, 1–8. [Google Scholar] [CrossRef] [PubMed]
- Vawdrey, D.K.; Wilcox, L.G.; Collins, S.A.; Bakken, S.; Feiner, S.; Boyer, A.; Restaino, S.W. A tablet computer application for patients to participate in their hospital care. In Proceedings of the AMIA Annual Symposium Proceedings Archive, Washington DC, USA, 22–26 October 2011; pp. 1428–1435. [Google Scholar]
- Higgins, J.P. Smartphone applications for patients’ health and fitness. Am. J. Med. 2016, 129, 11–19. [Google Scholar] [CrossRef] [PubMed]
- Paglialonga, A.; Patel, A.A.; Pinto, E.; Mugambi, D.; Keshavjee, K. The healthcare system perspective in mHealth. In m_Health Current and Future Applications; Springer: Berlin/Heidelberg, Germany, 2019; pp. 127–142. [Google Scholar]
- Roumeliotis, K.; Tselikas, N.D. Evaluating progressive web app accessibility for people with disabilities. Network 2022, 2, 350–369. [Google Scholar] [CrossRef]
- Gupta, N.; Castillo-Laborde, C.; Landry, M.D. Health-related rehabilitation services: Assessing the global supply of and need for human resources. BMC Health Serv. Res. 2011, 11, 276. [Google Scholar] [CrossRef]
- Tangcharoensathien, V.; Witthayapipopsaku, W.; Viriyathorn, S.; Patcharanarumol, W. Improving access to assistive technologies: Challenges and solutions in low-and middle-income countries. WHO South-East Asia J. Public Health 2018, 7, 84–89. [Google Scholar] [CrossRef] [PubMed]
- Lu, Y.; Wu, Y.; Liu, J.; Li, J.; Zhang, P. Understanding health care social media use from different stakeholder perspectives: A content analysis of an online health community. J. Med. Internet Res. 2017, 19. [Google Scholar] [CrossRef] [PubMed]
- Schwartz, D.A.; Schofield, K.A. Utilization of 3D printed orthoses for musculoskeletal conditions of the upper extremity: A systematic review. J. Hand Ther. 2021, 36, 166–178. [Google Scholar] [CrossRef]
- Schwartz, J.K.; Fermin, A.; Fine, K.; Iglesias, N.; Pivarnik, D.; Struck, S.; Varela, N.; Janes, W.E. Methodology and feasibility of a 3D printed assistive technology intervention. Disabil. Rehabil. Assist. Technol. 2020, 15, 141–147. [Google Scholar] [CrossRef] [PubMed]
- Janson, R.; Burkhart, K.; Firchau, C.; Hicks, K.; Pittman, M.; Yopps, M.; Hatfield, S.; Garabrant, A. Three-dimensional printed assistive devices for addressing occupational performance issues of the hand: A case report. J. Hand Ther. 2020, 33, 164–169. [Google Scholar] [CrossRef]
- Rasmussen, K.-A.M.; Stewart, B.C.; Janes, W.E. Feasibility of customized 3D-printed assistive technology within an existing multidisciplinary amyotrophic lateral sclerosis clinic. Disabil. Rehabil. Assist. Technol. 2021, 18, 1466–1472. [Google Scholar] [CrossRef]
- Thorsen, R.; Cugnod, D.; Ramella, M.; Converti, R.; Ferrarin, M. A parametric 3D printed assistive device for people with cerebral palsy–assessment of outcomes and comparison with a commercial counterpart. Assist. Technol. 2023, 36, 16–21. [Google Scholar] [CrossRef] [PubMed]
- Patterson, R.M.; Salatin, B.; Janson, R.; Salinas, S.; Mullins, M.J. A current snapshot of the state of 3D printing in hand rehabilitation. J. Hand Ther. 2020, 33, 156–163. [Google Scholar] [CrossRef] [PubMed]
- Best 3D Printing Software. Available online: https://www.g2.com/categories/3d-printing?selected_view=grid&segment=all#grid (accessed on 15 September 2023).
- Developer Documentation. Available online: https://customizer.makerbot.com/docs (accessed on 15 September 2023).
- Getting Started. Available online: https://models.makewithtech.com/help (accessed on 15 September 2023).
- Getting Started. Available online: https://learn.cadhub.xyz/docs/ (accessed on 15 September 2023).
- Keller, J. Stack Overflow’s Crowdsourcing Model Guarantees Success. 2010. Available online: https://www.theatlantic.com/technology/archive/2010/11/stack-overflows-crowdsourcing-model-guarantees-success/66713/ (accessed on 15 September 2023).
- Majchrzak, A.; Malhotra, A. Towards an information systems perspective and research agenda on crowdsourcing for innovation. J. Strateg. Inf. Syst. 2013, 22, 257–268. [Google Scholar] [CrossRef]
- Karachiwalla, R.; Pinkow, F. Understanding crowdsourcing projects: A review on the key design elements of a crowdsourcing initiative. Creat. Innov. Manag. 2021, 30, 563–584. [Google Scholar] [CrossRef]
- Benham, S.; San, S. Student Technology Acceptance of 3D Printing in Occupational Therapy Education. Am. J. Occup. Ther. 2020, 74, 7403205060p1–7403205060p7. [Google Scholar] [CrossRef] [PubMed]
- Benham, S.; Bush, J.; Curley, B. 3D printing applications through peer-assisted learning and interprofessional education approaches. Focus Health Prof. Educ. A Multi-Prof. J. 2022, 23, 81–95. [Google Scholar] [CrossRef]
- Slegers, K.; Krieg, A.M.; Lexis, M.A.S. Acceptance of 3D Printing by occupational therapists: An exploratory survey study. Occup. Ther. Int. 2022, 2022, 4241907. [Google Scholar] [CrossRef] [PubMed]
- Giesbrecht, E.; Carreiro, N.; Mack, C. Improvement and retention of wheelchair skills training for students in entry-level occupational therapy education. Am. J. Occup. Ther. 2021, 75, 7501205160p1–7501205160p9. [Google Scholar] [CrossRef] [PubMed]
- Benham, S.; Milstrey, B.; Stemple, J.; Davis, J.; Scatena, D.; Bush, J.; Kolakowsky-Hayner, S.; Amy, K. Mobile device accessibility with 3D printed devices for individuals with physical disabilities. Disabil. Rehabil. Assist. Technol. 2023, 1–6. [Google Scholar] [CrossRef]
- Geneva World Health Organization. International Classification of Functioning, Disability and Health ICF. 2001. Available online: https://icd.who.int/dev11/l-icf/en (accessed on 15 September 2023).
- Aigner, S. Compose Multiplatform for iOS Is in Alpha. 18 May 2023. Available online: https://blog.jetbrains.com/kotlin/2023/05/compose-multiplatform-for-ios-is-in-alpha/ (accessed on 15 September 2023).
- Jake-Schoffman, D.E.; Silfee, V.J.; Waring, M.E.; Boudreaux, E.D.; Sadasivam, R.S.; Mullen, S.P.; Carey, J.L.; Hayes, R.B.; Ding, E.Y.; Bennett, G.G.; et al. Methods for Evaluating the Content, Usability, and Efficacy of Commercial Mobile Health Apps. JMIR Mhealth Uhealth 2017, 5, e190. [Google Scholar] [CrossRef]
- Abras, C.; Maloney-Krichmar, D.; Preece, J. User-Centered Design. In Encyclopedia of Human-Computer Interaction, Sage Publications; Bainbridge, W., Ed.; Sage: Thousand Oaks, CA, USA, 2004; pp. 445–456. [Google Scholar]
- Benner, P. From novice to expert. Menlo Park 1984, 84, 10–97. Available online: https://files.eric.ed.gov/fulltext/ED384695.pdf#page=130 (accessed on 30 April 2024). [CrossRef]
- Patton, M.Q. Qualitative Research & Evaluation Methods: Integrating Theory and Practice, 4th ed.; Sage: Thousand Oaks, CA, USA, 2014. [Google Scholar]
- Bowen, D.J.; Kreuter, M.; Spring, B.; Cofta-Woerpel, L.; Linnan, L.; Weiner, D.; Bakken, S.; Kaplan, C.P.; Squiers, L.; Fabrizio, C.; et al. How we design feasibility studies. Am. J. Prev. Med. 2009, 36, 452–457. [Google Scholar] [CrossRef] [PubMed]
- Lagan, S.; Emerson, M.R.; King, D.; Matwin, S.; Chan, S.R.; Proctor, S.; Tartaglia, J.; Fortuna, K.L.; Aquino, P.; Walker, R.; et al. Mental health app evaluation: Updating the American Psychiatric Association’s framework through a stakeholder-engaged workshop. Psychiatr. Serv. 2021, 72, 1095–1098. [Google Scholar] [CrossRef] [PubMed]
- Erlingsson, C.; Brysiewicz, P. A hands-on guide to doing content analysis. Afr. J. Emerg. Med. 2017, 7, 93–99. [Google Scholar] [CrossRef] [PubMed]
- Hsieh, H.-F.; Shannon, S.E. Three approaches to qualitative content analysis. Qual. Health Res. 2005, 15, 1277–1288. [Google Scholar] [CrossRef] [PubMed]
- Elo, S.; Kyngäs, H. The qualitative content analysis process. J. Adv. Nurs. 2008, 62, 107–115. [Google Scholar] [CrossRef] [PubMed]
- Pope, C.; Ziebland, S.; Mays, N. Qualitative research in health care. Analysing qualitative data. BMJ 2000, 320, 114–116. [Google Scholar] [CrossRef]
- Saunders, B.; Sim, J.; Kingstone, T.; Baker, S.; Waterfield, J.; Bartlam, B.; Burroughs, H.; Jinks, C. Saturation in qualitative research: Exploring its conceptualization and operationalization. Qual. Quant. 2018, 52, 1893–1907. [Google Scholar] [CrossRef]
- Jaspers, MW A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int. J. Med. Inform. 2009, 78, 340–353. [CrossRef]
- Petrie, H.; Hamilton, F.; King, N.; Pavan, P. Remote usability evaluations with disabled people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada, 22–27 April 2006; pp. 1133–1141. [Google Scholar] [CrossRef]
- Kushniruk, A.W.; Patel, V.L. Cognitive and usability engineering methods for the evaluation of clinical information systems. J. Biomed. Inform. 2004, 37, 56–76. [Google Scholar] [CrossRef]
- Principles for Digital Development. Available online: https://digitalprinciples.org/ (accessed on 30 April 2024).
- Pereira-Azevedo, N.; Carrasquinho, E.; Cardoso de Oliveira, E.; Cavadas, V.; Osório, L.; Fraga, A.; Castelo-Branco, M.; Roobol, M.J. mHealth in Urology: A Review of Experts’ Involvement in App Development. PLoS ONE 2015, 10, e0125547. [Google Scholar] [CrossRef] [PubMed]
- WHO Guideline Recommendations on Digital Interventions for Health System Strengthening; World Health Organization: Geneva, Switzerland, 2019. Available online: https://www.ncbi.nlm.nih.gov/books/NBK541902/ (accessed on 30 April 2024).
Phase 1: Expert feedback on initial usability and recommended app modifications |
|
App Modification: Developers addressed the issues and suggestions that emerged from the expert feedback. | |
Phase 2: Usability testing with care partners, PWDs, and clinicians |
|
App Modification: Developers addressed the issues that emerged from the usability testing. | |
Phase 3: Efficacy testing of 3DAdapt with PWDs and novice clinician pairs |
|
App Modification: Developers addressed the issues identified from the efficacy testing. |
Expert | Specialization and Employment Setting | Relevant Experience and Expertise |
---|---|---|
Expert A1 | United States (US)-based licensed occupational therapist (OT) employed by their state’s designated AT Act Program in the northeast. |
|
Expert B2 | US-based licensed OT and a faculty member at a public R1 institution in the midwest. |
|
Expert M3 | A US-based licensed OT for over 25 years for a large non-profit healthcare system in the midwest operating 12 hospitals and 90 clinics. |
|
Themes | Sub-Themes |
---|---|
1. Utility and relevance of 3DAdapt | Acceptability |
Demand | |
Implementation | |
Practicality | |
Adaptation | |
Integration | |
2. Actionable steps for modification | See Table 4 below for a detailed list of actionable steps |
3. Areas for improvement in the future | Different levels of permissions for various users Need for and ways to market the app |
Suggestions for enhancing user experience | |
Sustaining user interest via community engagement |
Actionable Steps for Modification | Efforts Made to Improve the App |
---|---|
Confusion between printset (a word invented for this mobile app) and files | Made it so that the average user is not even exposed to this level of detail but an advanced user will see definitions and purpose explanations when they look for the details. |
Horizontal scroll is not intuitive in several places | Improved displays to not require horizontal scrolling at all. |
Uniform style of description. The name and purpose of each design is not obvious in the user experience | When looking through designs, users need to see the curated title and purpose first rather than the original source title and description, which may even be in a different language. The original information can still shown but is initially hidden. |
More pediatric options | Added more options for children and added pre-sized printsets with smaller sizing. |
Liability concerns and not comfortable recommending | Enhanced the legal safeguards in the “terms and conditions” so healthcare personnel would not be responsible for users’ interactions in the app. |
Fonts used are too large | Tested to make sure the phone’s accessibility settings for font size are used. |
Account creation needs to be clarified with passwords | Added information about password requirements and clarity to error messages. |
Customization | Added the ability to customize parametric models and clarified that sizing is typically in millimeters. |
Q&A crowdsource | Encouraging comments with the description of how to use comments and added tab for “FAQ/Help” overall for the app. |
Notification and email frequency | Added frequency options to user preferences. |
Privacy protection concerns | Added prompt in photos not to include any personal, identifiable information, and checked for scrubbing of geotagging in photos. |
Complex post-processing, filtering, and comments | Added search filters for customizable and more than one file to assemble. Added prompts in the reviews to write information if significant post-processing modifications were required. |
ICF categories may not be helpful for non-professional users | Simplified keywords used for ICF categories but kept full descriptions when performing a “hard press”. |
Private messaging is not a priority | Removed “messages” to encourage open public comments and reviews. |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bush, J.; Benham, S.; Kaniamattam, M. Development and Evaluation of an mHealth App That Promotes Access to 3D Printable Assistive Devices. Technologies 2024, 12, 114. https://doi.org/10.3390/technologies12070114
Bush J, Benham S, Kaniamattam M. Development and Evaluation of an mHealth App That Promotes Access to 3D Printable Assistive Devices. Technologies. 2024; 12(7):114. https://doi.org/10.3390/technologies12070114
Chicago/Turabian StyleBush, Jeffrey, Sara Benham, and Monica Kaniamattam. 2024. "Development and Evaluation of an mHealth App That Promotes Access to 3D Printable Assistive Devices" Technologies 12, no. 7: 114. https://doi.org/10.3390/technologies12070114