Next Article in Journal
Class-Wise Classifier Design Capable of Continual Learning Using Adaptive Resonance Theory-Based Topological Clustering
Previous Article in Journal
Acute and Chronic Effects of Static Stretching on Neuromuscular Properties: A Meta-Analytical Review
Previous Article in Special Issue
Experiences of the Telemedicine and eHealth Conferences in Poland—A Cross-National Overview of Progress in Telemedicine
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 13
Issue 21
10.3390/app132111982
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Remote, Touchless Interaction with Medical Images and Telementoring in the Operating Room Using a Kinect-Based Application—A Usability Study

by
Wojciech Michał Glinkowski
1,2,*,
Tomasz Miścior
3 and
Robert Sitnik
3,*
1
Center of Excellence “TeleOrto”, Telediagnostics and Treatment of Disorders and Injuries of the Locomotor System, Department of Medical Informatics and Telemedicine, Medical University of Warsaw, 00-581 Warsaw, Poland
2
Polish Telemedicine and eHealth Society, 03-728 Warsaw, Poland
3
Institute of Micromechanics and Photonics, Warsaw University of Technology, 8 Sw. Andrzeja Boboli Str., 02-525 Warsaw, Poland
*
Authors to whom correspondence should be addressed.
Appl. Sci. 2023, 13(21), 11982; https://doi.org/10.3390/app132111982
Submission received: 14 October 2023 / Revised: 31 October 2023 / Accepted: 1 November 2023 / Published: 2 November 2023
(This article belongs to the Special Issue Pattern Recognition and Medical Data Analytics in Telemedicine)
Browse Figures
Versions Notes

Abstract

:
Innovative technologies can improve user usability and satisfaction in computer-based activities in the work environment, including surgeons working in the operating room (OR). A scrubbed surgeon must ask unscrubbed staff to retrieve medical images from a surgical PACS system on a monitor in a hybrid operating room. The study aimed to check users’ usability and satisfaction with the designed and developed Ortho_Kinect_OR application, which enables contactless control of access to medical images during surgery. The application also facilitates access to telemedicine applications such as intraoperative telementoring during orthopedic procedures or tele-education. The application’s usability was tested by assigning standardized tasks to surgeons for PACS and teleconferencing in the operating room. Hand gestures were recognized and translated to function like mouse buttons. Field tests conducted among orthopedic surgeons showed high usability and user satisfaction. PACS access in the operating room did not distract the orthopedic surgeon during orthopedic surgery procedures. All surgeons completed the tests and tasks without any problems. OR field test results showed high agreement among users and were very satisfactory. Junior surgeons and residents pointed out that people with average computer knowledge could use the application. It has been shown that the contactless system designed and built based on the Kinect sensor available on the shelves meets the requirements of the operating room environment and is easy to use. Touchless controller technology provides the opportunity to increase the use of intraoperative imaging previews and improve the safety of surgical patients by improving sterility and reducing unnecessary staff in the operating room. Using the Ortho_Kinect_OR application and the Kinect sensor, it is possible to provide contactless access to videoconference telementoring and PACS in the operating room without an orthopedic surgeon’s unnecessary distraction in the operating room environment.

1. Introduction

Gesture-based computer interfaces are well-known to many researchers [1]. The widespread use of input video game consoles (such as Microsoft, Xbox Kinect, Nintendo Wii, and PlayStation Move) has taken human–computer interaction (HCI) to entirely new, unexplored levels. There has been extensive research into developing this type of human–computer interaction [2,3,4]. The Picture Archiving and Communication System (PACS) has become a standard in the handling and archiving of medical images displayed in digital form. Imaging is needed on a computer screen in an operating room (OR) environment. The success of the last elective surgery is based on quick access to the preoperative and intra-operative images and the provision of preoperative planning using digital templates [5,6,7,8,9]. Viewing medical images during surgical procedures and turning on teleconferencing applications in a hybrid operating room may be necessary to verify surgical indications or seek the opinion of a consultant remotely as part of telementoring. In the operating room, where the environment requires maintaining sterility, running applications on monitors should also be available to scrubbed surgeons. Viewing patients’ images on a PACS workstation is usually limited to a traditional computer mouse, touchpad, and keyboard. A surgeon wishing to use the computer independently would have to be unscrubbed. Another way is to cooperate with an additional unscrubbed assistant, i.e., an additional person is needed temporarily or permanently in the operating room to operate the PACS workstation. Increasing the number of staff in the operating room may increase the risk of environmental contamination. Almost 50 years ago, the operating room staff was shown to be the primary source of bacteria [10]. Therefore, limiting traffic in the OR is highly recommended [11].
Moreover, additional staff can distract the operating surgeon, increasing operation time. The surgeon can communicate verbally and express the needs of the screen image and the appropriate templates and resources. The procedure itself can be time-consuming, interrupting the work and distracting the surgeon, leading to an increase in the frequency of surgeries [12,13].
Grätzel [14] has shown that, on average, communication with the assistant may consume up to 7 min to click on the exact location indicated by the surgeon. They improve diagnostic imaging during the surgical procedure and maintain appropriate field sterility during the surgery. There has been considerable interest in touch-free image manipulation interfaces. In the current literature, few papers address the problem of the touchless operation of PACS browsers, including Osirix [1,15]. There are some reports concerning such devices used for medical purposes [16,17,18,19,20,21]. There is no evidence in the literature for using a touchless interface that allows viewing and handling medical images and connecting with an associate or supervising surgeon for telementoring sessions or educational purposes directly from the operating room using hand gestures in orthopedics. As necessary from the point of view of functionality, the use of a contactless interface was considered to increase the cleanliness of the operating room environment, reduce the necessary number of medical personnel needed to ensure the proper course of the surgery while meeting the requirements, and provide at least the same performance compared to current procedures.
The study aimed to conduct field tests with end users (orthopedic surgeons) and assess the feasibility and suitability of the developed application for contactless control of medical image visualization and switching on the surgical telementoring session.

2. Materials and Methods

The prototype of a virtual mouse application controlled with hand gestures captured from a Kinect-based sensor was developed for surgical purposes. The application can handle the majority of Windows applications. The Kinect device was used as a motion sensor. Microsoft launched the device in 2010 with gaming as its primary implementation. However, it should be mentioned that non-gaming applications and Kinect for Windows appeared around 2011 and began to be used in medicine [22,23,24], frequently in rehabilitation [25,26,27,28,29], posture assessment [30,31], and gait analysis [32,33,34]. The Kinect is no longer produced in its former form. In 2020, Microsoft released a continuation of the technology as Azure Kinect by integrating with the Microsoft Azure cloud computing platform, enabling the compatibility of the solution [35,36,37].
It was devoted to recognizing body gestures and serves as a game controller. The device has a high-resolution color camera and a depth sensor with a lower resolution. It allows for capturing RGBD (RGB + depth) images at a frequency of 30 frames per second. Four microphones have a noise-filtering function.
Preliminary tests facilitated the choice of the specific hand gestures used in the Ortho_Kinect_OR application for some Kinect sensor gaming experiences and were selected by the authors considering the circumstances and specific environment of the OR.
Gesture Recognition Algorithms (GRA) enabled control based on three modes: Mode 1 (Figure 1a) is created by combining a gesture with two fingers raised and simulating a mouse button. Moving the cursor was like moving the operator’s hand; finger movements allowed the operator to imitate a raised click. Mode 2 (Figure 1b) simulates the action of the mouse rollers by pushing the open hand up and down, which allows for fast scrolling. Mode 3 (Figure 1c), making a fist, was used by moving the cursor while holding the left mouse button. The above approach allowed objects to be rotated and moved in many applications for graphics. The back-and-forth movements imitate the action of a roller. It is possible to recognize these gestures when using a computer, browsing the web, and using various applications.
The functionality described above was implemented in C++ and installed as a virtual mouse controller on the OR PACS station. The OR PACS station has an easy-to-disinfect medical keyboard, an integrated touchpad, and an antibacterial coating. After placing the switchboard in it, the keyboard served as the basis for the Kinect. Kinect is connected using one of the USB connectors. Testing was conducted on the OR-PACS station using PACS software (Philips IntelliSpace Release 3.2.SP1) used routinely at the facility to view DICOM files. This setup allowed the user to evaluate the usability and user-friendliness of such a system.
Field testing and feasibility studies aimed to provide an early version of the application to a small sample of the target audience. By testing the application in a “real” environment, app developers can gather valuable information that will help them improve survey items. The feasibility study aimed to objectively and rationally reveal the strengths and weaknesses of the developed solution. Generally, feasibility studies precede the project’s technical development and final implementation. A feasibility study evaluates the project’s potential for success. The authors conducted the study as objectively and impartially as possible.
The study was conducted from 1 October 2014 to 30 December 2014. Participants were first familiarized with the procedure and purpose of the study. After formal acceptance, they were prepared to enter the OR to perform experimental tests. The recruited orthopedic surgeons invited to participate in the Center of Excellence “TeleOrto” field tests presented different experience levels and demographic characteristics. Participants were given a few minutes of instruction (five minutes per person) to use the gesture learning program. After this time, they proceeded to a series of practical tests. At first, each participant was asked to switch on the PACS client without using a standard keyboard. During this operation, entering the username and password was necessary using the virtual keyboard available in Windows 7 Pro. The next part of the test was finding a set of PACS X-rays of the patient’s hip to take measurements. Test the use of the software for measurements based on the measurement of the size of the base of the head of the hip prosthesis. It was found that measuring such a small element would require the respondents to simultaneously use several functions offered by the system (Figure 2). The PACS station is operated by the surgeon contactless, remotely in the operating room, as seen in Figure 3, so the participant had to start the system, retrieve the medical image, use the measurement system menu, and then measure the object. The time to complete all tasks, the time to complete each task, and the measurement time and their precision were assessed.
Switching on other applications like videoconferencing for telementoring purposes supplemented the final task.
After completing all the tasks, the participants were asked to complete a special questionnaire. The Ortho_Kinect_OR User Satisfaction and Satisfaction Assessment Form consists of 14 closed-ended questions that allow self-assessment by taking the test. Answers to questions were based on a Likert scale or affirmative/negative responses. The questions and statements were related to program initialization issues, the ability to initialize the App, the ease of Kickstarter, remembering all movements and gestures required to use the App, ease and problems when using the numeric keypad, the ability to recover patient data (images) from PACS, intuitive movements/gestures required to operate the App, understandability of the App, assessing the quality of navigation in the App, ease of use, and having the skills to run the App. When evaluating the App, surgeons were asked to answer questions such as the following: Did they enjoy using the App? Would the surgeon recommend the use of the App to other surgeons? Is it better to use applications rather than traditional PACS viewers? Is it better to use the numeric keypad or keyboard remotely? Did sterile gowns make it challenging to use the App? Did using the App make it easier to view PACS in the OR? Did the use of the App affect the effectiveness of activities? Did it facilitate the execution of planned tasks? Was the App helpful during the operation? Was using the App a good idea? Did the surgeon like the App and should additional training to use the App better be necessary?
The presented setup allowed us to evaluate the usability and user-friendliness of such a system.
Orthopedic surgeons with different experience levels and demographic characteristics were recruited for the field test. Participants were given a few minutes of instruction (five minutes per person) to use the gesture learning program. After this time, they proceeded to a series of practical tests. At first, the participant was asked to enable the PACS client without using a standard keyboard. During this operation, entering the username and password was necessary using the virtual keyboard available in Windows 7 Pro. The next part of the test was finding a set of PACS X-rays of the patient’s hip to take measurements. Test the use of the software for measurements based on the measurement of the size of the base of the head of the hip prosthesis. It was found that measuring such a small element would require the respondents to simultaneously use several functions offered by the system (Photo 5). The participant will need to start the roll to zoom in on the image by clicking on both mouse buttons to navigate the menu of the PACS measurement system and then measure the object. The time to complete all tasks, the time to complete each task, and the measurement time and their precision were assessed.
After completing all the tasks, the participants were asked to complete a special questionnaire. The Ortho_Kinect_OR User Satisfaction and User Satisfaction Assessment Form consisted of 14 questions, 7 open ones, which allowed self-review by entering the test, and 3 close-ended on a Likert scale from 1 to 10, where 1 is the lowest and 10 is the highest level in the two answers. There were also several embodiments for 3 and 4 with possible selection variants. Seven questions were answered based on the Likert scale. Others were closed-ended questions with affirmative or negative responses.
The operating room used for the study was accessible after the scheduled surgical procedures were performed and did not disrupt the surgical plan in any respect.

3. Results

Five participants finalized the study tasks. Two were specialists; the remaining part of the group were residents of varying degrees of advancement in the specialization (PGY-4 -1, PGY-3 -1, and PGY-1 -1). All participants followed the scheduled protocol. The mean age of the study participants was 35.6 years (range: 28–53). Only two people reported little or no experience with video games, including those controlled with Kinect. Others admitted playing occasionally. The average testing time was 5:19 min (7:25–3:25). The mean measurement time for the hip base was 29.8 s (range: 37.0–25.0).
Using a designated number of approaches to accurately measure results, study participants took an average of three approaches (range: 1–5). None of the examiners bothered to perform surgical tasks while wearing a suit in the operating room. Four respondents stated that the gestures to operate the program are intuitive. When asked about the legibility of the program on a scale from 1 to 10, the average respondent selected 7 (range: 5–10). When asked about the quality of navigation, 1–10 respondents obtained an average of 5.2 (range: 3–8). All respondents recognized the need for further training in using the program, which will allow them to use the tools more effectively. One person felt stressed during the test. Among the advantages of the presented solution, the respondents indicated at least three issues. The most frequently indicated advantages are responses regarding the touchless use of PACS, decreased distance walking in the operating room and the number of interpersonal interactions during the procedure, and reduced staff in the operating room. The most frequently reported opinion about the shortcomings of the presented solution was a complaint about slower PACS handling. Two respondents reported needing more skills to operate the program.
One person reflects on the handler that was challenging for him. Answering questions about the commands used by different surgeons, all surgeons decided to recommend ORTO_KINECT_OR to other surgeons. Only one of the respondents stated that ORTO_KINECT_OR lacked a traditional form of viewing imaging examinations. In the surveys, only one person raised the issue of insufficient precision during a given operation due to errors in the performance of tasks. Four out of five respondents reported a proposal to improve work precision in future device versions as part of the conclusions drawn from the system performance improvement. The agreement between the evaluators on the selected questions is presented in Table 1. The calculated overall agreement coefficients for all raters are presented in Table 2.
No one reported problems with launching and logging into the system—the most negative opinions concerned typing on a virtual keyboard. Three surveyed physicians identified hand discomfort and fatigue as the main problems when using the App, which required a unique, unusual lifting of the hands and forearms in space. All respondents pointed to the disadvantage that the accuracy and precision of performing tasks on the screen could have been higher. Most respondents replied that they would prefer to use traditional touchscreen photo viewers because they do it daily. All study participants enjoyed using the program and would recommend it to others.

4. Discussion

The development of the presented App confirmed the possibility of writing a fully operational computer program based on open-source code. A Kinect motion capture sensor is economically feasible for any medical facility providing orthopedics and orthopedic trauma services [20]. The strength and universality of the proposed solutions is the ease of implementing the device and software at any workstation for viewing imaging studies, with any viewer of images saved in the DICOM format [17,19]. In the past, similar conclusions were drawn by a few authors who used Kinect devices in the OR. Tan et al. [17] developed a program to use Kinect to view computed tomography images intraoperatively for interventional radiology. Guilherme et al. [38] co-developed a program that uses Kinect in urological surgery. After reviewing many PUBMED, MEDLINE, and EMBASE databases, the authors claimed they were the first to create a program applicable to orthopedic surgery.
Madapana et al. [39] did not find any support in the literature for developing a standard for gestures for the Picture Archiving and Communication Systems (PACS) operation in the operating room and proposed and tested the originally developed solutions.
It should be noted that Grätzel [14] is considered a pioneer of contemporary research on the construction of a device that enables hands-free operation of the operating room—years before the Kinect market appeared. Since 2004, he has been conducting experiments in the operating room using the Videre Mega-D camera. The quality of the detection system settings and the use of the system on OR equipment were examined [14]. The undoubted advantage is the low cost and the lack of calibration, allowing easy integration with existing approaches [17]. No issues have been reported with the installation tool designed to view the available images of the station in the operating room where the tests were performed. He invented a program using the Kinect device and unequivocally demonstrated its usefulness without tactile handling equipment in the OR field. It reduces the possibility of contamination in the OR field [40].
Most participants reported that they rarely play video games, especially with Kinect. Despite our lack of experience, we can all complete tasks after just a few minutes of training. In the survey, none of the respondents reported any problems with gestures to use Kinect. It only raised uncertainty about their gesture’s execution. The respondents also indicated a need for more precision when typing on a virtual keyboard, which extended the test duration. The users also stressed that further trials would be necessary to improve the typing module with the Kinect sensor.
Controlling the test participants’ gestures was encountered as the main usability problem. Three out of five respondents reported problems typing on the virtual keyboard, which they felt took longer than expected. Improving text typing will undoubtedly increase the usability of the program. Attention should also be paid to the relationship between the number of X-ray measurement attempts and the age of the users. This phenomenon may be related to younger respondents’ previous experience with computer games. The oldest participant obtained a positive result only after the fifth attempt. At best, two tries were enough.
Interestingly, no significant difference was observed in the time required to complete the interaction. Ma et al. [41] concluded from their studies that both user satisfaction and service speed were achieved at a reliable level. In the current research, we also recorded satisfactory results in terms of user satisfaction.
Tracking user gestures with more personnel in the operating room in the camera’s field of view may cause disruptions to the touchless interface. This observation confirms the conclusions described in other works: there is a need to improve and completely adapt tracking algorithms to OR conditions [1]. So far, the authors who have developed similar programs in the past have also noted the difficulty of designing gestures to best fit the computer screen by controlling the mouse on the other. They reported that gestures were intuitive for people using the device [14,17,40].
In this study, respondents—surgeons emphasized the most significant advantage of the present system, PACS hands-free access in the OR, which, in their opinion, can lead to a significant reduction in staff present in the operating room and will be an additional factor in reducing the possibility of contamination spreading [42,43]. Traffic reduction in the OR is commonly considered a factor that reduces the risk of infection [43,44,45]. Eliminating the causes of the lack of concentration loss during the work of an orthopedic surgeon contributes to increasing the safety of the patient operated on in the operating room while reducing the working time. This conclusion is in line with that provided by Grätzel et al. [14], describing the dependence of the duration of the operation on the amount of verbal interaction between the surgeon and the other people in the operating room. Analyzing the answers, the surgeons pointed out the importance of contactless computer use in the OR for retrieving images stored in the PACS and videoconferencing systems while working there. Surgeons participating in the ORTHO_KINECT_OR application tests confirmed that they would recommend this device to other orthopedic surgeons.
The weak point of the current project is the minimal number of surveyed surgeons, which does not allow for more advanced statistical analyses. However, given the very high agreement of the opinions obtained after completing the ORTHO_KINECT_OR application tests in the operating room, it may be assumed that the opinions among a larger population of orthopedic surgeons would be similar.
The number of participants in similar studies was never higher than thirty and usually around ten or less [14,17,40]. Other researchers did not report the number of study participants [38]. The research demonstrated the usability and satisfaction of users using the ORTHO_KINECT_OR prototype application with the Kinect sensor. It is reasonable to assume that Microsoft Kinect technology may increase the use of intraoperative imaging previews. Touchless performance of tasks on monitors providing access to PACS and videoconferencing systems improves the safety of surgical patients in the operating room by better maintaining the sterility of the operating room, eliminating staff traffic, and reducing the distraction of the operating team at the workplace.
The readability, transparency, and quality of use of the created program on a scale from 1 (the lowest rating) to 10 (the highest rating) were rated on average as 7. The navigation quality was rated indirectly due to the lack of need for many tools enabling contactless, self-service devices in the operating room environment. The aspect of reducing the risk of contamination in the OR field was understood. The development of human–computer interaction should be continued to improve the quality and safety of work in the operating room.
Some project vulnerabilities differ from threats because they are internal to the project and can be controlled or eliminated.
This feasibility study also aimed to determine whether the project should continue or be redesigned. It was stated that for the preparation of the feasibility study, selected fragments of the study were sufficient to determine the application’s usefulness in surgical practice due to the complementary nature of the usability provided by the designed application. It should be noted that the production of the Kinect sensor has been discontinued in its current form and is now called Kinect Azure, which is available in stock. This device can be considered an extended version on which the next updated application version can be based. Several medical applications are continuing on Kinect Azure [35,36,37]. The solution described can be considered valuable and worth continuing. However, changing the sensor can pose challenges and constitute a significant limitation to the research. Nevertheless, it is essential to emphasize that such challenges do not detract from the importance of the research but rather highlight the need for further research. The potential future integration with other advanced technologies, such as augmented reality [26,46,47,48] or artificial intelligence [48,49,50], may improve usability and functionality.

5. Conclusions

It has been demonstrated that it is possible to design and construct a contactless, easy-to-use application that allows contactless access to medical images and videoconferencing while maintaining complete sterility in the operating room. The system’s essential elements are justified economically and in terms of implementation. The proposed mobile solution can be quickly transferred to another operating room if necessary. The ORTHO_Kinect_OR application and the Kinect motion capture sensor as an optical tracking technology are suitable for contactless computer use in the OR, medical image retrieval from the PACS, its analysis, and videoconferencing use for telementoring and teleeducational purposes in the OR. In the future, the developed application will be helpful for the remote control of all programs used in the OR, including electronic medical records.

Author Contributions

Conceptualization, W.M.G., T.M. and R.S.; methodology, W.M.G.; software, T.M. and R.S.; validation, W.M.G. and R.S.; formal analysis, W.M.G., T.M. and R.S.; investigation, W.M.G.; resources, W.M.G. and R.S.; data curation, W.M.G.; writing—original draft preparation, W.M.G., T.M. and R.S.; writing—review and editing, W.M.G. and R.S.; visualization, W.M.G., T.M. and R.S.; supervision, W.M.G. and R.S.; project administration, W.M.G. and R.S.; funding acquisition, W.M.G. and R.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of the Medical University of Warsaw for the project entitled “A three-dimensional computer system for designing the functional evaluation of orthopedic equipment in patients with disorders of statics and kinematics of the body in the course of civilization diseases of the musculoskeletal system” (Protocol code KB 22/2012; date of approval 17 January 2012).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Feied, C.; Gillam, M.; Wachs, J.; Handler, J.; Stern, H.; Smith, M. A real-time gesture interface for hands-free control of electronic medical records. AMIA Annu. Symp. Proc. 2006, 2006, 920. [Google Scholar]
  2. Rose, J.; Kenton, H.; Abigail, S.; Claire, C.; Antonio, C. Exploring the potential for touchless interaction in image-guided interventional radiology. In Proceedings of the ACM Conference on Computer-Human Interaction, Vancouver, BC, Canada, 7–12 April 2011; pp. 1–10. [Google Scholar]
  3. Kadobayashi, R.N.K.; Mase, K. Design and evaluation of gesture interface of an immersive walk-through application for exploring cyberspace. In Proceedings of the 3rd IEEE International Conference on Digital Object Identifier, Nara, Japan, 14–16 April 1998. [Google Scholar]
  4. Keskin, C.A.O.; Akarun, L. Real time gestural interface for generic applications. In Proceedings of the 2005 13th European Signal Processing Conference, Antalya, Turkey, 4–8 September 2005. [Google Scholar]
  5. Savov, P.; Windhagen, H.; Haasper, C.; Ettinger, M. Digital templating of rotating hinge revision and primary total knee arthroplasty. Orthop. Rev. 2018, 10, 7811. [Google Scholar] [CrossRef]
  6. Huang, A.B.; Qi, Y.S.; Song, C.H.; Zhang, J.Y.; Yang, Y.Q.; Yu, J.K. Novel customized template designing for patellar resurfacing in total knee arthroplasty. J. Orthop. Res. 2016, 34, 1798–1803. [Google Scholar] [CrossRef]
  7. Ettinger, M.; Claassen, L.; Paes, P.; Calliess, T. 2D versus 3D templating in total knee arthroplasty. Knee 2016, 23, 149–151. [Google Scholar] [CrossRef]
  8. Hsu, A.R.; Gross, C.E.; Bhatia, S.; Levine, B.R. Template-directed instrumentation in total knee arthroplasty: Cost savings analysis. Orthopedics 2012, 35, e1596–e1600. [Google Scholar] [CrossRef] [PubMed]
  9. Kosashvili, Y.; Shasha, N.; Olschewski, E.; Safir, O.; White, L.; Gross, A.; Backstein, D. Digital versus conventional templating techniques in preoperative planning for total hip arthroplasty. Can. J. Surg. 2009, 52, 6–11. [Google Scholar] [PubMed]
  10. Ritter, M.A.; Eitzen, H.; French, M.L.; Hart, J.B. The operating room environment as affected by people and the surgical face mask. Clin. Orthop. Relat. Res. 1975, 111, 147–150. [Google Scholar] [CrossRef] [PubMed]
  11. Florschutz, A.V.; Fagan, R.P.; Matar, W.Y.; Sawyer, R.G.; Berrios-Torres, S.I. Surgical site infection risk factors and risk stratification. J. Am. Acad. Orthop. Surg. 2015, 23, S8–S11. [Google Scholar] [CrossRef]
  12. de Lissovoy, G.; Fraeman, K.; Hutchins, V.; Murphy, D.; Song, D.; Vaughn, B.B. Surgical site infection: Incidence and impact on hospital utilization and treatment costs. Am. J. Infect. Control 2009, 37, 387–397. [Google Scholar] [CrossRef] [PubMed]
  13. Hartmann, B.; Benson, M.; Junger, A.; Quinzio, L.; Rohrig, R.; Fengler, B.; Farber, U.W.; Wille, B.; Hempelmann, G. Computer keyboard and mouse as a reservoir of pathogens in an intensive care unit. J. Clin. Monit. Comput. 2004, 18, 7–12. [Google Scholar] [CrossRef] [PubMed]
  14. Gratzel, C.; Fong, T.; Grange, S.; Baur, C. A non-contact mouse for surgeon-computer interaction. Technol. Health Care Off. J. Eur. Soc. Eng. Med. 2004, 12, 245–257. [Google Scholar] [CrossRef]
  15. Wachs, J.P.; Stern, H.I.; Edan, Y.; Gillam, M.; Handler, J.; Feied, C.; Smith, M. A gesture-based tool for sterile browsing of radiology images. J. Am. Med. Inform. Assoc. JAMIA 2008, 15, 321–323. [Google Scholar] [CrossRef] [PubMed]
  16. Anton, D.; Goni, A.; Illarramendi, A. Exercise Recognition for Kinect-based Telerehabilitation. Methods Inf. Med. 2014, 53, 145–155. [Google Scholar] [CrossRef]
  17. Ebert, L.C.; Hatch, G.; Ampanozi, G.; Thali, M.J.; Ross, S. You can’t touch this: Touch-free navigation through radiological images. Surg. Innov. 2012, 19, 301–307. [Google Scholar] [CrossRef]
  18. Lahner, M.; Musshoff, D.; von Schulze Pellengahr, C.; Willburger, R.; Hagen, M.; Ficklscherer, A.; von Engelhardt, L.V.; Ackermann, O.; Lahner, N.; Vetter, G. Is the Kinect system suitable for evaluation of the hip joint range of motion and as a screening tool for femoroacetabular impingement (FAI)? Technol. Health Care 2015, 23, 75–81. [Google Scholar] [CrossRef] [PubMed]
  19. Nouei, M.T.; Kamyad, A.V.; Soroush, A.R.; Ghazalbash, S. A comprehensive operating room information system using the Kinect sensors and RFID. J. Clin. Monit. Comput. 2015, 29, 251–261. [Google Scholar] [CrossRef]
  20. Wipfli, R.; Dubois-Ferriere, V.; Budry, S.; Hoffmeyer, P.; Lovis, C. Gesture-Controlled Image Management for Operating Room: A Randomized Crossover Study to Compare Interaction Using Gestures, Mouse, and Third Person Relaying. PLoS ONE 2016, 11, e0153596. [Google Scholar] [CrossRef]
  21. Gleason, R.L., Jr.; Yigeremu, M.; Debebe, T.; Teklu, S.; Zewdeneh, D.; Weiler, M.; Frank, N.; Tolentino, L.; Attia, S.; Dixon, J.B.; et al. A safe, low-cost, easy-to-use 3D camera platform to assess risk of obstructed labor due to cephalopelvic disproportion. PLoS ONE 2018, 13, e0203865. [Google Scholar] [CrossRef]
  22. Schiltz, M.; Goudman, L.; Moens, M.; Nijs, J.; Hatem, S.M. The diagnostic value of physical examination tests in adhesive capsulitis: A systematic review. Eur. J. Phys. Rehabil. Med. 2023. Online ahead of print. [Google Scholar] [CrossRef]
  23. Silverstein, E.; Snyder, M. Implementation of facial recognition with Microsoft Kinect v2 sensor for patient verification. Med. Phys. 2017, 44, 2391–2399. [Google Scholar] [CrossRef]
  24. Kurillo, G.; Han, J.J.; Nicorici, A.; Bajcsy, R. Tele-MFAsT: Kinect-Based Tele-Medicine Tool for Remote Motion and Function Assessment. Stud. Health Technol. Inf. 2014, 196, 215–221. [Google Scholar]
  25. Mousavi Hondori, H.; Khademi, M. A Review on Technical and Clinical Impact of Microsoft Kinect on Physical Therapy and Rehabilitation. J. Med. Eng. 2014, 2014, 846514. [Google Scholar] [CrossRef] [PubMed]
  26. Kiani, S.; Rezaei, I.; Abasi, S.; Zakerabasali, S.; Yazdani, A. Technical aspects of virtual augmented reality-based rehabilitation systems for musculoskeletal disorders of the lower limbs: A systematic review. BMC Musculoskelet. Disord. 2023, 24, 4. [Google Scholar] [CrossRef]
  27. Lange, B.; Chang, C.Y.; Suma, E.; Newman, B.; Rizzo, A.S.; Bolas, M. Development and evaluation of low cost game-based balance rehabilitation tool using the Microsoft Kinect sensor. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2011, 2011, 1831–1834. [Google Scholar] [CrossRef] [PubMed]
  28. Da Gama, A.; Fallavollita, P.; Teichrieb, V.; Navab, N. Motor Rehabilitation Using Kinect: A Systematic Review. Games Health J. 2015, 4, 123–135. [Google Scholar] [CrossRef]
  29. Anton, D.; Nelson, M.; Russell, T.; Goni, A.; Illarramendi, A. Validation of a Kinect-based telerehabilitation system with total hip replacement patients. J. Telemed. Telecare 2016, 22, 192–197. [Google Scholar] [CrossRef]
  30. Shum, H.P.; Ho, E.S.; Jiang, Y.; Takagi, S. Real-time posture reconstruction for Microsoft Kinect. IEEE Trans. Cybern. 2013, 43, 1357–1369. [Google Scholar] [CrossRef]
  31. Castro, A.P.G.; Pacheco, J.D.; Lourenço, C.; Queirós, S.; Moreira, A.H.J.; Rodrigues, N.F.; Vilaça, J.L. Evaluation of spinal posture using Microsoft Kinect™: A preliminary case-study with 98 volunteers. Porto Biomed. J. 2017, 2, 18–22. [Google Scholar] [CrossRef] [PubMed]
  32. Bawa, A.; Banitsas, K.; Abbod, M. A Review on the Use of Microsoft Kinect for Gait Abnormality and Postural Disorder Assessment. J. Health Eng. 2021, 2021, 4360122. [Google Scholar] [CrossRef]
  33. Pfister, A.; West, A.M.; Bronner, S.; Noah, J.A. Comparative abilities of Microsoft Kinect and Vicon 3D motion capture for gait analysis. J. Med. Eng. Technol. 2014, 38, 274–280. [Google Scholar] [CrossRef]
  34. Springer, S.; Yogev Seligmann, G. Validity of the Kinect for Gait Assessment: A Focused Review. Sensors 2016, 16, 194. [Google Scholar] [CrossRef]
  35. Cattaneo, A.; Fragasso, M.; Magni, M.; Mostachetti, I.; Perri, E.; Vitali, A. Assessing the Accuracy of the Azure Kinect for Telerehabilitation After Breast Cancer Surgery. Stud. Health Technol. Inf. 2023, 301, 83–88. [Google Scholar] [CrossRef]
  36. Büker, L.; Quinten, V.; Hackbarth, M.; Hellmers, S.; Diekmann, R.; Hein, A. How the Processing Mode Influences Azure Kinect Body Tracking Results. Sensors 2023, 23, 878. [Google Scholar] [CrossRef] [PubMed]
  37. Belotti, N.; Bonfanti, S.; Locatelli, A.; Rota, L.; Ghidotti, A.; Vitali, A. A Tele-Rehabilitation Platform for Shoulder Motor Function Recovery Using Serious Games and an Azure Kinect Device. Stud. Health Technol. Inf. 2022, 293, 145–152. [Google Scholar] [CrossRef]
  38. Ruppert, G.C.; Reis, L.O.; Amorim, P.H.; de Moraes, T.F.; da Silva, J.V. Touchless gesture user interface for interactive image visualization in urological surgery. World J. Urol. 2012, 30, 687–691. [Google Scholar] [CrossRef]
  39. Madapana, N.; Gonzalez, G.; Rodgers, R.; Zhang, L.; Wachs, J.P. Gestures for Picture Archiving and Communication Systems (PACS) operation in the operating room: Is there any standard? PLoS ONE 2018, 13, e0198092. [Google Scholar] [CrossRef]
  40. Tan, J.H.; Chao, C.; Zawaideh, M.; Roberts, A.C.; Kinney, T.B. Informatics in Radiology: Developing a touchless user interface for intraoperative image control during interventional radiology procedures. Radiogr. A Rev. Publ. Radiol. Soc. N. Am. Inc. 2013, 33, E61–E70. [Google Scholar] [CrossRef]
  41. Ma, M.; Fallavollita, P.; Habert, S.; Weidert, S.; Navab, N. Device- and system-independent personal touchless user interface for operating rooms: One personal UI to control all displays in an operating room. Int. J. Comput. Assist. Radiol. Surg. 2016, 11, 853–861. [Google Scholar] [CrossRef]
  42. Birgand, G.; Azevedo, C.; Toupet, G.; Pissard-Gibollet, R.; Grandbastien, B.; Fleury, E.; Lucet, J.C. Attitudes, risk of infection and behaviours in the operating room (the ARIBO Project): A prospective, cross-sectional study. BMJ Open 2014, 4, e004274. [Google Scholar] [CrossRef]
  43. Hijji, F.Y.; Schneider, A.D.; Reeves, J.T.; Wilson, M.L.; Nye, L.; Lyons, J.G.; Prayson, M.J.; Rubino, L.J. Reduction in Operating Room Airborne Particle Burden and Time-Dependent Contamination of Sterile Instrument Trays With the Use of a Novel Air Filtration System. Cureus 2022, 14, e26864. [Google Scholar] [CrossRef]
  44. Stauning, M.T.; Bediako-Bowan, A.; Andersen, L.P.; Opintan, J.A.; Labi, A.K.; Kurtzhals, J.A.L.; Bjerrum, S. Traffic flow and microbial air contamination in operating rooms at a major teaching hospital in Ghana. J. Hosp. Infect. 2018, 99, 263–270. [Google Scholar] [CrossRef] [PubMed]
  45. Alverdy, J.C.; Hyman, N.; Gilbert, J. Re-examining causes of surgical site infections following elective surgery in the era of asepsis. Lancet Infect. Dis. 2020, 20, e38–e43. [Google Scholar] [CrossRef] [PubMed]
  46. Li, C.; Fahmy, A.; Sienz, J. An Augmented Reality Based Human-Robot Interaction Interface Using Kalman Filter Sensor Fusion. Sensors 2019, 19, 4586. [Google Scholar] [CrossRef] [PubMed]
  47. Ha, J.; Parekh, P.; Gamble, D.; Masters, J.; Jun, P.; Hester, T.; Daniels, T.; Halai, M. Opportunities and challenges of using augmented reality and heads-up display in orthopaedic surgery: A narrative review. J. Clin. Orthop. Trauma. 2021, 18, 209–215. [Google Scholar] [CrossRef] [PubMed]
  48. Pauly, O.; Diotte, B.; Fallavollita, P.; Weidert, S.; Euler, E.; Navab, N. Machine learning-based augmented reality for improved surgical scene understanding. Comput. Med. Imaging Graph. 2015, 41, 55–60. [Google Scholar] [CrossRef]
  49. Guerra, B.M.V.; Ramat, S.; Beltrami, G.; Schmid, M. Recurrent Network Solutions for Human Posture Recognition Based on Kinect Skeletal Data. Sensors 2023, 23, 5260. [Google Scholar] [CrossRef]
  50. Khan, I.U.; Afzal, S.; Lee, J.W. Human Activity Recognition via Hybrid Deep Learning Based Model. Sensors 2022, 22, 323. [Google Scholar] [CrossRef]
Applsci 13 11982 g001
Figure 1. Hand gesture modes for imitating computer mouse behavior: (a) mode 1, (b) mode 2, and (c) mode 3.
Figure 1. Hand gesture modes for imitating computer mouse behavior: (a) mode 1, (b) mode 2, and (c) mode 3.
Applsci 13 11982 g001
Applsci 13 11982 g002
Figure 2. The orthopedic surgeon is seen using the App in the OR.
Figure 2. The orthopedic surgeon is seen using the App in the OR.
Applsci 13 11982 g002
Applsci 13 11982 g003
Figure 3. The PACS station is touchless, operated by the surgeon at a distance in the OR.
Figure 3. The PACS station is touchless, operated by the surgeon at a distance in the OR.
Applsci 13 11982 g003
Table 1. This table presents the agreement of answers regardless of the value of the response.
Table 1. This table presents the agreement of answers regardless of the value of the response.
Question/StatementAgreement
Did you have problems initializing the program?100%
Were you able to initialize the ORTHO _KINECT_OR?40%
Was the Kickstarter easy for you?100%
I remembered all the movements and gestures60%
Using the numeric keypad was easy for me30%
Have you had a problem with the numeric keypad?40%
Were you able to recover patient data from PACS?100%
Are the movements/gestures in ORTHO _KINECT_OR intuitive?60%
How understandable is Ortho_Kinect_OR to you?10%
How do you rate the quality of ORTHO_KINECT_OR navigation?10%
Using ORTHO_KINECT_OR was easy for me60%
I was skilled enough to use ORTHO_KINECT_OR40%
Did you enjoy using ORTHO_KINECT_OR?100%
Would you recommend ORTHO_KINECT_OR to another surgeon?100%
Is it better to use ORTHO_KINECT_OR than a traditional PACS viewer?60%
Is it better to use ORTHO_KINECT_OR than a numeric keypad?40%
Did sterile gowns make it difficult to use ORTHO_KINECT_OR?100%
Did ORTHO_KINECT_OR make it easier to view PACS in the OR?30%
Did ORTHO_KINECT_OR affect the efficiency of the operation?30%
Did ORTHO_KINECT_OR make it easier to complete scheduled tasks?30%
Was ORTHO_KINECT_OR useful during the operation?20%
Was using ORTHO_KINECT_OR a good idea?40%
How are you enjoying ORTHO_KINECT_OR?40%
Is additional training needed to use ORTHO_KINECT_OR better?60%
Table 2. This table presents the overall agreement coefficients for all raters. AC—Gwet’s coefficient and BP—Brennan–Prediger coefficient.
Table 2. This table presents the overall agreement coefficients for all raters. AC—Gwet’s coefficient and BP—Brennan–Prediger coefficient.
MethodEstimateStd Error95% C.I.
AC0.550.061(0.425: 0.675)
Kappa0.430.057(0.322: 0.556)
BP0.540.061(0.417: 0.665)
Conger0.440.057(0.325: 0.559)
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Glinkowski, W.M.; Miścior, T.; Sitnik, R. Remote, Touchless Interaction with Medical Images and Telementoring in the Operating Room Using a Kinect-Based Application—A Usability Study. Appl. Sci. 2023, 13, 11982. https://doi.org/10.3390/app132111982

AMA Style

Glinkowski WM, Miścior T, Sitnik R. Remote, Touchless Interaction with Medical Images and Telementoring in the Operating Room Using a Kinect-Based Application—A Usability Study. Applied Sciences. 2023; 13(21):11982. https://doi.org/10.3390/app132111982

Chicago/Turabian Style

Glinkowski, Wojciech Michał, Tomasz Miścior, and Robert Sitnik. 2023. "Remote, Touchless Interaction with Medical Images and Telementoring in the Operating Room Using a Kinect-Based Application—A Usability Study" Applied Sciences 13, no. 21: 11982. https://doi.org/10.3390/app132111982

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop