Multimodal Natural Human–Computer Interfaces for Computer-Aided Design: A Review Paper

Abstract

Computer-aided design (CAD) systems have advanced to become a critical tool in product design. Nevertheless, they still primarily rely on the traditional mouse and keyboard interface. This limits the naturalness and intuitiveness of the 3D modeling process. Recently, a multimodal human–computer interface (HCI) has been proposed as the next-generation interaction paradigm. Widening the use of a multimodal HCI provides new opportunities for realizing natural interactions in 3D modeling. In this study, we conducted a literature review of a multimodal HCI for CAD to summarize the state-of-the-art research and establish a solid foundation for future research. We explore and categorize the requirements for natural HCIs and discuss paradigms for their implementation in CAD. Following this, factors to evaluate the system performance and user experience of a natural HCI are summarized and analyzed. We conclude by discussing challenges and key research directions for a natural HCI in product design to inspire future studies.

1. Introduction

Computer-aided design (CAD) systems have traditionally been used to aid the processes of product design and manufacturing [1]. Recent advances in CAD technology have led to a greater efficiency of 3D modeling in multiple industrial domains, such as mechanical and aerospace engineering [2,3]. Yet, there are some significant obstacles preventing further advancements in the research on human–computer interfaces (HCIs) for CAD, slowing the same pace of development. The current CAD systems still heavily rely on the conventional WIMP (Windows, Icons, Menus, Pointer) interface, using keystrokes and pointer movements as input. The behavioral patterns providing the system with input in this way show a mismatch in mapping between the user’s mental model and the computer interface. This mismatch forces the user to perform complex mental transformations between the 3D modeling space and 2D input devices, resulting in a high lead-time and hindered design creativity [3,4]. Moreover, novice users need to master a dense set of toolbars and menus, which takes considerable time to learn [5]. It is therefore necessary to move to user-centered natural HCI technology that can make CAD modeling more natural for users to operate and more intuitive to learn.

Efforts to mitigate this need have led to the development of multimodal interfaces that use eye tracking [6], gesture recognition [7], speech recognition [8], and brain signal recording techniques such as electroencephalograms (EEG) [9]. Standing at the crossroads of several research areas including psychology, computer vision, signal analysis, artificial intelligence, and cognitive science, the field of multimodal HCIs for CAD is in need of a solid theoretical foundation [10]. With the assistance of neighboring disciplines, researchers have developed a variety of novel interfaces for CAD [8,11,12]. Unfortunately, new input modalities, such as gesture, gaze, and EEG signals, are currently processed separately and integrated only at the application stage. Not only does this ignore the many issues regarding the application of each separate modality but also, more importantly, the roles and functions of multiple modalities for CAD and their interplay remain to be quantified and scientifically understood. In addition, there is a one-size-fits all attitude with respect to these modalities, in that an application fails to take individual differences into account. To elaborate on these issues, we review the state-of-the-art interaction patterns in CAD systems and discuss likely future research trends.

The remainder of the paper is organized as follows: Section 2 delineates the review methodology, including the methods adopted for the literature search and literature scope. In Section 3, the requirements and effects of a natural HCI for traditional CAD are presented. Based on this, Section 4 illustrates the state-of-the-art interaction techniques to implement a natural HCI, including eye tracking, gesture recognition, speech recognition, and EEG recognition. Section 5 and Section 6 summarize the measurements and devices of natural HCIs for CAD, respectively. A discussion of future research directions is provided in Section 7, and Section 8 summarizes this review and highlights its contributions.

2. Methodology

We performed a two-step literature search in this study. Firstly, the major engineering databases, including Web of Science, Engineering Village, and IEEE Xplore Digital Library, were retrieved by using specific keywords, such as “Computer aided design”, “Conceptual design”, “Human computer interface”, “Multimodal”, and “Natural interaction”, to obtain a list of publications in English dating from 1980 to 2022. To this, results from a Google Scholar search with the same keywords were added. Secondly, a manual screening was conducted by reading the titles and abstracts of these papers, so as to retain those related to the field of computer-aided design in engineering while excluding those focused on other areas such as architecture or medicine.

Ninety-five papers were identified through this search. Other papers cited in our study deal with general concepts and devices, for example, eye tracking, gesture recognition and EEG acquisitions.

It should be emphasized that this review does not claim to comprehensively and exhaustively cover all aspects of CAD systems for product design in engineering. We limit ourselves to providing an overview of the state-of-the-art studies related to multimodal interfaces in the application of CAD and to discussing future directions for the implementation of natural HCIs for CAD.

3. Natural HCI for CAD

The concept of natural HCIs was first introduced by Steve Mann [13,14], in an effort to bridge the gap between human and computer systems. As a continuously expanding field, natural HCIs are difficult to describe exactly [15]. When we think of user interfaces that are natural and easy to use, we most often think of one where the interaction is direct and consistent with our natural behaviors [4]. As such, these behaviors can be considered as media for everyday social communication through voice, gestures, and emotional expressions. Exploring the world through locomotion and the use of our senses, especially vision, is also prominent in natural behavior. Whenever we visually explore the world, we move our eyes to keep the information of interest in focus. Hence, natural HCIs visually explore virtual environments in manners analogous to how we visually explore the environment through eye movement and the manipulate virtual models in ways analogous to how we manipulate physical objects. The key focus of natural HCIs is to allow humans to interact with computer systems as much as possible in the same way as human–human communication in everyday life [14,16].

The requirements and effects of natural HCIs for CAD, as identified in this section, cover product design activities and aim to emphasize the importance of natural HCI technology for CAD development.

3.1. The Reasons Why Natural HCI Is Required for CAD

In the early 1960s, product design processes evolved from manual to computer-aided drawing, in parallel with the emergence of computer graphics display technology [17]. The initial technology based on 2D computer graphics evolved into 3D CAD systems around the 1970s, and the subsequent evolution of computer software and hardware allowed a rapid development of CAD technology [18,19,20]. CAD functions have incrementally been standardized and integrated, and artificial intelligence has been incorporated into their design [20,21]. However, a key component of CAD systems has lagged in development: interactive interface technology still reflects the earlier stages of CAD technology.

In most CAD applications, a text-based Command Line Interface (CLI) and WIMP-based Graphical User Interface (GUI) are still the mainly used components to date [20,22,23]. These traditional interaction modes have some obvious limitations for CAD applications as users have to create, modify, and manipulate 3D models with 2D devices (mouse and monitor), which make it difficult to meet users’ needs for natural interactions.

On the one hand, users are forced to frequently toggle certain on/off modules, and traverse across hundreds of menu items and toolbar buttons in the current interface, just to execute the desired CAD command [24]. Moreover, users need to memorize complicated hotkey combinations to control specific software products. Different CAD software programs usually have a variety of ways to represent the same command functions. This becomes even more challenging when designing across different 3D modeling applications, such as FreeCAD, AutoCAD, and SolidWorks [11]. As a result, current 3D CAD systems require a “very steep learning curve” for novice users to master the complex task of using a dense set of tool bars and menus [25].

On the other hand, the visuomotor requirements of using a mouse and keyboard may interfere with the visuospatial resources of the design process. The user needs to switch between the 3D modeling environment to the 2D input and output space [3]. In this process, the cognitive load is further enhanced by planning ahead for the manipulations needed to build the model [26,27]. Thus, the design intentions of users are frequently interrupted since considerable attention must be paid to finding the right commands via different button clicks [24].

Therefore, to ease learning and increase modeling efficiency, there is a strong demand for natural HCIs that bypasses these difficulties and enables users to interact with the system in a natural manner.

Natural HCIs for CAD maximally encapsulate the characteristics of human–human interactions (HHIs) in the modeling process [14,16,28]. As the next generation of natural HCIs, capturing original signals, such as human EEG, eye movements, gestures, and voices, is the premise and foundation for understanding users’ model operation intentions accurately [29,30]. Note that human–human communication and exploration behavior involve visual, auditory, and tactile processes, and thus are intrinsically multimodal. Multimodal interfaces therefore become an effective approach for realizing natural HCIs for CAD.

3.2. The Effects of Using Natural HCI in CAD

Natural HCIs will drive a gradual shift in the product design process from a traditional system-centered design to user-centered design, and provide a more intuitive model operation experience for users [31]. Thus, research on natural HCIs will become a main development trend of CAD interaction technology. While natural HCI technology is not yet commonly used in current CAD systems in industry, it has been explored extensively in academic research and areas where natural HCIs could bring significant positive effects have been identified [32,33]. The potential effects of natural HCIs in CAD are mainly reflected in three aspects described below.

1.

They can shorten the learning cycle and reduce the difficulty of learning for novice users.

Since users traditionally have to spend a huge amount of energy on learning and memorizing dense software menus and complicated operation instructions in modeling, it is not easy for the novice user to learn and master a new CAD application. To overcome this shortcoming, a natural HCI provides users with a novel interactive pattern where users just need to model in the way that is most comfortable and natural for them. The users’ intention of modeling will be automatically sensed and accurately recognized by the multimodal natural HCI without the user having to remember a lot of rules and instructions embedded in CAD systems. For instance, in order to manipulate a target model, users can select it just by directing their gaze at it and, at the same time, instruct the system to move the model by making a move gesture with their hands [34]. This interactive pattern is easy for novices to learn and could be quickly mastered regardless of experience level.

A natural HCI also adds cross-platform features to CAD systems that enable users to control 3D models in different software environments. As a result, users do not need to relearn a large number of new commands when faced with a new CAD application. This could significantly shorten the learning cycle and quickly turn users from novices to experts.

2.

They enable a more intuitive and natural interaction process.

Traditional WIMP interfaces use 2D subspace inputs such as mouse and keyboard to create and operate 3D models, which disconnect the input space from the modeling space and make the interaction process tedious and non-intuitive [3]. A natural HCI circumvents the limitations of traditional interfaces in terms of interaction space and dimensions, and provides a new end-to-end channel for direct interaction between users and 3D models. By capturing and recognizing the users’ EEG, gestures, speech, and other original signals in 3D modeling, natural HCIs can provide a more familiar and direct interaction for users, meaning that the interaction between the user and model can be performed more intuitively and naturally [35].

3.

They can enhance the innovative design ability of designers.

Current CAD systems have difficulty in supporting innovative design as they have been developed with too much emphasis on graphical representation techniques, and not enough on exploring how to support the processes that are essential for creative design [36,37]. Central to innovative design are ambiguity, uncertainty, and parallel lines of thought. All of this is limited by the circumscribed thinking and premature fixation from current CAD systems [38], which are readily supported by natural HCIs.

For natural HCIs, the input and output devices that separate the user and the system should be as unobtrusive as possible or hidden, so that users do not notice them and instead pay more attention to the product design process [3]. In addition, a natural HCI gives designers the freedom to creatively express their ideas and allows them to rapidly generate a wide variety of concepts, which will increase designers’ ability to innovate.

4. Ways to Implement Natural HCI for CAD

Multimodal interfaces provide a promising solution to realize natural interactions for CAD applications. In this study, we consider the following single-mode components: (1) eye tracking, (2) gesture recognition, (3) speech recognition, and (4) brain–computer interfaces (BCIs). We discuss each of them briefly before turning to the key topic of their multimodal integration in this section. An overview of core references is presented in

Table 1. Overview of core references for natural HCIs for CAD.

Category of Interfaces	Descriptions	References
Unimodal	Eye Tracking	[39,40,41]
	Gesture	[5,7,25,27,42,43,44,45,46,47]
	Speech	[24,48,49,50,51,52,53,54,55]
	BCI	[9,28,56,57,58,59,60,61]
Multimodal	Gesture + Speech	[8,11,12,62,63,64,65,66,67,68]
	Gesture + Eye Tracking	[6,34,69]
	Gesture + BCI	[70]
	Gesture + Speech + BCI	[3]
	Others	[30,71,72,73,74,75]

.

4.1. Eye Tracking-Based Interaction for CAD

The human gaze is defined as the direction to which the eyes are pointing in space, which could indicate where the attention of the user is focused [76,77]. Eye tracking is a technique in which eye-trackers record a user’s eye movements to determine where on the screen the user’s attention is directed when looking at the screen [78,79,80]. Ever since 1879, eye tracking has been used extensively in psychological science in the field of reading and image perception, and more recently in neuroscience and computer applications, especially HCI applications [80].

In the area of HCIs, eye tracking offers an important interface for a range of input methods including pointing and selecting [81], text input via virtual keyboards [82], gaze gestures [83], and drawing [84]. By tracking the direction and position of human eyes, the user’s intention could be judged and analyzed directly and rapidly [80,85,86]. For example, Connor et al. [87] developed two eyeLook applications (i.e., seeTV and seeTXT) based on a gaze-aware mobile computing platform. For seeTV, a mobile video player, gaze is used as a trigger condition such that content playback is automatically paused when the user is not looking. seeTXT, as an attentive speed-reading application, can flash words on a display, advancing the text only when the user is looking. Similarly, Nagamatsu et al. [88] integrated an eye-tracking-based interface into a handheld mobile device, through which the user is able to move the cursor and interact with a mobile device through their gaze. Furthermore, Miluzzo and Wang [89] demonstrated the capability of a smart phone to track the user’s eye movements across the phone. They presented a prototype implementation of EyePhone on a Nokia phone, enabling the user to operate a phone in touch-free manner. Incorporating such a facility in the process of interaction enhances the entertainment value and practicability, and benefits the development of natural HCIs.

For CAD applications, eye tracking could be used to interpret the user’s intentions for modeling and operating a 3D model effectively in a touch-free manner [34,39]. For example, model selection is one of the fundamental operations and also the initial task for the most common user interactions in CAD modeling [41]. Using eye-tracking technology, the user could select models in 2D/3D environments directly and quickly. Ryu et al. [6] and Pouke et al. [69] adopted eye tracking as an extra channel of input and employed the user’s gaze to find the intersection point with the surface of the target to select 3D objects in a virtual environment.

In addition, eye tracking could also be used in certain CAD operation tasks as an independent pointing interface, such as rotation, zooming, and translation. In traditional CAD systems, free rotation is obviously difficult, requiring additional commands for setting the rotation center [34]. With an eye-tracking-based interface, the user in the CAD environment can easily choose the rotation center without much effort just by fixating their gaze on it. For the zooming task, in order to explore detailed information through zooming, the user needs to preset the point of interest to the target that the model view camera moves towards in CAD applications. To ensure intuitiveness of operation, the location of interest is fixated and recorded by the eye-tracker, while the zooming operation could be controlled by another modality, e.g., a hand gesture [34]. Furthermore, the direction of translation operation in CAD could also be designed to be controlled by gazing. During all of the above operations, the application of eye tracking could decrease the user’s effort in position error feedback and reduce the fatigue of users, since it involves moving their eyes instead of hands. Thus, an eye-tracking-based interface in CAD can be a more natural and intuitive interface than traditional mouse and keyboard-based interfaces.

In sum, eye-tracking technology presents itself as a potentially natural interface for CAD applications. However, eye tracking has rarely been used independently as an interactional modality for CAD modeling. To achieve more complex modeling tasks, researchers have focused on combining eye tracking with other modalities [32,69,79], as described in Section 4.5.

4.2. Gesture Recognition-Based Interaction for CAD

Gestures are a form of body language that offer an effective way of communicating with others. People use a variety of gestures ranging from simple ones (such as pointing and pinching) to more complex actions for expressing feelings [90]. In recent years, the maturity of gesture-recognition technology based on depth vision has promoted the development of gesture interaction technology in applications of CAD [91]. Gestural interfaces aim to offer highly intuitive and free-form modeling modes that emulate interactions with physical products [42] and allow designers to create 3D conceptual models quickly, while just requiring a minimal amount of CAD software experience [43]. Approaches and modes using gestures to execute CAD modeling manipulation tasks will be covered in this section.

According to their spatiotemporal characteristics, gestures used in CAD interactions can mainly be divided into static and dynamic ones [25]. Research on static gestures considers the position information of a gesture, while that on dynamic gestures needs to consider not only the change in spatial position of the gesture, but also that in the expression of a gesture during the time sequence. Therefore, static gestures could be mainly used to invoke CAD commands for model-creation tasks while dynamic gestures could be used for model manipulation and modification [27]. In some studies [3,34,92], static gestures and dynamic gestures are combined for CAD manipulation instructions. For example, users can translate a model by closing the left hand into a fist (i.e., static gesture) and moving the right hand in the direction they want to translate it (i.e., dynamic gesture) [92,93]. In this process, the left-hand gesture plays the role of a trigger signal for the translation task while the right-hand gesture controls the direction and distance of the translation.

In our literature review, we found that researchers designed various sets or repositories of gestures for carrying out various CAD model tasks in the past [7,8,25,44]. The gestures in these repositories could, in principle, readily be used to test some gesture recognition algorithms and also directly as resources to select gestures in the application of CAD interfaces. However, most gestures in these repositories are typically chosen by researchers rather than users, often for the sake of ease of implementation with existing technology, while ignoring users’ preferences. It could be difficult for users to remember this information and, as a result, their cognitive loads are enhanced rather than reduced [94]. To advance from this situation, user-based research has been established as an effective gesture elicitation method [7,95]. Vuletic [7] and Thakur [25] developed user-based gesture vocabulary for conceptual design via evaluating the user’s activities in the modeling process in order to achieve a higher adoptability of gestures by inexperienced users and reduce the need for gesture learning.

Moreover, some authors explored the use of simple prescribed, even if free-form, gestures, such as pinching or grasping with one’s hand, to quickly create a variety of constrained and free-form shapes without the need for extensive gesture training [45,46,96]. Their evaluations demonstrated that it is possible to enable users to express their intents for modeling without the need for a fixed set of gestures. Therefore, a potential future gesture system for CAD could allow individual designers to use non-prescribed gestures that will support rather than inhibit their conceptual design thinking. Meanwhile, further research into both the procedural structure of CAD-based activities and the ways in which they might change depending on the shape being created could be conducted to explore better gesture-based design patterns that adapt to users and their specific workflows as they are used [7].

4.3. Speech Recognition-Based Interaction for CAD

With the rapid development of speech synthesis and recognition technologies in recent years, speech-based HCIs have been extensively employed in a variety of household, automobile, office, and driving applications [48,97]. In particular, voice-enabled intelligent personal assistants (IPAs), such as Amazon’s Alexa, Apple’s Siri, Google Assistant, and Microsoft Cortana, are widely available on a number of automatic devices [97]. Using a speech-based interface, routine operations can be efficiently executed with intuitive voice commands and the ease of use of the system can be improved. As well as in the household, speech-recognition technology can be applied in the context of CAD.

CAD modeling involves the development of a virtual object, always using step-by-step commands of instantiating primitives and using operations such as move, copy, rotate, and zoom for model manipulation and assembly [11]. These CAD commands could be realized naturally and directly through a speech-based interface. Therefore, some studies tried converting the names of menus and icons into voice-controlled commands [48,49,50,51,52,53]. Most of these studies were aimed at using fixed expressions to perform modeling operations. For example, Gao et al. [98] developed a prototype system with more than 80 speech commands which were primarily used to activate different CAD operations, for instance, selecting various operations of the system or switching from one mode to another. Similar research was also carried out by [50,52,53].

However, for all these speech-based CAD systems, CAD commands can only be activated when users utter the exact corresponding words or expressions [51]. So, novice users still have to devote considerable time to familiarize themselves with these systems and remember all the fixed vocabularies or preset expressions, which limits the utility of these speech-based CAD systems [54]. In order to implement a more flexible modeling operation, X.Y. Kou and S.K. Xue [24,54] proposed integrating a semantic inference approach into a speech-based CAD system, so users will no longer be constrained by predefined commands. In such a system, instead of using one-to-one mapping from word expressions to model actions, users can, for example, employ various expressions, such as “Rectangle”, “Draw a Rectangle” and “Create a Rectangle”, all with the same effect, namely, to generate a rectangle. This frees users from the need to memorize a host of predefined voice commands [24]. Whereas it is plain to see how this would work for simple commands, recognition methods should be further advanced for a speech-based CAD system to deal flexibly and intelligently with complex verbal commands [24,55].

Note that, as an effective interaction modality for CAD modeling, speech has been mostly used in conjunction with other interaction interfaces, such as “speech and gesture” [62], “speech and EEG” [3], and “speech and eye tracking” [32]. For this topic, more detailed studies will be reviewed in Section 4.5.

4.4. BCI-Based Interaction for CAD

Brain–computer interfaces (BCIs) provide a novel communication and control channel from the brain to an output device without the involvement of users’ peripheral nerves and muscles [99]. Typically, brain activity can be detected using a variety of approaches, such as EEG, magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), electrocorticography (ECoG), and near-infrared spectroscopy (NIRS) [100]. Among them, the EEG signal is considered as the input of choice in most BCI systems to relate patterns in brain signals to the users’ thoughts and intentions [28]. The very first advancements in this respect were made by Pfurtscheller [101], who for the first time demonstrated that it is possible for users to move through a virtual street just by imagining their feet movements. Lécuyer [102] and Zhao [103] also developed some prototypes allowing users to navigate in virtual scenes and manipulate virtual objects just through EEG signals. Furthermore, Trejo [56] and Li [104] further extended the application of BCI to cursor movement. These studies show that BCI, as a new non-muscular channel, could be used in many applications involving human cognition, such as computer games, vehicle driving, assistive appliances, and neural prostheses [99].

In CAD systems, BCI could offer a more intuitive and natural pattern of interaction between the user and CAD application. Esfahani and Sundararajan [57] conducted the first study that investigated the possibility of using BCI for geometry selection. They used the evoked potential component P300-based BCI for selecting different target surfaces of geometrical objects in the CAD systems. Some other important functions for CAD application such as creating models or manipulating models via BCI have been studied by other researchers [9,28,58,59,105]. Esfahani and Sundararajan [58] also carried out experiments to distinguish between different primitive shapes based on users’ EEG activity, including cube, sphere, cylinder, pyramid and cone shapes. The EEG headset was used to collect brain signals from 14 locations on the scalp and a linear discriminant classifier was trained to discriminate between the five basic primitive objects with an average accuracy of about 44.6%, significantly above the chance level of 20%. Postelnicu [60] conducted similar studies where the user was able to create and modify geometrical models by using EEG signals. As an important task in CAD applications, model manipulation-based BCI was also realized by [9,59,60]. In another recent work, Sree [61] used EEG and electromyogram (EMG) signals from facial movement using the Emotiv headset to enable users to carry out certain tasks in the CAD environment of Google SketchUp. Meanwhile, a human-factors study assessing the usability of the EEG/EMG interface was performed with participants from different backgrounds. The results suggested that the BCI-based system could help to lower the learning curve and has high usability as a generalized medium for CAD modeling [61].

In sum, BCI shows great potential in allowing users in CAD applications to create, modify and manipulate models directly. However, there are still some common limitations on the practical applicability of BCI resulting from the difficulty of brain signal acquisition and the limited accuracy of instruction discrimination. Future, lightweight EEG acquisition equipment and high-precision classification algorithms will become the new frontier for developing BCI-based CAD applications. Additionally, detecting the emotional state and user’s satisfaction from EEG signals could be used in CAD systems to correct for errors and to strengthen proper classifications, which will make the system more reliable [106].

4.5. Multimodal HCI for CAD

Multimodal interfaces aim to construct interactive systems that leverage natural human capabilities to communicate through different modalities such as speech, gesture, gaze, and others, bringing more accurate and robust intention recognition methods to human–computer interaction [29]. Thanks to advances in the development of pattern-recognition techniques (in natural language processing, machine vision, etc.) and hardware technologies of input and output (cameras and sensors, etc.), there has been a significant increase in multimodal HCI research [107]. Turk [29], Jaímes [33], and Dumas [108] performed valuable surveys on the current status of multimodal HCI research. From these studies, we can easily see that the main goal of research on multimodal interactions is to explore a more transparent, flexible, efficient, and natural interface that can remove existing constraints on what is possible in the field of HCIs, and progress towards the full use of human communication and interaction capabilities. The realization of this aim depends on collaboration among researchers from different disciplines, such as computer scientists, mechanical engineers, cognitive scientists, and other experts [29].

Meanwhile, multimodal interfaces are considered to offer an improved user experience and better control than a unimodal input in the application of CAD modeling. The pioneering study by Bolt [63] in creating a “Put That There” system showed the potential and advantages of combining gesture and speech inputs for the selection and displacement of virtual models. This system proposed the use of commands such as “Move that to the right of the green square” and “Delete that”, allowing users to employ vague language to activate CAD functions while disambiguating them with gestures. It is noteworthy that none of these commands can be interpreted properly from just a single input modality—at least two were required—but this multimodal interface created simple and expressive commands that are natural to users.

The work by Bolt [63] led to a rapid development of many multimodal interfaces that typically used gesture, speech, eye-tracking, or BCI-based input modals for creating and manipulating CAD models, as is shown in Figure 1. From the survey results, “Gesture and Speech” is the most widely used interface in the application of CAD and drove the majority of multimodal interface research. Alternative formulations have also been followed, bringing new multimodal interfaces such as “Gesture and Eye Tracking” [34,69], “Gesture and BCI” [70], “Gesture, Speech and BCI” [3] and others [30,71,72,73,74,75]. For these multimodal interface-based CAD applications, there are three main types of combined modalities, as shown below.

• Different modalities can be used to execute different CAD commands. For example, in [69], eye tracking is used to select CAD models and gesture is used to activate different types of manipulation commands.
• Different modalities can be used to execute the same CAD commands at the same time. In another words, there is overlap in the set of CAD commands that can be implemented through different modalities. For example, the task of creating models can be completed both by BCI-enabled commands and speech commands [3]. In this case, users can choose different interfaces according to their preferences. Moreover, to a certain extent, this kind of overlap in commands improves the robustness and flexibility of the interfaces.
• Different modalities can be combined to execute a CAD command synergistically. In this respect, the advantages of different interfaces can be leveraged to work together on a model manipulation task. For example, in most CAD prototype systems combined with speech and gesture [8,12,64], speech is used to activate specific model manipulation commands (translation, rotation, zooming, etc.), while gestures are used to control the magnitude and direction of specific manipulations.

Most studies on multimodal interaction have concluded that using eye tracking, gesture, speech, and BCI for conceptual CAD modeling is easy to learn and use [3,8,12]. However, they mainly focused on the feasibility of the multimodal interface developed, considering aspects such as the accuracy and efficiency of user intent recognition, rather than user experience [3]. Notably, one of the most important goals of multimodal interfaces is to provide a natural interactive experience for CAD users. Hence, it is essential to carry out human factor studies for multimodal systems. Song [34] tested the intuitiveness and comfort levels of a multimodal system, i.e., GaFinC, by means of user interviews. Participants reported that it was very natural to pick the point of interest using their own gaze. However, they felt more uncomfortable and fatigued using two hands as opposed to one during the process of CAD modeling. In addition, in most gesture-based multimodal systems, the gestures used are always prescribed in advance and users have to remember complex gesture vocabulary before modeling, which significantly increases the additional cognitive load imposed on users [7]. All the issues mentioned above will benefit from the development of future multimodal-based CAD systems.

4.6. Other Interfaces for CAD

In addition to the interfaces listed in the previous sections, a short overview of other interfaces developed is given in this section.

Touch-based interfaces are those in which designers can input design information on a touchscreen with their fingers. Sharma et al. [71] developed a multimodal interface, “MozArt”, which combines speech and touch for creating and interacting with CAD models. This system features a minimalist user interface and uses a touch table whose orientation can be changed depending on the needs of the user. While MozArt provides users with efficient results, the main problem with this interface is that the touch-based system is subject to ambiguity since it is two dimensional in nature. To resolve these ambiguities, EEG is considered here as the main method to resolve these ambiguities. Therefore, a novel system combining touch and BCI was proposed by Bhat [30] in order to reduce the ambiguity associated with 2D touch to perform 3D model manipulation. The experimental results have shown that BCI could help to identify the true intention of the users while performing the modeling tasks by touch [30].

Virtual reality (VR) is providing a new perspective for human interaction with CAD models, allowing for the stereoscopic visualization of virtual objects on one-to-one scale within the working space of users [74]. In a VR-based system, the user can directly create, modify, and manipulate models without learning complex protocols and procedures [109]. Haptic interface, as a promising 3D HCI for various applications, could allow the user to feel the contact force in the process of free-form design and more realistically convey virtual reality to humans so as to aid the design process [110,111,112]. To construct more immersive design environments for users, Damien Chamaret [113] and Simon Kind [114] combined haptic and VR in multimodal systems and found that haptic feedback can provide a more accurate manipulation of virtual objects and significantly improve the working efficiency of CAD systems. On this basis, a novel multimodal application, “VRSolid”, was proposed by Mogan [73], further extending the input modalities within the VR environment. The VRSolid system consists of five modalities, including two for output (haptic feedback and 3D visual) and three for input (gesture, speech, and motion tracking). Following this, Bourdot [72] and Toma [35] constructed some similar multimodal systems for CAD in a VR environment. All applications mentioned above are designed to provide an intuitiveness and natural environment for interaction with CAD models, while creating a common multimodal interface that can be integrated with the existing commercial CAD software, such as SolidWorks, Creo, and Parasolid [35,72,73].

Additionally, Stark et al. [74] tested “pen and VR” in an application called VR-cave, which has a physical pen for free-hand sketching in 2D and 3D immersive modeling environments. Xiangshi et al. [75] investigated the performances of four modes of CAD applications, such as, “mouse”, “pen”, “pen and speech”, and “pen, speech and mouse”. They found that “pen, speech and mouse” was the best multimodal combination in terms of drawing time, modification time, and subjective preferences among these four modes [75].

All the studies reviewed in this section further emphasize the diversity of CAD interface development research and the difference in the outcome of the interface evaluations. Further exploring more natural interfaces and integration methods for CAD application will be an important issue for future studies.

5. Evaluation of Natural HCI for CAD

To evaluate the performance and usability of natural interfaces for CAD, we need to establish whether the interface is able to carry out tasks effectively and efficiently and if it is also easy to understand and use for novices in the field of CAD [35]. Generally, each novel interaction system needs to pass several assessment and modification cycles during its development so as to reach a high quality and ensure user satisfaction [115]. Consequently, the evaluation of the natural HCI is one of the indispensable parts during system development.

From the literature review, the evaluation factors can be broadly categorized into two groups, from the system performance and user experience perspectives [115,116]. For the system performance, Effectiveness and Efficiency are the two main evaluation metrics. For the user experience, evaluation metrics include Learnability, User preference, Cognitive load, and Physical fatigue. The specific description about these metrics is as follows:

• Effectiveness: This metric refers to the recognition accuracy (or error rate) and completeness of modeling tasks while using the novel CAD interaction system. To compute the accuracy parameter, Song et al. [34] carried out a manipulation test for the developed multimodal interface and the errors of translation, rotation, and zooming were recorded. The test results showed that the user performed the zooming task with higher accuracy than the translation task in the novel interaction system.
• Efficiency: This metric is mainly established in terms of the task completion time and includes the system response time. In the view of [117], the time period, when the user is planning how to operate a generate/modified task through the user interface in a CAD system, should also be included as a component of task completion time in addition to the operation execution time. From the time test results, Chu et al. [117] found that using the novel multimodal interface results in a speedup of 1.5 to 2.0 times over the traditional interface (mouse and keyboard) in the process of building simple models, such as block, cylinder, sphere, etc. For more complex models, modeling with a traditional interface takes three to four times longer than with the novel interface [117]. The main factor contributing to these differences is that the number of operations required for traditional interface increases significantly relative to the novel interface with the increase in model complexity. Additionally, the total task completion time is proportional to the total number of operations for modeling. It is worth noting that only users with the same level of experience for CAD were considered in the study of Chu et al. [117]. Future research is needed to conduct a comparative experimental analysis of system performance using users with different experience levels.
• Learnability: This metric describes how easy it is for novice users to interact with CAD systems and accomplish modeling tasks. For multimodal HCIs, learnability is a kind of quality that allows novice users to quickly become familiar with these interfaces and make good use of all their features and capabilities. Generally, the use of user questionnaire is a common method for evaluating the learnability of interaction systems. In the studies of [11,64], users were asked to rate the task on a five-point Likert scale in terms of the ease of performing tasks and satisfaction with the modeling results after the completion of tasks. The results showed that for a multimodal interface-based CAD system, users’ perceptions of the ease of performing tasks were high for all tasks, except for rotation and modeling tasks [64]. On the other hand, some quantitative parameters can also reflect the learnability of a system. For example, the improvement of user performance associated with task repetitions was used to evaluate learnability in [113], including the task completion time and number of collisions. Meanwhile, Chu et al. [117] focused on the number of design steps for modeling and the idle time period when the user knows exactly what to do and is planning how the interaction approach may be performed to achieve the design tasks. In the study of [117], compared to traditional interfaces, a VR-based multimodal system required fewer design steps and less idle time in the modeling process, which indicates that the user could quickly understand what operations need to be performed and complete the modeling tasks.
• User preference: This metric refers to the subjective evaluation of the user’s pleasure in using the novel interface based on the user characteristics, including static characteristics (e.g., age, gender, and language) and dynamical characteristics (e.g., motivation, emotional status, experience, and domain knowledge). This metric is also an important reflection of user-centered natural interaction design. In terms of multimodal systems, it can offer a set of modalities for user input and output depending on the interactional environment and functional requirements [115]. User experience and preference should be fully taken account in the process of the choice of different modalities and the design of a unimodal interface. In the studies of [3,118,119], participants were questioned about their preferred interaction interface and most of them appreciated the use of an immersive multimodal interface for CAD modeling, which can offer a more intuitive interactive environment than traditional interfaces where the user does not have to navigate through a series of windows and menus in order to achieve a desired action. Additionally, the unimodal interfaces, such as gesture-based interfaces [7] and speech-based interfaces [24], should be designed taking into account user preference, not just based on the expert opinion.
• Cognitive load: This metric refers to the amount of mental activity imposed on working memory in the process of modeling tasks, including the necessary information processing capacity and resources. In terms of cognitive psychology [120], the product designer is considered as a cognitive system utilizing CAD systems for modeling efficiently [35]. The cognitive interaction between the designer and system is performed by various types of interfaces. In general, the use of multimodal interfaces could make better use of the human cognitive resources and will thus result in a lower cognitive load on the user during the process of modeling [115,121]. To verify this point, Sharma et al. [71] collected subjective task load data with the NASA TLX method after each task, and evaluated the cognitive load of users in modeling with multitouch and multimodal systems. Following this, the qualitative self-reported data and EEG signals were used to analyze the user’s cognitive load during the process of modeling in the multimodal interface-based system [26]. The results showed that compared with the results of expert users, novice users’ alpha band activity increased when using a multimodal interface, which indicates that their cognitive load decreased [26]. This result was confirmed in the questionnaire responses. Consequently, it was easy for novice users to use a new set of multimodal inputs for modeling compared to expert users. Additionally, some performance features (e.g., reaction time and error rate) and psycho-physiological parameters (e.g., blink rate and pupil diameter) can also be used to evaluate the cognitive load of user [122].
• Physical fatigue: This metric refers to the physical effort needed for users while interacting with the CAD system in order to perform modeling tasks. Even if the multimodal interface puts users in the loop of an intuitive and immersive interactive environment, the repeated execution of body movements (e.g., arm, hand, eye, etc.) for a long time can lead to fatigue and stressful factors for users [35]. Therefore, it is necessary to evaluate multimodal interface-based CAD systems assessing the body motion in the performance of modeling tasks so as to reduce body fatigue and increase user satisfaction in the design process. Administering a questionnaire to users is a mainstream method adopted to evaluate the physical fatigue caused by using multimodal interfaces [6,64]. Meanwhile, the overall body posture, movement pattern, and hand movement distance can also be used as analytical factors for physical fatigue. Toma [35] used the distance covered by hand movement in the performance of a task to compare and evaluate the VR-CAD system with a traditional desktop workspace, and found that multimodal interfaces lead to higher physical fatigue, with the hand movement distance being on average 1.6 times greater than the desktop interface for the modeling process.

The presented metrics are not yet exhausted, and some other characteristics can still be considered to evaluate a multimodal interface-based CAD system, such as the number and type of interactive modalities; the ability to use modes in parallel, serially, or both; the size and type of recognition vocabularies; and methods for integrating multiple sensors and the types of applications supported [29].

As the modeling requirements become more complex, evaluating the system must become more formalized. Evaluating metrics will become more quantitative and evaluating methods will become more objective. The ultimate goal of analyzing these metrics is to create a prototype system that users like and can be used to perform modeling tasks in a natural and intuitive approach [123]. After conducting a system evaluation, it is important to record what was observed in addition to why such a behavior occurred and modify the interfaces according to the results. It is noteworthy that an effective evaluation may not generate a direct solution to the problems, but can provide modified design guidelines for continued testing [124].

6. Devices for Natural HCI

Although these interfaces are unlikely to completely replace traditional WIMP-based interfaces, the importance of multimodal natural HCIs is growing due to advances in hardware and software, the benefits that they can provide to users, and the natural fit with the increasingly ubiquitous mobile computing environment [125].

In the process of CAD modeling, signals that can represent the design intention should first be obtained to identify the user intention. Thus, signal acquisition and identification devices are important for achieving a natural HCI. In this section, the devices used for eye tracking, gesture recognition, speech recognition, and BCI are introduced in detail, as presented in

Table 2. Overview of devices for natural HCIs.

Signal Modalities	Categories	Devices
Eye tracking	Head-mounted	Dikablis Glass 3.0 [126], Tobii Glass 2 [127], Pupil Labs Glasses [128]
	Tabletop	Gazepoint GP3 [129], Tobii Pro Spectrum 150 [130], EyeLink 1000 Plus [131]
Gesture	Sensor-based	MoCap Pro Glove [132], Cyber Glove [133], Vrtrix Glove [134]
	Vision-based	Kinect [135], Leap Motion [44,136]
Speech	DP-100 Connected Speech Recognition System (CSRS) [63], Microsoft Speech API (SAPI) [54], CMUSphinx [137]
EEG	Saline electrode	Emotiv Epoc+ [3,58]
	Wet electrode	Neuroscan [56]
	Dry electrode	Neurosky MindWave [138], InteraXon Muse [139]
Others	Haptic	SensAble PHANTom [110], SPIDAR [113]
	VR/AR	HTC Vive [140], Oculus Rift [141], HoloLens 2 [142]

. In additional, some other devices related to natural HCIs are also briefly described.

6.1. Devices for Eye Tracking

Eye tracking technology is used to track the eye movement state of the user and recognize where the user is looking on a computer screen through some advanced optical recognition methods, such as the pupil–corneal reflection method [143], pupillary-canthus method [144], HN method [145], and so on. The first eye-tracker that could measure eye movements quantitatively was developed by Dodge and Cline in 1901 [146]. With the dramatic improvement in eye-tracking technology, more advanced and easy-to-use hardware has been developed on the market in recent years [147]. Currently, there are two main categories of eye-trackers available as interactive input devices, including head-mounted devices and tabletop devices.

The head-mounted eye-tracker, which usually needs to be attached to the eyes as special contact lenses or headsets, is composed of a scene camera, which records the user’s first-person view, and an eye camera, which continuously records the changes of the sight line by using the infrared ray of cornea and pupil reflection [148]. Some common commercial head-mounted eye-trackers can be used for HCIs, as shown in Figure 2. Obviously, these head-mounted devices are inconvenient and involve some physical load for users. Therefore, the lightweight design of head-mounted eye-trackers is needed to make them more widely used in natural HCIs.

Compared to the head-mounted tracer, the tabletop eye-tracker is more comfortable and flexible for users as it can be integrated with a monitor and does not require the user to wear any equipment, as shown in Figure 3. However, most of these commercial eye-trackers are designed to target the PC environment, with a short distance of 50 to 80 cm between users and eye-trackers, which limits their application in the field of HCIs [32]. Moreover, complex calibration between the eye-tracker and display coordinates is required and the user’s freedom of head movement is limited owing to the fixed camera. All the mentioned drawbacks need to be improved and optimized to promote the future development of tabletop eye-trackers.

6.2. Devices for Gesture Recognition

Gesture recognition involves algorithms to detect and recognize the movement of fingers, palms, arms, or the entire body so as to interpret the user’s interaction intent. Currently, the devices for gesture recognition can be broadly divided into sensor-based devices, such as data gloves, and vision-based devices, such as normal video cameras or depth-aware cameras.

Sensor-based devices can directly capture the motion and position of hand gesture by using data gloves. Commonly used sensors mainly include EMG sensors [149], bending sensors [150], pressure sensors [151], and acceleration sensors [152]. By using these glove sensors, we can easily obtain various gesture signals and accurately identify the hand pose and movement. Representative products in data gloves for gesture recognition are MoCap Pro Glove, Cyber Glove, and Vrtrix Glove, as shown in Figure 4. However, these data gloves are too expensive and their wired sensors restrict natural hand movement [47]. To overcome these limitations, the vision-based approach came into existence [93].

The vision-based device is less constrained and moves more naturally than the sensor-based one, and does not require users to wear anything over the hands. This device is relatively cheap, simple, natural, and convenient to use in the application of CAD. At present, representative vision-based products used in the process of CAD are Kinect and Leap Motion, as shown in Figure 5. Vinayak [45] developed a novel gesture-based 3D modeling system and used Kinect to complete the recognition and classification of human skeletal and hand posture in the process of modeling. However, the recognition performance of vision-based devices is sensitive and easily affected by environmental factors such as lighting conditions and cluttered backgrounds [153]. Therefore, the illumination change, multiuser cluster, partial or full occlusions are important challenges for vision-based gesture input devices to be addressed in the future study [93].

6.3. Devices for Speech Recognition

Speech recognition-based interfaces provide a natural method to operate and modify the CAD models for users. Along with the development of signal input devices and natural language-processing technology, the accuracy and effectiveness of speech recognition has been significantly improved, which further promotes the application of speech-based interfaces for CAD.

Generally, the devices for speech recognition consist of two main modules, i.e., an input microphone and speech recognition module. After converting the analog signal with speech information into a digital signal through the input microphone, the speech recognition module integrated with natural language-processing algorithms completes the final recognition of semantics [154]. Obviously, the speech recognition module is a cord part of this system as an internal algorithm processor. Additionally, many companies have developed a variety of speech-recognition modules. For example, NEC (Nippon Electric Company) developed a system called DP-100 Connected Speech Recognition System (CSRS) [63] which is able to recognize a limited number of connect voices without pause. Microsoft also introduced its own voice-recognition interface for computers, Microsoft Speech API (SAPI) [54]. Carnegie Mellon University developed a speech-recognition toolbox, CMUSphinx, that can be used for microphone speech recognition [137].

In addition, machine learning and deep learning algorithms such as the Hidden Markov Model (HMM) [155], Convolutional Neural Network (CNN) [156] and Long Short-term Memory (LSTM) [157] have been widely used in the core part of speech recognition modules to improve the accuracy of speech recognition.

6.4. Devices for EEG Acquisition

The EEG acquisition device is a piece of equipment that can record the electrical activity of the brain through some specific electrodes placed on a user’s scalp [158]. In 1924, the first EEG detection device in the world was developed by Berger in Jena, Germany [159]. Following the development of signal processing and microelectronics technology, the structure and function of EEG-acquisition devices have gradually matured. There are numerous EEG devices from different companies that are able to cater to the specific needs of EEG users, such as Neuroscan, Emotiv, and Neurosky MindWave.

Generally, EEG-acquisition devices can be divided into invasive, semi-invasive, and non-invasive categories according to the connection with the brain [160]. Among them, a non-invasive EEG device is mainly used in the area of natural HCIs due to the damage of the scalp caused by the invasive device and the semi-invasive device. For non-invasive EEG devices, the types of electrodes used can be categorized into saline electrode, wet electrode, and dry electrode [160]. Saline electrodes use salt water as a conductive medium, which is easy to carry and low in cost [161]. Wet electrodes usually need to use gel as a conductive medium and can collect high-quality EEG signals. Dry electrodes can directly contact the user’s scalp to achieve conduction without any conductive medium which is more natural and convenient to the user, but the signal acquisition accuracy is limited [162].

In current studies in the literature, Emotiv Epoc+ is commonly used as an EEG-acquisition device in the application of CAD [3,57,58,59], as shown in Figure 6. Emotiv Epoc+ is a small and low-cost device used to record and recognize EEG and facial muscles’ movements (EMG) with 14 saline electrodes. This device can communicate wirelessly with computer via a USB dongle and provide the Emotiv API, a C++ API, which can be used to easily obtain the EEG/EMG data and transform the user-specific action command into an easy-to-user structure [3].

Additionally, some other lightweight and low-cost EEG devices, such as Neurosky MindWave and InteraXon Muse, may also be used to build future BCI for CAD.

6.5. Other Devices for Natural HCI

In addition to the main devices mentioned above, some other devices for natural HCIs are briefly introduced in this part.

A haptic device is an interaction device which can deliver force feedback to a user, while interacting with the virtual environment (VE) or real environment (RE) [114]. This device could provide a more realistic interaction experience by making users feel the movement and touch the interaction models. The SensAble PHANTom [110,163,164] is one of the most common haptic devices used in CAD applications, as shown in Figure 7, which consists of a mechanical robotic arm with joints and a pen-shaped end-effector as manipulator. The arm follows the movement of the manipulator and is capable of generating a force equivalent to that applied to the manipulator [10]. Additionally, the wearable haptic device, SPIDAR, is also used to interact with virtual mock-ups [113], as shown in Figure 8. Compared to SensAble PHANTom, SPIDAR has a more flexible workspace and allows the user to grasp the virtual models with natural experience.

From the studies of [10,73,114], haptic devices are always applied in a virtual environment (VR) for CAD. In another word, VR devices are also needed and important for natural HCIs of CAD. Currently, the most popular VR/AR devices on the market are HTC Vive, Oculus Rift, HoloLens, and so on, as shown in Figure 9.

Additionally, some other interactive devices such as touchscreen and EMG-based devices, are not listed here due to their limited application in the area of CAD.

7. The Future of Natural HCI for CAD

Multimodal natural HCIs for CAD constitute a growing research area and lots of research efforts are expected. To develop more natural and intuitive multimodal systems for CAD with better performance characteristics than traditional interfaces, many fundamental scientific issues and multidisciplinary research challenges still remain to be addressed. In this section, the challenges and potential research directions of multimodal natural interfaces for CAD are discussed.

7.1. Changing from a Large Scale and Independence to Lightweight, Miniaturization and Integration for Interaction Devices

Interaction devices that can accurately capture physiological signals such as the EEG, gaze, and gestures of the user are the technological basis for building natural HCIs in the application of CAD. Most existing interaction devices in this review are designed to be wearable and independently recognize the user’s signals. However, these wearable devices’ excessive size and weight can make the user easily fatigued in the process of using. Therefore, lightweight and miniaturized designs will be a mainstream trend in the future, which can enhance the user’s comfort and decrease their physical fatigue by reducing the size and weight of interaction devices. For instance, Figure 10 shows the evolution of the EEG headset from conventional to in-ear EEG. Furthermore, the need for compact wearables also means that actuators and sensors should be nimble enough to fit into tight crevices on different bodies, especially for gloves, glasses, and EEG headsets [165]. With such requirements, fitting the actuation for signal acquisition of different modes could also be a challenge. In other words, instead of employing different devices to capture distinct signals, researchers should focus more on integrating multiple acquisition functions in one device. This trend is exacerbated by the popularization of multimodal interfaces [10].

7.2. Changing from System-Centered to User-Centered Interaction Patterns

Historically, the development of human–computer interfaces for CAD application has been a system-centered pattern, driven by the functional requirements of the system and the current technological capability enabling their implementation. This pattern mostly focuses on organizing the functionality of the system and builds the product in the designer’s own interpretation and through the implementation of systems thinking, rather than the user’s. This means that the system users have to memorize a large number of instructions in order to interact in a manner that matches system processing rules, which is not typically user friendly and contrary to the original intention of natural HCIs.

To build a natural HCI, the user’s preferences should be fully considered in the design process, thus forming a user-centered interaction pattern. A user-centered pattern is mainly based on a user’s abilities and real preferences, and focusing on the user rather than forcing the users to change their behavior to accommodate the interface. In the future, further empirical work is needed to build the user-centered interaction pattern for CAD, such as (a) developing natural and efficient unimodal interface for the CAD applications driven by what the users would naturally do instead of defined by the researchers developing the studies, just due to ease of application or alignment with the technology used, so that users do not need to learn too many particular commands or procedures and can focus more on the product design process, (b) giving users free control over the selection or combined use of input modes in a multimodal interface according to their preference to further reduce the cognitive load and improve their comfort, and (c) proposing a blended interaction style that combines a passive input mode (e.g., gaze tracking) and active input mode (e.g., BCI or gesture) toward supporting the recognition of users’ natural activities in context [31,167].

7.3. Changing from Single Discriminant to Multimodal Fusion Analysis for Interaction Algorithms

One of the most fundamental issues for multimodal HCIs is to integrate input information from different sensing modalities during the process of modeling. However, different input modalities yield disparate data forms and rates, which makes the integration of such signals a difficult and challenging task. For most current systems, multimodal data are typically processed separately and only combined at the end without taking full advantage of the complementary and redundant information from multimodal signals [168]. Therefore, to achieve the desired naturalness and robustness of the multimodal interface, more efficient and effective technologies of data fusion are needed to study.

Firstly, the temporal dimension is an important aspect in the process of multimodal fusion because different modalities may have different temporal constraints and semantic features. Some modal integrations are intended to be interpreted in parallel, which others may typically be offered sequentially [169]. In addition, not all multimodal systems are needed to fuse information to form a joint interpretation, which is more closely tied with the environment context and modeling task context [31]. Therefore, the questions about when to integrate and which needs to be combined should be solved according to a context-dependent model, including the temporal features and modeling tasks so as to provide better recognition results. Furthermore, based on the type of input modalities in a multimodal interface, different levels of integration methods should be defined, mainly including feature-level fusion and decision-level fusion [170]. For example, EEG and eye tracking have a strong correlation in the process of model selection, which can be fused at the feature level to increase the robustness and accuracy of user intent discrimination. Depending on the chosen level of fusion, the future integration can be performed using varies methods, ranging from simple feature concatenation to complex deep fusion algorithms. In particular, future work needs to continue to study the adaptive multimodal fusion architectures for realizing error handling and coupling multiple modalities in a natural manner [170,171].

7.4. Changing from Single-Person Design to Multiuser Collaborative Design for Interaction

Most of current multimodal interfaces for CAD have been developed as standalone single-user solutions, which unfortunately, are rarely designed to meet complex collaboration requirements for multiple people. Now more than ever, there is a growing need for collaborative CAD solutions, essentially moving away from the high-performance single user solutions of the past and towards more multiuser systems [172]. Therefore, in future interactive environments of product design, one important research goal will be allowing multiple users to interact together in larger physical spaces and ubiquitous forms, such as pointing to a shared display surface, gesturing concurrently, and talking in a natural manner [167]. For this target, some specific technological issues will need to be addressed because the current interfaces generally assume that a particular device provides input from a single user at a given moment, for example, how to capture data from multiple users at the same time and identify the contribution of an individual user to the overall multiuser interactive environment, as well as switching between different interactive subjects. The amount or size of multimodal devices that a user has to wear or carries will also need to be minimized.

In addition, due to the rapid development of network and VR/AR technologies, distributed collaborative design will be another important trend in the future. Technology for data sharing and communication allows users to share a design resource efficiently in co-located or distributed situations [173]. Multimodal interface-based AR/VR should also be designed and developed to support solid modeling and collaborative design activities in the distributed environment, where multiple users can simultaneously view and interact with virtual/real objects [174,175].

In sum, in order to meet the future requirements of the multiuser collaborative design of products, the multimodal natural HCI still faces an array of long-term research challenges.

8. Conclusions

To better understand the state-of-the-art research of multimodal natural HCIs for CAD, this paper summarizes the related research works and advanced techniques in the literature. The method of literature retrieval and research scope were determined. By analyzing the process of traditional product concept design from the retrieved papers, the requirements and effects of natural HCIs for CAD are presented. Then, the mainstream techniques for implementing natural HCIs, evaluation indicators and supported devices are summarized. Finally, challenges and potential directions for natural HCIs are identified for further discussion.

From the studies reviewed in this paper, we observed that although many issues are being addressed in the field of multimodal natural HCIs for CAD, such as novel interface designs, interface performance evaluations, and device development for natural HCIs, the field is still young and needs further research to establish reliable multimodal natural HCI and mature applications for CAD. In particular, the current novel interfaces for CAD are mainly used in the conceptual design phase of products and it is difficult to meet the interactive functional requirements of the other design process, such as detailed design, simulation, and analysis. Therefore, research of new multimodal natural HCIs needs to be considered to support multiple phases of product design, not just the conceptual design.

Finally, this paper contributes to the development of multimodal natural HCIs for CAD in two main aspects: summarizing the up-to-date software- and hardware-supported techniques for natural HCIs in the application of CAD in many disciplines, and proposing key future research directions to be addressed in the field. Hopefully, this paper can provide an insightful informative reference to researchers for the development of multimodal natural HCIs of CAD and inspire future studies on natural HCIs.