Interactive Rhythm Learning System by Combining Tablet Computers and Robots

Abstract

This study proposes a percussion learning device that combines tablet computers and robots. This device comprises two systems: a rhythm teaching system, in which users can compose and practice rhythms by using a tablet computer, and a robot performance system. First, teachers compose the rhythm training contents on the tablet computer. Then, the learners practice these percussion exercises by using the tablet computer and a small drum set. The teaching system provides a new and user-friendly score editing interface for composing a rhythm exercise. It also provides a rhythm rating function to facilitate percussion training for children and improve the stability of rhythmic beating. To encourage children to practice percussion exercises, a robotic performance system is used to interact with the children; this system can perform percussion exercises for students to listen to and then help them practice the exercise. This interaction enhances children’s interest and motivation to learn and practice rhythm exercises. The results of experimental course and field trials reveal that the proposed system not only increases students’ interest and efficiency in learning but also helps them in understanding musical rhythms through interaction and composing simple rhythms.

1. Introduction

Music has three components, namely rhythm, melody, and harmony, of which rhythm is the most crucial [1,2]. Music loses its dynamics and becomes a combination of monotonous tones in the absence of rhythm, regardless of the presence of a beautiful melody or rich harmony. Music education experts have indicated that the visual and auditory nerves of children undergo the most rapid development between the ages of 4 and 12 years. Therefore, this period is optimal for introducing children to music. Music education is generally categorized into two types based on the teaching methods [1]:

• Teaching through physical movements: Teachers play music and guide children in moving their bodies to music, thus enabling the children to experience rhythms through various physical movements [2].
• Teaching using musical instruments: This is the most common method. By playing with various musical instruments, children learn about aspects of music (e.g., scales, notes, musical scores, melody, and rhythm).

Playing a percussion instrument may be relatively difficult for young children. Teachers must select appropriate instruments for young children according to their age and degree of muscle development. Moreover, children of this age usually face difficulties in remaining focused for a period of time. Learning outcomes may also be undermined if the teaching materials are uninteresting. Therefore, an ideal teaching style should enable children to learn in a pleasurable environment and apply appropriate teaching aids, in addition to increasing the learning motivation and outcomes of children.

To improve children’s learning motivation, interactive systems that involve the music instruments with electronic techniques are developed for learning music in recent years [3,4,5,6,7,8,9,10,11,12,13,14]. Bae et al. developed “Drum On” to reduce boredom and other limitations [3]. Drum On projects animated images on the drum and directly provides rhythm cues to the drummer. Kanke et al. developed a learning system for enabling users to efficiently play percussion instruments [4]. This system uses haptic and visual indications to improve users’ rhythm internalization performance. Li and Fu developed an augmented reality system for practicing or performing a musical instrument [5]. In this system, users can play a musical instrument through a visual prompt. Baba et al. developed a scanning device called “Gosen” that can play handwritten musical notes [6]. Gosen can help a user read musical scores, even for users who are not adequately versed in reading such scores. Raymaekers et al. developed “Game of Tones”, which projects an animated image on a piano [7]. This image enables a user to play the piano without knowing how to read a musical score. Tsui et al. developed a smartphone-based platform for performing a one-man orchestra [8]. On this platform, a user uses traditional bimanual conducting gestures with leap motions to direct the orchestra. Bengler and Bryan-Jinns developed “Polymetros”, an interactive system for musical experiences [9]. This system supports multiple players, and the users need not have musical skills. Yazdi and Lee developed “E-Santur”, an interactive game-based application with music instructions [10]. This application helps children learn music through interesting and engaging games. Waranusast et al. developed “muSurface”, an interactive surface with ability to interact with tangible musical symbols [11]. Users place tangible music symbols on the virtual musical staffs displayed on the playing surface, and muSurface plays the resulting melody according to the symbols. Hoffman and Weinberg developed “Shimon”, an interactive music system [12]. Users use choreographic gestures to direct a jazz-playing robot. Rasamimanana et al. developed “Urban Musical Game”, which uses sports balls [13]. This system plays a variety of sounds with identifying physical dynamics according to how the ball is manipulated. Lee et al. developed the interactive “TuneTrain” app to help children learn music using interesting games [14].

Since the release of the iPhone in 2008, mobile devices have become prevalent and integral to the lives of numerous people. Various music applications are currently available on mobile devices [15,16,17,18,19]. Toothy Buddies developed “Carnival of Animals”, which combines animals and interactive learning with an audiobook [15]. This app can enhance children’s musicality in terms of duration, rhythm, melody, phrase, and music form. Apple Inc. developed “GarageBand” [16], an app comprising many editable music instruments such as a drum, guitar, and piano producing realistic instrument sounds. A user can play up to 32 tracks individually or with another person. This app also realistically simulates instruments such as guitar strings vibrating while playing. Marion Wood developed “Woodchuck Rhythm” [17], a rythm game mobile app featuring seven rhythm patterns, each of which has corresponding scores, cartoon graphics, and audio files. Lei Chi Tak developed “Rhythm Training” [18], a rhythm training mobile app featuring a metronome function. Learners can understand the note value of each piece of notation using the metronome function and practice their rhythm through virtual drumheads. Flippfly LLC developed “Monkey Drum” [19], a music-making mobile app featuring virtual musical instruments such as drums and string instruments. Children can tap the mobile device in whatever rhythm they desire and this app will identify and playback using the created rhythm. These applications can simulate musical instruments, enabling children to use touchscreen interfaces to play instruments and generate immediate sound feedback. For learners, these applications save the cost of purchasing actual musical instruments. In addition, playing instruments on mobile devices is much easier than playing the actual instruments. This motivated us to design a percussion learning app on a tablet computer for studying rhythm.

Children’s inability to remain focused while studying music constitutes another challenge. Furthermore, uninteresting teaching materials may undermine learning outcomes. To solve these problems, this study proposes a percussion learning device that combines tablet computers and robots. This device comprises two systems: a rhythm teaching system, in which users can compose and practice rhythms by using a tablet computer, and a robot rhythm performance system. The rhythm teaching system provides a new and user-friendly score editing interface for composing a rhythm exercise, in addition to providing a rhythm rating function to facilitate percussion training for children and improve the stability of rhythmic beating. The robot rhythm performance system interacts with children to attract their attention during learning, performs percussion exercises for children to listen to, and assists them in practicing exercises. Such interactions increase children’s interest and motivation to learn and practice rhythm exercises.

2. Rhythm Teaching System

In basic music education, cultivating a sense of rhythm is essential. An effective means of achieving this is to beat on percussion instruments (e.g., drums, tambourines, and castanets). Nevertheless, this type of learning requires the presence of music teachers, and children face difficulties in practicing on their own at home. Therefore, this study proposes a rhythm teaching system for assisting children to practice rhythms through a tablet computer. This system can increase children’s learning interest and learning outcomes, in addition to enabling them to create simple musical scores through interactions. The rhythm teaching system contains two main functions: musical score editing and playing, and rhythm rating.

2.1. Musical Score Editing and Playing Functions

Understanding musical scores is a challenge for music beginners. Figure 1 shows a drum score consisting of six bars [20]. The first three bars primarily involve the hi-hat, snare drum, and bass drum. The note positions are at the top, middle, and bottom parts of the score. Specifically, the hi-hat should be played in quarter notes in a consistent rhythm, whereas the snare and bass drums join in various beats. The bass drum should be played in a quaver of fast rhythm. In the final three bars, the tenor drum and cymbal are introduced, and the rhythm becomes complicated. If a learner wants to play a piece of music as shown in Figure 1, he or she must be able to understand the score in advance. However, this is a difficult task for a beginning learner.

To relieve the burden on children, a relatively intuitive and user-friendly score interface was designed for the rhythm teaching system. Figure 2 shows a piece of the rhythm edited by the proposed system. The score involves five instruments, and teachers can edit and adapt the melody for various instruments. The score of each instrument comprises continuous vertical grids. Four continuous grids represent a bar, and each bar is played within two seconds. The light blue bar represents a beating point. For each grid, four types of beat points can be edited by tapping the grid on the tablet computer, as shown in Figure 3. The four beat point types are a quarter note (Figure 3a), an eighth note (Figure 3b), two eighth notes (Figure 3c), and no beating point (Figure 3d).

When an edited score is played by the rhythm teaching system (Figure 4), a cue line appears on the screen of the tablet computer; the score is then scrolled from top to bottom over time. If the beat points reach the cut line, learners must beat the corresponding instruments in time. For example, in Figure 4, a bass drum is beaten. In addition, teachers can insert background music, which can improve the learner’s sense of rhythm when practicing rhythm exercises. After the completion of a score edition, the teaching system enables teachers to save the edited score and load it in the next rhythm exercise. Compared with conventional drum scores, the scores shown on this interface are easier for beginners to read. According to field trials, most children understand the rhythms in the score within a short period. After some practice and with a certain level of skills, most children can play a piece of music completely on their own. In addition, children can more easily create their own drum scores for listening and practicing on their own.

By applying musical score editing function, teachers can immediately edit scores of various difficulties (Figure 5) corresponding to students’ learning progress and adjust the difficulty according to the practice or test outcome. The drum score content includes both basic and advanced rhythm practices for a single drum, two drums, or multiple percussion instruments. In addition, students could also conveniently edit or create their own scores using this musical score editing function.

2.2. Rhythm Rating Function

In the proposed teaching system, a rhythm rating function was also designed to facilitate rhythm training for children and improve the stability of rhythmic beating. As shown in Figure 6, five instrument buttons are located at the bottom. When the teaching system plays a percussion exercise, the score scrolls from top to bottom over time. If the beat points reach the cut line, learners must then tap on the corresponding instrument buttons. As a learner presses the buttons, the rating function determines whether the tapping occurs on the correct beats and records the number of correct taps; concurrently, the corresponding instrument sounds are generated for simulating this tapping action. In addition, if the learners want to adjust the difficulty level of a percussion exercise, they can disable some instruments by tapping the check buttons at the top of the teaching system. Because playing musical instruments on mobile devices is relatively easy for children, we provide another learning style for children to improve their beating stability and effectively assess the accuracy of practicing rhythms.

3. Interactive Robot Performance System

This study developed a portable and low-cost percussion performance system, as shown in Figure 7. The design idea of this performance system refers to a jazz drum set. To attract children’s attention, the performance system applies daily necessities (e.g., a plastic bucket and a glass) as beating instruments and robots as drummers. The drum set includes a plastic bucket, iron lid, sugar sachet, glass, and bell on which the robots beat. These instruments can be divided into the following four categories according to their sound effects: (1) low frequency; (2) high frequency; (3) with some fermata; and (4) particularly loud. These instruments are respectively the bass drum, snare drum, hi-hat, and crash cymbal in a drum set (

Table 1. Instrument Selections.

Sound Type	Object Selected in Our Drum Set	Instrument Type
Low-Frequency Sound	Plastic Bucket	Bass Drum
High-Frequency Sound	Iron Lid	Snare Drum
Sound with Some Fermata	Sugar Sachet	Hi-Hat
Loud Sound	Glass, Bell	Crash Cymbal

). When the robots perform a percussion exercise, these daily necessities can produce sounds similar to those made by a jazz drum set. Simultaneously, the learner can listen to the rhythm music produced by the robot performance system and follow it to practice the percussion exercise using a small drum set, as depicted in Figure 8.

In the robot performance system, five robots, constructed with Lego blocks, act as drummers to perform the percussion exercise. Each robot contains a servo motor to generate a beating action (Figure 9). An embedded system (i.e., Arduino UNO [21]) serves as the control core of the robot. This system receives score information sent from the tablet computer and then controls the servo motor to beat on the instruments. Compared with a traditional jazz drum set, our robot performance system is smaller and portable. In additions, its construction cost is US$40. According to the field trials, children enjoy practicing with the robot performance system. This interactive learning style can improve their sense of rhythm when practicing rhythm exercises.

4. Experimental Course and Field Trials

To demonstrate the effectiveness of the proposed system, an experimental course was designed. This course incorporated the rhythm teaching system into traditional rhythm education to improve students’ sense of rhythm and musical performance skills. Ten students aged five to six years participated in four 50-min classes over four weeks. Appendix A shows the details of the teaching plan. In the first two classes, the children studied the basic sense of musical rhythm with traditional rhythm education. In the next two classes, the children used small snare drums to engage in interactive rhythm practice. First, robots played according to a score, and the children listened to them and observed their movements. Subsequently, the children were asked to replicate the rhythms according to the score by using small snare drums and repeat practicing what they have learned. To demonstrate the effectiveness of the proposed system, a test was administered to each student before and after rhythm education. The test methods are described as follows:

• The proposed rhythm rating function was used for assessment. This function played a test melody, during which the students were required to perform rhythms along with the melody. The system then recorded the number of beats that the students played accurately.
• When the students used snare drums to play songs along with the melodies, their performance was observed and assessed.

The test melody used in this experiment consisted of 25 beats that were to be tapped using one hand (Figure 10).

Table 2. Testing results of 10 students.

Student ID	Testing before the Rhythm Education			Testing after the Rhythm Education
	Beating Number	Correct Beating	Accuracy of Beating	Beating Number	Correct Beating	Accuracy of Beating
A	38	18	47.37%	52	19	36.54%
B	26	11	42.31%	27	23	85.19%
C	31	14	45.16%	25	19	76.00%
D	26	21	80.77%	26	24	92.31%
E	26	18	69.23%	27	20	74.07%
F	27	12	44.44%	25	21	84.00%
G	26	18	69.23%	25	22	88.00%
H	61	24	39.34%	24	16	66.67%
I	25	23	92.00%	26	25	96.15%
J	34	21	61.76%	26	25	96.15%
Average Beating Accuracy			59.16%			79.51%

lists the detailed test results of 10 students. Before the course, the students scored an average beating accuracy of 59.16%; after the course, the accuracy improved to 79.51%. This improvement verifies the benefit of our teaching method. Most students achieved superior beating accuracy after the course; only student A played poorly after the course. Next, we investigated the performance of each student. In the pretest, the beating accuracy of student F was 44%; we observed that he moved his body along with the rhythm when music was played, but he was unable to tap the rhythm on time (he was often too slow) when he performed. However, after the course, we observed that his sense of rhythm and musical performance skills improved significantly; he always tapped the rhythm on time, and his beating accuracy increased from 44% to 84%.

Student J showed a beating accuracy of 61.76% during the pretest. We observed that she was unfamiliar with musical rhythms and had difficulty following them. Consequently, she often tapped rhythms too early. However, she showed great interest in the course content. Subsequently, when performing an interactive musical performance with the robot, she could maintain her focus and imitate the rhythms played by the robot. In addition, she indicated her strong desire to practice playing again after training. Her beating accuracy increased from 61.76% to 96.15% after the course.

To further test our system, field trials were performed at Maker Faire Tokyo and Maker Faire Taipei (Figure 11). We designed a questionnaire to evaluate the user interaction satisfaction of the proposed system. The following conclusions were derived from the field tests and questionnaire results:

• Children face difficulty performing complex rhythms on their own. However, through our system, children can learn to play complex rhythms using simple system operations.
• Children who have a tendency to hit items with a stick show a willingness to spend more time editing their own rhythm-based sheet music after using our system.
• Because the robot plays self-created music, after listening to a particular piece of music several times, most children learn about how to edit the sheet music to produce the songs they want to play.
• The robot performance attracts children’s attention and increases their willingness to learn.
• In the field trials, the loud music and actions produced by the robot performance system drew attention from numerous adults and children. The audience gave positive feedback regarding the system’s novel and interesting design concept. They agreed that our system provides the users with a pleasant experience through its interactive features. Because the score could be easily edited, this also prompts them to adjust the input score in order to create more amusing rhythms.

Finally, by applying musical score editing function, teachers can teach students how to edit a better score by themselves. For example, the score in Figure 12a is edited by a student. However, the student’s score composes inconsistent rhythm and edits too many loud sounds, then the teacher could immediately teach the student how to reform a better score as the example shown in Figure 12b.

5. Comparison of Rhythm Apps

In this section, four rhythm Apps were applied for comparison, including Carnival of Animals [15], Woodchuck Rhythm [16], Rhythm Training [17], and Monkey Drum [18].

Table 3. Comparison of our system and four rhythm apps.

Heading	Our System	Carnival of Animals [15]	Woodchuck Rhythm [16]	Rhythm Training [17]	Monkey Drum [18]
Interactivity	✓	✓			✓
Notation Learning			✓	✓
Note Value Learning	✓	✓	✓	✓
Variety of Musical Instruments	✓		✓		✓
Create and Edit Drum Scores	✓				✓
Difficulty Adjustment	✓		✓
In-app Simulation of Percussion Instruments	✓			✓	✓

lists the functions of our system and four rhythm Apps. According to the comparison results in Table 3, we could find that our system contains more interactivity and functions than other apps. In another way, an experimental field trail of applying the four rhythm apps was processed. Three students of ages 6, 7, and 9 years participated the experimental field trail over 10 days. A test the same to Section 4 was administered to three students before and after this field trail.

Table 4. Testing results of three students.

Student ID	Testing before the Field Trail			Testing after the Field Trail
	Beating Number	Correct Beating	Accuracy of Beating	Beating Number	Correct Beating	Accuracy of Beating
A	25	18	72%	25	21	84%
B	25	17	68%	25	19	76%
C	25	22	88%	25	24	96%
Average Beating Accuracy			76%			85.33%

lists the detailed test results of three students. Before this field trail, the students scored an average beating accuracy of 76%; after the field trail, the accuracy improved to 83.33%. This improvement is not significant. The following conclusions were derived from this field trail:

• During the field-trail process, most participating parents did not like having their children spend excessive time on these apps. Some parents were concerned that long-term usage of mobile devices would impair their children’s vision.
• During the field-trail process, we found that children were spending a small amount of time on rhythm training and an excessive amount of time on in-app minigames unrelated to rhythm training.
• Although the Carnival of Animals and Rhythm Training apps had interesting and attractive interfaces, these apps had monotonous audio feedback and operation, and learners may become bored after one or two uses. In addition, these apps allowed practice with only minimal rhythm combinations and do not allow customization.
• For the Monkey Drum app, children could tap the mobile device in whatever rhythm they desire. However, this app had poor interactivity and monotonous playback and was unable to keep children’s long-term interest.
• Although the Woodchuck Rhythm app applied seven rhythm patterns, each of which had corresponding scores, cartoon graphics, and audio file. However, this app had only a small rhythm library and did not include rhythms with long beats (e.g., half notes)

These observations revealed that although these apps provided tremendous entertainment value, they did not facilitate favorable learning outcomes. By contrast, our system not only provides excellent interactive and entertainment features, but also enables desirable learning outcomes.

6. Conclusions and Future Works

This study proposes an interactive percussion learning device for helping children create and study musical rhythms. This system combines Lego robots and a tablet PC running the developed interactive system. Through this system, which is easy to manage for young music learners, children can edit musical scores and control the robots’ movement. During this process, they acquire the basic concepts of musical rhythms and show interest and strong motivation to use this system. The rhythm learning system can improve children’s sense of rhythm and motivate them to follow rhythms and play the correct beats. During actual teaching, children can first listen to rhythms played by the robots and understand the arrangements of each instrument in the music. The results of the experimental course and field trials indicate that the proposed system not only increases students’ interest and efficiency in learning but also helps them in understanding musical rhythms through interaction and composing simple rhythms.

Although a drum machine can synthesize drum sounds, the sound it generates is not as authentic as that produced using actual percussion instruments. In performances with real instruments, various sound effects can be generated using different strike strengths, and users can clearly feel the changes of musical volume and level. Striking a drum at differently location can produce various sounds that cannot be replicated by a drum machine. In the experimental course, we discovered that the students naturally followed the robot’s striking actions and rhythm to generate their own rhythmic body movements.

In the future, we expect to project hints regarding beat points onto the practice drumhead. Users will thus be signaled in advance regarding which drum to strike and where it should be struck. Additionally, vibratory sensors will be installed on each drum to identify the beating accuracy and note the score sections where the user is prone to making a mistake or is unable to keep up with the rhythm. For the robot performance system, we will provide other items as robotic percussion instruments. For example, cups made of various materials can generate different loud tones, whereas wood veneers can be used to produce high-frequency sounds. Learners can also cultivate their ability to distinguish sounds by listening to percussion using different items. In the rhythm teaching system, we will introduce basic music theory (e.g., notes, score, and rhythm) and the various elements of a drum set (e.g., bass drum, snare drum, and hi-hat).