1. Introduction
Intangible cultural heritage (ICH) includes traditions or living expressions that are inherited from our ancestors and passed on to our descendants; for example, oral traditions, performing arts, and skills required to make traditional crafts. Further, ICH is not a cultural manifestation, but rather the wealth of knowledge and skills passed on from one generation to the next. In recent decades, the lifestyle of people has changed drastically, and the influence of traditional culture on everyday life is diminishing [
1]. Now, we can find many traditional cultures and products mostly in documentary films or in museums.
Many governments and organizations are attempting to preserve their ICH, which is in crisis. In particular, UNESCO listed out the ICH of the world and operated multimedia archives, eServices, to record these ICH [
2]. However, such methods only record the external appearance of the performance of an expert, and higher-level data are required to revive the missing heritage information without any help from other experts.
Meanwhile, multisensor-based systems have been employed in various fields. Lei et al. introduced a fault detection method based on multisensor data fusion [
3]. Dong et al. presented an overview of recent advances in multisensor satellite image fusion [
4]. Yuan et al. designed a wearable multisensor system to obtain high-accuracy indoor heading estimations, according to a quaternion-based unscented Kalman filter algorithm [
5]. Han et al. introduced a comprehensive approach for context-aware applications that utilize the multimodal sensors in smartphones [
6]. Choi et al. proposed a multimodal sensor-based method for evaluating fear based on nonintrusive measurements [
7].
Recently, novel approaches based on multisensors have been suggested to rigorously preserve ICH. In the European Union, experts in various fields including computer science, education, medical science, and physiology, from 12 organizations in seven countries, conducted the “i-Treasures Project”. They captured multimodal data such as dance motions, craft motions, brainwaves, and facial expressions in traditional singing, dance, craft, and composition. Further, they designed educational game-like applications for practicing different types of ICH expressions, based on the data [
8,
9]. Grammalidis et al. introduced the dataset created by conducting an i-Treasures project, including traditional dance moves captured with multiple Kinects or optical markers, and human beatboxes captured with a hyper-helmet and audio equipment. Magnenat-Thalmann et al. [
10] also digitized folk dances originating from several regions of Europe, using an optical motion capture system as a recording device, and provided a learning framework for folk dances. Protopapadakis et al. [
11] used Kinect to capture depth images and videos of six traditional Greek dances in order to identify key movements and gestures. Lombardo et al. presented a solution for drama preservation in terms of a formal encoding through computational ontology [
12]. In addition, some researchers measured the learning effect of learners using a haptic device by capturing the process of manufacturing traditional paper [
13].
Painting is one of the essential heritages for humankind; the history of painting dates back to prehistoric times and spans most cultures. This is also true for traditional Korean painting (
Figure 1). Incidentally, the number of the successors has been decreasing on economic grounds, and we are experiencing difficulties in its transmission. However, there are few studies on capturing painting work to aid its transmission.
In this paper, we propose a novel multisensor-based acquisition system that records traditional Korean painting work with minimal interference. Painting work comprises the interactions between the following essential components; painter, canvas, brush, and pigment. They produce large amounts of different types of information, such as the painter’s pose, hand gesture, grasping power, brush pose, stroke shape, canvas material, and pigment concentration. Among them, we selected the painter’s action, canvas image, brush pose, and pigment information as the essential information for reviving traditional Korean painting. Thus, the proposed system captures this information.
Completing a traditional Korean painting requires a large amount of time, which implies that the proposed system should be sufficiently robust to unexpected external impacts. There have also been studies that acquired the data of brush motion by using a haptic device or attaching wired sensors to a brush [
14,
15]. Their objective was data acquisition for brush modeling, and they did not attempt to prevent interference with the painter’s work during acquisition. In contrast, we have utilized contactless sensors to avoid disturbing the painter’s work. As an exception, we had to attach a small marker tool, rather than a relatively heavy sensor, to a brush, because it was difficult to track the pose of the brush without any tools attached.
Further, we have encountered various difficulties in the process of building the proposed system. One of them is to configure the initial pose of a brush well. We have attempted to hold a brush upright by hand for the initial pose set-up, because canvases, in general, are laid out flat on a floor in traditional Korean painting, but there would always be an error of a few millimeters. Another problem is the time mismatch in the data captured from the sensors of the proposed system. Even if we started the sensors simultaneously, the recording time for the same event would differ from sensor to sensor. To analyze multimodal data properly, the sensors must be time-synchronized.
We utilized the proposed system to capture the painting work performed by two experts, and we visualize the captured data. Further, we show the results of statistical analysis, such as the total working time, drawing time, and the number of strokes.
The main contributions of this paper are as follows.
We propose a new robust system for recording painting work using contactless sensors, which has not been attempted so far.
We address the issue of the initial pose setup of a brush, which arises from the difficulty in holding the brush accurately upright.
We address the time synchronization problem between two heterogeneous sensors.
As a result, we lay the groundwork for preserving traditional Korean painting, which can easily be applied to other intangible cultural heritage related to painting.
The rest of this paper is organized as follows. We describe our proposed system in
Section 2. We show and discuss the experimental results from the painting work by two experts in
Section 3. Finally, we provide some concluding remarks in
Section 4.
3. Experimental Results
We requested two professionals in traditional Korean painting to draw Siwang, one of the major gods of Buddhism. We captured their activities and tracked their brush stroke.
Figure 17 shows the series of images captured in the painting work by Expert 1 and Expert 2. The top line shows the canvas images acquired from the canvas camera located at the top of the head. The middle and bottom lines show the color and depth images, respectively, captured by the Kinect sensor.
We divided the brush poses into two classes—strokes and others. Strokes are marks made by drawing a brush in one direction across the canvas. Recall that we set the tip of the brush as the pivot point. If a series of pivot points is observed under the canvas, those points compose a stroke. By doing so, we can extract useful information such as the number of strokes and the time interval for each stroke.
Table 1 summarizes the experimental results. In this experiment, we analyzed both sketching and coloring, respectively. Cho is the phase of the sketch using a relatively thin brush and a black pigment. Chaesaek is the phase of applying colors and drawing patterns. The column
Number of strokes denotes the number of strokes found during the two experts’ cho and chaesaek work of the two experts, respectively. The column
Stroke length denotes the length of a stroke, which is calculated as
where
is the coordinate of the
ith point in the stroke. We derived several statistics, such as the mean, standard deviation, maximum, minimum, and sum from the stroke lengths.
The column “Stroke time” denotes the time it takes to draw each stroke. If we denote by
the creation time of the
ith point in a stroke, we can calculate stroke time as
where
and
mean the start and end times, respectively, in the the stroke. Similarly, we determined the mean, standard deviation, maximum, minimum, and sum for stroke times. Moreover, we added the column “Work time total”, which includes all the time for drawing, preparing pigments, or waiting for the pigment to dry.
The column “Stroke speed” denotes the stroke drawing speed, which is the ratio of its total stroke length and total stroke time. We figured out the standard deviation and the maximum stroke speed.
Let us compare Expert 1 and Expert 2. The average stroke lengths of Expert 1 are 20.71 mm and are 7.51 mm longer than those of Expert 2 in the cho and chaesak phases, respectively. Expert 1 tends to draw longer strokes than Expert 2. The average stroke times of Expert 1 are 7.28 s; 5.17 s longer than those of Expert 2 in the cho and chaesak phases, respectively. Expert 1 spends more time to draw longer strokes than Expert 2. Expert 1 is slower than Expert 2 in terms of stroke speed. From the analysis of the results, we could distinguish the work styles of the two experts even though they did not draw the same picture.
Let us compare cho and chaesaek. In terms of stroke speed, both experts drew chaesaek faster than cho. In the work of Expert 2, the mean stroke length in the chaesaek phase is even almost three times longer than that in the cho phase. It implies that chaesaek is easier to draw than cho.
Using the data, we can obtain the ratio of the total work time to the actual drawing time. In addition to the actual drawing time, the total work time involves preparing pigments, applying pigment to the brush, and waiting for the canvas to dry. The ratios of chaesaek are larger than those of cho. For Expert 1, this ratio of cho is ~1.5; however, for chaesaek, it gets closer to 2. For Expert 2, this ratio of cho and chaesaek is approximately 2.1 and 4.9, respectively. The reason why this trend is stronger in chaesaek than in cho is that the experts use more pigments and brushes in this phase.
Because we acquired time series data, we can identify the order in which the painter draws the picture.
Figure 18 visualizes the painting sequence. We plot the x- and y-axes of the stroke trajectory. The series of data is filtered by the z-axis with a threshold of the height from the ground. This implies that we filtered the data; we omit the data recorded when the brush hair is on the canvas. Colors denote the progress of the drawing in the time axis: black, red, and yellow indicate the early, middle, and final sections during painting, respectively.
In the cho phase, Expert 1 went down from top to bottom while Expert 2 climbed from bottom to top. The case of chaesaek looks somewhat complicated. In fact, it consists of various small steps. In general, an expert covers primary colors throughout the canvas, shades some parts, and finally, adds some patterns in the chaesaek phase. However, it is not clear how to classify these steps and the work order of the steps varies from expert to expert. In the chaesaek phase of Expert 1, a yellow circular pattern stands out because he adds gold patterns in his final step. On the other hand, in Expert 2’s work, the end of the person’s clothing appears to be highlighted because he partly shaded it at the end of the work.
Figure 19 shows the resulting images of their work. Previously, we only can guess the process of painting work. However, we have now shown that we could analyze the details of the work progress with the proposed system.
Archiving the data acquired in this way is very meaningful. Even if traditional Korean paintings were not passed on in the future, the data could be used as a stepping stone to revive again. In addition, it can be widely used in various fields.
Figure 20 shows a case where the robot redraws the drawing using Expert 1’s brush pose data obtained by the proposed system.
Figure 21 shows the viewer replaying Siwang drawn by Expert 1 in a virtual reality environment. In this case, we can observe their work at a point where it was never possible before, and make the reproduction work a cultural commodity.
We also expect to be able to analyze and use the acquired data for a variety of problems, as we showed in this section. For example, the data can be used to judge the falsification of a drawing by analyzing the style and strokes of a painter. Besides, we expect to classify the proficiency of a painter by capturing and analyzing work of various levels of painters, such as how to control a brush consistently, how to keep the angle between canvas and brush constant.
As we have applied the proposed system to the field, we found some unexpected limitations. One example is the issue of the coloring technique called barim. Painters use several brushes during painting. However, they usually use only one brush in a hand. The proposed system determines the active brush by considering the speed and position of the brushes detected by the motion-tracking sensor. Barim is a technique for creating gradation effects by applying pigments and spreading them with water. At this time, painters hold two brushes in a hand and use a pigment on one brush and water on another brush. Then, they draw their drawings using the two brushes alternately. The proposed system did not consider this situation, and as a result, the experts had to abandon their habit and work uncomfortably.
4. Conclusions
In this paper, we proposed a novel multisensor-based acquisition system that captures the core work of traditional Korean painting while minimizing interference in an expert’s work. The proposed system records the information of four components of painting work—painter, canvas, brush, and pigment. To record the information, the proposed system consists of a sensor mounting frame, contactless sensors to capture data, a marker tool, and a pigment selection tool. We described how to solve the issues of the initial brush pose setting and time synchronization between the two sensors.
We utilized the proposed system to capture the painting work from two experts. We then visualize the acquired data, and we show some statistics. As a result of the investigation of the acquired data, it was found that there were both similar and different attributes between the two professional painters. We also discussed another usability of acquired data.
Our future works are as follows. We are attempting to solve new issues discovered during the acquisition of the two experts’ painting work. We plan to remove the painter’s body or hands to acquire clear canvas images by applying background estimation methods, such as the authors of [
19]. We believe that capturing the information of moving hands, such as in [
20], is important, so we will make efforts to derive the information by combining depth and canvas images.