Graph-Based Framework on Bimanual Manipulation Planning from Cooking Recipe

Abstract

It is difficult to effectively operate a dual-arm robot using only the information written in a cooking recipe. To cope with this problem, this paper proposes a graph-based approach on bimanual cooking motion planning from a cooking recipe. In our approach, we first decompose the cooking recipe into graph elements. Then, we try to connect the graph elements taking into account the attributes of the input/output nodes. If two graph elements cannot be connected to each other, we search for a graph element that can be inserted between them from a motion database. Since the constructed graph includes the whole sequence of the robot’s motions to perform the cooking task, we can generate a task sequence of a dual-arm manipulator simultaneously performing two different tasks by using two arms. Through experimental study, we show that it is possible to generate robot motions from a cooking recipe and perform the cooking motions while simultaneously moving the left and right arms.

1. Introduction

One of the ultimate goals of robotics research is to replace humans with robots in everyday household chores. This research focuses on cooking as one such household chore performed by humans. If a robot is to cook a meal, it must understand cooking recipes that were written for humans and must process ingredients appropriately based on these recipes. In order to deal with this problem, we propose a graph-based approach to convert a cooking recipe into an executable format for a robot and plan the robot’s motion based on the recipe.

The information described in the recipe is not sufficient for a robot to perform a cooking task. To control a robot based on a cooking recipe, it is necessary for a robot to add the actions which are not explicitly described in the recipe that a human performs unconsciously. For example, if “cut carrots” is described in a recipe we need to execute the action of “placing the carrots on the cutting board” beforehand. In addition, we need the action of “grasping the knife” before the cutting operation. However, these instructions are usually not described in the cooking recipe.

In this study, we propose a motion planner for the robotic cooking tasks by using the information from a cooking recipe. The planner can automatically determine objects and motions that are not explicitly described in the cooking recipe. As shown in Figure 1, our method represents a cooking task using a graph structure that can express the relationship among actions, tools, hands and objects with their state changes. Before a robot motion is planned, the elements of the network structure are stored in a database named the “action library” where each graph element includes a set of arguments defining the status of a task. From the arguments of the input node, we can detect the lacking information of the cooking recipe. The lacking information is added by the corresponding graph element obtained from searching the database. In addition, by introducing the hand node, our proposed planner automatically determines the bimanual coordination by assigning the extracted motion from a cooking recipe to one of the hands.

In this paper, after explaining the related works in Section 2, we explain each element of the proposed network structure in Section 3. We explain how to compose the network structure from a cooking recipe in Section 4. We show the results of experiments in Section 5.

2. Related Works

To plan the motion of a robot performing a task, Wolfe et al. [1] proposed hierarchical structured planning problems including the task and the motion layers, our proposed framework can also be categorized into the task and motion planning problems. For the task planning, a series of logic operations has been proposed such as in [2,3]. For the motion planning, the motion of a robot is planned by using several methods, such as the topological optimization [4,5] and graph search [6]. Recently, some extensions of the task and motion planning have been proposed, such as partially observable environment [7], symbolic method [8], real-time planning [9] and regrasp planning [10,11].

In addition, dual-arm manipulation has been receiving attention by several researchers such as [12,13]. In recent years, motion planning for dual-arm manipulation methods has been proposed, such as sequential planning [14,15], coordinated planning [16], assembly planning [17] and assembly planning with regrasp [18].

In addition, some researchers have focused on the robotic cooking task. Yamazaki et al. [19] performed robotic cooking operations like cut and peel by recognizing the foods and the cutting board. Mu et al. [20] analyzed the mechanics of the cutting operation for cooking ingredients. Yamaguchi et al. [21] created a robot to learn a pouring operation.

Recently, some studies have been conducted on robotic motion planning from linguistic information such as recipes. Inagawa et al. [22] propose a method to analyze a cooking recipe and generate a cooking behavior that can be executed by a robot based on the obtained behavior code. Beetz et al. [23] applied the natural language processing to recipes uploaded in the web and performed the action planning of robots. Lisca et al. [24] proposed a framework for conducting chemical experiments from linguistic information. However, In these studies, information that is not explicitly described in the linguistic information cannot be used to generate the robot’s motion. Kazhoyan et al. [25], however, prepared reasoning rules written in action descriptions and used them for supplying information which is not contained in the language. In addition, large-scale recipe data have been trained with RNN to build inference models [26]. Paulius et al. [27,28,29] proposed a representation framework called FOON (Functional Object-Oriented Network) to model the relationship between actions and objects and the state changes of objects in a task. On the other hand, we have proposed the framework on task and motion planning of dual-arm manipulator from a cooking recipe composed of food image and cooking instructions. In our previous research [30], we explained the entire structure of robot motion planning and detailed the recognition of ingredients from the food image and coordination of two arms. This paper focuses on the motion planning framework based on the graph structure, which solves the mentioned problem of planning the robot’s cooking task from a cooking recipe. We detail the method for generating the graph element, named the functional unit, and add the actions not explicitly described in the recipe. Our graph-based method clearly visualizes the action sequence and can obtain several information items from the graph structure. If two actions are parallel to each other, they can be executed by two arms at the same time. In addition, the connection condition of two functional units is clearly defined using the input/output arguments. To add the missing information, instead of preparing the reasoning rules, our method uses the graph element named the functional unit with placeholders. With our graph definition, the condition for completion of missing information can also be clearly defined by using the input/output arguments.

3. Proposed Method

In this paper, we represent a cooking task using a graph structure including the actions associated with objects and the state changes of objects. This graph structure enables us to identify actions and objects that are necessary for executing a cooking task but are not described in a cooking recipe. Our proposed planner generates a sequence of cooking task motions while complementing it with actions that are not explicitly described in a cooking recipe. The graph structure used in this study extends the FOON (Functional Object-Oriented Network) [27,28] to handle tools like cooking utensils and to consider the robot states. Section 3.1 describes the network structure used in this study and Section 3.2 describes the procedure for creating a graph based on a cooking recipe.

3.1. Graph Structure

The graph structure used in this study is a directed graph consisting of edges and three types of nodes, i.e., the object, hand and motion nodes. The nodes have attributes and can represent their state change in the manipulation task. This network structure is created by connecting multiple functional units where a functional unit denotes the smallest unit of the graph structure, representing the information contained in a single action. A functional unit also consists of edges and three types of nodes. In the following, we first describe the details of the nodes and functional units included in the graph structure.

3.1.1. Object Node

An object node represents the object in action. The types of objects in cooking include ingredients, seasonings, tools and containers. An object node includes attributes of type, name, place and state (see

Table 1. Examples of objects’ name and attributes.

Type	Name	Attribute1 (Place)	Attribute2 (State)
Food	Potato, Beef	Storage space, Cutting board	Whole, Cut, Chopped
Seasoning	Sugar, Salt, Water	Storage space, Cup, Bowl	Whole, Mixed
Tool	Knife, Spatula, Ladle	Storage space, Hand	Clean, Dirty
Container	Cutting board, Bowl	Storage space, Work space	ingredient inside (Potato)

). If two instances of the same object type have two different attributes, they can be treated as different object nodes. This makes it possible to determine the action needed for executing the task. For example, as shown in Figure 2, it is possible to distinguish between “a knife that exists in a storage location” and “a knife that is being grasped by the robot”. The knife needed for the cutting operation can be specified as the “grasped knife” on the right side of Figure 2. If the knife is in the storage area, we can determine that the grasping action is required before the cutting action.

3.1.2. Hand Node

A hand node is assigned for each robotic hand used in the cooking task. It includes the usage status of the robot hand as an attribute. Specifically, it has the names of the objects grasped by the hand. With these attributes, it is possible to determine the task sequence according to the number of robot arms used. For example, we can plan the motion of a robot stir frying a food while simultaneously holding a spatula in one hand and a cooking pan in the other.

3.1.3. Motion Node

A motion node represents an action. In a cooking task, there are two types of motions: main motion and sub-motion (

Table 2. Type of motion node.

Motion	Name
	Cut (half)
Main motion	Poar
	Boil
	Pick & Place
Sub-motion	Grasp
	Release

). The main motion includes cooking motions like cut, pour and boil. These actions are usually described in a cooking recipe. The sub-motion is used to prepare for a cooking motion, e.g., pick and place, grasp and release. This node does not have any attributes.

3.1.4. Functional Unit

A functional unit consists of a motion node, hand nodes and object nodes where object and hand nodes work as inputs and outputs for a motion node. The attributes of the input objects and hand nodes are updated according to the changes caused by the execution of the cooking task. The functional unit represents the objects needed for executing the motion and its changes during the motion execution. An example of the operation “cut potato in half” is shown in Figure 3. To execute this operation, the cutting board, the potato and the knife need to be in the input objects and hand nodes. We also show that, according to the motion execution, the state of the potato and the knife changes, highlighted in yellow in Figure 3.

3.1.5. Discussion

Different from FOON, our proposed network structure can take into account tools like cooking utensils and possible orders of actions by considering the attributes of a hand node. For example, consider the case where a knife or a chopping board is in a storage location when the robot performs a cutting operation. In such a case, our method can include actions such as “grasping the knife” and “preparing the cutting board” in the task planning (Figure 4).

In addition, the execution order is determined based on the status of the robot hand. In the example shown in Figure 4, there are multiple execution sequences, but by taking into account the status of the robot hands, we can determine the execution order, i.e., grasping of a knife should be carried out after the picking and placing of a potato; this is because the hand should not grasp anything before executing the pick and place.

3.2. Graph Construction

By using the graph structure described in Section 3.1, we can identify the missing actions and objects in a cooking recipe. We store the functional units on basic operations used for cooking in a database. If missing actions or objects are identified, we search in the database and make adequate modifications to a functional unit stored in the database. We use this modified functional unit to add the missing actions and objects. In this section, we first describe the motion database and then explain the procedure for creating a network adding the missing actions and objects.

3.2.1. Motion Database

In this subsection, we explain the motion database (DB) used in this study. In this study, we use two types of motion DBs for main motion and sub-motion, which will be referred to as the main motion DB and the sub-motion DB, respectively. In each DB, functional units in which the parameters have not been set yet are stored. We call such functional units the functional units with placeholders. By setting adequate parameters to the arguments, an appropriate functional unit can be created.

The main motion DB stores the functional units with placeholders corresponding to the main motion, i.e., cooking motion explicitly described in a cooking recipe. The type of object required, included in a functional unit, is determined for each main motion. For example, the objects required for the “pouring motion” are the “ingredient” to be poured, the “container” in which the ingredient is placed before pouring and the “container” into which the ingredient is poured. As shown in this example, even if an object cannot be uniquely defined, the type of object can be defined for each operation and objects that cannot be uniquely defined can be expressed as functional units with placeholders as shown in Figure 5 where placeholders are marked in yellow. Such functional units with placeholders are constructed for each main motion and stored in the main motion DB.

In the sub-motion DB, functional units for each sub-motion, i.e., the motion which is not described in the cooking recipe, are stored. Sub-motions are related to the handling of objects and include “Pick & Place,” “Grasp,” ”Release “Pick & Place”, “Grasp”, “Release”, etc. The types of objects required for these motions can also be defined and they can be expressed as functional units with placeholders as shown in Figure 6 where placeholders are marked in yellow. Such functional units with placeholders are constructed for each sub-motion and stored in the sub-motion DB.

3.2.2. Graph Connection

In this subsection, we explain the procedure for creating a whole graph structure to achieve the cooking task from a recipe. Figure 7 and Figure 8 show the outline of the creation procedure using the motion DB.

The input is a sentence described in a cooking recipe. First, it is parsed to obtain the relation between the object and the motion (Figure 7➀), where the i-th motion obtained here is denoted as the main motion (i) ($i=1,2,\cdots ,N$). Then, we search for the functional unit with placeholders stored in the main motion DB corresponding to the main motion (i), and substitute the extracted object and motion from the recipe for the arguments and create a functional unit (Figure 7➁). The next step is to recognize the object initially placed in the robot’s working environment. We obtain the name, state and place of the object and create the object node. Then, the functional unit is created in which the object node obtained here becomes the output node (the input and motion nodes are assumed to be empty nodes) (Figure 7➂). Finally, the functional units obtained here are connected based on the merging algorithm to complete the network (Figure 7➃). The details of this merging algorithm are described in the following paragraph.

The flowchart of the network connection is shown in Figure 9. In the connection algorithm, we first create a list of input nodes (Figure 9(A)) where the elements of this list are the input of the functional unit for the main motion i (abbreviated as the functional unit (i)) ($i=1,\cdots ,N$). Then, we create a list of output nodes (Figure 9(B)) where the elements of this list are the output nodes of a functional unit to be connected to the functional unit (i). For example, when a robot starts a cooking task, the elements of the output node list are the output of a functional unit, the input of which includes the objects’ initial configuration.

Next, the elements of the input/output node lists are compared with the input/output of functional units to see if they have the same names and attributes (Figure 9(C)). For example, if the elements of the output node list match the input of the functional unit (j) as shown in Figure 10, we connect the functional unit (j) to the network structure by deleting these elements from the output node list and appending the output of the functional unit (j) to the output node list (Figure 9(C-1)).

Furthermore, if the elements of the output node list do not match the input of any functional units for the main motion (abbreviated as the main-functional units) as shown in Figure 11, we create a new functional unit for the sub-motion (abbreviated as the sub-functional unit) and try to connect it to the network structure (Figure 9(C-2)). We substitute the elements of the output node list for the arguments of a sub-functional unit stored in the sub-motion DB. Then, the sub-functional unit is connected to the network structure by deleting the elements of the output node list and appending the output node of the sub-functional unit to the output node list (Figure 9(B)). The functional units are iteratively connected by comparing the elements of the input/output node lists with the input/output of functional units (Figure 9(C)). At each connection, we check if the input node list is empty (Figure 9(D)). If the input node list is empty, i.e., all the input nodes of the main motion are combined with functional units (Figure 9(D)), the actions necessary to execute the main motion are added and the main motion is ready to be executed. Finally, the order to execute each motion is determined, taking into account the number of available arms (Figure 9(E)). Here, an executable sequence will not exist if the arm is grasping an object and cannot perform any other action. If there is no executable sequence, we create and combine sub-functional units that eliminate such cases to obtain an executable task planning (Figure 9(E-1)).

After the above process is performed for the main motion (i) ($i=1,2,\cdots ,N$), we obtain a whole network structure that represents the cooking task.

3.3. Bimanual Task Planning

We introduce a dual-arm robot to efficiently perform the cooking task. Introduction of the hand nodes to a functional unit enables us to consider the number of hands used for each cooking action and to plan the cooking task considering multiple hands performing multiple tasks in parallel. In order to reduce the execution time, we plan the cooking task while taking into account whether two actions can be simultaneously performed with two different hands. An example is shown in Figure 12 where we explain the method of determining which hand to be used and explain the method of simultaneously executing multiple tasks.

First, the hand to be used in each action is determined by considering the distance from the hand to the object to be grasped. Figure 13 shows an example where the objects placed in the storage space (Right) and (Left) are grasped by the R-Hand and L-Hand, respectively. If there are multiple candidates, the hand to be used is determined by evaluating the length of the trajectory to execute the action. Figure 13 shows an example where the objects placed in the work space are grasped by the hand with shorter length of trajectory.

From the graph structure, we can extract two motion nodes which can be simultaneously performed. If two different hands can be assigned for these two actions, we plan these two actions to be executed simultaneously.

3.4. Motion Planning

In this section, we describe the motion planning method to realize the planned task sequence. Before executing the cooking task, we define multiple grasping poses to stably grasp a tool as shown in Figure 14. When executing the cooking task, we use a vision sensor to detect the pose of the tool. Once we obtain it, we select the highest priority grasping pose where the IK is solvable. The priority for grasping poses is set in a heuristic fashion. After the grasping pose is determined, the trajectory of the robot is planned by using RRT-connect.

4. Experiment

To verify the effectiveness of our approach, we conducted the experiments using a real dual-arm manipulator.

4.1. Experimental Setup

As a cooking recipe, we consider two simple Japanese sentences that are equivalent to “Cut potatoes in half and pour them into a bowl” and “Heat oil in a frying pan over high heat and fry pork” in English. We used CaboCha [31] to obtain the dependency structure of Japanese sentences written in a recipe. In this experiment, we prepared a work space and a storage space in a kitchen environment. We assume that the objects are stored in the left or right storage spaces before the cooking task starts (Figure 15). We use two UR-3 robot arms in which a 2-Fingered 85 mm Robotiq Gripper is attached at the tip. UR-3 has 6DOF and can reach the area with 500 mm radius. Its payload is 3 kg. For more detailed specifications, please refer to the website of UR robots [32]. Since we use two robot arms with independent controllers, we approximately realized the coordination of two arms. When two arms move at the same time, we send the independent motion command to both arms at the same time. By using RRT-connect, we can specify the via points between the initial and final configurations. The trajectory between two via points is determined by using the native function prepared in the controller of the UR-3 robot.

We constructed the cutting board, spatula, knife and stove by ourselves using a 3D printer.

We attached an ArUco marker to each tool. The pose of the tool is detected by capturing the image of the ArUco marker through a 2D RGB camera (Figure 16). We prepared eight grasping poses for the cutting board and one grasping pose for other tools. On the other hand, the relative pose of ingredients from the tools is assumed to be known. After detecting the position/orientation of the tool, the target wrist pose can be obtained. Then, the manipulator is commanded by using the open-loop control.

4.2. Results

We first show some examples of functional units and corresponding robot motion. We show two main motion (cut in half and pour) and four sub-motion (two pick and place, grasp and release) actions in Figure 17. When cutting a potato in half (a), the cutting board is placed on the work space with a potato placed on it. By cutting, the state of the potato changes from whole to cut. In addition, the state of the knife changes from clean to dirty and the hand is always holding the knife. On the other hand, by grasping the knife (e), its place changes from stand for knife to hand.

We then generated a graph structure corresponding to each sentence. The graph structure corresponding to “Cut potatoes in half and pour them into a bowl” is shown in Figure 18 where the motion node of the functional unit stored in the main-motion DB is cut (half) and pour. To realize the cooking task, we need five functional units stored in the sub-motion DB where their motion nodes are three pick and place, a grasp and a release. Here, the pick and place of a potato and grasping of the knife are performed in parallel. Figure 19 shows the motion of a robot performing this cooking task.

The graph structure corresponding to “Heat oil in a frying pan over high heat and fry pork” is shown in Figure 20 where motion nodes of functional unit stored in the main-motion DB are pour, turn on, heat (high heat) and stir fry. To realize the cooking task, we need two functional units stored in the sub-motion DB where their motion nodes are pick and place, grasp. Figure 21 shows the robot’s motions while performing this cooking task. As a result, the robot was able to cook autonomously and we confirmed the effectiveness of our method for motion planning from cooking recipes consisting of simple sentences. In both cases, the calculation time of the motion planning is between 1 and 2 min.

5. Conclusions

In this paper, we proposed a method for planning a cooking task using a dual-armed robot by representing the cooking task in a graph structure. Our proposed method can deal with actions and objects that are not explicitly described in the cooking recipe and automatically add them to the task contents. We applied the proposed method to sentences of a simple cooking recipe and confirmed that the dual-armed robot can perform the cooking task based on the result of task planning.

In the future, we will verify the effectiveness of the proposed method for cooking recipes with more complex sentences. In addition, when cutting food, the manipulator has to carefully adjust the force applied to the food from the knife. Such closed-loop control is not considered in the current paper and is our future research topic. Furthermore, we have manually defined each functional unit. The automatic construction of functional units is considered to be our future research topic.