Handcrafting Objects made with Machine Learning: An Object Design Approach with Computer Vision ^†

Abstract

Many of today’s computational design systems based on explicit or graphic programming software require designers to determine relationships for morphogenesis based on computational thinking supported by the abstraction process. This computational thinking process can reduce the ability to generate analogies in design development and adverse vision related to computational tools. It also reduces the innovation capacity of small companies that produce handicrafts and design teaching in a customized way. This research promotes a computational model based on machine learning combined with an analog creation process. Machine learning engines determine the objects similarity percentage between students’ objects and master objects through a forecasting model. There is a proposal to combine parametric design systems, such as Grasshopper3D, with cloud computing and an edge computing device.

1. Introduction

A digital process in design is supported by computational thinking; therefore, by a process of abstraction. Abstraction generates two main mental processes. The first is the reduction of the variables that are present in any phenomenon of reality. An example is the production of algorithms unlike the traditional design process based on analog objects. The second is a generalization with the aim of generalizing reduction using the software [1].

To transform an analogic process into an industrialized one depends on the automatization of the process that can reduce the skill to generate analogs and change the design teaching process for handcrafted objects.

Nowadays, computational thinking in digital design is often underestimated because there is the misconception in the current practice where computational design is equal to superficial toll knowledge without algorithms knowledge [2]. The actual design practice uses sketches, models, or another way to express ideas process, is closer to ancient practice of art, there, creative achievement was linked to in-depth knowledge of tools. Also, the computational thinking process is not part of the traditional practice with analogic tools because the implementation cost is huge, and it needs professionals with higher education training [3]. In this context, another creative practices such as craftsman process cannot incorporated digital technologies based on computational thinking very well where the craft master knowledge is the key piece to improve digital technologies.

There are contradictory points between the analog process of handicrafts and industrialized production based on digital technologies. For instance, the production speed of craft objects depends on the master craftsman time, which is finite. On the other hand, digital technology has the advantage of reaching a lot of people around the world in a few seconds. The handcrafted design can imprint the designer’s personality on each object, which is something that the industrial process cannot do.

The challenge to be overcome, in many cases, is the belief that being creative with software is knowing the commands associated with the production of geometry and not with the production of new strategies in the use and customization of software in original ideas; the over-dependence on programming logic can reduce the intuition and sensitivity.

Figure 1 shows a handcraft master workshop with different objects made with clay. The picture on the bottom shows the modeling platform; in this case, it is the workshop floor.

2. Crafting Inference Engine

2.1. Research Site

This research was developed in Lamas city, San Martin, Peru, in the Wayku indigenous community (Figure 2) and includes images taken in an artisan’s workshop located in the community and images from a laboratory.

2.2. Inference Engine Architecture

The system architecture was: Frontend in Rhinoceros 3D-Grasshopper (1), Backend in Amazon Web Services (2), Frontend with Nvidia Jetson Nano (3), AI in edge computing model (4), AI engine on cloud computing server (5) The five parts work in the Nvidia Jetson Nano (3 and 4), Amazon Web Services (2 and 5), and Rhinoceros 3D-Grasshopper (1) (Figure 3).

The data collection starts in the Nvidia Jetson Nano and the ML engine-computer vision, both works to send information to Amazon Web Services (AWS), which stores the data and runs another ML engine for prediction, after Rhinoceros3D-Grasshopper receives this prediction from AWS using an API Rest.

2.2.1. Frontend in Rhinoceros3D-Grasshopper

A connection system with AWS was made to transmit parametric design data and know the inference results from the Nvidia Jetson Nano board (Figure 4).

This means the new component receives data from variables such as parameters and produces results such as outputs. Once the user sends information through inputs, the component uses a method to obtain information from AWS; in this way, the JSON message format was used.

2.2.2. Backend in Amazon Web Services

The AWS backend was made to locate complementary microservices, such as databases, computing without servers, unstructured data storage, support connections to foreign AWS, and the forecasting engine model.

• AWS API Gateway (API connections)
• AWS DynamoDB (database)

AWS DynamoDB stores the data to be transferred from Grasshopper to Jetson, and vice versa. A database is used to query and write information at high speeds;

3.

AWS S3 (unstructured data storage)

It is used to store images that come from the Nvidia Jetson Nano device and allows us to obtain this information not from the AWS application;

4.

AWS Lambda (computation without servers)

To execute code without turning on a server, providing viability to the prototype, it is used to process images, save them, and save information in a database;

5.

AWS IoT Core (to connect edge device to AWS)

It is used to transfer the edge device results to the cloud using MQTT and HTTP protocols;

6.

AWS Forecast (ML engine to predict)

It is used with the DeepAR+ predictor that allows for a performance prediction based on time series; this time series was previously saved using edge computing.

APIs are built based on the information transfer needs between Rhinoceros 3D-Grasshopper and Nvidia Jetson Nano under the REST protocol.

2.2.3. Frontend with Nvidia Jetson Nano (Edge Computing)

This allows code execution for taking and sending images and also image processing with a neural network.

2.2.4. ML Engine in Edge Computing Device

1.

Finished object.

An ML engine recognizes that a ceramic work looks so much like one completed well and another finished with deficiencies. This computer vision is a classification task built on the Nvidia board with the ResNet 18 algorithm. The objective is to identify how much the apprentice object resembles that of the master craftsman through a bank of photographs (see Figure 5).

2.

Object making process.

To recognize the step in the object making process, a regression task was used with the goal of detecting the position of the last ceramic mass with a real process (Figure 6). It was also used to detect imperfections that can damage the final object (wrinkles and cracks).

In machine learning, the classification task is associated with engine training based on categorical values as labels, where an object can be categorized as one type or another. A regression task is also related to supervised learning, but the labels are not categorical and are values that change according to a trend; this ML engine can recognize a specific point in the image (Figure 7).

The reason for choosing an edge device was the need for ease-of-use in artisan communities, where they do not usually have a personal computer with which to carry out their activities; the other reason was the ability of the edge device to run high-intensity computing tasks.

Regarding the connectivity of the device, the workshop has a telephone signal which allows the Wi-Fi network to be generated from a mobile phone to speed up the training of neural networks. In the case of not having a connection that allows the use of a mobile phone, it is possible to use a GPRS/GSM device, which allows access to the telecommunications chip and, through it, the internet.

Electricity is important for the project; in this prototype, the artisans provided us with the electrical connections, and the use of batteries is suggested.

2.2.5. ML Engine on Cloud Computing Server

The methods are divided into two sets of microservices, to respond to (1) information sent from the edge computing device, and (2) information sent from Rhinoceros3D-Grasshopper. For both, the following procedure was used (Figure 8).

After the ML forecasting was trained, it was important to connect this microservice with AWS.

2.3. Data Capture and Processing

For the collection of information, the creation process stages suggested by the master craftsman were taken. These stages were stored in the Jetson device, such as pictures to be consumed, and a bank of images was developed to have labels related to the position of the last clay mass (Figure 9). For this reason, it is possible to associate the advance percentage with the position of the mass. These images were taken from various angles and under different lighting conditions.

2.4. System Features

The system uses two user interfaces—first, on the edge computing device, and second in Rhinoceros 3D through a Grasshopper3D component. The edge computing device shows that it looks very much like the object of the master (Figure 10). The Grasshopper interface shows the prediction vector made by the Amazon Web Services server (Figure 11).

Figure 12 shows the image analysis using the custom Grasshopper component, this sample image was attached to the Grasshopper route component, and after that, the custom component sends the image to AWS as a base 64 string to activate the forecasting machine learning engine on AWS, and the Nvidia Jetson to execute the second-, and third-machine learning (ML) engine. The custom component has two input parameters, the image route and the activation button. There are six output parameters; “cloudRes” shows the complete response from AWS; “%_ToComplete” shows the percentage for completing the craftsman object, this comes from Nvidia Jetson Nano with the ML classification Engine; “CloudAI-Recog-X” and “CloudAI-Recog-Y” parameters show the point at X and Y coordinates on the sample picture, this information comes from Nvidia with the ML regression engine; Nvidia engines use AWS to send data; “CloudAI-Suggestion-X” and “CloudAI-Suggestion-Y” come from a forecasting model located on AWS, and show the suggested vector that the apprentice needs to get (green arrow) (Figure 11).

3. Conclusions

Usually, teaching design is a process of accompanying a teacher and apprentices. This process is linked to the time and space in which the teacher can develop. In this research, it is proposed that the prediction system allows the teacher’s design process to be captured as a continuous process of geometry development to increase the scope of the teaching process and provide better education with the teacher’s guidance in the computer system.

Designers can reduce the time taken with the suggestions and avoid losing the analogies made by analog design. This means that the designer can learn to create forms without explicit programming and with an assistant who suggests how to continue the development of the physical object in a vector.

Analog design allows for the flexibility to incorporate different ideas into the design and make quick decisions. The system captures the imprint of the designer or master craftsman in the creation of physical objects to reduce the time it takes to learn the master design process.

4. Discussion and Future Work

Small craft workshops in Latin America sell their products to tourists. Their productive capacity and income are related to the time that the master craftsman can dedicate to the production of new pieces. To develop the skills of apprentices, years of accompaniment are needed for education, and understand that the digital resources as a means for digital craftsmanship to bring together visual thinking with manual dexterity and practical knowledge [4].

For the development and modernization of infrastructures, contemporary design tools would be incorporated into a traditional process to improve the workshop’s production capacity. Likewise, as it is connected to a server, the work can be viewed by potential buyers on a website to evaluate which pieces are being developed as well as the connection with other artisan workshops around the world to transmit their imprint through the machine learning model.

How can the processes of machine learning be integrated into the manufacturing creation process?

The apprentice needs the master’s guidance. This guidance is crucial to learning about the development of physical objects and is delivered through suggestions made by machine learning. In addition, the process of advancing the object and the detection of possible faults in its creation are delivered by the Nvidia Jetson Nano through the user interface.

This research proposes the complementation of the parametric–generative system to the analog process to use parameters and generative relations, such as application for computational thinking, in explicit or visual programming with machine learning suggestions made by an analog process.