**3. Materials and Methods**

#### *3.1. Dough Volume Monitoring*

To create a topology of the inside of the fermentation chamber, we developed a three-dimensional measuring system consisting of a linear motor (Figure 3: Linear motor), a holder with adjustable angle around the x-axis (Figure 3: Holder), a two-dimensional light detection and ranging (LIDAR) scanner (RPLidar A1M8) (Figure 3: LIDAR), and a time of flight distance sensor (Adafruit VL53L0X) (Figure 3: Time of flight distance sensor) to extend the LIDAR data to the third dimension. The system needed to be as rigid as possible to obtain highly accurate measurements. We integrated the measurement system to the back of the chamber interior. Figure 3 illustrates the system schematically on the left and shows the real module on the right side.

**Figure 3.** Schematical illustration of the measurement system (**left**) and real measurement module (**right**).

The two sensor outputs were fused by means of an Arduino microcontroller to create a single data stream containing the X, Y, and Z-coordinate of a measurement point. Therefore, the polar coordinates that are generated by the LIDAR and represented by the particular angle θ and the corresponding distance r are transformed into Cartesian coordinates by means of the transformation:

$$\mathbf{x} = r \ast \cos(\theta) \tag{1}$$

$$y = r \ast \sin(\theta) \tag{2}$$

The distance sensor directing vertically to the bottom of the fermentation chamber returns measurements of the current height of the measurement sensor system. By moving the system along the Z-axis, a complete topology of one row of the observed situation is captured and serves as the basis for further image processing that detects and investigates the volume of each dough piece. The tilting of the LIDAR scanner by the angle α enables the possibility of monitoring multiple rows of dough pieces laying behind each other. In Figure 4, the general setup of the measurement system is illustrated. This represents the real setup of bread dough pieces within a typical fermentation setup on the left and the schematic overview of the measurement process on the right side.

**Figure 4.** Test setup (**left**) and schematic (**right**).

In the example of one object (Figure 4 right), the measurement can be described as follows. Due to the known height of the sensor system z and the tilt angle α, the height of an object *zdough* at a certain position can be calculated as equation 3 by using simple trigonometric relationships:

$$z\_{\text{dough}} = z - r \times \cos(90^\circ - \alpha) \tag{3}$$

The motion of the linear motor in the Z-direction is performed automatically by means of a continuous loop moving between the maximal top and bottom positions with a step speed of 2 *mm <sup>s</sup>* to generate a sufficiently dense point cloud. In order to analyze the gradient of the volume development of the dough pieces, the topology of the fermentation chamber is measured continuously in five-minute steps. The underlying setup of the measurements in this paper generates point cloud files of approximately 6500 three-dimensional coordinate points in that way with an accuracy of 0.5-1% of the range, respectively 0.5 mm when measuring within 1.5 m, which is mostly the case in our setup. In comparison to the use of a camera, we are able to obtain depth information of the whole fermentation chamber interior, which allowed us to perform three-dimensional object recognition directly.

### *3.2. Data Processing*

The information extraction of the measured topography of the setting is performed by executing the procedure illustrated in Figure 5. The difficulty is in obtaining a highly accurate segmentation with the number of outliers reduced to a minimum. Otherwise, false models are fitted to the segmented points leading to an inaccurate volume estimation.

**Figure 5.** Processing Sequence.

Initially, the point cloud is segmented by means of a machine learning approach using a Mask Region-based convolutional neural network [15]. This neural network simultaneously performs the tasks of classification, bounding box regression, and mask estimation. The architecture of the Mask R-CNN is illustrated in Figure 6 in an abstracted form. For detailed information the reader is referred to [15].

**Figure 6.** Architecture of Mask R-CNN (own illustration based on [15]).

Classification means the ability to recognize and understand different objects. In this paper, we use this function to predict if any bread dough is present in the current topography or not. The bounding box regression determines the rectangle that frames an object, in our case one piece of dough. By means of the estimated mask, the bounding of the particular object is precisely detected. To fit the neural network to our task, a training dataset containing example topographies that were generated during a real fermentation process was built-up and projected into depth maps by taking the x- and z-coordinates as the particular pixel positions and by normalizing the y-value to an 8-bit linear scale representing the depth. These depth maps were masked with an image annotation tool by defining the regions, and hence all pixels that contain bread. In that way, a dataset of 50 images with the corresponding ground-truth was generated. Due to the small amount of training data, the so-called 'transfer learning method' was applied [21]. Therefore, a weight file that was pre-trained on the huge COCO-dataset (Common Objects in Context dataset) containing 2.5 million labeled instances in 328.000 images was used [22]. With this status, the neural network was already able to recognize common image features. Afterward, the dataset was fed with training material for the bread instance segmentation. The output of the network was a set of images containing the particular segmented bread instances. The images were transformed back to a point cloud by converting the depth value back to the y-coordinate representation. The results of this process step are the segmented points belonging to a particular instance that can be used for the further processing steps.

After the segmentation, model fitting by means of superellipsoids is applied to achieve an implicit representation of the object that allows an analytical volume calculation. The surfaces of superellipsoids fulfill the equation

$$F(X,Y,Z) = \left( \left(\frac{X}{A}\right)^{\frac{2}{\epsilon\_2}} + \left(\frac{Y}{B}\right)^{\frac{2}{\epsilon\_1}} \right)^{\frac{\epsilon\_2}{\epsilon\_1}} + \left(\frac{Z}{C}\right)^{\frac{2}{\epsilon\_1}} = 1\tag{4}$$

where *A*, *B* and *C* define the axis-scaling and ε<sup>1</sup> and ε<sup>2</sup> the deformation parameters. By means of that equation a large amount of different bodies can be represented. To achieve a complete pose estimation, the translation and rotation by quaternions are added which leads to [23]:

$$
\begin{bmatrix} x \\ y \\ z \end{bmatrix} = R\_{\text{Quat}} \begin{bmatrix} X - X\_0 \\ Y - Y\_0 \\ Z - Z\_0 \end{bmatrix}
$$

$$
= \begin{bmatrix} a^2 + b^2 - c^2 - d^2 & 2bc - 2ad & 2bd + 2ac \\ 2ad + 2bc & a^2 - b^2 + c^2 - d^2 & -2ab + 2cd \\ -2ac + 2bd & 2ab + 2cd & a^2 - b^2 - c^2 + d^2 \end{bmatrix} \begin{bmatrix} X - X\_0 \\ Y - Y\_0 \\ Z - Z\_0 \end{bmatrix} \tag{5}
$$

The parameters *a*, *b*, *c* and *d* represent the quaternion components. In order to find the superellipsoid that fits best regarding a given point cloud, we minimize the objective function

$$\min\_{\lambda} \sum\_{i=1}^{n} \left( \sqrt{\lambda\_1 \lambda\_2 \lambda\_3} \left( F^{\lambda\_4} \left( p\_{i\nu} \lambda\_1 \right) - 1 \right) \right)^2 \tag{6}$$

where the set λ defines the parameters *A*, *B*, *C*, ε1, ε2, *X*0, *Y*0, *Z*0, *a*, *b*, *c*, *d* of the 12-dimensional optimization problem and *pi* with *<sup>i</sup>* <sup>=</sup> <sup>1</sup> ... , *<sup>n</sup>* a particular point of the point cloud. The factor <sup>√</sup> λ1λ2λ<sup>3</sup> accelerates the convergence of the minimization algorithm by increasing the gradients around deep minima of the fitting function parameter space [19]. To achieve better numerical value stability, the fitting function is raised to the power of ε<sup>1</sup> as Whaite and Ferrie propose in [24]. In contrast to most of the other researches regarding superellisoid-, or the more general superquadric-model-fitting, where the Levenberg-Marquardt algorithm is used for solving the nonlinear optimization problem ([25–27]), we chose the interior point optimizer (IPOPT) software library. The underlying method applies a primal-dual interior-point algorithm with a filter line-search method and is specially designed for solving large-scale problems [28]. For a detailed explanation of the mathematical background, reference to the literature is provided.

On the basis of the model parameters determined, the volume of the particular superellipsoid is calculated using the equation:

$$V = 2ABC\varepsilon\_1\varepsilon\_2\beta\left(\frac{\varepsilon\_1}{2} + 1, \varepsilon\_1\right) \beta\left(\frac{\varepsilon\_2}{2}, \frac{\varepsilon\_2}{2}\right) \tag{7}$$

where β(*x*, *y*) represents the beta-function defined by [29]:

$$\beta(x,y) = 2 \int\_0^{\frac{\pi}{2}} \sin^{2x-1}\phi \cos^{2y-1}\phi d\phi \tag{8}$$

#### *3.3. Preparation of the Dough Pieces and Fermentation Process*

We produced standard pieces of dough made of wheat flour. It is of great importance in this process to produce equal pieces of dough with the same rheological properties. Accordingly, the dough production parameters have to be followed accurately. The following percentages of the particular ingredients were used for our recipe: 61% wheat flour Type 550, 34% water, 1.8% gold malt, 1.8% yeast, and 1.4% salt. Wheat flour Type 550 has a mineral content between 0.51% and 0.63% in dry mass and is well suited for fine-pored dough. All ingredients were combined and kneaded with a kneading machine (Diosna Dierks & Söhne GmbH Type SP24F/TEU) for two minutes at first and after a short break for an additional six minutes at room temperature. After kneading, the temperature of the dough was 24 ◦C, and it was left to rest for ten minutes. The dough was divided into equal dough pieces and set to the intermediate proofing for ten minutes before the final molding of the single dough pieces that were performed by using a dough press (Fortuna). Inside a fermentation chamber (MIWE GR)

#### *Appl. Sci.* **2019**, *9*, 4266

the temperature was set to 35 ◦C and the humidity to 95%, which are common process parameters to optimize the growth of the yeast species *Saccharomyces cerevisiae* [30].
