**4. Results**

We evaluated the functionality of our proposed method firstly by means of an experimental setup consisting of a measurement chamber, a CNC (Computerized Numerical Control) positioning unit and a compressor nebulizer (Hangsun CN560) to simulate the real ambient conditions existing in real fermentation chambers. To obtain static measurement objects, we produced salt dough pieces, which were hardened and hence keep their shape during the whole measurement procedure. We measured the topology of the chamber interior containing different dough shapes for two minutes continuously, first without the addition of aerosol, and segmented the points belonging to a certain dough object by means of our instance segmentation neural network. Figure 7 shows the extracted dough surface measurements of two different dough shapes laid over images taken manually from the same perspective as from the sensor system. For better visualization, we reconstructed the surface structure by means of a subdivision surface modifier to obtain a denser mesh.

**Figure 7.** Surface measurement laid over scene images.

The illustration clarifies the limitations of our measurement system regarding the measurable surface part of the objects. Due to the surface curvature of the dough pieces, the reflections of the emitted light pulses outside of the red area do not reach the sensors receiving optics and hence do not generate a measurement point.

To analyze the impact of a present aerosol atmosphere, we measured dough objects with different shapes ones with and ones without induced aerosol. We again segmented the measured dough surfaces and apply a subdivision surface modifier whereby we increased the number of points with a factor of approximately 100. The resulting surface meshes allowed a superimposition to examine the deviations. Figure 8 shows the meshes of three different dough pieces.

**Figure 8.** Superimposition of dough surface with (pink) and without induced aerosol (purple) of three different dough objects (**a**–**c**).

It is observable that the aerosol particles do not have an impact on the quality of the measurement. Both measurements map a comparable surface area of the dough pieces without large deviations. By applying the iterative closest point algorithm of the point clouds generated with and the one without aerosol addition, we obtained a quality measure of the similarity. All point cloud pairs achieved a Euclidean fitness score smaller than one, which proves the high congruence of measurements with and without added aerosol. Due to measurement inaccuracies of our measurement system, a certain degree of variance is not avoidable. So-called Mie scattering does not occur because the wavelength of the used LIDAR is, with ~785 nm, much smaller than the aerosol particle size of around 10 μm [31]. The effect of optical scattering is not observable in the measurements, which could stem from the weak signal strength of the beams reflected from the particles.

In comparison to our proposed method, the existing monitoring methods using optical measurement techniques are not able to work properly under the influence of aerosol. For example, the method used in [12] would not be able to catch an image by a digital camera because the light of the object would not reach the camera sensor. The structured light technique described in [4] might be able to obtain some information due to the strength of the line laser but only when being used on a short distance.

After proving the suitability of our measurement system, we evaluated the quality of our model fitting algorithm to determine the volume of dough. Therefore, we performed ten measurements of our chamber interior with ten different hardened dough pieces, each a member of either the class round or long, and determine the real volume values by means of the water displacement method to obtain a reference value. We fit superellipsoidal models to the segmented dough surface parts and compared the accuracies to the determined reference values. Thereby, the rheological properties of dough have to be taken into consideration. The dough represents a non-Newtonian fluid and dough pieces, that are placed on a surface, show a flow behavior which results in a flat base area [32]. Measurements on the ten dough pieces prepared show that the relation between height and width of the minor-axis is approximately 0.5. Because the surface area that is measured by our measurement system does not contain information about that change of shape, our algorithm would fit a model of a whole superellipsoid to the points. Therefore, we multiplied the calculated volume with the factor 0.5 to obtain the final value. Table 1 represents the reference and the calculated volume values of the ten examined dough pieces and, additionally, the deviation between the two values. In Figure 9, we compare the statistical distributions of the volume deviations of the round and the long dough test pieces by means of boxplots.


**Table 1.** Volume calculation.

**Figure 9.** Boxplots for different dough shapes.

One can observe the higher median volume deviation of the elongated dough pieces (12.2% for the long compared to 8.6% for the round dough pieces), although the interquartile and the absolute ranges are smaller. The reason for that is that the surface area, which is captured by our measurement system is not unambiguous and too small to allow proper and accurate modeling. The fact that the volume of long dough test pieces is always calculated lower than the reference values supports this assumption. Nevertheless, the volumes determined represent a good approximation for our purpose.

After verifying the applicability of our volume calculation algorithm, we initiated the fermentation of three dough pieces of round shape inside a fermentation chamber and captured the topology of the interior continuously for 90 min. We determined a time step of five minutes to be appropriate to create a topology that is dense enough for a proper segmentation and model-fitting. Each five minutes, the measured points are assigned to a new topology state cloud to reproduce the volume expansion. In Figure 10, the processing steps performed are illustrated.

**Figure 10.** Processing steps: (**a**) Measured topography point cloud of the fermentation chamber inside; (**b**) Segmented dough pieces; (**c**) Model-fitted dough pieces.

Proper segmentation and plausible model-fitting of the points by means of superellipsoids could be seen. The metallic interior surface of the fermentation chamber does not disturb the measurement because the light scatters and bounces in uncontrollable directions. This fact improves the function of the segmentation neural network because of the more precise point cloud structure. The processing time results from the sum of the segmentation and the model fitting time. Currently, the former amounts to less than one second due to the use of an NVIDIA GeForce GTX 1060 GPU. The latter highly depends on the number of points that have to be taken into consideration during the minimization of the objective function. With the current configuration and a number of approximately 100 points per dough piece, a processing time of fewer than three seconds on average was achieved.

To obtain a ground-truth representation of the dough development, we measured the width and height of each dough piece once every five minutes and calculated the volumes assuming an ellipsoidal shape with circular base area by means of:

$$V = \frac{4}{3} \times \pi \times \left(\frac{W}{h}\right)^2 \times h \tag{9}$$

Figure 11 represents our ground truth values of the width, height, the quotient width/height and the volume to be compared with our results.

**Figure 11.** Course of width (**upper left**) and height (upper right) of three dough objects, the quotient W/h (**lower left**) and calculated volume (**lower right**).

The results contradict the assumptions about the determination of the optimal fermentation state made in the introduction of this work. According to a baking expert who was consulted during the experiment, the fermentation would have been stopped after 55 min to obtain an optimal fermentation state. Both the widths and the heights of the dough samples increased approximately linearly also after the designated optimal time and no notable change in the courses, which also caused a nearly linear volume increase. Figure 12 shows the difference between the reference and the calculated volume using our method and the relative deviation with regard to the dough volume.

**Figure 12.** Volume Difference: absolute (**left**) and relative (**right**).

We obtained median volume differences of approximately 10 cm<sup>3</sup> for the three dough pieces during the fermentation period of 90 min, which resembles our measurement of the static test objects. Furthermore, the relative volume difference with regards to the dough volume decreased with increasing fermentation time, which resulted in more accurate volume estimation when the process approaches the optimal state. The method for estimating the volume of watermelons by means of image processing proposed in [33] achieves a deviation of approximately 7.7 % on average. Approximately the same deviation (7.8 %) is stated by the authors in [34] while estimating the volume of kiwi fruits using image processing techniques. Compared to those, our method performs slightly

weaker at the beginning with a small object volume, but with increasing fermentation time we obtain more accurate volume estimations because the relative volume error decreases.

The results show that, contrary to the assumptions, the optimal fermentation state cannot be determined only by considering the dough volume gradient. Nevertheless, a pre-defined volume growth factor, which is set by an expert before the start of fermentation, that signifies an optimal state, can be detected properly using our method. In our case, we obtain a factor of approximately three that signifies the desired state.

#### **5. Conclusions and Future Work**

In this paper, we proposed a novel method for the continuous monitoring of the volumetric parameters of dough piece during fermentation. We showed that a proper segmentation of a three-dimensional topography and a reliable volume estimation could be achieved by the combination of machine learning and 3D model fitting. Using our proposed method, an essential and promising advancement of the current state of the art has been proposed, which finally enables the possibility of parallel monitoring of multiple dough pieces, instead of single ones, to get a better overview of the complete fermentation interior. In that way, certain planes of fermenting dough pieces can be analyzed individually and independently from each other. We found out that the optimal fermentation state cannot be determined only by considering the volume gradient. It remains to be examined whether other parameters like the surface moisture of the dough can be analyzed to obtain the optimal time automatically. With our method, a predefined desired final volume can be detected automatically, which already can relieve the baking staff. Due to the fact that, using our system, the fermentation chamber does not have to be opened during the fermentation to check the state, a much smoother process without interruptions can be established.

Future work will concentrate on optimizing the model fitting method and minimizing the processing time, which currently takes approximately three seconds on average per object. That is of great interest, since a fermentation chamber contains several dozens of dough pieces whose volumes have to be monitored simultaneously. A difference of one second of computation time for one object applied to 50 dough pieces would lead to a reduction of 50 s altogether, which could make the difference between optimal and over-fermentation. For the acceleration of the calculation, the complete transfer of the process to hardware, like field programmable gate array (FPGA), is conceivable. Another future research task lies in the concurrent detection of different kinds of dough object types like pretzels, donuts, and so on. Therefore, more data has to be captured and our neural network has to be trained to be able to perform the classification of different classes with a high classification rate and accuracy. It should be examined if it is possible to use simulated point clouds for the training, which would lead to an unlimited amount of training data and hence to a much more precise network.

#### **6. Patents**

A patent application for the system for the automatic capturing of the fermentation chamber topology and the determination of the volume of dough pieces has been lodged [8].

**Author Contributions:** Conceptualization, L.A.G. and A.-K.R.; methodology, L.A.G.; software, L.A.G.; validation, L.A.G.; writing—original draft preparation, L.A.G.; writing—review and editing, M.L., A.-K.R. and M.F.; visualization, L.A.G.; supervision, M.L. and M.F.; project administration, L.A.G., M.L., A.-K.R. and M.F..; funding acquisition, M.L., A.-K.R. and M.F.

**Funding:** This work is part of the research project F.I.T. Fully-Automated Fermentation System, funded by the German Federal Ministry of Economic Affairs and Energy (BMWi, funding code 16KN057228).

**Conflicts of Interest:** The authors declare no conflict of interest.
