3.1. Image Classification Module
In order to obtain the growth state of leafy greens and infer their maturity date, this paper analyzes the phenotype of leafy greens based on a deep convolution neural network to determine their growth stage. In the field of computer vision, the technology of object detection and instance segmentation is relatively mature. A naive idea is to calculate the average leaf area of leafy greens by case segmentation and compare the growth curve to determine its growth stage. However, in the actual planting environment, the planting of leafy greens is crowded, and the occlusion between vegetable leaves is very serious, so the effect of this kind of model is not good. Considering that the color, texture, and other different features of leafy greens can also be used to judge their growth stages, this paper directly uses image classification algorithms to classify the collected leafy greens images.
When a convolutional neural network classifies leafy greens images, it mainly relies on convolutional layers to extract the features of images at different growth stages. Given the input image, the convolutional neural network performs image preprocessing, feature extraction and classification and finally outputs the label of the prediction category. Among them, the most critical step is extracting the effective feature information. Common convolutional neural network models for phenotypic analysis of crops include VGGNet [
16], Inception V3 [
17], ResNet34 [
18], etc. ResNet [
19] effectively solves the phenomenon of gradient disappearance in neural network training by introducing a residual module. In this paper, we adopt a 34-layer ResNet network, which is composed of a series of residual modules, a full connection layer, and a lower sampling layer, as shown in
Figure 3.
The parameters in ResNet are gradually updated by comparing the error between the predicted value of the model and the ground truth, and the loss function is used to measure the degree of inconsistency between the predicted value and the true value. In this paper, in order to predict the growth stage of leafy greens, the final loss function value is calculated using the cross-entropy function, which is a weighted average of the cross-entropy of each sample in the input network, and the specific calculation formula is
where
represents the ground truth of sample
i in class
j,
represents the prediction probability of sample
i on class
j, and n is the number of samples.
In the classification task of leafy greens planted in the greenhouse, the prediction results are easily affected by soil and weeds, and our expected algorithm is to focus on the characterization of leafy greens. Therefore, the attention mechanism is introduced in this paper. By embedding the existing SE channel attention mechanism [
20] into the end of each residual structure of the original ResNet network, the network feature extraction ability is improved to capture significant feature information, such as shape, color, etc., on the leafy greens image. The basic module of SENet attention and residual network integration is shown in
Figure 4. Specifically, we adopt the network structure of Global Average pooling-FC-Relu-FC-Sigmoid. SENet has two fully connected (FC) layers, and the two functions are opposite. First, the FC layer of the first layer will reduce the number of channels in the network. On the contrary, the FC layer of the second layer will increase the number of channels. The weight will be generated in this process, and the weight is equal to the number of channels, which can still provide the weight value for the corresponding channel. The core of SENet is to use the loss of the network to understand the weight of the features and assign a larger weight to efficient feature graphs and a smaller weight to useless or less-efficient feature graphs so as to train the model and obtain better results. In general, the SE channel attention mechanism reduces the error rate of the previous model to a large extent, and the new parameters and calculation amount are small. Therefore, we add this attention mechanism to the ResNet network.
3.2. Causal Effect Estimation Model of Environmental Factors
Crop growth is closely related to its environment, so the role of environmental factors should be considered when predicting the maturity date of leafy greens. At present, most studies predict the maturity date by building crop growth models based on environmental factors such as temperature and light. These methods have some limitations; not only the selection of environmental variables is limited, but also the multicollinearity and causal relationship between variables are ignored. In this paper, the causal inference technology is used to find the causal relationship between environmental factors in the greenhouse and the growth rate of leafy greens. On the one hand, the accuracy of the model prediction is improved. On the other hand, the causal mechanism between environmental factors and the growth of leafy greens is analyzed to a certain extent, which is conducive to the accurate control of the greenhouse environment in the future. Causal inference has achieved fruitful results in econometrics, sociology, computer science, and other fields [
21,
22,
23]. Since this is a new research direction and is limited by the collection and analysis of data, causal inference technology has less application in the field of agriculture and still has great potential.
Causal inference usually refers to the search for causality, while causality refers to the relationship between one event (i.e., “cause”) and another event (i.e., “effect”), in which the latter event is considered the outcome of the previous event. Generally speaking, causality can also refer to the relationship between a series of factors (causes) and a phenomenon (effect). Any event that affects the outcome is a factor in the outcome [
24]. It is generally recognized that the most effective causal inference method is random controlled trials (RCT). However, it is often inoperable, and its scope of the experiment cannot cover all real scenes. Therefore, we consider the causal inference method based on observation data. At present, the mainstream causal analysis methods based on observational data are mainly divided into two categories: a potential consequence model and the structural causal model. Among them, the potential outcome model constructs a new analytical framework for causal inference by drawing on the concepts of randomized controlled trials and potential outcomes in statistics. Compared with the potential outcome model algorithm, the structural causal model is used to find the causal relationship between multiple variables. Based on Bayesian networks, Pearl [
25] proposed the concept of intervention and expressed causality in a formal way, which pioneered the method of extracting causality and the data generation mechanism from data.
According to the data characteristics of various environmental factors in a greenhouse planting environment, this paper adopts the methods of potential outcome model and structural causal model for different environmental factors and calculates the causal effect by observing the length of the growth cycle of leafy greens in different environments.
First, we mine and collect causal environment variables and then use xgboost [
26] to detect the correlation between input variables and outcome variables, including whether there is a correlation and the strength of the correlation. Seven variables are used to rank the importance of features, as shown in
Figure 5.
After confirmation with greenhouse planting professionals, these variables with obvious correlation are basically consistent with empirical judgment. Based on the data distribution characteristics and empirical judgment, we treat soil temperature as a continuous variable to calculate its continuous causal effect on the growth of leafy greens, and other environmental factors are regarded as binary variables to calculate relatively rough causal effects. We use graph theory as a mathematical tool to formalize the causal hypothesis behind the leafy green growth model, infer the causal relationship of environmental factors other than soil temperature through the structural causal model, and calculate the more accurate causal effect of soil temperature through the nonlinear model.
The structural causal model adopted in this paper is based on directed acyclic graphs (DAGs) [
25]. Therefore, the causal diagram is established based on the PC algorithm [
27] and prior knowledge of the relationship between environmental factors and crop growth [
28], as shown in
Figure 6.
The graph model provides a visual representation of causality, and joint distribution can be obtained by statistical analysis of observed data, but it is difficult to identify causality from these correlations. Therefore, we need to intervene, which means changing the existing data distribution to identify causality. Pearl [
25] proposed to express intervention with do-calculus, and the variables to be studied were
X and
Y, the vector composed of other covariates is
. For simplification, suppose
X is a binary variable; that is, the value can only be 0 or 1. A set of data
is observed. For example,
represents the probability distribution of
under the condition of
.
represents the probability distribution of
when
is achieved through intervention. Because of the existence of confounders, usually
, in order to analyze the causal effects of pairs in the causal model, we need to adjust for all confounders. For example, confounders can be removed through backdoor adjustment. Given a pair of ordered variables
in the model, if there is no descendant of
X in confounder set
Z and all backdoor paths between
X and
Y are blocked by confounder set
Z, then confounder set
Z meets the backdoor adjustment, and backdoor paths pointing to
X can be blocked to find the causal effect, as shown in the following formula
If all the elements of the blocking path are unobservable, the backdoor path cannot be calculated. However, if all the paths from
X and
Y have elements
z, and there is no open path from
z to
Y, we can use the set
Z of
z to measure
, i.e., the frontdoor adjustment. The following formula converts the expression containing do-calculus into the expression without do-calculus by taking the variable set
Z on the frontdoor path as a condition:
Using the backdoor adjustment and the frontdoor adjustment to intervene the causal diagram with the elimination of the do-calculus, the causal relationship between variables can be inferred only through known observation data and variable distribution. After identifying the causality between each node on the causal diagram of environment and leafy greens, we estimate the causal effect, that is, use the historical data recorded by greenhouse environmental sensors to calculate the specific value of the causal effect. First, the causal effect of environmental factors other than soil temperature is calculated by using propensity score matching (PSM) [
29]. Since we set these environmental factors as binary variables, the treatment for each environmental factor is binary, and the treatment
. Then our outcome is the growth cycle
y of leafy greens. In addition, feature
X is composed of each environmental factor, where
represents the feature vector of sample
i. Our goal is to estimate the causal effect of
t on
y, called the average treatment effect (
ATE)
where
represents the individual treatment effect (
ITE) and is defined as
.
For PSM, it must find an object for each instance. It assumes that treatment
t is a random variable extracted from a certain distribution, i.e.,
, and then a propensity score
will be calculated. This propensity score is to calculate the given
x. It tends to receive the probability of a certain treatment, that is, a binary score. Generally, this score is fitted with LR. After obtaining the propensity score, we will find an object different from its treatment for each sample. Therefore, for sample
i, we find sample
j s.t
. The obtained
ATE is
As the environmental factor that has the greatest impact on the maturity of leafy greens, we regard soil temperature as a continuous variable and take 2 °C as an interval to calculate the continuous treatment effect. The double machine learning (DML) [
30] method is used to realize the unbiased estimation of the influence of the target on the feature. The random forest algorithm is used to fit
Y and
T, respectively, and each fitting is combined to obtain the residual.
where
and
are the fitted residuals. After we get the residual model, treatment effect
can be obtained by fitting a simple regression model
. In fact, the residual is the change quantity, so
is the description of the relationship between the two residual quantities, that is, the causal effect
ATE.
is an equation related to
X. We can solve it by the least square method.