ROC Curves, Loss Functions, and Distorted Probabilities in Binary Classification

Abstract

The main purpose of this work is to study how loss functions in machine learning influence the “binary machines”, i.e., probabilistic AI models for predicting binary classification problems. In particular, we show the following results: (i) Different measures of accuracy such as area under the curve (AUC) of the ROC curve, the maximal balanced accuracy, and the maximally weighted accuracy are topologically equivalent, with natural inequalities relating them; (ii) the so-called real probability machines with respect to given information spaces are the optimal machines, i.e., they have the highest precision among all possible machines, and moreover, their ROC curves are automatically convex; (iii) the cross-entropy and the square loss are the most natural loss functions in the sense that the real probability machine is their minimizer; (iv) an arbitrary strictly convex loss function will also have as its minimizer an optimal machine, which is related to the real probability machine by just a reparametrization of sigmoid values; however, if the loss function is not convex, then its minimizer is not an optimal machine, and strange phenomena may happen.

1. Introduction

The aim of this paper is to study loss functions that are used in machine learning for training binary classification machines. We refer to [1,2,3,4,5] for an introduction to machine learning. Loss functions play an extremely important role in differential machine learning, and many different loss functions have been created for each problem, based on some general principles (see, e.g., [4,6,7,8,9]). However, most people use some available loss functions without paying much attention to the question of “how the choice of a loss function will affect the outcome of a machine learning problem”. Recently, researchers started paying more attention to the properties of the loss functions, which would help the stochastic gradient flows in differential machine learning converge to the desired values of the parameters; see, e.g., [10,11,12,13,14,15,16,17,18,19]. Our present work is also a contribution in this direction.

The question that we want to address in this paper is how to design the loss functions so that the machines that are loss minimizers are also optimal in terms of prediction accuracy. In order to study this question, we first need to study the ways to measure and compare the accuracy of different binary machines. This led us to the notions of information spaces, real probability machines, optimal machines, the convexity of the ROC curve, and natural inequalities relating different metrics of accuracy. We then discovered that the convexity condition is the main condition on the loss function in order to ensure that its minimizer will be an optimal machine in terms of precision. In general, those loss-minimizing machines will give distorted probabilities instead of real probabilities. (When people interpret sigmoid values as probabilities in the artificial intelligence literature, this is not accurate in general, as those numbers are not real probabilities.) However, this distortion may be viewed positively, as a feature instead of a bug, and one can go back from distorted probabilities to real probabilities by performing large-scale testing if one wishes.

The main results presented in this paper are the following:

• (Section 2, Proposition 1) Different measures of accuracy such as the area under the curve (AUC) of the ROC curve, the maximal balanced accuracy, and the maximally weighted accuracy are “topologically equivalent” in the sense that if one of them is high (i.e., close to 1), then the other ones are also automatically high, with natural inequalities relating them.
• (Section 3, Proposition 2) The so-called real probability machines with respect to given information spaces are the optimal machines, i.e., they have the highest precision among all possible machines, and moreover, their ROC curves are automatically convex.
• (Section 4, Proposition 3) The cross-entropy and the square loss are the most natural loss functions in the sense that the real probability machine is their minimizer.
• (Section 5 and Section 6, Proposition 4) An arbitrary smooth strictly convex loss function will also have as its minimizer an optimal machine, which is related to the real probability machine by just a reparametrization of the sigmoid values. However, if the loss function is not convex, then its minimizer is not an optimal machine, and strange phenomena may happen.

Propositions 1 and 2 have been announced by us in a recent talk at AICI 2022 [20].

2. Binary Machines, ROC Curves, and Accuracy Metrics

Let us fix some notations for this paper. Denote by $\Omega$ an input space together with some probability measure $P_{\Omega }$, and

$$Y:\Omega \to \{0,1\}$$

(1)

a binary classification problem on $\Omega$. For example, $\Omega$ is the population, and Y is COVID-positive (1) or COVID-negative (0). Y is often called the ground truth.

We want to build a binary machine:

$$M:\Omega \to [0,1]$$

(2)

(a test, whose values are in the interval [0,1]) that predicts the value of Y. Given a threshold $\sigma \in ]0,1[$, for each element $x\in \Omega$, we put

$$Y_{\sigma }\left(x\right)=1\mathrm{if}M\left(x\right)\ge \sigma \mathrm{and}{Y}_{\sigma }\left(x\right)=0\mathrm{if}M\left(x\right)<\sigma$$

(3)

The performance (i.e., precision) of the predictor $Y_{\sigma }$ with respect to the ground truth Y can be measured by two basic performance indicators, called the sensitivity (=true positive rate) $TP\left(\sigma \right)$ and specificity (=true negative rate) $TN\left(\sigma \right)$, defined by the following formulas:

$$TP\left(\sigma \right)=P(Y_{\sigma }=1|Y=1)=\frac{P_{\Omega }(M\left(x\right)\ge \sigma ,Y\left(x\right)=1)}{P_{\Omega }(Y\left(x\right)=1)},$$

(4)

$$TN\left(\sigma \right)=P(Y_{\sigma }=0|Y=0)=\frac{P_{\Omega }(M\left(x\right)<\sigma ,Y\left(x\right)=0)}{P_{\Omega }(Y\left(x\right)=0)}.$$

(5)

The curve $ROC:[0,1]\to [0,1]\times [0,1]$ given by the formula

$$ROC\left(\sigma \right)=(1-TN(\sigma ),TP(\sigma \left)\right)$$

(6)

is called the receiver operating characteristic (ROC) curve of the machine M in the literature and is very widely used in many fields; see, e.g., [21,22,23,24,25,26] and the references therein. The number $FP\left(\sigma \right)=1-TN\left(\sigma \right)$ is called the false positive rate at the threshold $\sigma$.

The ROC curves goes “backward” from the point $ROC\left(0\right)=(1,1)$ to the point $ROC\left(1\right)=(0,0)$ in the unit square, and the higher the curve, the more accurate the machine is. The so-called AUC is the area of the region under the ROC curve in the unit square and is a popular measure for the accuracy of the machine. See Figure 1 for an illustration.

Another popular measure of accuracy is the maximally weighted accuracy, denoted here by MWA (see, e.g., [23]): given a weight $w\in [0,1]$ (determined by the ratio between the cost of a false negative and the cost of a false positive), we put

$$\begin{array}{cc}\hfill WA\left(\sigma \right)& =w·TP\left(\sigma \right)+(1-w)·TN\left(\sigma \right)=w·TP\left(\sigma \right)-(1-w)·FP\left(\sigma \right)+(1-w),\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill MWA& =\underset{\sigma \in [0,1]}{max}WA\left(\sigma \right).\hfill \end{array}$$

(8)

In particular, when $w=0.5$, then one obtains the so-called maximal balanced accuracy:

$$MBA=\underset{\sigma \in [0,1]}{max}BA\left(\sigma \right),\mathrm{where}BA\left(\sigma \right)=\frac{TP\left(\sigma \right)+TN\left(\sigma \right)}{2}$$

(9)

We remark that $0\le AUC,MWA,MBA\le 1$ for any machine, and if any of these numbers is equal to 1, then it means that the machine is perfect, 100% accurate. To borrow a notion from topology and functional analysis, we can say that the AUC, MWA, and MBA are different metrics of accuracy, but they are topologically equivalent, in the sense that if one of these numbers is close to 1, then the other two numbers must also be automatically close to 1, i.e., if the machine is highly precise with respect to one of these metrics, then it is also highly precise with respect to the other metrics. More precisely, we have the following simple inequalities relating these metrics of accuracy:

Proposition 1.

With the above notations:

(i) For any binary machine M, we have

$$1-2{(1-MBA)}^2\ge AUC\ge 2MBA-1.$$

(10)

If, moreover, the ROC curve of the machine M is convex, then we have

$$AUC\ge MBA.$$

(11)

(ii) For any given weight $w\in ]0,1[$ and any given binary machine M, we have

$$1-{\displaystyle \frac{{(1-MWA)}^2}{2w(1-w)}}\ge AUC.$$

(12)

If, moreover, the ROC curve of the machine M is convex, then we have

$$AUC\ge 1-{\displaystyle \frac{(1-MWA)}{2min(w,1-w)}}.$$

(13)

Proof. 

(See Figure 1). W remark that a number $\sigma \in [0,1]$ is a threshold where the machine M attains the highest weighted accuracy if and only if the straight line through the point $ROC\left(\sigma \right)$, consisting of the points $\left(FP\right(\sigma )+wt,TP(\sigma )+(1-w\left)t\right),t\in \mathbb{R},$ lies above the ROC curve. Indeed, the lines $\left\{\right(FP\left(\sigma \right)+wt,TP\left(\sigma \right)+(1-w)t),t\in \mathbb{R}\}$ of slope $(w,1-w)$ are simply “lines of constant weighted accuracy”. If the point $B=\left(FP\right(\sigma ),TP(\sigma \left)\right)$ of the ROC curve gives the maximally weighted accuracy, then no point of the ROC curve can lie above its corresponding line of slope $(w,1-w)$, because lying above means higher weighted accuracy.

The line $\ell =\left\{\right(FP\left(\sigma \right)+wt,TP\left(\sigma \right)+(1-w)t),t\in \mathbb{R}\}$, where $\sigma$ gives the maximally weighted accuracy for the machine M, cuts the boundary of the unit square at two points $A=(0,1-{\displaystyle \frac{1-MWA}{w}})$ and $C=({\displaystyle \frac{1-MWA}{1-w}},1)$. The triangle $▵ACF$, where $F=(0,1)$, is disjoint from the region under the ROC curve, which implies that $AUC+\mathrm{area}(▵ACF)\le 1$. Since $\mathrm{area}(▵ACF)={\displaystyle \frac{FA·FC}{2}}={\displaystyle \frac{{(1-MWA)}^2}{2w(1-w)}},$ we obtain the inequality

$${\displaystyle AUC\le 1-{\displaystyle \frac{{(1-MWA)}^2}{2w(1-w)}}.}$$

On the other hand, the region under the ROC curve contains the rectangle whose vertices are $\left(FP\right(\sigma ),0),\left(FP\right(\sigma ),TP(\sigma ),(1,TP\left(\sigma \right)),(1,0)$. The surface area of this rectangle is $TP\left(\sigma \right)·(1-FP(\sigma \left)\right)=TP\left(\sigma \right)·TN\left(\sigma \right)=TP\left(\sigma \right)+TN\left(\sigma \right)-1+(1-TP(\sigma \left)\right)(1-TP(\sigma \left)\right)\ge TP\left(\sigma \right)+TN\left(\sigma \right)-1=2BA\left(\sigma \right)-1$ (for every $\sigma$). Hence, we obtain the inequality

$$AUC\ge 2MBA-1.$$

If the ROC curve is convex, then the region below it contains the quadrilateral $OBDE$, where $O=(0,0)$, $B=\left(FP\right(\sigma ),TP(\sigma \left)\right),D=(1,1),E=(1,0)$ (for any $\sigma )$. The surface area of this quadrilateral is exactly equal to $BA\left(\sigma \right)$, i.e., to MBA; hence, we obtain the inequality $AUC\ge BA\left(\sigma \right)$ for any $\sigma$, i.e., we have $AUC\ge MBA.$

Finally, the inequality $AUC\ge 1-{\displaystyle \frac{(1-MWA)}{2min(w,1-w)}}$ in the case when the ROC curve is convex and the weight w is arbitrary is a direct consequence of the inequalities

$$AUC\ge area\left(OBDE\right)\ge min\left(area\right(OADE),area(OCDE\left)\right)$$

and the equalities $area\left(OCDE\right)=1-{\displaystyle \frac{(1-MWA)}{2(1-w)}},$ $area\left(OADE\right)=1-{\displaystyle \frac{(1-MWA)}{2w}}$.   □

Remark 1.

For some inequalities in Proposition 1, we assume the ROC curve to be convex. The (near-)convexity of ROC curves has been observed empirically in many monographs and articles for long time; see e.g., [21,22,23,24,25,26] and the references therein; it has been shown in [22] (Theorem 3) that the convexity of the ROC curve is equivalent to some other natural reasonable “rationality” conditions on the machine (namely, the higher the “sigmoid value” is, the higher the probability of the event being true is).

In Section 3, we show that the so-called real probability machine, which is in a sense the most natural machine, is also the most precise machine, and its ROC is automatically convex. Our result helps explain why most ROC curves met in practice are nearly convex (because the machines must be nearly optimal in some sense if the machine learning processes for creating them are efficient).

Remark 2.

If we reparametrize a sigmoid function Σ by composing it with an arbitrary increasing bijection $f:[0,1]\to [0,1]$, then we obtain a new sigmoid function ${\Sigma }^{\prime }=f\circ \Sigma$ whose ROC curve is the same as the ROC curve of Σ, up to a reparametrization by f. Namely, $RO{C}_{\Sigma }\left(\sigma \right)=RO{C}_{{\Sigma }^{\prime }}\left(f\left(\sigma \right)\right)$ for all $\sigma \in [0,1]$. In particular, a reparametrization allows us to change the sigmoid values without changing the performance metrics AUC, MBA, and MWA of a system.

Remark 3.

Some authors (see., e.g., [27]) also use the geometric mean $GM=\sqrt{TP.TN}$ of sensitivity ($TP$) and specificity ($TN$) as a measure of accuracy for binary prediction problems. The obvious arithmetical inequalities ${(a+b)}^2/4\ge ab\ge {(a+b)}^2/2-1$ (for any positive numbers $a,b\le 1$) relate (in two ways) the geometric mean accuracy with the balanced accuracy $BA=(TP+TN)/2$. In particular, it means that the (maximal) geometric mean accuracy is also as good a measure of accuracy as MBA and the AUC, in the sense that they are topologically equivalent.

Remark 4.

Given an original probability distribution $P=P_{\Omega }$ on the data space Ω, which is imbalanced in the sense that $P(Y=0)\ne P(Y=1)$ (imbalance between negative and positive cases), we may change it to a new, balanced, probability distribution $\widehat{P}$, defined by the following formula:

$$\widehat{P}\left(A\right)={\displaystyle \frac{1}{2}}\left[{\displaystyle \frac{P(A\cap \{Y=0\left\}\right)}{P(Y=0)}}+{\displaystyle \frac{P(A\cap \{Y=1\left\}\right)}{P(Y=1)}}\right].$$

It is easy to verify that the parametrized ROC curve (for a given machine $M=\Sigma \circ \varphi :\Omega \to [0,1]$) with respect to $\widehat{P}$ coincides exactly with the ROC curve with respect to P. Indeed, for any $\sigma \in [0,1]$, the true positive level at σ is $TP\left(\sigma \right)={\displaystyle \frac{P(\Sigma >\sigma ,Y=1)}{P(Y=1)}}=2\widehat{P}(\Sigma >\sigma ,Y=1)={\displaystyle \frac{\widehat{P}(\Sigma >\sigma ,Y=1)}{\widehat{P}(Y=1)}}=\widehat{TP}\left(\sigma \right)$, and similarly for $TN\left(\sigma \right)$. Thus, in the study of accuracy, without loss of generality, one may suppose that the probability distribution is balanced with respect to Y in the sense that $P(Y=0)=P(Y=1)=0.5$.

3. Information Projection, Sigmoid Functions, and Optimal Machines

Conceptually, we can describe a binary machine M as a composition of two steps:

$$M=\Sigma \circ \varphi ,$$

(14)

where

$$\varphi :\Omega \to \Phi$$

(15)

may be called the information projection map from the original data space $\Omega$ to a certain “distilled features space” or information space $\Phi$, and

$$\Sigma :\Phi \to [0,1]$$

(16)

is a function from the information space $\Phi$ to the interval $[0,1]$, which we will call a (generalized) sigmoid function, in analogy with the classical sigmoid function $sigmoid\left(z\right)=exp\left(z\right)/(exp(z)+exp(-z\left)\right)$ often used in the last layer of a neural network in deep learning, even though our $\Sigma$ is a function of many variables in general.

The idea is that, in most cases, we cannot know everything about an element $x\in \Omega$; we can know only some information about x, and that information is given by the projection map $\varphi$. Based on the available information $\varphi \left(x\right)$ about x, we have to decide, via the value $M\left(x\right)=\Sigma \left(\varphi \right(x\left)\right)$, whether x is “negative” or “positive”. Even when we know everything about x (e.g., when x is a digital image), the information contained in x may be too big (millions of bits for an image), so we have first to “distill” that information into something smaller, which we call $\varphi \left(x\right)$ and which we can control more easily.

In this paper, we assume that the information space $\Phi$ and the projection map $\varphi :\Omega \to \Phi$ are fixed, and what we want to choose is just the sigmoid function $\Sigma :\Phi \to [0,1]$. The probability measure $P_{\Phi }$ on $\Phi$ is the push-forward of the probability measure $P_{\Omega }$ on $\Omega$ via the projection map $\varphi$.

In the artificial intelligence (AI) literature, the number $M\left(x\right)=\Sigma \left(\varphi \right(x\left)\right)$ is often called the probability (of x being positive given the information $\varphi \left(x\right)$), even though it is not true in general, because $\Sigma$ can be chosen rather arbitrarily, and as we will see in the following sections of this paper, even optimal machines obtained by machine learning methods usually give what we call distorted probabilities instead of real probabilities.

Nevertheless, among all the possible machines M (all the possible sigmoid functions $\Sigma :\Phi \to [0,1]$), there is one that is more natural than the others, which we call the real probability machine (the probability sigmoid function). The probability sigmoid function is simply the following conditional probability function:

$${\Sigma }_{proba}\left(\phi \right)=P(Y\left(x\right)=1|\varphi \left(x\right)=\phi )$$

(17)

for each $\phi \in \Phi$.

We remark that, if we change the sigmoid function $\Sigma$ by composing it with another function, ${\Sigma }^{\prime }=\theta \circ \Sigma ,$ where $\theta :[0,1]\to [0,1]$ is a strictly increasing bijective function, then $\Sigma$ and ${\Sigma }^{\prime }$ give the same ROC curve up to a reparametrization by $\theta$. In other words, we can change a sigmoid value to any other value by composing it with a function, without changing the accuracy of the system, and this is one more reason why the sigmoid values should not be called “probabilities” in general.

Proposition 2.

With the above notations:

(i) If $\Sigma :\Phi \to [0,1]$ is an arbitrary sigmoid function, then the real probability machine $M_{proba}={\Sigma }_{proba}\circ \varphi$ is more precise than (or at least as precise as) the machine $M=\Sigma \circ \varphi$, in the sense that the ROC curve of $M_{proba}$ lies above the ROC curve of M. In other words, for any false positive level $\alpha \in ]0,1[$, if σ and ${\sigma }_{proba}$ are the corresponding thresholds such that

$$F{P}_{M_{proba}}\left({\sigma }_{proba}\right)=F{P}_M\left(\sigma \right)=\alpha ,$$

(18)

then

$$T{P}_{M_{proba}}\left({\sigma }_{proba}\right)\ge T{P}_M\left(\sigma \right).$$

(19)

(ii) The ROC curve of the real probability machine $M_{proba}$ is convex.

Proof. 

Fix an arbitrary false positive level $\alpha \in ]0,1[$ and ${\sigma }_{proba},\sigma \in ]0,1[$ the two corresponding threshold values as in the statement of the proposition. Then, we have the following formula for $\alpha$:

$$\alpha =\frac{{\int }_{\{\phi \in \Phi |{\Sigma }_{proba}\left(\phi \right)\ge {\sigma }_{proba}\}}(1-{\Sigma }_{proba}\left(\phi \right))d\phi }{{\int }_{\Phi }(1-{\Sigma }_{proba}\left(\phi \right))d\phi }=\frac{{\int }_{\{\phi \in \Phi |\Sigma \left(\phi \right)\ge \sigma \}}(1-{\Sigma }_{proba}\left(\phi \right))d\phi }{{\int }_{\Phi }(1-{\Sigma }_{proba}\left(\phi \right))d\phi },$$

(20)

(where ${\int }_{\Phi }(1-{\Sigma }_{proba}\left(\phi \right))d\phi =P_{\Omega }(Y=0)$ is the probability measure of the negative set $\{x\in \Omega |Y\left(x\right)=0\}$), which implies that

$${\int }_{\{\phi \in \Phi |{\Sigma }_{proba}\left(\phi \right)\ge {\sigma }_{proba}\}}(1-{\Sigma }_{proba}\left(\phi \right))d\phi ={\int }_{\{\phi \in \Phi |\Sigma \left(\phi \right)\ge \sigma \}}(1-{\Sigma }_{proba}\left(\phi \right))d\phi .$$

(21)

To simplify the notations, put

$$A=\{\phi \in \Phi |{\Sigma }_{proba}\left(\phi \right)\ge {\sigma }_{proba}\}\mathrm{and}B=\{\phi \in \Phi |\Sigma \left(\phi \right)\ge \sigma \}.$$

(22)

Then, we have ${\displaystyle {\int }_A(1-{\Sigma }_{proba}\left(\phi \right))d\phi ={\int }_B(1-{\Sigma }_{proba}\left(\phi \right))d\phi ,}$ which implies that

$${\int }_{A\backslash B}(1-{\Sigma }_{proba}\left(\phi \right))d\phi ={\int }_{B\backslash A}(1-{\Sigma }_{proba}\left(\phi \right))d\phi .$$

(23)

Since $(1-{\Sigma }_{proba}\left(\phi \right))\le 1-{\sigma }_{proba}$ on $A\backslash B$ while $(1-{\Sigma }_{proba}\left(\phi \right))>1-{\sigma }_{proba}$ on $B\backslash A$, we must have that $P(A\backslash B)\ge P(B\backslash A)$, which implies that

$${\int }_{A\backslash B}{\Sigma }_{proba}\left(\phi \right)d\phi \ge {\sigma }_{proba}P(A\backslash B)\ge {\sigma }_{proba}P(B\backslash A)\ge {\int }_{B\backslash A}{\Sigma }_{proba}\left(\phi \right)d\phi ,$$

(24)

which implies that

$${\int }_A{\Sigma }_{proba}\left(\phi \right)d\phi \ge {\int }_B{\Sigma }_{proba}\left(\phi \right)d\phi .$$

(25)

This last inequality means exactly that the true positive level of ${\Sigma }_{proba}$ at the false positive level $\alpha$ is greater than or equal to the true positive level of $\Sigma$ at the same false positive level. In other words, the ROC curve of the probability sigmoid function ${\Sigma }_{proba}$ lies above the ROC curve of $\Sigma$ everywhere, i.e., ${\Sigma }_{proba}$ is the optimal sigmoid function.

The two ROC curves coincide if and only if, in the above formulas, B coincides with A (up to a set of measure zero) for every false positive level $\alpha$, and it basically means that $\Sigma$ is obtained from ${\Sigma }_{proba}$ by composing it with a monotonous function. In other words, up to a reparametrization of the sigmoid values, the probability sigmoid function is the only optimal sigmoid function.

The convexity of the ROC curve of the probability sigmoid function ${\Sigma }_{proba}$ follows directly from its construction, which ensures that the conditional event probability is nondecreasing (the higher the sigmoid value $\sigma$, the higher the conditional probability value is, which is obvious because this value is equal to $\sigma$ in our construction). See Theorem 3 of [22]. Indeed, denote by

$$\alpha \left(\sigma \right)=\frac{{\int }_{\{\phi \in \Phi |p\left(\phi \right)\ge \sigma \}}(1-{\Sigma }_{proba}\left(\phi \right))d\phi }{{\int }_{\Phi }(1-{\Sigma }_{proba}\left(\phi \right))d\phi }\mathrm{and}\beta \left(\sigma \right)=\frac{{\int }_{\{\phi \in \Phi |{\Sigma }_{proba}\left(\phi \right)\ge \sigma \}}p\left(\phi \right)d\phi }{{\int }_{\Phi }{\Sigma }_{proba}\left(\phi \right)d\phi }$$

(26)

the false negative and false positive levels at threshold $\sigma$ for the probability sigmoid function ${\Sigma }_{proba}$, then we have

$$\frac{d\beta }{d\alpha }=\frac{{\int }_{\Phi }(1-{\Sigma }_{proba}\left(\phi \right))d\phi }{{\int }_{\Phi }{\Sigma }_{proba}\left(\phi \right)d\phi }·\frac{\sigma }{1-\sigma },$$

(27)

which is an increasing function in $\sigma$, but a decreasing function in $\alpha$, because $\alpha$ itself is a decreasing function in $\sigma$. Hence, $\beta$ is a concave function in $\alpha$, which means that the ROC curve is convex.   □

4. Differential Machine Learning and Loss Functions

The main idea of machine learning is that we have not just one, but a large family of machines $M_{\theta }:\Omega \to [0,1]$ that depend on some vector parameter $\theta \in \Theta$, where $\Theta$ is a multi-dimensional space, and the learning process consists of changing $\theta$ step by step, e.g.,

$$\theta ={\theta }_0\mapsto {\theta }_1\mapsto {\theta }_2\mapsto \cdots \mapsto {\theta }_n\mapsto \cdots$$

(28)

in order to improve the performance or the precision of $M_{\theta }$. In differential learning, one constructs a loss function:

$$L:\Theta \to \mathbb{R}$$

(29)

which acts as a proxy for the precision of the machines (the lower the loss $L\left(\theta \right)$, the higher the precision of the machine $M_{\theta }$ in some sense) and uses the stochastic gradient descent method to find a minimal point $\theta$ (or a near-minimal point) for the loss function L. That (near-)minimal point $\theta$ would logically correspond to a (near-)optimal machine $M_{\theta }$. Theoretically, L is equal to the integral over the whole data space $\Omega$ of a pointwise loss function ℓ:

$$L\left(\theta \right)={\int }_{x\in \Omega }\ell (M_{\theta }\left(x\right),Y\left(x\right))d{P}_{\Omega }$$

(30)

A priori, one can choose the loss function ℓ as one pleases, the only natural restriction being that the further $m=M_{\theta }\left(x\right)$ is away from the ground truth $y=Y\left(x\right)$, the higher the loss $\ell (m,y)$ should be, and if $m=y$, then there is no loss, i.e., $\ell (y,y)=0$. The two most popular loss functions are the quadratic loss:

$${\ell }_{quadratic}(m,y)={(m-y)}^2$$

(31)

and the so-called binary cross-entropy, which corresponds to the function

$${\ell }_{crossentropy}(m,y)=-ln(1-|m-y\left|\right)$$

(32)

However, one can choose many other loss functions. For example, the following quartic loss function will work very well in many problems:

$${\ell }_{quartic}(m,y)={(m-y)}^2+{(m-y)}^4$$

(33)

One can even try to use non-smooth, non-convex loss functions, for example

$${\ell }_{broken}(m,y)=min\left(\right|m-{y|,0.5)}^2+max\left(\right|m-y|-0.25,0)$$

(34)

Below, we give a theoretical explanation of the following facts:

(i)

The quadratic loss and the cross-entropy are the two most natural loss functions;

(ii)

Convex loss functions such as ${\ell }_{quartic}$ are good loss functions in the sense that their minimizers are optimal machines in terms of accuracy);

(iii)

Nonconvex loss functions such as ${\ell }_{broken}$ may lead to very erratic results (stochastic traps) in machine learning.

One can rewrite the loss $L\left(M\right)={\int }_{x\in \Omega }\ell (\Sigma \left(\varphi \left(x\right)\right),Y\left(x\right))d{P}_{\Omega }$ of a binary machine $M=\Sigma \circ \varphi$ as an integral on the information space $\Phi$ and then call it the loss of the sigmoid function $\Sigma$, as follows:

$${\displaystyle L(\Sigma )={\int }_{\Phi }^{}\left[(1-{\Sigma }_{proba}\left(\phi \right))·\ell (\Sigma \left(\phi \right),0)+{\Sigma }_{proba}\left(\phi \right)·\ell (\Sigma \left(\phi \right),1)\right]d\phi .}$$

(35)

(For each given $\phi$, the value of $\ell (\Sigma (\varphi \left(x\right)),Y(x\left)\right)$ under the condition $\varphi \left(x\right)=\phi$ will be equal to $\ell (\Sigma (\phi ),0)$ with probability $(1-{\Sigma }_{proba}\left(\phi \right))$ and equal to $\ell (\Sigma (\phi ),1)$ with probability ${\Sigma }_{proba}\left(\phi \right)$. The integrand $(1-{\Sigma }_{proba}\left(\phi \right))·\ell (\Sigma \left(\phi \right),0)+{\Sigma }_{proba}\left(\phi \right)·\ell (\Sigma \left(\phi \right),1)$ in the above formula is nothing but the integral of $\ell (\Sigma (\varphi \left(x\right)),Y(x\left)\right)$ over the space $\{x\in \Omega ,\varphi (x)=\phi \})$ with respect to the conditional probability measure on that space; that is why we have the above formula).

For example, in the case of the cross-entropy loss, we have the integral formula, whose integrand is really a cross-entropy:

$$L_{crossentropy}(\Sigma )={\int }_{\Phi }^{}-\left[(1-{\Sigma }_{proba}\left(\phi \right))·ln(1-\Sigma \left(\phi \right))+{\Sigma }_{proba}\left(\phi \right)·ln(\Sigma \left(\phi \right))\right]d\phi .$$

(36)

The above examples of loss functions are symmetric, in the sense that they treat the losses in negative cases ($Y=0$) and the losses in positive cases ($Y=1$) on an equal footing. However, due to huge data imbalance in some problems (for example, when the number of positive cases is just 1/1000 the number of negative cases), in practice, it is sometimes better to use asymmetric loss functions instead of symmetric loss functions. Given a function:

$$f:[0,1]\to \mathbb{R}$$

(37)

which is increasing and such that $f\left(0\right)=0$, we can create a family of asymmetric loss functions ${\ell }_c$ depending on an asymmetry coefficient$c>0$ by the following formula:

$${\ell }_c(m,y)=(1-y)f\left(m\right)+cyf(1-m).$$

(38)

Since our ground truth admits only two values $y=0$ and $y=1$, the above formula simply means that the loss is equal to $f\left(m\right)$ if $y=0$ and is equal to $cf(1-m)$ if $y=1$, so the negative cases and the positive cases are treated differently in the total loss. For example, when $c=100$, then it is like every positive case is counted one hundred times while every negative case is counted only once. As such, the asymmetry coefficient can be used to offset data imbalances.

The two Formulas (35) and (38) give us the following formula for the loss of a machine $M=\Sigma \circ \varphi$ with respect to a given generating function f and asymmetry coefficient c:

$$L(\Sigma )={\int }_{\Phi }^{}\left[(1-{\Sigma }_{proba}\left(\phi \right))·f(\Sigma \left(\phi \right))+c{\Sigma }_{proba}\left(\phi \right)·f(1-\Sigma \left(\phi \right))\right]d\phi .$$

(39)

For each given $\phi \in \Phi$, the integrand $(1-{\Sigma }_{proba}\left(\phi \right))·f(\Sigma \left(\phi \right))+c{\Sigma }_{proba}\left(\phi \right)·f(1-\Sigma \left(\phi \right))$ in the above integral formula can be written as a function of one variable $\sigma =\Sigma \left(\phi \right)$ and one parameter $p={\Sigma }_{proba}\left(\phi \right)$ (we cannot change p, but can choose our sigmoid function $\Sigma$, i.e., choose $\sigma$, in order to minimize the loss):

$$g\left(\sigma \right):=(1-p)f\left(\sigma \right)+cpf(1-\sigma ).$$

(40)

Minimizing the loss $L(\Sigma )$ means minimizing $g\left(\sigma \right)$ for each $\phi$. In other words, a sigmoid function $\Sigma$ is a minimizer of the loss function $L(\Sigma )$ given by Formula (39) if and only if (up to a set of measure zero) for each $p\in [0,1]$ and each $\phi \in \Omega$ such that ${\Sigma }_{proba}\left(\phi \right)=p$, we have

$$\Sigma \left(\phi \right)=\underset{\sigma }{\mathrm{argmin}}\left[(1-p)f\left(\sigma \right)+cpf(1-\sigma )\right]$$

(41)

This last equation leads us to the following very interesting result about the naturality of the classical quadratic loss function (the case with $f\left(\sigma \right)={\sigma }^2$ and $c=1$) and the binary cross-entropy (the case with $f\left(\sigma \right)=-ln(1-\sigma )$ and $c=1$):

Proposition 3.

With the above notations, we have:

(i) The real probability machine is the only loss minimizer for the quadratic loss function.

(ii) The real probability machine is also the only loss minimizer for the binary cross entropy function.

Proof. 

(i) The quadratic loss case. As discussed above, a sigmoid function $\Sigma :\Phi \to [0,1]$ is a minimizer of the quadratic loss function if and only if

$$\Sigma \left(\phi \right)=\underset{\sigma }{\mathrm{argmin}}\left((1-p){\sigma }^2+p{(1-\sigma )}^2\right).$$

(42)

for each $p\in$ [0,1] and each $\phi \in \Omega$ such that ${\Sigma }_{proba}\left(\phi \right)=p$.

The quadratic function $g\left(\sigma \right):=(1-p){\sigma }^2+p{(1-\sigma )}^2$ has its derivative equal to $g^{\prime }\left(\sigma \right)=2(1-p)\sigma +2p(\sigma -1)=2(\sigma -p)$, and the equation $g^{\prime }\left(\sigma \right)=0$ has a unique solution $\sigma =p$. This point $\sigma =p={\Sigma }_{proba}\left(\phi \right)$ is the unique minimal point for the function $g\left(\sigma \right)$. It follows that the loss $L(\Sigma )$ achieves its minimal at (and only at) the function $\Sigma \left(\phi \right)={\Sigma }_{proba}\left(\phi \right),$ i.e., when the machine is the real probability machine.

(ii) The cross-entropy case. In this case, the minimizer of the loss function satisfies the equation

$$\Sigma \left(\phi \right)=\underset{\sigma }{\mathrm{argmin}}\left(-(1-p)ln(1-\sigma )-pln\left(\sigma \right)\right).$$

(43)

for each $p\in [0,1]$ and each $\phi \in \Omega$ such that ${\Sigma }_{proba}\left(\phi \right)=p$.

The logarithmic function $h\left(\sigma \right)=-(1-p)ln(1-\sigma )-pln\left(\sigma \right)$ tends to infinity at both ends (when $\sigma$ tends to 0 and when $\sigma$ tends to 1) and has its derivative equal to $h^{\prime }\left(\sigma \right)={\displaystyle \frac{1-p}{1-\sigma }}-{\displaystyle \frac{p}{\sigma }}={\displaystyle \frac{\sigma -p}{\sigma (1-\sigma )}}$, and the equation $h^{\prime }\left(\sigma \right)=0$ has a unique solution $\sigma =p={\Sigma }_{proba}\left(\phi \right)$. This value of $\sigma$ is the unique minimal point of $h\left(\sigma \right)$, so similar to the previous case, the probability function $\Sigma \left(\phi \right)={\Sigma }_{proba}\left(\phi \right)$ is also the unique minimizer of the cross-entropy loss function.   □

5. Convex Loss Functions and Distorted Probabilities

The following proposition shows that, not only the cross-entropy and the quadratic loss function can lead to the real probability machine (which is the most natural and most precise machine according to Proposition 2), but all the other convex loss functions can also lead to this optimal machine, up to a reparametrization (distortion of the probabilities).

Proposition 4.

Let $f:[0,1[\to {\mathbb{R}}_{+}$ be an arbitrary strictly convex increasing continuously differentiable function such that either $f^{\prime }\left(0\right)=0$ or ${lim}_{\sigma \to 1}{f}^{\prime }\left(\sigma \right)=+\infty$, $c>0$ be an arbitrary positive number (the asymmetry efficient), and

$$L\left(M\right)={\int }_{x\in \Omega }\left[(1-Y(x\left)\right)·f\left(M\right(x\left)\right)+cY\left(x\right)·f(1-M(x\left)\right)\right]dx$$

(44)

be the loss of a machine M for a given binary classification problem Y, measured by f and c.

Then, the minimizer $M=\Sigma \circ \varphi$ for the loss function $L\left(M\right)$ is just a reparametrization of the real probability machine. In other words, there is an increasing bijection $g:[0,1]\to [0,1]$ such that the machine with the sigmoid function:

$$\Sigma \left(\phi \right):=g\left({\Sigma }_{proba}\left(\phi \right)\right)$$

(45)

has the minimal loss with respect to L.

Proof. 

Recall from (35) that the loss $L\left(M\right)$ for a machine M can be written as a loss for its sigmoid function $\Sigma$ as follows:

$${\displaystyle L\left(M\right)=L(\Sigma )={\int }_{\Phi }^{}\left[\left(1-{\Sigma }_{proba}\left(\phi \right)\right)f(\Sigma \left(\phi \right))+c{\Sigma }_{proba}\left(\phi \right)f\left(1-\Sigma \left(\phi \right)\right)\right]d{\mu }_{\varphi }}$$

(46)

In order to minimize the loss $L(\Sigma )$ over all functions $\Sigma :\Phi \to [0,1]$, we have to minimize $(1-p)f\left(\sigma \right)+cpf(1-\sigma )$ for each given $p={\Sigma }_{proba}\left(\phi \right)$ over all $\sigma$ and put

$$\Sigma \left(\phi \right)=\underset{\sigma }{\mathrm{argmin}}\left[(1-p)f\left(\sigma \right)+cpf(1-\sigma )\right]$$

(47)

to obtain the optimal sigmoid function with respect to the loss function L.

Under our assumptions about the function f, the minimal value of the function $(1-p)f\left(\sigma \right)+cpf(1-\sigma )$ (for a given p) is attained at the point $\sigma$ where its derivative vanishes, i.e., $(1-p)f^{\prime }\left(\sigma \right)-cp{f}^{\prime }(1-\sigma )=0$, or

$${\displaystyle \frac{f^{\prime }\left(\sigma \right)}{f^{\prime }(1-\sigma )}}={\displaystyle \frac{cp}{1-p}}.$$

(48)

Notice that $h\left(\sigma \right):={\displaystyle \frac{f^{\prime }\left(\sigma \right)}{f^{\prime }(1-\sigma )}}$ is a strictly increasing continuous function, with $h\left(0\right)=0$ and $\underset{\sigma \to 1}{lim}h\left(\sigma \right)=+\infty$; hence, for each p ($p\in [0,1]$), there is a unique $\sigma$ such that $h\left(\sigma \right)={\displaystyle \frac{cp}{1-p}}$, and moreover, this value of $\sigma$ increases when p increases. In other words, there is an increasing bijection

$$g:[0,1]\to [0,1]$$

(49)

such that $\sigma =g\left(p\right)$ will satisfy the above equation, i.e., $h\left(g\left(p\right)\right)={\displaystyle \frac{cp}{1-p}}.$ This implies that $\Sigma \left(\phi \right):=g\left(p\right)=g\left({\Sigma }_{proba}\left(\phi \right)\right)$ is the sigmoid function whose corresponding machine $M=\Sigma \circ \varphi$ has the minimal loss with respect to the loss function L.

Of course, this machine is just a reparametrization of the real probability machine by the reparametrization function g.   □

Remark 5.

If the loss function is not convex (i.e., the function f in the formula of the loss function is not convex), then Equation (48) may have zero solution or many solutions instead of a unique solution in the interval $[0,1]$. If there is no solution, then it means that the minimal value of $(1-p)f\left(\sigma \right)+cpf(1-\sigma )$ falls at $\sigma =0$ or $\sigma =1$, and the optimizer for L will be an kind of extreme machine whose values will be mostly just 0 or 1 instead of some kind of probability numbers; such a machine will not be very useful. When there are many solutions, then a minimizer Σ for the loss function will not be a reparametrization of the probability sigmoid function either, i.e., the machine that minimizes the loss function will not be an optimal machine.

Remark 6.

In practice, hyper-convex loss functions such as the quartic loss are often preferable to the square loss and the cross-entropy, even though they manifestly lead to machines that give distorted probabilities. One of the reasons is their focality (see, e.g., [10,16]), which allows the machine learning process to concentrate more of its learning on the difficult cases instead of “revising” too much the easy cases.

Remark 7.

Every probability is in fact a conditional probability and can be distorted one way or another. For example, we do not know what the real probability distribution on the dataset Ω is, especially when Ω is very large, and the samples that we have just a small subset of Ω. Therefore, even if we use the cross-entropy or the square loss and a very good machine learning method, there is no guarantee that the obtained machine will give real probabilities, and it is better to assume that all the obtained sigmoid values will be just distorted probabilities. To go back from distorted probabilities to real probabilities, i.e., to find the reparametrization function g such that $\Sigma \left(\phi \right)=g\left({\Sigma }_{proba}\left(\phi \right)\right)$, one may make large-scale post-training tests of the machine (similar to clinical studies for medical products). This problem of going from distorted probabilities to true probabilities is a well-known problem of the calibration of probabilities in machine learning; see, e.g., [28,29].

Remark 8.

Distorted probabilities may actually be a good thing. For example, if we have a cancer detection machine that gives the sigmoid value $0.1$ for a patient, how should this number be interpreted? Since this number is very small, one may want to dismiss it as “very low cancer risk”. However, if it is a real probability number, i.e., the chance of having cancer is 10%, then that number is already high enough to be taken very seriously. It would be better if the sigmoid value 0.1 would correspond to the cancer probability of just 0.001 for example (and to the probability of having a late-stage cancer much smaller than that).

6. Non-Convex Loss Functions and Stochastic Traps

We performed many experiments of deep learning with both “good” (hyper-convex loss functions, such as the quartic loss function in Formula (33)) and “bad” loss functions (non-convex functions, such as the broken loss function in Formula (34) and other functions, which the reader can probably invent easily by herself/himself). We used only well-known neural networks such as VGG16 [30] and standard data augmentation methods, absolutely nothing fancy. The purpose was not to achieve the best-performing AI models, but to study phenomena created by different loss functions.

For example, we performed hundreds of experiments with the binary classification problems such as “cat versus not-cat”, “dog versus not-dog” on the well-known public dataset called CIFAR-10 collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton (see [31,32]). This dataset consists of 60 thousand $32\times 32$ color images divided into 10 equal classes: cat, dog, ship, plane, etc. The experiments confirmed our theoretical reasoning that hyper-convex loss functions lead to good results in general, while non-convex loss functions very often (more than half of the time in our experiments) lead to stochastic traps in the parameter space: the stochastic descent of the machine learning process falls into those places of low accuracy, gets trapped there, and cannot get out.

Due to the stochastic nature of machine learning, sometimes, the machine does break out of the stochastic trap after being stuck there for many epochs (learning steps). Sometimes, the trap is so big or so strong that the machine breaks out of it only to fall back into it again after some machine learning epochs. An illustration of a stochastic trap that we observed is shown in Figure 2.

The problem of describing precisely the mechanisms for stochastic traps in machine learning is a very large and interesting problem, but it is outside of the scope of this paper. Here, we just wanted to show our observation that ill-designed non-convex loss functions may be responsible for such traps.