*4.3. Evaluation of the Proposed Scheme*

As we have proven in former works the significant difference in verification performance between PE categories [28,40–43], we naturally adopted a methodology assessing the impact of our strategy on each PE category separately. This methodology consists in the following steps: first we computed PE values of the 74 persons considering only their usual signatures. Then, we generated three user categories based on the obtained PE values, by a Hierarchical Clustering (as explained in Section 4.1). Finally, we assessed, per user category, verification performance on usual signatures, and compared it to performance when considering the other signature types: initials, name-surname, SI, SN, NDP, SDP, and SIDP.

For performance assessment, we considered, for each person and each signature type, the remaining 15 genuine signatures (the other 10 genuine signatures were used for PE computation) and the 10 available skilled forgeries. For each person, the HMM classifier was trained on five genuine signatures among the 15, and tested on the remaining 10 genuine instances and the 10 skilled forgeries. The same signature type is considered in the training and testing phases.

Five random samplings were carried out on the training signatures. The false acceptance rate (FAR) and false rejection rate (FRR) are computed relying on the total number of false rejections and false acceptances obtained on all the five random samplings.

#### 4.3.1. Results on High PE Category

We analyze in this section the results obtained on the so-called "problematic" users in the literature [28,43,45], which are the main target of our strategy for enhancing signature security in uncontrolled mobile conditions.

Figure 11 and Table 3 display the system performance on problematic users, those with the highest PE. The EER reaches 7.17% when considering their usual signature (see Table 3 and the blue curve in Figure 11). We first notice a significant degradation of performance when persons sign with their initials (green curve in Figure 11). A relative degradation of 93% at the EER is observed even if the usual signature is already simple and variable. This highlights the importance of the ballistic aspect of the signing process in terms of resistance to attacks. Note that the vulnerability of initials is also predictable by their very high PE values observed in Figure 12.

**Figure 11.** System performance on users of the highest PE category considering the 8 signature types.


**Table 3.** System performance on users of the highest PE category in terms of EER.

**Figure 12.** Boxplots of PE values per signature type for users of the highest PE category.

Moreover, we notice a significant improvement in performance when persons sign with their name-surname (red curve in Figure 11). The FAR is in this case bounded around 10%. Also, the hybrid type SN, which combines the usual signature and name-surname, improves significantly performance (black curve in Figure 11): at the EER, the relative improvement is of 63% compared to the usual signature. This result confirms the robustness of this hybrid type to attacks, predictable by its low PE values displayed in its corresponding boxplot in Figure 12.

Besides, adding the date and place information clearly enhances performance. Indeed, the NDP type (magenta curve in Figure 11) improves performance of 83.68% at the EER when compared to the usual signature. But the SDP type outperforms the NDP: the relative improvement is of 98% at the EER, when compared to the usual signature (see Table 3 and black dotted curve in Figure 11). Moreover, it leads to a bounded FAR at 0.2% and a bounded FRR at 0.5%. Interestingly, we notice that this could not be predicted by PE since SDP type has higher PE values than NDP (see Figure 12). This result shows that the ballistic gesture inherent to the usual signature remains more discriminant than the name-surname, when being combined to an alphanumeric information (the date) and handwriting (the place), even in the case of a very simple problematic signature.

Finally, the SIDP type does not perform significantly better than the SDP type. This may be explained by the fact that in this particular category of users, the usual signature is simple and variable, and thus close to initials in terms of information content.

#### 4.3.2. Results on Low PE Category

Figure 13 and Table 4 show system performance on persons with low PE, whose signatures are the most complex and stable, and the most robust to attacks. The EER reaches 6.93% (see Table 4) when considering their usual signature (blue curve in Figure 13).

**Figure 13.** System performance on users of the lowest PE category considering the 8 signature types.


**Table 4.** System performance on users of the lowest PE category in terms of EER.

Some trends observed on problematic users in the previous section are here confirmed. First, a significant degradation of 116% is obtained at the EER with initials relatively to the usual signatures. PE predicts this trend in Figure 14 (higher PE values for initials). Besides, as expected, this relative degradation of 116% is higher in the case of complex signatures of this category, compared to problematic users (relative degradation of 98% as reported in Section 4.3.1). Figure 14 shows the significant gap between initials and usual signatures for the low PE category, compared to that obtained on problematic users (high PE category).

**Figure 14.** Boxplots of PE for users with highest (left) and lowest (right) PE values, considering their usual signature and initials.

Moreover, the hybrid types SI, SN, NDP, SDP and SIDP outperform significantly the usual signature; we note a relative improvement of 100% at the EER for NDP, SDP and SIDP types. For this reason, the three associated DET-curves are not visible in Figure 13. This confirms again their resistance to attacks, predictable by their low PE values, as shown in Figure 14.

On the other hand, some trends differ from those observed on problematic users. We notice that the name-surname type (red curve in Figure 13) gives similar performance to that of the usual signature (blue curve in Figure 13); while for problematic users, the name-surname outperforms by 40% the usual signature (see Figure 11 and Table 3). This means that in this category of persons, if we consider the name-surname as a possible signature for identity verification, although it has higher complexity (low PE in Figure 15), performance would not be improved relatively to the usual signature.

**Figure 15.** Boxplots of PE values per signature type for users of the lowest PE category.

In conclusion, this result shows on one hand that the complexity criterion is not sufficient to enhance the security of a signature. On the other hand, it highlights the importance of the ballistic process for identity verification.
