2.4.1. Subset Analysis

To detect the signal for the interaction between *drug D*1 and *drug D*2, we created subsets of the patient group using *drug D*1 (or the patient group using *drug D*2) (Table 1).

**Table 1.** The 4 × 2 contingency table for signal detection: AE: adverse event; n: the number of reports (e.g., *n*+++: the number of all reports, *n*111: the number of *drug D*1- and *drug D*2-induced target AE reports).


The following equations (Equations (1) and (2)) were used to calculate the ROR and 95% confidence interval (95% CI) of the target AE caused by *drug D*1 (or *drug D*2) from the generated subset, respectively. For the signal of a patient group on *drug D*1 that takes *drug D*2, the number of each report can be expressed as follows: *N*11 = *n*111, *N*10 = *n*110, *N*01 = *n*101, *N*00 = *n*100. On the other hand, for the signal of a patient group on *drug D*2 that takes *drug D*1, the number of each report can be expressed as follows: *N*11 = *n*111, *N*10 = *n*110, *N*01 = *n*011, *N*00 = *n*010.

$$\text{ROR} = \frac{\text{N}\_{11}/\text{N}\_{00}}{\text{N}\_{01}/\text{N}\_{10}} \tag{1}$$

$$\text{ROR } (\text{95\% CI}) = \mathcal{e}^{\ln\left(\text{ROR}\right) \pm 1.96} \sqrt{\frac{1}{N\_{11}} + \frac{1}{N\_{10}} + \frac{1}{N\_{01}} + \frac{1}{N\_{00}}} \tag{2}$$

In previous studies [11–13], if the signal for *drug D*2 was detected in the subset of a patient group using *drug D*1 or if the signal for *drug D*1 was detected in the subset of a patient group using *drug D*2, this signal was considered the drug–drug interaction signal. The criterion that a signal only needs to be detected from a subset of either patient group is ambiguous, highlighting the two shortcomings mentioned earlier. Therefore, for the newly proposed subset analysis, a case was redefined as the drug–drug interaction signal if a signal was detected in both subsets of a patient group using *drug D*1 and a patient group using *drug D*2.

#### 2.4.2. Ω Shrinkage Measure Model

The Ω shrinkage measure model [16] is based on a measure calculated as the ratio of the observed reporting ratio of the AE associated with the combination of two drugs and its expected value; this model is used by the Uppsala Monitoring Center (UMC) and the World Health Organization (WHO) Collaborating Centre for International Drug Monitoring for signal analysis of drug–drug interactions (Table 1, Equations (3)–(7)).

$$
\Omega = \log\_2 \frac{n\_{111} + 0.5}{E\_{111} + 0.5} \tag{3}
$$

where *n*111 is the reported number of AEs caused by the combination of two drugs, and *E*111 is the expected value of AEs caused by the combination of two drugs.

φ(0.975) is 97.5% of the standard normal distribution and Ω025 > 0 is used as a threshold to screen for signals under the combination of two drugs (Equation (4)).

$$
\Omega\_{025} = \,\,\Omega - \frac{\phi(0.975)}{\log(2)\,\sqrt{n\_{111}}}\tag{4}
$$

To calculate *E*111, we used the following Equations (5)–(7).

$$f\_{00} = \frac{n\_{001}}{n\_{00+}}, f\_{10} = \frac{n\_{101}}{n\_{10+}}, f\_{01} = \frac{n\_{011}}{n\_{01+}}, f\_{11} = \frac{n\_{111}}{n\_{11+}} \tag{5}$$

$$g\_{11} = 1 - \frac{1}{\max\limits{(\frac{f\_{00}}{1 - f\_{00}}, \frac{f\_{10}}{1 - f\_{10}})} + \max\limits{(\frac{f\_{00}}{1 - f\_{00}}, \frac{f\_{01}}{1 - f\_{01}}) - \frac{f\_{00}}{1 - f\_{00}} + 1}}\tag{6}$$

When *f* 10 < *f* 00 (which denotes no risk of AE caused by drug *D*1), the most sensible estimator *g*11 = max (*f* 00, *f* 01) is yielded and vice versa when *f* 01 < *f* 00.

Norén et al. re-expressed the observed and expected RRR *f* 11 and g11 in terms of the observed number of reports n111 and expected numbers of reports *E*111 = *g*11 × *n*11+, respectively:

$$\frac{f\_{11}}{g\_{11}} = \frac{n\_{111}/n\_{11+}}{E\_{111}/n\_{11+}} = \frac{n\_{111}}{E\_{111}}\tag{7}$$

#### *2.5. Evaluation of Detection Models*

#### 2.5.1. Using Evaluations of Classification in Machine Learning

The evaluation indicators that we set were *Accuracy* (Table 2, Equation (8)), *Precision* (*Positive predictive value*; *PPV*) (Table 2, Equation (9)), *Recall* (*Sensitivity*) (Table 2, Equation (10)), *Specificity* (Table 2, Equation (11)), *Youden's index* (Table 2, Equation (12)), *F*-measure (Table 2, Equation (13)), and *Negative predictive value* (*NPV*) (Table 2, Equation (14)).

$$Accuracy = \frac{TP + TN}{TP + FP + TN + FN} \tag{8}$$

$$\text{Precision} \left( \text{Positive predictive value}; \text{PPV} \right) = \frac{\text{TP}}{\text{TP} + \text{FP}} \tag{9}$$

$$\text{Recall (Sensitivity)} = \frac{TP}{TP + FN} \tag{10}$$

$$Specificity = \frac{TN}{FP + TN} \tag{11}$$

$$\text{Youden's index} = \text{Sensitivity} + \text{Specificity} - 1\tag{12}$$

$$F-measure = \frac{2 \times Recall \times Precision}{Recall + Precision} \tag{13}$$

$$\text{Negative predictive value (NPV)} = \frac{TN}{TN + FN} \tag{14}$$



#### 2.5.2. Cohen's Kappa Coefficient

The commonality of the signals detected by each statistical model was evaluated using *Cohen's kappa coe*ffi*cient* (κ), proportionate agreemen<sup>t</sup> for positive rating (*P*positive), and proportionate agreemen<sup>t</sup> for negative rating (*P*negative), as reported in a previous study [14,15]. In this study, we investigated the similarities with Ω shrinkage measure model for the previous/newly proposed subset analysis.
