Retention Time Prediction with Message-Passing Neural Networks

Abstract

Retention time prediction, facilitated by advances in machine learning, has become a useful tool in untargeted LC-MS applications. State-of-the-art approaches include graph neural networks and 1D-convolutional neural networks that are trained on the METLIN small molecule retention time dataset (SMRT). These approaches demonstrate accurate predictions comparable with the experimental error for the training set. The weak point of retention time prediction approaches is the transfer of predictions to various systems. The accuracy of this step depends both on the method of mapping and on the accuracy of the general model trained on SMRT. Therefore, improvements to both parts of prediction workflows may lead to improved compound annotations. Here, we evaluate capabilities of message-passing neural networks (MPNN) that have demonstrated outstanding performance on many chemical tasks to accurately predict retention times. The model was initially trained on SMRT, providing mean and median absolute cross-validation errors of 32 and 16 s, respectively. The pretrained MPNN was further fine-tuned on five publicly available small reversed-phase retention sets in a transfer learning mode and demonstrated up to 30% improvement of prediction accuracy for these sets compared with the state-of-the-art methods. We demonstrated that filtering isomeric candidates by predicted retention with the thresholds obtained from ROC curves eliminates up to 50% of false identities.

1. Introduction

Liquid chromatography–mass spectrometry is the primary technique for untargeted small molecule analysis. High-resolution tandem instruments annotate detected compounds in untargeted metabolomics experiments that rely on experimental or in silico fragmentation data. Mass spectral databases cover less than 1% of known organic compounds [1,2]. In silico fragmentation predictors [3,4,5] may have a restricted applicability domain due to being trained on small mass spectral datasets. Complementary retention information may refine compound annotation, reduce the search space and eliminate false identities.

In targeted assays, retention time of a detected peak is compared with the value obtained for reference material in the same separation conditions. Wide-scope untargeted experiments, such as forensic or environmental analysis, aim to cover as diverse a range of molecular sets as possible, making such comparisons ineffective because of the lack of experimental reference data. Retention time modeling techniques [6,7] provide an opportunity for high-throughput retention prediction for thousands of candidates retrieved by accurate mass search. These predicted values may be used as reference values to filter false identities.

There have been numerous attempts to estimate retention times of small molecules based on machine learning or non-learning approaches [8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]. However, such models lack accuracy because of limited training data and the complicated nature of retention in HPLC. The breakthrough in retention time prediction became possible due to significant progress in developing deep learning techniques. The introduction of the METLIN Small Molecule Retention Time (SMRT) dataset [26] made it possible to train deep learning models to achieve unprecedented performance. Gradient boosting [27], deep neural networks with fully connected layers [26], graph neural networks [28,29,30] and convolutional neural networks have been deployed with the SMRT data. Recent works have indicated accuracy comparable with the experimental error of retention time measurements [31].

The variety of experimental setups is the major problem associated with retention prediction. Even though a model can provide very accurate predictions for SMRT, these predictions should somehow be transferred to the separation conditions used in a particular experiment. The correlation of retention on various systems is usually unclear. Changes in column dimensions, stationary phase or eluent composition may result in shifted times or even retention order. Complicated regression models may be used to recalculate predictions between two systems, but this requires shared molecules between two datasets [26,27,32,33,34]. A more intelligent way is to apply the two-step transfer learning approach [35]. This includes model pretraining on a large dataset, and fine-tuning the model for the task of interest on a smaller collection using weights obtained during the first step. Recent work [36] demonstrated that a retention dataset of several hundreds of compounds, not necessarily from SMRT, is needed for transfer learning. A deep learning model is first pretrained on SMRT or even a non-labeled dataset [29,37]. At this step, the model learns low-level molecular features common for different molecular tasks. Then, the model is initialized for target training with pretrained first layers that correspond to such features. Since these first layers are often frozen during fine-tuning, the generalization ability of a model significantly influences the overall performance. Thus, overfitted models may demonstrate low prediction error on the pretraining set but poor performance after transfer learning. Indeed, the prediction error normalized to the chromatographic runtime for small datasets is much worse than that obtained for the SMRT conditions.

Therefore, the evaluation of new architectures may contribute to accurate predictions on small datasets. This work aims to evaluate the message-passing neural network (MPNN), a kind of graph neural network, on retention data. MPNNs were initially designed for handling molecular data [38] and demonstrated outstanding performance on various chemical tasks [39,40,41,42]. To our knowledge, MPNN has only been used to model retention time in liquid chromatography on a tiny dataset [43]. We implement a transfer learning pipeline with pretraining MPNN on the SMRT dataset and fine-tuning on several available and in-house retention collections. We also propose a workflow to eliminate false identities based on predicted retention values.

2. Materials and Methods

Datasets. The METLIN SMRT dataset of 80,038 molecules was used for pretraining and was processed with the rdkit library. Retention times were collected in reversed-phase separation conditions with mean and median retention time variability of 36 and 18 s, respectively [26]. SMRT has a bimodal retention distribution (Figure S1) due to non-retained molecules with less than 2 min retention time. Following previous work [30,31], we excluded these molecules from the training set. The final set contained 77,977 compounds.

The model was fine-tuned and evaluated on small reversed-phase datasets from the PredRet database [33] (FEM_long, LIFE_new, LIFE_old, Eawag_XBridgeC18) and the RIKEN Retip dataset [12]. The retention distributions of these sets can be found in Figure S1.

Message-passing neural network. Following the pipeline for constructing the message-passing neural network from the original paper on MPNNs [38], our model included a featurizing step, message-passing, readout and a set of fully-connected layers. We took the implementation from the Keras tutorial on MPNNs with several changes of molecular features and hyperparameters. First, molecules were converted into undirected graphs, with atoms as nodes and bonds as edges. Graph nodes and edges were encoded with atom and bond features (Table S1) to generate input for the model. The message-passing step updated the node state considering its neighborhood and edge features. Previous states were handled with a gated recurrent unit (GRU) [44]. At the readout phase, the highly-dimensional updated node states were reduced to embedded vectors with a combination of a transformer and average pooling layers. These embeddings were then passed through 3 dense layers with ReLU activation and a linear layer to get predictions. The overview of the workflow is presented in Figure 1. The detailed model architecture with all input/output layer dimensions can be found in Figure S2.

Model training and evaluation. The model was trained with Adam optimizer, mean absolute error as a loss function, batch size of 64 and initial learning rate 0.0002. The learning rate was decreased by a factor of 0.25 if validation loss did not improve for ten epochs. Overfitting was controlled via an early stopping mechanism. Fitting was stopped if the validation loss did not improve for twenty epochs.

Model evaluation was performed with a 5-fold cross-validation protocol. Additionally, a separate 20% test set was extracted randomly and was common for all cross-validation iterations. Mean (MAE), median absolute error (MedAE), root mean squared error (RMSE), mean absolute percentage errors (MAPE) and R² score were used as metrics for evaluation. Datasets were randomly split into training and validation sets at an 80:20 ratio to train the final models. In the transfer learning mode, the weights of all the layers except the dense and the linear were frozen and loaded from a pretrained model before training. The batch size was reduced to 8. All other parameters were kept the same.

Elimination of false identities. To estimate the proposed approach’s practical utility, we applied it to distinguish between the isomers of compounds in the evaluation sets by predicted RT values. To choose the threshold to filter compounds, we plotted the receiver–operating characteristic (ROC) curve. For that, for all the molecules in the test sets that were unseen by the model, we downloaded all the isomers from PubChem using PUG requests [45]. For each isomer, RT was predicted by the model and compared with the experimental RT for the initial molecule from the test set to consider its isomer as a true-negative or false-positive for a certain threshold.

For each threshold value taken from the range 0–200% with step 2.5%, we calculated the number of true-positives (TP) (the correct identity’s predicted RT was within this threshold from experimentally measured value), false-positives (FP) (the false identity’s predicted RT was within this threshold from measured value), the total number of positive cases (P) (equal to the total number of compounds in the evaluation set) and negative cases (N) (total number of isomers of all compounds from test sets without positive cases).

For example, if the relative difference between the experimental and predicted values for a molecule in the test set was 5%, it was considered TP for all thresholds above 5% and FN for all bellow. Its isomers (that are presumed false identities) were considered FP and TN, respectively, for the same thresholds.

To obtain ROC, we plotted TP/P vs. FP/N at different thresholds. The best threshold corresponds to the highest value of TPR–FPR to compromise between the numbers of FN and FP.

All the code was written in Python and is available with the pretrained models via the following link https://github.com/osv91/MPNN-RT (Accessed on 3 October 2022).

3. Results and Discussion

Message-passing neural network. Since their introduction in 2017, message-passing neural networks have outperformed other deep neural networks and traditional machine learning methods in many molecular regression and classification tasks [39]. MPNNs are implemented in DeepChem, a popular chemistry-oriented deep learning framework [46]. They are available for general deep learning frameworks in official or non-official tutorials or add-ins. We have chosen an MPNN implementation from Keras [47] code examples and adjusted the model architecture, increasing the number of message-passing steps and adding dense layers to the decoder part of the model.

The results of the 5-fold cross-validation (Figure 2) of the MPNN are listed in

Table 1. Mean metrics for 5-fold cross-validation of MPNN on SMRT dataset compared with previously published results.

Model	MAE, s		MedAE, s		MAPE, %		RMSE		R2
	Validation	Test	Validation	Test	Validation	Test	Validation	Test	Validation	Test
MPNN	32.1 ± 0.6	31.5 ± 0.1	16.2 ± 0.2	16.0 ± 0.2	4.1 ± 0.1	4.0 ± 0.01	62.8 ± 1.9	60.5 ± 0.3	0.872 ± 0.008	0.879 ± 0.001
1D-CNN [31]	34.7 ± 1.2		18.7 ± 1.3		4.3 ± 0.1		65.5 ± 2.7		No data
GNN [30]	39.87		25.24		5		No data		No data

. The MPNN outperforms the recently proposed state-of-the-art models based on 1D-CNN [31] and GNN [35] architectures. More importantly, it provides mean and median absolute errors less than the experimental values for the SMRT dataset, which were reported to be 36 and 18 s, respectively. The difference between the test and the validation metrics is minor, indicating moderate model overfitting.

Transferring predictions to small datasets. Predictions made with a model trained on the SMRT dataset cannot be directly applied for small molecule analysis because of the wide variety of chromatographic setups. Changing a column or mobile phase gradient program leads to substantial retention time changes for a given analyte. Therefore, a new collection of retention times for the chosen separation conditions is required to get actual predicted values. These retention times can be obtained with mappings between SMRT and the new dataset, constructing a model from scratch with the new dataset, or using transfer learning. The last approach is the most promising. To reduce overfitting and get more accurate predictions, it may benefit to have a large data collection, such as SMRT or non-labeled public chemistry databases. We estimated the capability of the MPNN pretrained on the SMRT to improve predictions for five publicly available datasets and compared the results with MPNN trained from scratch and with some previously reported results. We reduced the batch size and froze all the weights except dense and linear layers to account for small dataset sizes. The results are summarized in

Table 2. Mean absolute error (in seconds) for 5-fold cross-validation of MPNN compared with previously published results.

Model	RIKEN Retip		FEM_long		Eawag_XBridgeC18		LIFE_new		LIFE_old
	Validation	Test	Validation	Test	Validation	Test	Validation	Test	Validation	Test
MPNN Transfer Learning	34.7 ± 3.3	38.2 ± 3.2	214.4 ± 25. 6	204.6 ± 23.0	79.5 ± 10.3	80.9 ± 6.5	23.3 ± 3.8	22.1 ± 3.3	16.9 ± 2.9	16.9 ± 2.0
MPNN From Scratch	56.2 ± 13.6	57.2 ± 13.2	299.4 ± 51.0	317.7 ± 25.8	137.1 ± 24.7	135.8 ± 14.1	37.5 ± 17.2	39.1 ± 15.2	27.2 ± 8.3	23.2 ± 2.1
1D-CNN Transfer learning [31]	32.4 ± 3.0		No data		No data		23.6 ± 5.1		15.5 ± 2.8
GNN Transfer learning [30]	No data		235.01		112.78		29.38		17.10

and

Table 3. Median absolute error (in seconds) for 5-fold cross-validation of MPNN compared with previously published results.

Model	RIKEN Retip		FEM_long		Eawag_XBridgeC18		LIFE_new		LIFE_old
	Validation	Test	Validation	Test	Validation	Test	Validation	Test	Validation	Test
MPNN Transfer Learning	22.2 ± 2.4	25.0 ± 3.5	93.5 ± 14.9	125.2 ± 12.0	56.0 ± 4.7	57.4 ± 9.3	12.2 ± 1.1	9.7 ± 3.6	9.9 ± 3.1	9.5 ± 0.7
MPNN From Scratch	38.0 ± 9.5	40.8 ± 12.4	162.4 ± 51.2	193.1 ± 41.3	105.3 ± 20.7	115.1 ± 18.1	20.4 ± 21.5	24.0 ± 22.2	18.11 ± 8.6	19.5 ± 3.6
1D-CNN Transfer learning [31]	22.2 ± 3.6		No data		No data		14.7 ± 4.7		11.8 ± 4.2
GNN Transfer learning [30]	No data		94.66		83.79		15.16		12.88

and Figure 3. MPNN behaves comparably with or better than 1D-CNN and GNN on most datasets. For example, for the Eawag_XBridgeC18 dataset, the mean and median absolute errors improved by about 30%. In total, MPNN outperforms GNN on 3 of 4 evaluated sets by median absolute error and all datasets by mean absolute error. Only for the RIKEN Retip dataset the mean absolute error of MPNN was higher by more than 2 s than that reported for 1D-CNN.

An example of eliminating false identities. The main goal of retention modeling is to provide a filter for non-targeted LC-MS workflows. Compound annotation in untargeted metabolomics is done by accurate mass, providing long isomeric candidates. Their fragmentation spectra may partially distinguish them, but MS/MS data are unavailable for most isomers, and annotation relies on in silico fragmentation. Although these methods are powerful, they require computational resources, and filtering the list of candidates by retention times may reduce the overall time for data processing.

For example, filtering candidates by retention time may be done using a simple filter based on the prediction standard deviations. However, estimating thresholds from receiver–operation characteristics (ROC) may reduce the number of false results. To test MPNN’s ability to aid in compound annotation, we plotted ROC curves with the test splits of small datasets to choose the optimum threshold and filtered out candidates using these values. The results are shown in Figure 4. The best thresholds established for the Retip, FEM_long, Eawag_XbridgeC18, LIFE_new and LIFE_old datasets were 22.5, 10, 20, 17.5 and 20%, respectively, and facilitated eliminating on average 53, 23, 35, 33 and 31% of false identities. Although the average number of eliminated false positives is relatively small, it was obtained for most complete lists of isomeric candidates from a general chemistry database. Targeting the search space to more specific databases, such as HMDB [48], or combining filtration by retention with in silico fragmentation [49] may improve the results.

4. Conclusions

Message-passing neural networks applied to the task of predicting retention time in liquid chromatography behave comparably or outperform state-of-the-art graph neural networks and 1D convolutional neural networks when trained on the METLIN SMRT dataset. Transferring the knowledge by two-step training allows accurate predictions on datasets collected from custom HPLC systems benefiting from pretraining on an extensive retention time collection. Investigating ROC curves allows adjusting the thresholds for efficient filtering of isomeric candidates by their predicted retention. Such filtering may facilitate the annotation process in untargeted small molecule applications.