A Point Cloud-Based Deep Learning Model for Protein Docking Decoys Evaluation

Abstract

Protein-protein docking reveals the process and product in protein interactions. Typically, a protein docking works with a docking model sampling, and then an evaluation method is used to rank the near-native models out from a large pool of generated decoys. In practice, the evaluation stage is the bottleneck to perform accurate protein docking. In this paper, PointNet, a deep learning algorithm based on point cloud, is applied to evaluate protein docking models. The proposed architecture is able to directly learn deep representations carrying the geometrical properties and atomic attributes from the 3D structural data of protein decoys. The experimental results show that the informative representations can benefit our proposed method to outperform other algorithms.

1. Introduction

Protein and protein interaction can form protein complexes in cells, which are capable of regulating enzyme activity and various biological processes. In recent years, more and more protein complexes have been crystallized through X-ray crystallization [1,2,3]. The rapid accumulation of the 3D structures of these protein complexes provides many insights into protein-protein interactions, facilitating rational drug development and the treatment of disease [4,5]. However, experimentally determined protein structures is costly and takes a lot of time, so various computational techniques for protein complex structure have been developed [6,7].

Protein docking is one of the most promising approaches to predict the structure of a complex between two proteins based on the independently crystallized structures of the components [8,9,10]. Sampling and scoring are two basic processes in protein docking. The sampling process can generate many binding models, called decoys, by docking two individual proteins [11,12]. Scoring is used to evaluate these decoys and identify near-native decoys as docking results [13,14,15].

An accurate scoring function for selecting near-native decoys is crucial to the success of docking, especially considering that such decoys make up only a small fraction of the entire decoy set. In recent years, a number of scoring functions have been developed for this purpose, falling into four broad categories: physical energy-based, knowledge-based, machine learning-based and deep learning-based methods.

Physical energy-based scoring functions use several physical energy terms to rank generated decoys. The docking programs using physical energy-based scoring functions include HADDOCK [16], pyDock [17], SwarmDock [18], etc. The scoring function of HADDOCK consists of intermolecular electrostatic and van der Waals energy terms combined with an empirical desolvation potential. PyDock uses electrostatics and desolvation energy to score docking poses generated with FFT-based algorithms. SwarmDock calculated the binding energy of the binding interface to evaluate the generated decoys.

Knowledge-based scoring function, on the other hand, are related to pair potentials between residue and residue. SIPPER [19] extracted new statistical potentials from intermolecular pairs of exposed residues in known complex structures, which were then used to score protein-protein docking poses. PIZSA incorporates residue-residue contact preference on the interface with per residue-pair atomic contributions and accounts for clashes [20]. Huang et. developed a distance-dependent knowledge-based scoring function to predict protein-protein interactions [21]. 3D-Dock [22] and DFIRE [23] transform distance-based pairwise contact distributions into potentials through an inverse Boltzmann relationship. The popular protein docking tool ZDOCK employs physical energy and potentials to rank their docking models [24]; hence, it is a representative of these two types of approaches.

The major limitation of the physical energy-based and knowledge-based scoring functions is that they only consider the simple relationship between interacting energies. In contrast, machine learning-based scoring functions can learn non-linear mappings from protein complexes, which can lead to better performance compared to traditional scoring functions. PROCOS [25] utilizes a support vector machine (SVM) to calculate a probability-like measure to be native for a given complex. ProQDock [26] also uses SVM to predict the quality of protein docking models. The difference is ProQDock predicts the absolute quality of a docking model measured by a novel protein docking quality score (DockQ). iScore [27] transforms the protein-protein interfaces into graph representation and combines HADDOCK energy terms to score the docked conformations.

The traditional machine learning-based scoring function cannot deal with 3D protein structure well, but deep learning-based algorithms have recently revolutionized traditional machine learning approaches by shifting from “feature engineering” to “architecture engineering”. This allows for the incorporation of multiple features using a specifically designed deep architecture to acquire a higher precision representation of 3D structures. Deep learning models have the ability to implicitly analyze data characteristics and correlations without any domain expertise, making them beneficial for docking model assessments. Due to the lack of consensus on the appropriate use of features from various perspectives, including physico-chemical properties and 3D geometric properties for distinguishing near-native models from other docking models, deep learning-based approaches are preferred. The deep architecture is designed to work on 3D voxels or graphic data derived from 3D structural data to capture fine-grained 3D information, making them best suited for docking model assessments [28,29,30,31]. TRScore [28] and DOVE [29] voxelize the protein–protein interface into a 3D grid, and considers atom type or physicochemical properties as input features applied to the neural network. However, one major limitation is that the atoms from 3D protein structures do not always align perfectly with the 3D grid voxels, causing some loss of useful information and introducing meaningless calculations. Since docking decoys can be naturally represented as graphs according to their 3D structures, in our previous work, we proposed a Deep Graph Attention Neural Network (DGANN)-based evaluation method [30], which formulate scoring function as a graph classification problem. This method treated residues as nodes and residue-residue contacts as edges and learned node latent representations across the global topology of a graph. GNN-Dove [31] is also a Graph Neural Network–based Docking decoy evaluation score and used the chemical properties of atoms and the inter-atom distances as features of nodes and edges in the graph respectively. However, the creation of a graph mainly relies on the distance to determine if two atoms have an edge. Different distance thresholds may result in different graphs that will eventually affect the final prediction result. In addition, the graph neural network only features learned topology but ignores geometrical features.

In this work, we proposed a point cloud-based deep learning model for protein docking decoys evaluation. Point cloud is a commonly used digital format, which contains x, y and z coordinates along with some point features. It was often used to depict a 3D object for its downstream analysis. PointNet, a widely used deep learning-based algorithm to learn the properties of point cloud data [32,33], has recently been successfully applied to protein–ligand binding affinity prediction [34,35,36]. It is able to adaptively detect the local geometric properties and atomic interactions from the protein structure data in a data-driven way. This benefits for mimicking the microenvironment to protein-protein interaction, hence, we leveraged PointNet to generate the abstractive representations of docking decoys and assess their binding strengths. Different from deep learning-based approaches, our method needs less preprocessing time for protein decoy structures since we directly input the atomic coordinates and their properties into a neural network after simple normalization. Additionally, our method learns spatial representation in a more efficient way. It directly learns the deep representation embedding 3D coordinates and physico-chemical properties of atoms, which can achieve a better trade off of computational effectiveness and efficiency. To the best of our knowledge, this paper is the first work that applies the point cloud-based method for docking decoy evaluation. Compared to other well-known algorithms, this method demonstrated more promising results in terms of Success Rate and Hit Count.

2. Materials and Methods

In this section, we mainly introduce the Benchmark dataset, encoding of protein decoys and deep learning architecture.

2.1. Benchmark Dataset

We trained and tested our method by using the dataset Benchmark 4.0 provided by ZDOCK [37]. It collected 176 protein complexes for benchmarking docking decoy quality, each of which contains 54,000 decoys generated by 6 degrees sampling in docking.

For each decoy, we calculated iRMSD (root mean square deviation of interface residues), fnat (fraction of receptor-ligand residue contacts in natural structures replicated in the decoys) and lRMSD (root mean square deviation of ligand residues). According to the CAPRI quality assessment criteria, Incorrect, Acceptable, Medium and High decoys were defined as follows:

Incorrect: fnat < 0.1 or (lRMSD > 10.0 and iRMSD > 4.0);

Acceptable: ((fnat >= 0.1 and fnat < 0.3) and (lRMSD <= 10.0 or iRMSD <= 4.0) or (fnat >= 0.3 and lRMSD > 5.0 and iRMSD > 2.0));

Medium: ((fnat >= 0.3 and fnat < 0.5) and (lRMSD <= 5.0 or iRMSD <= 2.0) or (fnat >= 0.5 and lRMSD > 1.0 and iRMSD > 1.0));

High: (fnat >= 0.5 and (LRMS <= 1.0 or iRMSD <= 1.0)).

We defined the Acceptable, Medium and High decoys as near-native decoys and Incorrect decoys as non-native decoys. However, the number of near-native and non-native decoys in ZDOCK dataset was extremely unbalanced. The non-native decoy to near-native decoy ratio is 1 to 19. To construct a balanced dataset, we reduce the number of non-native decoys to the number of near-native decoys. The selection of non-native decoys is according to the value of DockQ, which is used to measure the quality of the decoys. Generally, non-native decoys often have a value of DockQ less than 0.23. Thus, we chose the non-native decoys as follows: one-third of non-native decoys were selected from those with a DockQ less than 0.1, another third came from those with a DockQ between 0.1 and 0.2, and the rest came from those with a DockQ between 0.2 and 0.23. Figure 1 shows the procedure of dataset splitting.

In this study, we employed 5-fold cross-validation to evaluate the effectiveness of the model. In this process, all the protein complexes were partitioned into 5 equal parts. The ratio of the training, validation and testing sets was 7:1:2.

Table 1. The number of training, validation and testing set in different folds.

	Training Set	Validation Set	Testing Set
Fold1	18,452	2420	6440
Fold2	20,228	2420	4664
Fold3	19,940	2342	5030
Fold4	18,756	2872	5684
Fold5	18,972	2846	5494

shows the number of training, validation and testing set in different folds.

Besides 5-fold cross validation, we also conducted an independent evaluation via a brand new ZDOCK Benchmark 5.5 and DockGround 1.0. Benchmark 5.5 that included 81 protein complexes that differed from those of the Benchmark 4.0 dataset. After an initial check for the new protein complexes, we found that some of them do not contain any near-native decoys. Hence, only 43 protein complexes with near-native decoys were selected to form the independent set. DockGround 1.0 (http://dockground.compbio.ku.edu/downloads/unbound/decoy/decoys1.0.zip, accessed on 15 November 2022) includes 61 target complexes, and the decoys are generated by Gramm-X. The number of protein decoys in Benchmark 5.5 and DockGround is 86,000 and 6596 separately.

2.2. Point Cloud Preprocessing

In our study, we treated the atoms of a protein complex as points in a point-cloud, and a protein complex naturally was converted into a point cloud. On each point cloud, the following preprocessing steps were made before sending them to our architecture (Figure 2).

Firstly, we extracted the interface of each protein decoy. The interface was defined as the combination of all the residues within 10.0 Å from any residues of its counterpart. Since among all possible decoys of two proteins, the most distinctive and informative region is the interface, we will learn the deep representations of each interface through our architecture.

Secondly, we aligned the atomic coordinates of the protein complex to ligand’s center and normalized the 3D coordinates of all atoms according to the distance from the center. Ligand is the smaller one of the two proteins in one protein complex.

Finally, each atom of the protein decoy is represented as a single point containing seven atomic properties, including spatial coordinates, van der Waals radius, atomic weight, atomic type, atomic valence, atomic charge and atomic chain. For quantifying these properties, we use the raw values of spatial 3-dimensional coordinates, van der Waals radius, atomic weight of each atom in a 5-dimensional numerical vector. Atomic type is encoded into a 23-dimensional one-hot vector considering all possible atoms. Atomic valence is denoted by a 6-dimensional one-hot vector due to its range from 0 to 5. The possible atomic charge ranges in [−2, 2], hence, a 5-dimensional one-hot vector is employed to represent it. The atomic chain indicates the atom locates at ligand or receptor. Thus, a 2-dimensional one-hot vector is used to encode it. Totally, each point/atom is featured by a 41-dimensional feature vector for the following representation learning. If the total number of atoms is fewer than 700, we created all parameters set to zero for additional points.

2.3. Deep Learning Architecture

Our deep learning architecture was built based on PointNet [32]. It contains three modules, i.e., an encoder, a maximum pooler and a classification head.

The encoder takes the 41-dimensional feature of 700 points as input and then employs two Transform Nets to adjust the input point cloud to a suitable position for better revealing their docking surfaces. Three 1D convolution layers were followed to fuse point features and transformed coordinates into a deep representation. The deep representation embeds the spatial information and physio-chemical properties of each atom. A series of 1D convolution layers are assigned to further learn deep point representation according to neighboring points falling into convolution kernels. Eventually, the generative deep representation is fully containing local geometric properties and physio-chemical properties to mimic the microenvironment around each atom. Then, a maximum pooler is designed to squeeze the deep representations of all atoms at interfaces into a feature vector describing the interface. The classification head formed two fully connected layers, a normalization layer and an activation function RELU to identify the near-native ones from decoys based on their feature vectors. The 1-channel output layer yields a score for assessing each input docking model, whose score is close to 1 denoting that it is more likely a native-like docking model. With all predicted scores for candidate models, top ranks can be chosen to apply in any downstream analysis and wet experiments.

Given a docking model, let its atom number be N and the convolution kernel is K. In practice, K << N. For each atom, we need to find its neighborhoods for further massage passing, this search costs O(logN) if we chose a quick sort to its potential neighbors. After message passing, the atomic embedding will have several linear operations including weighted sum and activation function, which may ignore their computational complexity. Since all atoms require such neighboring search, the computational complexity of our deep architecture can be O(NlogN) [38]. In the point cloud model, we used 1,959,496 parameters in total.

Table 2. The specific number of different layers.

Layers		Number of Parameters
Encoding layer	3D Coordinates transformation	656,832
	Convld (41,64)	64
	Data transformation	656,832
	Convld (64,128)	128
	Convld (128,1024)	1024
Maximum pooling layer		0
Fully connected layer		644,616

shows the specific number of different layers.

The deep architecture is shown in Figure 3.

2.4. Modeling Setting

The deep architecture was trained using the training set split from 5-fold cross-validation. We utilized the SGD optimizer with a learning rate of 0.00005 to conduct training. The iterative epoch number was set to 100, and the batch size is fixed to 100. The early stop strategy was adopted to control the possible reductant training, which monitored the loss value of the validation set and stopped training when the model did not show any improvements in consecutive 50 epochs.

The binary cross-entropy was chosen as the loss function to guild the training. To evaluate our model, we ranked all decoys by their predictive logits and calculated two indicators Success Rate (SR) and Hit Count (HC) over the top N ranks. SR is the percentage with at least one near-native decoy and HC is the average number of near-native decoys over the number of all protein complexes.

We implemented our algorithm using Python language and some prebuilt Python packages were also utilized to facilitate our implementation, such as Pytorch, which provided a deep neural network framework, Torch_geometric which gave a graph neural network toolkit, Biopython which parsed the docking models, PSSMGen which produced PSSM features of each residue and SciPy which offered necessary scientific calculations.

3. Results

3.1. Comparative Results among Our Model and Other Representative Algorithms on ZDOCK Dataset

We trained and evaluated our model on Benchmark 4.0 based on the 5-fold cross-validation. As the representatives of physical energy/knowledge-based, machine leaning-based, and deep learning-based approaches, ZDOCK, GNN-DOVE, our previously proposed DGANN-based algorithms and TRScore were also tested on this dataset to compare with our model. ZDOCK is a popular traditional protein docking tool, which employs physical energy and potentials to rank their docking models [24]. GNN-DOVE [31] and our previously proposed DGANN-based algorithms [30] are representatives of graph neural network-based scoring functions. TRScore [28] is a recently proposed deep learning-based scoring method for ranking protein–protein docking models based on a 3D RepVGG network.

We selected these representatives considering their code availability and modeling efficiency. To keep fairness, all the comparisons were built on modeling with the same training data and evaluation with the same testing data. The Success Rate and Hit Count in top N from 5-fold cross-validation were calculated and plotted in Figure 4. The two indicators can well reflect whether these algorithms can pick up near-native decoy faster and better.

As shown in Figure 4a, for the top 50, SR of our model was higher than those of the other three algorithms. The value of SR is close to 1, which means that all the protein complexes in our dataset have at least one near-native decoy in the top 50 predictions. SRs in top 50 from the two GNN-based algorithms were close, which showed that at least one near-native decoy was detected in the top 50 predictions for the over 80% target complex. The figure showed that our method outperformed the other scoring function in terms of success rate except in the top 1. For the top 1, our method obtained an SR of 30%, which was a little bit lower than the SR of 42% by ZDOCK. Overall, our method obtained the success rates of 81% and 87% for top 5 and top 10, respectively, which were better than other scoring functions.

As for Hit Count, the performance of our method was also comparable to the other three scoring functions. As shown in Figure 4b, our method obtained the HC of top 50 nearly to 24. While for the top 5 and top 10, Hit Counts of our method were 1.99 and 4.08, respectively, compared with 1.73 and 3.68 for the TRScore. This trend was basically consistent with the results of SR.

Figure S1 shows the distribution of the area under the receiver operating characteristic curve (ROC-AUC) and the precision-recall area under the curve (PR-AUC) per complex. Our method achieved high ROC-AUC values and PR-AUC values according to the boxplots. Table S1 is the average precision and recall of all complexes from the threshold of 0.7. The results also show that our method has better performance.

Next, we also would like to report the details in the five-fold cross-validation as shown in Figure 5. From the figure, we can see that our proposed model outperformed other methods in each fold. These results demonstrated our method achieved not only good performance but also excellent robustness.

3.2. Testing on Independent Dataset Benchmark 5.5

To exhibit the generalizability of our method, we validated our algorithm and the competitive tools on the independent test dataset ZDOCK benchmark 5.5. We ran ZDOCK 3.0.2 to generate the possible docking decoys and used the top 2000 decoys from each complex to test the performance of our method. Since we obtained five models from five-fold cross-validation as described in Section 2.1, the prediction of a given decoy is the average from the five models in this report. The other three methods were also evaluated on this independent dataset for comparisons.

As shown in Figure 6a, our method got the best success rate before the top 22. For example, our method achieved the success rates of 25.23% and 29.08% for the top 5 and 10 ranks, respectively, which are higher than the highest success rates of 19.23% and 26% among the other three scoring functions. Although our method did not keep its predominant SR over all other methods after 22 ranks, it still showed overall better success rates than DGANN-based, GNN-DOVE and TRScore. Similar to SR, our method performed best HC in most places among all of the four scoring functions as shown in Figure 6b. For example, our method achieved the HC of 1.02 and 1.48 over top 30 and 50 ranks, respectively, which were higher than the highest HC of 0.85 and 1.31 among the other three scoring functions. These consistent results on a new testing dataset revealed our method achieved more powerful generalization than other methods.

3.3. Testing on Independent Dataset DockGround

To further validate the reliability and generalization ability of our architecture, we used another dataset DockGround 1.0 (http://dockground.compbio.ku.edu/downloads/unbound/decoy/decoys1.0.zip, accessed on 15 November 2022) to test our scoring function. DockGround 1.0 includes 58 target complexes, and the decoys are generated by Gramm-X. The average number of near-native decoys and non-native decoys is 9.83 and 98.5. All the scoring functions are tested on this dataset. All five scoring functions are tested on DockGround.

As shown in Figure 7, our method achieved the best performance among all of the five scoring functions. For example, within the top 10 rank, our method achieved a success rate of 57%, while the next best method, TRScore, achieved 55%, and the third best method, GNN-DOVE, obtained 48%. Our method also achieved the hit count of 3.01 for the top 10, which was higher than the highest hit count of 2.5 among the other four scoring functions.

4. Discussion

In our work, point cloud-based algorithm is used to develop a deep learning model to evaluate protein docking decoys. Since 3D structural information of protein complex plays a vital role in deep learning-based scoring function, in our scoring function, we used point cloud-based deep learning algorithm PointNet to directly process the 3D coordinates and attributes of atom in protein docking decoy

Different from other deep learning algorithms, our method can learn not only topology features but also geometrical features of 3D protein structure, which adaptively detect the local geometric properties and atomic interactions from the protein structure data in a data-driven way. By comparing the results from some competitive tools, our superior Success Rate and Hit Count on 5-fold cross-validation and the independent dataset demonstrated the robustness and generalization of our method, which demonstrates that our deep architecture was able to adaptively detect effective geometric and physio-chemical descriptors for identifying near-native decoys from non-native decoys. The improved performance benefited from the informative learned representations of each atom mimicking the atomic interactions and their microenvironment. Meanwhile, the diversity of training data also contributed to enhancing the generalizability of our model.

Our point cloud-based algorithm works in a supervised learning way. Supervised learning requires sufficient and diverse labeled data. In this study, we employed the currently largest dataset Benchmark 4.0 to train our model. With the growth of available protein docking complexes and docking models, the generalization of our proposed method is expected to be further enhanced. Meanwhile, due to the complication of flexible docking, only rigid docking was considered in our formulation. Future studies will introduce the dynamics of the docking process into our model and generalize our model to address the flexible docking model assessment. We will also consider fusing more features, such as protein sequence, into our model. We can use contrastive learning to fuse multiple kinds of features and further improve performance. In addition, our protein decoy evaluation task can also be related to antibody and antigen binding prediction. We can embed our evaluation model into other docking tools to facilitate vaccine design.