Detection of Unknown Polymorphic Patterns Using Feature-Extracting Part of a Convolutional Autoencoder

Abstract

Background: The present paper proposes a novel approach for detecting the presence of unknown polymorphic patterns in random symbol sequences that also comprise already known polymorphic patterns. Methods: We propose to represent rules that define the considered patterns as regular expressions and show how these expressions can be modeled using filter cascades of neural convolutional layers. We adopted a convolutional autoencoder (CAE) as a pattern detection framework. To detect unknown patterns, we first incorporated knowledge of known rules into the CAE’s convolutional feature extractor by fixing weights in some of its filter cascades. Then, we executed the learning procedure, where the weights of the remaining filters were driven by two different objectives. The first was to ensure correct sequence reconstruction, whereas the second was to prevent weights from learning the already known patterns. Results: The proposed methodology was tested on sample sequences derived from the human genome. The analysis of the experimental results provided statistically significant information on the presence or absence of polymorphic patterns that were not known in advance. Conclusions: The proposed method was able to detect the existence of unknown polymorphic patterns.

1. Introduction

Recent trends in machine learning indicate an emerging need for methods and models capable of incorporating explicitly formulated knowledge. This is especially important when training data are scarce and a shortage of available information needs to be compensated for with other means. The integration of knowledge and learning is crucial for enabling progress in several challenging domains, such as in the field of medical research and, in particular, in motif detection [1]. Motif detection is one of the most challenging tasks in data analysis due to the polymorphic nature of patterns that can encode genomic information and the scarcity of data, which severely impairs learning feasibility.

The present paper is concerned with research on the detection of polymorphic patterns in sequences by utilizing prior knowledge to facilitate this difficult task. The polymorphic sequences considered in this paper are sequences generated by regular expressions with flexible rules, resulting in a high diversity of valid, alternative sequence instances. A particular research question that we attempted to address was the following: having a set of sequences, each comprising several polymorphic patterns, where some of these match known flexible rules (defined via regular expressions), is it possible to determine the presence of previously unknown ones?

In what follows, we present a method that enables polymorphic pattern detection in sequences. The key elements of the proposed method are the integration of learning with prior knowledge and the introduction of mechanisms that discourage the learning of concepts that are already known. We assumed that polymorphic pattern-describing rules can be, at least partially, represented as filter cascades of a convolutional neural network. To discourage the learning of known patterns, we defined an appropriate loss component that evaluates the similarity of filter-represented pattern detectors and used this information to guide training. We evaluated the method on a dataset extracted from human genome data and showed that it enabled the statistically significant detection of unknown polymorphic patterns.

A significant amount of research has been carried out so far on handling the problem of detecting patterns (motifs) in sequences, among which we can name methods that aim to disentangle patterns to be recognized by convolutional neural networks. Liang et al. [2] proposed a training method for classification models that makes convolutional filters class-specific. They used an additional mask that assigned filters to classes and, then, in two-phase training, optimized that mask. Although this was an interesting approach, its main contribution was the disentanglement of knowledge by minimizing the number of filters sharing class-specific knowledge, thus making a step towards better interpretability. However, it did not expand upon the internal representation of said knowledge.

Koo et al. [3] examined representations of genomic motifs in CNNs. They searched for motifs in first-layer convolutional filters, transforming them into position–weight matrices based on the filter’s response to specific samples. This work exemplified a common approach in the representation of genomic patterns in filters of convolutional networks. It should be emphasized that this approach focuses on one-layer representation, which constrains the complexity and in-place interaction analysis between the discovered patterns.

Zhang et al. [4] proposed a method for updating a specific subfilter cascade chosen dynamically during training to produce more diverse convolutional filters and reduce overlap in representation. A more general examination of this problem found solutions like structuring a network to resemble the knowledge base, which could be achieved either manually [5] or through a training process [6]. In the case of relational neural machines (RNMs) [7], knowledge is provided through optimization constraints, thus being a part of the parametric regularization and not the learner structure.

The use of convolutional neural networks has also emerged in omic studies, for example, in protein design [8]. Ingraham et al. [9] proposed a novel model named Structured Transformer based on the assumptions of a self-attention mechanism. They introduced prior knowledge into the model formulated as a graph of relations between amino acids. Sabban and Markowsky [10] used LSTM to generate novel proteins with the addition of a GAN-like classifier scheme to determine acceptable designs. Bepler and Berger [11] used BiLSTM to learn protein sequence embeddings based on structural similarity. They proposed a mechanism based on soft alignment and attention and successfully predicted the structural embeddings. Hie et al. [12] proposed a way of quantifying the uncertainty of prediction and used it in the training loop, thus reducing the volume of data needed. Ding et al. [13] proposed an interesting approach to transforming convolutional kernels into position–weight matrices.

The two widely used motif-detection methods that do not resort to deep learning are DREME (discriminative regular expression motif elicitation) [14] and MEME (multiple EMs for motif elicitation) [15]. The MEME method uses a probabilistic approach to generate a position-specific weight matrix of a recurring, ungapped motif, while DREME searches for consensus sequences using an enumeration approach [16]. Due to their frequent use, these methods are natural candidates for a baseline; however, they do not target the problem of finding polymorphic patterns, as opposed to our approach.

Among the methods that focus on finding motifs in sequences, algorithms such as ListMotif [17] and YMF [18] should be mentioned as those use the enumerative approach. In the family of probabilistic methods, the approaches worth noting are EXTREME [19] and STEME [20] (both use the same principle as the MEME method). Also, soft computing approaches, such as ant colony optimization, have been considered for handling the problem [21].

The main contribution of the proposed approach is in addressing the two limitations of existing methods, which are poor handling of the mutability of patterns to be detected and poor use of knowledge on known, possibly polymorphic patterns, which can exist in sequences to be analyzed. To cope with the first problem, we propose to use a flexible pattern representation scheme; whereas, to solve the second one, we propose to inject knowledge into the network (by means of preset convolutional filter cascades) and introduce a learning mechanism that enables the efficient capture of novelties (by discouraging learning of what is already known). The ideas proposed in the paper provide the first, to our knowledge, effective way of detecting the presence of unknown, latent data generation mechanisms that are difficult to discover due to the flexibility of their underlying data-generation rules.

2. Materials and Methods

2.1. The Proposed Polymorphic-Pattern Detection Methodology

As patterns to be detected are unknown, an unsupervised learning strategy was adopted to solve the problem. The framework of the proposed concept is provided by the convolutional autoencoder (CAE), and the core module involved in pattern detection is CAE’s cascade of convolutional filters. A part of this structure is preset to encode knowledge on known polymorphic sequences (we refer to it as the Fixed Convolutional Module—FCM), and the remaining part is expected to learn any previously unknown regularities that exist in data (we refer to it as the Learnable Convolutional Module—LCM).

An overall computational architecture used for polymorphic pattern detection, the adopted two-layer convolutional, pattern-detecting filter cascade, is depicted in Figure 1.

2.2. Input Sequences

Sequences to be analyzed are short samples extracted from the human genome of the fixed length L, composed of symbols from the four-element alphabet {a,c,g,t}. For the purpose of subsequent processing, each symbol is represented using one-hot encoding. The dataset contains sequences that match polymorphic patterns defined by regular expressions. Sample regular expressions along with the adopted encoding convention, where the expression-matching strings are highlighted, are shown in Figure 2. The regular expressions are formulated in accordance with the specification of the Python $re$ library (https://docs.python.org/3/library/re.html, (accessed on 1 June 2023)). The motif-detection scenario considered in the reported work assumes that sequences to be analyzed comprise patterns generated by two known, flexible rules and, in addition, can also contain an unknown, polymorphic pattern. We transfer knowledge on the known patterns by fixing FCM parameters of the proposed architecture.

To quantify a level of the pattern polymorphism, we propose the following metrics:

$$\begin{array}{c}m=\frac{{|P|}_0}{{|A|}_0^l}\end{array}$$

(1)

where ${|P|}_0$ is the zero-norm of a set of patterns that can be produced by the considered, underlying pattern-generating rule; A is the alphabet over which the pattern is constructed; and l is the length of the pattern. This metric is closely correlated with the difficulty of the problem at hand; as can be observed in

Table 1. Examples of six-element patterns generated from a four-element alphabet and their corresponding m measures: ‘.’ stands for any alphabet element, and $\left[xy\right]$ denotes any element of the subset $\{x,y\}$.

Pattern	m
$AAAAAA$	$0.0002$
$AA..AA$	$0.004$
$CC.\left[ACT\right].C$	$0.012$
$\left[ACT\right]\dots .T$	$0.19$

, its minimum value is $\frac{1}{{|A|}_0^l}$ for a non-mutable pattern and its maximum value is one for random noise.

2.3. Polymorphic Pattern Representation with Convolutional Filters

For the purpose of introducing knowledge to convolutional modules, a method for representing patterns, which are defined via regular expressions, employing convolutional filter parameters, was developed. Convolutional filters can be seen as pattern-detecting operators that produce the maximum output for inputs that match filters’ weights. It is reasonable to use a cascade of simple filters arranged in conventional convolutional layers to provide a flexible representation of polymorphic patterns of arbitrary length. The first-layer filters could be designed to capture different short patterns that comply with local rules defined by a given regular expression, whereas subsequent layers could combine these short chunks into longer strings in accordance with the considered rules.

The simplicity of defining filter cascades that specialize in detecting specific input patterns enables the simple incorporation of initial knowledge into a structure of convolutional neural networks. For example, to represent patterns generated by the provided sample regular expression provided in Figure 3, the first layer filters need to have a receptive field of size $w=3\times h=1\times d=4$, where the adopted width (w) enables capturing three-element strings and the adopted depth (d) reflects one-hot encoding used for the considered four-letter alphabet (height—i.e., h—is set to unity as the one-dimensional input is considered). To match symbol-generating rules used by a regex, three different values can be assigned to each weight of a filter. If some symbol is encouraged at a given position of a string, the corresponding filter’s weight is set to a value of ‘1’; if it is forbidden, the corresponding weight is set to ‘−1’. Otherwise, the weight is set to ‘0’ if it is neither encouraged nor forbidden.

The second-layer filters that merge short segments detected by first-layer filters into the longer chunks require a depth that enables integrating all relevant first-layer detectors. For the example presented in Figure 3, the required depth d = 2 allows for combining results produced by the two three-symbol string detectors (two first-layer filters), into a six-element string detector. The span of the receptive field in the second-layer filters must enable seamless string-stitching, which can be achieved by ensuring sufficient filter width (as shown in Figure 3) or by employing appropriate weight dilation. The rules for setting second-layer filters’ weights, which transform outputs of first-layer filters, are the same as presented for the first-layer filters. The method can be scaled both in terms of the number and size of patterns, as well as in terms of rule complexity, by adding new layers of filters.

It should be emphasized that the size of filters does not define the exact length of the pattern but only constrains its maximum length, as the proposed methodology allows for patterns that contain any character at each position, including the ends. Therefore, shorter patterns can be injected by filling the trailing positions with ‘zeros’ (any symbol is allowed).

2.4. Regular Expression Reconstruction

To search for unknown polymorphic patterns that are embedded in sequences comprising runs of random symbols together with known, possibly polymorphic strings, we initialize our algorithm with knowledge provided in the form of a cascade of appropriately preset filters, as explained in the previous subsection.

Several additional filters with parameters to be learned from data are also placed in convolutional layers of the proposed algorithm to detect unknown patterns. A structure of the convolutional pattern detectors can be bundled in the form of the position–weight tensor R [22] of the following form:

$$\begin{array}{c}R=\left[\begin{array}{ccccc}\left[\begin{array}{c}\left[\begin{array}{ccc}{\theta }_{11A}& {\theta }_{12A}& {\theta }_{13A}\\ {\theta }_{11C}& {\theta }_{12C}& {\theta }_{13C}\\ {\theta }_{11G}& {\theta }_{12G}& {\theta }_{13G}\\ {\theta }_{11T}& {\theta }_{12T}& {\theta }_{13T}\end{array}\right]\\ ⋮\\ \left[\begin{array}{ccc}{\theta }_{n1A}& {\theta }_{n2A}& {\theta }_{n3A}\\ {\theta }_{n1C}& {\theta }_{n2C}& {\theta }_{n3C}\\ {\theta }_{n1G}& {\theta }_{n2G}& {\theta }_{n3G}\\ {\theta }_{n1T}& {\theta }_{n2T}& {\theta }_{n3T}\end{array}\right]\end{array}\right]\odot \left[\begin{array}{c}{\gamma }_{1,1}\\ {\gamma }_{1,2}\\ {\gamma }_{1,3}\\ ⋮\\ {\gamma }_{1,n-2}\\ {\gamma }_{1,n-1}\\ {\gamma }_{1,n}\end{array}\right]& & \cdots & & \left[\begin{array}{c}\left[\begin{array}{ccc}{\theta }_{11A}& {\theta }_{12A}& {\theta }_{13A}\\ {\theta }_{11C}& {\theta }_{12C}& {\theta }_{13C}\\ {\theta }_{11G}& {\theta }_{12G}& {\theta }_{13G}\\ {\theta }_{11T}& {\theta }_{12T}& {\theta }_{13T}\end{array}\right]\\ ⋮\\ \left[\begin{array}{ccc}{\theta }_{n1A}& {\theta }_{n2A}& {\theta }_{n3A}\\ {\theta }_{n1C}& {\theta }_{n2C}& {\theta }_{n3C}\\ {\theta }_{n1G}& {\theta }_{n2G}& {\theta }_{n3G}\\ {\theta }_{n1T}& {\theta }_{n2T}& {\theta }_{n3T}\end{array}\right]\end{array}\right]\odot \left[\begin{array}{c}{\gamma }_{m,1}\\ {\gamma }_{m,2}\\ {\gamma }_{m,3}\\ ⋮\\ {\gamma }_{m,n-2}\\ {\gamma }_{m,n-1}\\ {\gamma }_{m,n}\end{array}\right]\end{array}\right]\\ {\theta }_{xyz}\mathrm{a}\mathrm{value}\mathrm{in}{x}^{th}\mathrm{filter},y^{th}\mathrm{position},\mathrm{and}{z}^{th}\mathrm{channel}\mathrm{of}{1}^{st}\mathrm{layer}\\ {\gamma }_{x,y}\mathrm{a}\mathrm{value}\mathrm{in}{x}^{th}\mathrm{position},y^{th}\mathrm{channel}\mathrm{of}{2}^{nd}\mathrm{layer}\mathrm{filter}\end{array}$$

(2)

where the symbol ⊙ denotes component-wise multiplication, i.e.,

$$\begin{array}{c}\left[\begin{array}{c}\left[\begin{array}{ccc}{\theta }_{11A}& {\theta }_{12A}& {\theta }_{13A}\\ {\theta }_{11C}& {\theta }_{12C}& {\theta }_{13C}\\ {\theta }_{11G}& {\theta }_{12G}& {\theta }_{13G}\\ {\theta }_{11T}& {\theta }_{12T}& {\theta }_{13T}\end{array}\right]\\ \left[\begin{array}{ccc}{\theta }_{21A}& {\theta }_{22A}& {\theta }_{23A}\\ {\theta }_{21C}& {\theta }_{22C}& {\theta }_{23C}\\ {\theta }_{21G}& {\theta }_{22G}& {\theta }_{23G}\\ {\theta }_{21T}& {\theta }_{22T}& {\theta }_{23T}\end{array}\right]\\ ⋮\end{array}\right]\odot \left[\begin{array}{c}{\gamma }_1\\ {\gamma }_2\\ ⋮\end{array}\right]=\left[\begin{array}{c}{\mathit{\theta }}_{\mathbf{1}}\\ {\mathit{\theta }}_{\mathbf{2}}\\ ⋮\end{array}\right]\odot \left[\begin{array}{c}{\gamma }_1\\ {\gamma }_2\\ ⋮\end{array}\right]=\left[\begin{array}{c}{\gamma }_1{\mathit{\theta }}_{\mathbf{1}}\\ {\gamma }_2{\mathit{\theta }}_{\mathbf{2}}\\ ⋮\end{array}\right]\end{array}$$

(3)

The first-layer filters $1..,i\dots n$, with weights arranged into matrices ${\mathit{\theta }}_{\mathit{i}}$ are combined by the second-layer ones (from 1 through m), with entries ${\gamma }_{i,j}$. The constructed PWM provides a basis for identifying a regular expression that emerges from the weights of a trained network.

The reconstruction follows the scheme provided in Equation (4), which involves weight normalization, aimed at converting learned weights into probabilities (transformation of R into R″) followed by thresholding, which is expected to produce an unambiguous basis for the reconstruction of the detected regular expression. Here, the symbols U and W denote the position-wise minimum of R and the position-wise sum of R′, respectively.

$$\begin{array}{c}{R}^{\prime }=R+2\ast |U|\mathrm{where}{U}_i={\displaystyle \underset{j\in J}{\min }}{R}_{ij},i=1,2\dots n\\ R^{″}=R^{\prime }/W\mathrm{where}{W}_i={\displaystyle \sum _{j\in J}}{R}_{ij}^{\prime },i=1,2\dots n\\ R^{‴}=f\left(R^{″}\right)\mathrm{where}f\left(r_{ij}\right)=\left\{\begin{array}{cc}1\hfill & r_{ij}>t{h}_{upper}\hfill \\ -1\hfill & r_{ij}<t{h}_{lower}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(4)

The reconstructed regular expression $RE$ is defined as a tree, built of shorter chunks $RS$, encoded by the first-layer filters:

$$\begin{array}{c}{\forall }_{s_i\in S}{s}_i\in R_i\\ R\in RE\\ RE=\left\{RS\right\}♢\{\vee ,\wedge \}\end{array}$$

(5)

where ♢ denotes a recursive tree operator, and $s_i$ is a symbol at the ith position of a string S that is admissible at this position of the regular expression R (i.e., $R_i$), which is a specific instance of an expression $RE$.

2.5. Learning Outcome Evaluation

We expect that throughout learning, any new polymorphic, unknown patterns present in input sequences will be learned by learnable weights of the convolutional module. As described in the previous section, these patterns can be subsequently reconstructed from the learned weights.

To measure the similarity of rules that generate patterns, we use a mean Levenshtein distance [23] applied to pairs of sequences produced by the considered regular expressions, i.e., target sequences $S^T=\{s_1^T,\dots s_n^T\}$ generated by a rule $R^T$ to be learned and sequences $S^L=\{s_1^L,\dots s_m^L\}$ generated by a rule that has actually been learned and is reconstructed from the learned filter cascade weights ($R^L$):

$$d(R^T\mapsto S^T,R^L\mapsto S^L)=\frac{1}{nm}\sum _{i=1}^n\sum _{j=1}^m{d}_{i,j}(s_i^T,s_j^L)$$

(6)

The Levenshtein (edit) distance $L_{i,j}$ between sequences $s_i^T,s_j^L$ is defined as

$$\begin{array}{c}{L}_{i,j}=Le{v}_{M,N}\\ Le{v}_{m,0}=m\\ Le{v}_{0,n}=n\\ Le{v}_{m,n}=min\left\{\begin{array}{c}Le{v}_{m,n-1}+c_{ins}\left(i_m\right)\hfill \\ Le{v}_{m-1,n}+c_{del}\left(j_n\right)\hfill \\ Le{v}_{m-1,n-1}+c_{sub}(i_m,j_n)\hfill \end{array}\right.\end{array}$$

(7)

where M = $|s_i^T{|}_0$, N = $|s_j^L{|}_0$ are dimensions of the distance matrix $Lev$. $c_{ins}$, $c_{del}$, and $c_{sub}$ are the insertion, deletion, and substitution costs, respectively. The distance between sequences i and j is placed at position M, N of the matrix $Lev$ and represents the minimum number of insertion, deletion, and substitution operations necessary to match one string to the other.

2.6. Disentanglement of Novel Rules from Preset Knowledge

Learning of rules that underlie new polymorphic patterns might be impaired by the influence of patterns already known to the network. Therefore, we consider an additional learning scenario, where learnable convolutional filters are discouraged from discovering knowledge already injected into the network via FCM filters.

To focus on learning new patterns, we use the adopted way of evaluating the proximity of regular expressions, which can also be seen as a way of measuring the similarity between filter cascades that encode regular expressions. Adopting this perspective, one can easily introduce additional mechanisms to control the training of filter cascades that are to learn unknown polymorphic patterns. In particular, one may hypothesize that learning entirely new concepts would be more efficient if learned weight representations were uncorrelated with existing knowledge, embedded in preset filter cascades. Therefore, we propose to augment the training procedure with an additional step aimed at maximizing dissimilarity between cascades that are to be learned and knowledge-encoding cascades.

To implement this objective, at each training iteration, we evaluate the proximity of filter cascades preset with prior knowledge and filter cascades that are to learn unknown patterns. Given some known regular expression $R_k^F$, encoded through some k-th filter cascade of FCM, and a regular expression $R_l^L$ that is decoded from some j-th filter cascade of LCM, we generate the corresponding two sets of sequences: $S^k$ and $S^l$. The similarity of the regular expressions $R_k^F$ and $R_l^L$ is then quantified using the mean edit distance (6), which, for the considered case, adopts the form

$$d(R_k^F,R_l^L)=\frac{1}{|S|}{\displaystyle \sum _{(s_i,s_j)\in S^k\times S^l}}d(s_i^F,s_j^L)$$

(8)

where × denotes a Cartesian product of the sets $S_k,S_l$.

The evaluated similarity is then used to modify filter weight updates to favor learning regular expressions dissimilar to each other, as presented in the listing of Algorithm 1. Training of convolutional filters proceeds differently for FCM and LCM neurons. The weights of neurons that make up knowledge-representing filter cascades are fixed. In the case of learnable module filters, weight updates are driven by scaled gradients, where scaling favors increasing dissimilarity between known and learned regular expressions. This is implemented using an iterative two-step procedure. An objective of the first, ‘trial’ step is to calculate new, candidate filter weights using basic backpropagation. A ‘virtual’ pattern $V_k^L$ reconstructed from the updated candidate filters is then confronted with all patterns generated by preset rules $R_l^F$. Suppose that between-pattern distances (i.e., $d(V_k^L,R_l^F)$) get increased; in that case, all gradients for a learnable cascade ‘k’ are amplified, i.e.,

$${\forall }_{g_{i,j}\in G}{g}_{i.j}^{\prime }=g_{i.j}\ast d(V_i^L,R_j^F)$$

(9)

and the final weight updates are made using old weights and modified gradients. However, if virtual updates increased the similarity between the learned and preset expressions (the distance $d(V_k^L,R_l^F)$ increases), the gradients get diminished, i.e.,

$${\forall }_{g_{i,j}\in G}{g}_{i.j}^{\prime }=\frac{g_{i.j}}{d(V_i^L,R_j^F)}$$

(10)

where G is the matrix of gradients backpropagated to the considered filter in the network, and i, j are specific positions in this filter.

Algorithm 1 Modification of Backpropagation algorithm

$ts\leftarrow weights-l_r\ast grad$ $V^L\leftarrow reconstruct\left(ts\right)$ for  $V_k^L$    in    $V^L$do     for $R_l^F$    in    $R^F$ do         if $d(V_k^L,R_l^F)>d(R_k^L,R_l^F)|$ then            $grad\left[V_k^L\right]\leftarrow amplify\left(grad\left[V_k^L\right]\right)$         else            $grad\left[V_k^L\right]\leftarrow attenuate\left(grad\left[V_k^L\right]\right)$         end if     end for end for

▷ ts - trial optimizer step

2.7. Experiments

The proposed polymorphic pattern detection computational architecture was trained on a dataset made up of one hundred 40-element-long sequences selected from human genomic data (the small number of samples was motivated by the scarcity of genomic data representing rare patterns). These samples were used as input for training the proposed CAE model, where the objective was to tune CAE’s weights (including LCM parameters) to maximize sequence reconstruction accuracy.

The datasets used in this work were extracted from GRCh38.p14 assembly by searching for sequences that contained two or three patterns exactly matching regular-expression-encoded rules. For the extracted sequences, additional analysis was performed to ensure no extra motifs were present in the data. This was achieved using DREME with the word length set to 3 and, if the test proved positive, the corresponding sample was removed from the dataset.

Different scenarios considered in experiments involved different combinations of regular expressions that are known to the network (two out of three were assumed to be known), the presence or absence of patterns generated by an unknown rule, and optional use of the proposed backpropagation algorithm modification.

2.8. Network Architecture and Training

The convolutional autoencoder of the architecture depicted in Figure 4 was used as the computational framework for polymorphic pattern detection experiments. The encoder consists of two Conv1D layers, with 9 and 6 filters, respectively, followed by ReLU activation. The latent representation is obtained using a 15-neuron dense layer.

The decoder mirrors the encoder’s architecture; however, it has a different number of neurons in dense and transpose convolution layers. The activation functions for dense layers are ReLU and Softmax for the network’s output. The size of the output sequence is the same as the input one.

Every network was trained for 500 epochs using the Mean Squared Error loss function and the RMSProp optimizer with a learning rate of 0.001.

The developed model was implemented using the PyTorch library [24] and Python 3.8. Experiments were run on a machine with an AMD Ryzen 9 3950X processor, 64 GB of RAM, and an NVIDIA GeForce RTX 2080 Ti graphics card.

3. Results

The target characteristic to be quantified at the completion of training was a mean distance between the learned filter cascades and the fixed, knowledge-representing cascades.

We were interested in learning whether statistically significant differences exist among results produced for four different experimental scenarios:

• An unknown pattern present and filter similarity discouragement turned on;
• An unknown pattern not present and filter similarity discouragement turned on;
• An unknown pattern present and no filter similarity discouragement;
• An unknown pattern not present and no filter similarity discouragement.

For each scenario (repeated 100 times), the resulting mean distances between pairs of regular expressions represented by LCM filters and FCM filters were evaluated. Results of the Levene [25] and Fligner [26] tests (widely accepted for equality of variance testing [27])—for which the null hypothesis is that samples are drawn from the same distribution—show significant outcomes when group 2 is tested against group 4. For groups 1 and 3, test results give no basis for rejecting the null hypothesis; in both training regimes, rules that are similarly distant from the preset ones are learned (

Table 2. Results of statistical tests for GRCh38 data. Statistically significant results support the conclusion that samples are drawn from different distributions.

	Levene Test		Fligner Test
♢	Statistics	p-Value	Statistics	p-Value
True	$1.89$	$0.09$	$3.74$	$0.24$
False	$3.56$	$0.02$	$9.57$	$0.01$

♢ Unknown regex present in data.

). In other words, statistical tests have significant results when the dataset does not contain unknown patterns. If the test result is insignificant, we can conclude that there is a new, unknown pattern in the data. The flowchart of this method is presented in Figure 5. It can also be observed that patterns learned by the trainable filter cascade are most similar to those actually present in the data (

Table 3. Averaged distances between regular expressions retrieved from filters and the unknown pattern. Columns $d_1$ and $d_2$ represent distances from FCMs with injected knowledge and $d_3$ represents LCM, which is supposed to reconstruct the pattern.

Backprop Modification	♢	${\mathit{d}}_1$	${\mathit{d}}_2$	${\mathit{d}}_3$
False	True	3.43	3.46	2.91
False	False	3.40	3.43	2.87
True	True	3.52	3.49	2.91
True	False	3.44	3.45	2.94

♢ Unknown regex present in data.

).

The datasets used in our study were also analyzed using the two previously indicated baseline methods: DREME and MEME. The analysis was performed in two scenarios—for two or three patterns present in the data. Both methods were set to search for six-element-long motifs—the same as used in our method.

Table 4. DREME results. Three motifs with the lowest E-value are presented for each dataset. Columns $d_1$, $d_2$, and $d_3$ represent distances from patterns considered in the study, with $d_3$ being the unknown pattern.

♢	Motif No.	E-Value	${\mathit{d}}_1$	${\mathit{d}}_2$	${\mathit{d}}_3$
False	1.	$1.4\times {10}^{-56}$	3.65	2.43	2.54
False	2.	$3.8\times {10}^{-44}$	4.76	2.54	4.85
False	3.	$8.4\times {10}^{-38}$	2.87	5.43	4.03
True	1.	$7.6\times {10}^{-45}$	3.63	2.52	2.91
True	2.	$4.7\times {10}^{-34}$	3.45	1.93	4.72
True	3.	$2.7\times {10}^{-32}$	2.97	2.16	2.59

♢ Unknown regex present in data.

and

Table 5. MEME results. Three motifs with the lowest E-value are presented for each dataset. Columns $d_1$, $d_2$, and $d_3$ represent distances from patterns considered in the study, with $d_3$ being the unknown pattern.

♢	Motif No.	E-Value	${\mathit{d}}_1$	${\mathit{d}}_2$	${\mathit{d}}_3$
False	1.	$3.2\times {10}^{-30}$	1.53	4.50	3.91
False	2.	$3.5\times {10}^{-27}$	1.76	2.93	3.34
False	3.	$4.7\times {10}^{-20}$	2.40	4.43	4.87
True	1.	$3.2\times {10}^{-30}$	3.43	3.75	3.91
True	2.	$6.0\times {10}^{-20}$	3.33	2.40	3.45
True	3.	$5.1\times {10}^{-18}$	4.40	2.80	2.92

♢ Unknown regex present in data.

show the results of experiments for DREME and MEME, respectively. As can be observed, both methods perform very well in discovering significant motifs; such a conclusion can be drawn by analysis of the E-value [14], which is responsible for motif significance. On the other hand, distances between the patterns discovered by these methods and the patterns considered in our experiments clearly show that neither of those two baselines can exploit prior knowledge, as the patterns they discover are unrelated to the target motifs. Moreover, both MEME and DREME admit very low pattern mutability, reporting multiple motifs that can be described with one regular expression using our methodology.

4. Discussion

The proposed unknown polymorphic pattern detection method introduces several novel elements. Firstly, we show how prior knowledge of rules, which generate polymorphic patterns, can be incorporated into a network and preserved during training. Another contribution concerns the proposal of measuring a distance between pattern-describing rules by evaluating Levenshtein distances between sequences generated by these rules.

The proposed network architecture is designed to enable the injection of complex knowledge. Furthermore, several constraints introduced to facilitate measurements (e.g., keeping unused parts of filters) can be removed for more general utilization of prior knowledge. It is also worth noting that the presented problem is complex and difficult to be solved by traditional approaches.

As can be seen from the results, the proposed data processing pipeline built upon the introduced methodology can answer whether new, unknown patterns are present in the data. The proposed model is not yet capable of reconstructing the exact pattern. However, it learns a pattern more similar to the original unknown one that the fixed cascades present in the network.

The introduced method’s main contribution and advantage over existing solutions is the ability to utilize prior knowledge about complex polymorphic patterns.

As can be seen from the results, the classical methods, although capable of motif finding, have two severe limitations. They cannot benefit from prior knowledge of regular expressions 1 and 2; so, they waste resources learning them on the same basis as in the case of the unknown pattern 3. Detection of another pattern can only be achieved by further analysis of motifs extracted from the data, which is additionally hindered by the inability to handle patterns’ mutability; again, as can be seen from the comparison, classical methods find multiple motifs whereas the proposed methodology correctly discovers only a single mutable motif. Also, baseline methods are not capable of showing that a discovered pattern is unknown.

The combination of prior knowledge and gradient-based learning is, at its current stage, capable of unknown pattern detection. An obvious direction for future development is modifying the method so that it is also capable of reconstructing the unknown pattern.

The proposed approach still has several limitations that need to be addressed. A matter for future work is generalizing results beyond regular expressions of structure considered in this study. New structures in the convolutional modules would have to be introduced for other regex operators. Also, patterns are not currently allowed to use common knowledge, which is a limitation introduced for experimental purposes; however, the behavior of the network with several constraints relaxed is unknown.

Despite several limitations of the method, it must be emphasized that it achieves the contribution declared at the beginning: it allows for both injection and extraction of knowledge on polymorphic patterns, which is difficult or impossible by means of other methods.