This section provides a background highlighting the collection and analysis of gamma radiation measurements, including the dataset used here and data augmentations that can be used for radiation spectra. The mathematical foundation for semi-supervised and contrastive self-supervised machine learning underpinning the analytical method used in this work is also introduced. A generic framework is presented which will be used in designing a contrastive machine-learning model for identifying nuclear material transfers from radiation spectra. A derivation follows the contrastive loss function used to optimize a representation model’s output. Finally, alternative contrastive machine methods and models are discussed.
2.1. Spectral Data Augmentations
The use of (randomly applied) data augmentations expands a finite training dataset and helps a contrastive model learn semantic labeling information that can be used in downstream detection tasks. Crucially, each augmentation in a set of transformations must be label-invariant, meaning augmented spectra preserve labeling information shared with their parent spectrum. If an augmentation is too extreme, labeling information might be obscured. Too light of an augmentation and the new spectrum is indistinguishable from its parent. Either scenario would limit contrastive learning for meaningful representations.
When radioactive nuclei decay, they can produce electromagnetic gamma radiation with discrete energy. When measured with gamma detectors (e.g., sodium iodide (NaI)), a spectrum of counts per energy level is constructed. Due to various interactions with the detector, measured gamma radiation will produce several unique patterns in a spectrum, including a Gaussian-shaped photopeak centered at the discrete energy levels. For a more detailed description of nuclear physics or gamma detection, refer to textbooks written by Kenneth Krane [
17] or Glen Knoll [
18], respectively. High-fidelity simulation of radiation detection is possible, but generating enough synthetic data for machine learning is computationally cost-prohibitive. Simulation is further complicated by the various configurations of materials, detectors, shielding geometry, etc. Instead, the focus here is on efficiently extracting meaningful information from real gamma radiation measurements (labeled and unlabeled) collected at a testbed facility. Whereas most contrastive learning is conducted on images (e.g., SimCLR) or text, where valid transformations are well studied, this work uses radiation spectra. An established set of radiation data augmentations—designed and detailed in prior work [
8]—maintain spectral label-invariance and are highlighted in
Figure 1.
Nuclear materials of interest to national security are radioactive, meaning they could potentially be measured by a radiation detector. Rapidly identifying these materials (e.g., with machine learning) when they are being transported has a variety of valuable applications, from enabling safety and security for nuclear operators to ensuring monitoring and safeguards for nuclear nonproliferation. Previous work [
11] highlighted how SSML could be used to determine—given a measured spectrum—whether a transfer event occurred. Here, given a material transfer spectrum, the classification task is to distinguish what type of material was transferred.
Each augmentation relies on manipulating combinations of decomposed components of the spectrum. Assume that an individual spectrum (binned as a function of energy) can be decomposed into three constituent parts:
where a spectrum (
) consists of Gaussian-shaped signals (
) specifically associated with various radioactive sources present during measurement, a slowly varying and characteristic background distribution (
), and Poissonian noise (
). The spectra in this analysis consist of vectors with 1000 components corresponding to the 1000 energy bins (3 keV each) collected during measurements. Each augmentation manipulates either one or more of a spectrum’s underlying components or an individual feature in that component.
It is necessary for some augmentations to first estimate the background distribution (
) before transforming the spectrum. This can be difficult to manually extract—even for a subject-matter expert—because background radiation is highly dependent on environmental factors that are unique to every measurement. For example, in a given region of interest (
) where one signature is present, a common procedure is to fit a Gaussian peak with a linear trend
where the additional fit parameters are the peak’s amplitude (
A), the peak’s standard deviation (
, related to the peak’s width), the peak’s centroid (
), the background’s linear slope (
m), and the background’s zero-point (
c). This assumes a region of interest sufficiently small enough to capture the peak and its shoulders, where the present background is approximately linear. Equation (2) omits the noise associated with counting statistics for detection setups (
) because modeling this noise could result in overfitting if successful. This fit procedure is used below when estimating signal peak parameters for manipulation.
Instead of measuring a region of interest’s fit parameters directly, an entire spectrum could be decomposed using an algorithm for Baseline Estimation and Denoising using Sparsity (BEADS) [
19]. BEADS was originally developed for chromatograph spectra and decomposes the spectrum using low- and high-pass operators with tunable hyperparameters. A simulated gamma spectrum from a sodium iodide detector was generated with known components of signal, baseline, and noise. The hyperparameters were then tuned to match the BEADS decomposed portions to the known distributions. Refer to
Table 1 for the BEADS hyperparameters used in this work when augmenting spectra (Hyperparameter optimization and the values presented in
Table 1 were provided by an experienced BEADS user.) Augmentations are then applied to one or a combination of the spectrum components: signals (
), background (
), or noise (
).
2.2. MINOS Data and Detection Task
For this analysis, gamma radiation spectra collected at the Multi-Informatics for Nuclear Operating Scenarios (MINOS) testbed are used. The MINOS testbed at Oak Ridge National Laboratory (ORNL) collects multimodal data streams relevant to nuclear nonproliferation. This is accomplished using a network of mature sensor technologies distributed around two points of interest at ORNL: the Radiochemical Engineering Development Center (REDC) and High Flux Isotope Reactor (HFIR). The reactor facility (HFIR) is used for scientific experiments (e.g., neutron scattering) and isotope production. Materials generated at HFIR are loaded into shielded casks and are transferred by a variety of methods (e.g., flatbed truck) to REDC. Once at REDC, materials are unloaded, stored, and processed in hot cells.
Material transportation can occur along several routes between HFIR and REDC. Nodes at the MINOS testbed are distributed beside the road along the possible routes. Radiation data are collected using NaI detectors capable of measuring gamma radiation emitted from nearby sources. Assume that all measured spectra have been measured for a 1-min integration time and undergone an energy calibration binned in 1000 channels of 3 keV per bin. A material transfer could occur on the order of tens of seconds or longer. A 1-min integration time ensures that the spectrum has enough statistics for conducting augmentations without being so long as to dominate transfer spectra with background radiation.
A breakdown of all data, unlabeled data, and labeled data can be seen in
Figure 2. At a 1-min integration time—with no temporal overlap in integration—there are approximately 9.5 million uncharacterized spectra collected over six nodes and 3 years at the testbed. Since most of the spectra measured over time will be of background, down-sampling will provide a training set more reflective of nuclear material transfers. A spectrum with a weakly anomalous, elevated count rate may be a transfer event or some other type of anomalous environmental radioactive event. Whereas there may be too many weakly anomalous spectra to arbitrate manually, SSML methods can further isolate events of interest, e.g., transfers [
11].
A training subset is collected using the hypothesis testing anomaly detection algorithm detailed in previous work [
11]. Briefly, assuming total counts are
, an anomalous (e.g., transfer) spectrum will have sufficiently different counts to be sampled from a different distribution than consecutive measurements. Radiation counts measured by a gamma detector are governed by a Poisson distribution because gamma radiation emitted from radioactive isotopes are independent events, with a specific probability of decay governed by the half-life of any given decay mechanism. With this discrete likelihood of decay occurring, the measurement is collected at an average interval. The test is then to accept or reject the hypothesis that two consecutive measurements are sampled from the same underlying distribution
using a binomial test
Here, a larger p-value captures more anomalous spectra by applying a looser threshold to changes in gross count rate over time. A p-value of yields 68,124 weakly anomalous spectra. This subset will include some unknown number of transfers, background, and other anomalous event spectra. An even larger p-value could be used, but as p increases, more and more of the entire dataset is captured as anomalous. For example, captures 252,854 spectra, but the increase from only includes spectra that are less anomalous, and therefore, less likely, but not impossible, to be from material transfers. Initial contrastive models that were trained and tested indicated no added benefit to classification accuracy by using over . Likewise, a smaller p-value would capture fewer anomalies, and therefore, have less unlabeled data to contrastively train from. Therefore, produces a large enough training dataset for contrastive learning while limiting the number of included background radiation spectra.
Seven types of transfers occur in the MINOS dataset that can roughly be subdivided into two mutually exclusive classes (see
Table 2 [
20]). To simulate nuclear nonproliferation scenarios, there are byproducts of the nuclear fuel cycle that may be radioactive or have alternative applications but are not necessarily materials of interest. Contrast these with nuclear material that might be tracked or monitored for proliferation resistance. To demonstrate the utility of contrastive machine learning, the model attempts to distinguish between these two classes. Certainly, many other nonproliferation detection tasks could be designed from this dataset (e.g., multiclass classification for identifying each type of material transfer).
A smaller subset of labeled data was prepared for training a supervised classifier. Labeled data were independently prepared and shared by a subject-matter expert, and cross-referenced with ground truth transfer records. In total, 261 spectra spanning the seven transfer subcategories were labeled. Timestamps are not available for these spectra, and the degree of overlap with the unlabeled subset is unknown. Crucially, some spectra may not qualitatively exhibit characteristic features (e.g., photopeaks) associated with their labeled transfer type. This can occur when ground truth data specifies a transfer, but various factors (e.g., background) obfuscate features. A contrastive learning framework and classifier may be able to classify these spectra if it is sensitive enough to transfer type, or these spectra may result in misclassifications or out-of-distribution behavior, resulting in limited classification performance.
2.3. Semi-Supervised Machine Learning
Consider the following dataset of observations (for this analysis, radiation spectra):
consisting of two finite subsets: labeled data
and unlabeled data
. For a given observable (
), its assigned label describes some environmental phenomenon captured by its measurement among a finite set of classes:
. The goal of machine learning, therefore, is to construct a model that approximates the relationship between
x and
y such that labels could be estimated for an unseen measurement:
An optimal model can be found using empirical risk minimization:
Using some form of optimization, a finite number of models can be tested. For supervised models,
M is presumably a sufficiently large set such that only labeled data are needed for training (and thus only the supervised loss term is used). In many domains,
, making it necessary to incorporate unlabeled data in training. Assuming some necessary [
21] relationship between labels and the unlabeled data distribution (e.g., the cluster assumption, manifold assumption, etc. [
22]), SSML can successfully learn decision boundaries and outperform supervised learning when labeled data are scarce and classes are sufficiently distinguishable [
23].
2.4. Contrastive Machine Learning
Contrastive learning, or self-supervision more broadly, is a set of SSML methods that considers the similarity between data instances. Like other graph-based methods (e.g., Label Propagation [
24]), contrastive machine learning uses a notion of distances for making label predictions.
Consider the framework in
Figure 3, based on SimCLR [
2]. Each unlabeled (or labeled) spectrum,
s, in a batch, is augmented twice using two random transformations, with replacement, from the set of transformations (i.e.,
). The two transformations randomly generate two augmented spectra,
. The representation model,
, takes each augmented spectrum as input and outputs a higher dimensional representation.
These representations become the input of a projection head, , which reduces the dimensionality (thereby distilling significant features or patterns in representations) to an output that is directly used in computing and optimizing a contrastive loss. The reduced dimensionality of projected representations mitigates the curse of dimensionality associated with Euclidean and other distance metrics like cosine similarity. However, this implies that no two representations will appear similar when dimensionality is large.
The projection head also helps to manage noise from data augmentations and encourages pairwise feature independence in resulting representations [
25]. An augmentation applied too severely might overwrite labeling information in a spectrum, therefore, breaking label invariance. In this case, it would be inappropriate (or at least suboptimal) to encourage representation similarity. Distilling a representation to a projection dimension dilutes the effect of this scenario by potentially acting as another barrier to overfitting. Finally, pairwise feature independence is valuable for preventing the collapse of output representations to a trivial solution [
26].
Once the encoder has been trained, the projection head can be discarded, and a classification model can be used to predict labels (here, byproduct or tracked nuclear material) given a spectral representation. Discarding the projection head for classification tasks has been shown to be beneficial for transfer learning by avoiding the model overfitting from the training [
25]. That is, the upstream task (contrastive learning of fundamental patterns in radiation spectra) differs from the downstream task (classifying nuclear material transfers) and pruning helps align the representation model to the downstream task.
Each , created by transforming s, should both have the same label because they contain the same intrinsic labeling information, assuming the transformation is label invariant. The optimization task of contrastive learning is then to ensure that representations are similar for positive pairs. These pairs could be the original spectrum, s, and its transformed value, , or two augmented instances, , from the same source instance. Negative pairs from dissimilar sources should be maximally dissimilar. Therefore, the model contrasts between similar, augmented views of samples—which implies information about an instance’s class—and dissimilar samples in a batch. Note, that the true or observed label is not necessary for contrastive learning (although labeled data could theoretically be used to regularize representations with known labels).
2.5. Alternative Contrastive Methods
Several contrastive and self-supervised models have already been proposed, mostly used on visual modalities. SimCLR [
2] does not directly use the original sample image (here, a spectrum) but rather trains a model using only the augmented pairs generated from the image. Alternatively, a bag of visual words representation [
27] could be constructed, comparing the original image to an augmented counterpart. The
BARLOW TWINS method [
4] tunes the representation model by optimizing the cross-correlation in output for augmented samples. One student–teacher method [
5] trains a smaller distillation model to effectively generalize. Others apply a stop-gradient [
6] so that optimization only occurs along one representation of the input sample. One implementation [
7] includes spectral clustering to avoid a requirement for conditional independence and more closely consider class similarity.
Empirical results [
10] suggest that self-supervised learning frameworks like contrastive learning can be robust to the class imbalance in select data streams. This is likely because the representations learned are strong enough to discriminate between different types of class distributions. Even though the experiments described above conduct binary classification as an ultimate downstream target task, these models would be beneficial for multinomial classification. Special nuclear material transfers tend to be rare in the MINOS data and potentially nonexistent for certain real-world monitoring applications. If so, a model needs to be tuned and prepared for severe class imbalance to the point of anomaly detection.
Contrastive learning, and self-supervised machine learning broadly, do not intrinsically require labeled data to optimize a model. Contrastive loss functions typically maximize agreement between positive pairs, where positive pairs are defined either as two augmentations of the same data instance or one augmented instance compared with the original. Optimization in this way can be successful if the augmentations—given a known set of transformations—are not severe enough to obscure labeling information present in the unaltered observation. If labeled data are available, fine-tuning can be used to “check answers” in model training or more importantly define the number of classes possible in the dataset. Semi-supervision has an advantage over supervised learning because it can handle unlabeled data and extrapolate labels via known classes.
How else could labeled data be used in self-supervised learning? Typical SSML models are rooted in supervised methods with consistency enforcing functions added to the loss term. Self-supervision is almost the opposite, reminiscent of unsupervised methods. One such example trains a larger model contrastively and then prunes the model, or makes it smaller, by a process of supervised fine-tuning [
28]. Another example, supervised contrastive learning [
29], uses labeled data as anchor points for the broader data distribution, further defining positive pairs and adding generalization weight to contrastive learning. Finally, another example uses the similarity between samples to gradually group classes: weakly supervised contrastive learning [
30].
2.6. Normalized Cross-Entropy Loss
The loss function used in SimCLR is a generalization of N-Pairs Loss, dubbed the normalized temperature-scaled cross-entropy loss (or
NT-XENT). Elsewhere, this loss function is titled
INFO NCE [
31,
32]. Given a set of data augmentations, samples are fed through a base encoder, which produces a representation. There are no constraints on this encoder in SimCLR, varying strategies include producing higher-dimensional representations, an encoder/decoder pair, convolution followed by average pooling, etc., SimCLR observed that attaching a projecting head consisting of a single hidden layer with a ReLU activation function is beneficial for applying and optimizing contrastive loss.
The projection head reduces the dimensionality of the representations to what is fed to the contrastive loss function. The training algorithm for SimCLR is as follows: For a given batch of batch size N, produce two augmentations for each, resulting in (augmented) samples. Each pair of augmented samples produces a pair of encoded representations, , derived from the same sample (), which are considered a positive pair; the other samples are assumed as negatives.
For two nonzero vectors, where
is the angle between the two vectors, their cosine similarity is
which varies in the range
. This serves as a distance metric, or difference measure, between the two vectors (i.e., two input samples).
NT-XENT loss is then the cross-entropy of cosine similarities for the projected representations of positive pairs
, normalized over all pairs
The temperature coefficient,
, acts as a fixed hyperparameter scaling the magnitude of similarities. Note, that the denominator sums over the first input,
, paired with all other samples but itself. If, for example,
, then their cosine similarity will be 1, and (assuming
) the first term of Equation (9) will be
. If
, then their cosine similarity will be 0, and the first term of Equation (9) will be
. In practice, many negative pairs will not be exactly orthogonal to the anchor point (i.e., the first input), and therefore, the cosine similarity in the second term will be nonzero, meaning the minimum might not be precisely
even if the positive pair is identical. However, the additional loss for negative pairs encourages them to be maximally dissimilar, i.e., assuming
, an optimal solution would have
for positive pairs and
for negative pairs. Therefore, minimizing the loss function achieves two goals: maximizing similarity between positive pairs and minimizing similarity between negative pairs. Finally, the loss is averaged for all positive pairs in each direction:
Even though cosine similarity is a symmetric distance metric (i.e., , because the denominator of Equation (9) will change depending on the anchor point of , resulting in both terms being computed in Equation (10).
In summary, the loss function optimizes the encoder to maximize similarity in output representations for augmented spectra derived from the same original spectrum and maximize dissimilarity for all other representations in a batch. Note, that this process does not strictly require knowledge of the true labels for augmented spectra and is, therefore, nominally a self-supervised method. If the augmentations used are label-invariant, then a positive pair of augmented samples are assumed to have the same underlying label as that of their parent, regardless of what that label may be. After successfully training a contrastive machine learning model, the representations of spectra sharing the same label (whatever that label is) should be clustered in the encoder’s state space (i.e., have minimal notional distance).