SimMolCC: A Similarity of Automatically Detected Bio-Molecule Clusters between Fluorescent Cells

Abstract

In the field of studies on the “Neural Synapses” in the nervous system, its experts manually (or pseudo-automatically) detect the bio-molecule clusters (e.g., of proteins) in many TIRF (Total Internal Reflection Fluorescence) images of a fluorescent cell and analyze their static/dynamic behaviors. This paper proposes a novel method for the automatic detection of the bio-molecule clusters in a TIRF image of a fluorescent cell and conducts several experiments on its performance, e.g., mAP @ IoU (mean Average Precision @ Intersection over Union) and F1-score @ IoU, as an objective/quantitative means of evaluation. As a result, the best of the proposed methods achieved 0.695 as its mAP @ IoU = 0.5 and 0.250 as its F1-score @ IoU = 0.5 and would have to be improved, especially with respect to its recall @ IoU. But, the proposed method could automatically detect bio-molecule clusters that are not only circular and not always uniform in size, and it can output various histograms and heatmaps for novel deeper analyses of the automatically detected bio-molecule clusters, while the particles detected by the Mosaic Particle Tracker 2D/3D, which is one of the most conventional methods for experts, can be only circular and uniform in size. In addition, this paper defines and validates a novel similarity of automatically detected bio-molecule clusters between fluorescent cells, i.e., SimMolCC, and also shows some examples of SimMolCC-based applications.

1. Introduction

In recent years, AI (Artificial Intelligence) technologies have started to become pervasive/ubiquitous in various situations of the real world (towards Society 5.0 [1], which was proposed by the Cabinet Office, Government of Japan): dialogue systems based on LLMs (Large Language Models) such as OpenAI’s ChatGPT and Google’s Gemini, Text-to-Image generation [2] such as Stable Diffusion and Midjourney, DX (Digital Transformation) in companies, more advanced ITSs (Intelligent Transport Systems) and automated driving, and a diverse array of AI technologies in education (such as EduTech [3]), medical care and nursing care (such as MedTech and SleepTech [4,5]), finance (such as FinTech), clothing, food, and housing [6,7], various forms of entertainment, such as sports [8] and video games [9,10], and so forth.

An Artificial Neural Network (ANN) [11], especially a Deep Neural Network (i.e., Deep Learning), which is playing a starring role in them, is a model for Machine Learning, inspired by the neuronal organization found in the biological neural networks in animal brains. An ANN is composed of connected units called artificial neurons, which loosely model the natural neurons in a brain and receive signals from connected artificial neurons, process the signals, and send signals to other connected artificial neurons. Artificial neurons are connected by edges, which model the “Neural Synapses” in a brain. Therefore, more advancement in Brain Science contributes to more advancement in ANNs and Brain Computing.

In such a field of studies as the “Neural Synapses” in the nervous system, its experts observe and manually (or pseudo-automatically) detect bio-molecule clusters (e.g., of proteins) in many TIRF (Total Internal Reflection Fluorescence) images of a fluorescent cell and analyze their static/dynamic behaviors. Most of the conventional methods for “Bright Spot Analysis” and “Particle Tracking” fit the point spread function of not multiple but a single fluorescent particle to a 2D Gaussian function and apply template matching to an input image [12], e.g., the Mosaic Particle Tracker 2D/3D [13].

As shown in Figure 1, information transmission in the nervous system occurs at the Neural Synapses, which conjugate neuron cells:

• In the presynaptic terminal of a neuron cell, which transmits information, synaptic vesicles are recruited to the release sites at the active zone and release message-carrying chemicals rapidly upon Ca²⁺ influx outside the neuron cell towards its corresponding postsynaptic site. Note that the docked vesicles at the release sites are considered to be the vesicles within the so-called RRP (Readily Releasable Pool) [14];
• In the postsynaptic site of the corresponding neuron cell, which receives information, the released message-carrying chemicals act on postsynaptic receptors and evoke postsynaptic responses.

Figure 1. Direct imaging bio-molecule clusters in a fluorescent cell, e.g., in the presynaptic terminal of a neuron cell, using TIRF (Total Internal Reflection Fluorescence) microscopy and detecting them manually by human experts or automatically by the proposed method.

Figure 1. Direct imaging bio-molecule clusters in a fluorescent cell, e.g., in the presynaptic terminal of a neuron cell, using TIRF (Total Internal Reflection Fluorescence) microscopy and detecting them manually by human experts or automatically by the proposed method.

TIRF microscopy is explained as follows in Chapter 13 [14] of the book whose title is “Exocytosis from molecules to cells”, published by IOP Publishing:

Zenisek et al. [15] pioneered the imaging of single-vesicle dynamics in dissociated goldfish retinal bipolar cells using total internal reflection fluorescence (TIRF) microscopy. TIRF microscopy has very good resolution in the z-axis (50–100 nm), which provides detailed vesicle dynamics near the plasma membrane. On the other hand, the resolutions of the x- and y-axes are diffraction-limited, which means that single vesicles (30–50 nm in diameter) appear as single dots. Sparse labeling of synaptic vesicles with FM1-43 (FM1-43 emits fluorescence when excited by TIRF microscopy’s Evanescent field.) allows one to look at the dynamics of synaptic vesicles before and during fusion. One can see the fusion of synaptic vesicles, which accompanies the loss of dyes in the center and a transient increase in the surrounding fluorescence, reflecting the diffusion of dyes along the plasma membrane.

Moreover, an example of observation (especially direct imaging) by TIRF microscopy of a target phenomenon, e.g., rapid tethering, of synaptic vesicles accompanying exocytosis at a fast central synapse is also shown as follows:

Miki et al. [16] applied TIRF microscopy to cerebellar mossy fiber terminals. They found that the RRP (Readily Releasable Pool) corresponds to those vesicles already resident and ready for fusion upon Ca²⁺ influx. Following depletion of the RRP, vesicles which are within the TIRF field (100 nm) are fused in response to sustained depolarization or a train of action potentials, suggesting that newly replenished vesicles are already close to the membrane. At the same time, vesicles are recruited to the TIRF field more rapidly than they are to the calyx synapse. In addition, newly tethered vesicles can be fused with maturation times of several hundreds of milliseconds, which is much faster than that of the calyx of Held. Therefore, cerebellar mossy fiber terminals have more efficient vesicle recruitment and priming processes than those of the calyx of Held synapse.

For more advances in Brain Science with more advances in ANNs and Brain Computing, in the nervous system, the following features, i.e., properties and static/dynamic behaviors, of bio-molecule clusters in many TIRF images of a fluorescent cell need to be observed and analyzed from various directions.

• Properties: each bio-molecule cluster’s size (e.g., the width and height of its detected Bounding Box), segmented area, shape (e.g., circle/spot-like, narrow, or odd-looking), 2D/3D position (x-, y-, and z-axes), fluorescence intensity, etc.
• Static/dynamic behaviors: each bio-molecule cluster’s change in state, tethering at the active zone, releasing message-carrying chemicals, vanishing from the active zone, moving in a cell, receiving and responding to message-carrying chemicals, and fusing with a membrane or with the other bio-molecule cluster(s), etc.

In addition, there are various methods of analysis, as indicated: temporal analysis, spatial analysis, spatio-temporal analysis, state analysis, similarity analysis (e.g., retrieving similar cells with similar features, or clustering cells based on a similarity between cells), network/community analysis, etc.

However, huge costs, e.g., a long time for manually detecting the bio-molecule clusters in many TIRF images of a fluorescent cell as well as a large sum of money for making them by TIRF microscopy, have been hindering speeding up the advancements in Brain Science, and there are other problems including biased detection and missing some of them. Therefore, as shown in Figure 1, this paper proposes a novel method for the automatic detection of the bio-molecule clusters in a TIRF image of a fluorescent cell to reduce the manual costs and solve the manual problems, and several experiments have been conducted on its performance, e.g., mAP @ IoU (mean Average Precision @ Intersection over Union) and F1-score @ IoU, as an objective/quantitative means of evaluation. The proposed method can automatically detect bio-molecule clusters that are not only circular and not always uniform in size, and it can output various histograms and heatmaps for novel deeper analyses of the automatically detected bio-molecule clusters, while the particles detected by the Mosaic Particle Tracker 2D/3D [13] can be only circular and uniform in size. In addition, this paper defines and validates a novel similarity of automatically detected bio-molecule clusters between fluorescent cells, i.e., SimMolCC, and also shows some examples of SimMolCC-based applications.

The remainder of this paper is organized as follows. Section 2 introduces two kinds of related studies and compares them with this paper. Section 3 describes the novel method in detail for the automatic detection of the bio-molecule clusters in a TIRF image of a fluorescent cell. Section 4 defines a novel similarity of automatically detected bio-molecule clusters between fluorescent cells, i.e., SimMolCC. Section 5 shows several experimental results to validate the proposed method. Finally, Section 6 concludes this paper.

2. Related Work

2.1. Object Detection on Cells

In such a broad field of studies on general-purpose/specific “Computer Vision”, many object detection and segmentation techniques, e.g., YOLO (You Only Look Once) [17], and their practical applications have been proposed [2,18,19,20,21,22]: automatic driving/traffic [23,24,25,26], maritime [27,28], aerial [29], remote sensing [30], agriculture [31,32], and power line infrastructure [33].

Meanwhile, many object detection and segmentation techniques that are not general-purpose but specific to (the region of) cells have also been proposed: from classical techniques [34] based on conditional opening and closing [35], Laplace edge features and SVM (Support Vector Machine) [36], HOG (Histogram of Oriented Gradients) features and SVM [37], SIFT (Scale-Invariant Feature Transform) features, Random Forests, and Hierarchical Clustering [38], or other features [39], to Deep Learning techniques [40,41] such as cellpose [42], Residual U-Net [43,44], which combines U-Net [45] and Residual-Net [46], and R2U-Net [47], which is a Recurrent Residual convolutional neural network based on U-Net.

However, few object detection and segmentation techniques to detect the micro-objects in a cell, e.g., a nucleus in a cell [36,48,49,50,51], and melanin [52,53,54] in a microscopic image of the stratum corneum for skin diagnosis, have been proposed, as shown in

Table 1. Comparison between methods to detect micro-objects in a cell.

	[51]	[52]	This Paper
Targets	Cell Nuclei	FM (Fontana-Masson) Stained Melanin	Bio-Molecule Clusters *1
	in Blue/red-stained Buccal Cells for Liquid Cytology	in Face Epidermal Corneocyte	in a Fluorescent Cell
	1280 × 1024 [pixels]	736 × 440 [pixels]	512 × 512 [pixels]
	21? nm per pixel *2	272 nm per pixel	65 nm per pixel
	TIFF (Full? Color)	BMP (24-bit RGB)	TIFF (16-bit Grayscale)
Input image
Main techs to detect	Sliding Window Method	Template matching	Filtering by thresholds
micro-objects in a cell	Mask-RCNN		Edge extraction
Output image
	red: cell nuclei with a	black: background	black: not bio-molecule clusters
	probability over 0.90	gray: laminated Corneocyte	gray: bio-molecule clusters
	yellow: cell nuclei with a	white: Corneocyte	with a fluorescence intensity
	probability over 0.50	pink: Melanin spots

^*1 The size of 1 target bio-molecule is about 10 nm, observed as 200–300 nm (2D Gaussian, σ = 120–130 nm) because the TIRF’s resolutions of x- and y-axes are diffraction-limited. Therefore, it is applied to Step 4(c) of the proposed method that a target bio-molecule cluster would occupy at least 5 pixels in an input TIRF image. ^*2 Maybe 21 nm per pixel = 271.7·736/300·40/1280, which is not clearly specified in [51] but is estimated from 40× in [51], while 300× in [52].

. This paper proposes novel methods to detect the nano-objects, e.g., bio-molecule clusters (of proteins) in a TIRF (Total Internal Reflection Fluorescence) image of a cell for neuroscience, and to analyze their size and fluorescence intensity, e.g., as various histograms (size/area → frequency, or intensity → frequency) and heatmaps (size/area × intensity → frequency).

2.2. Similarity on Cells

In the field of studies on general-purpose “Image Recognition” and “Content-Based Image Retrieval (CBIR)”, various graphic similarities between images based on their bag of features (i.e., visual words) have been defined [55]. The image features are divided into two kinds: global features, e.g., a color histogram [56], which are extracted by globally describing the features of an image, while the local features, e.g., SIFT (Scale-Invariant Feature Transform) [57], SURF (Speeded Up Robust Features) [58], HOG (Histogram of Oriented Gradients) [37], and LBP (Local Binary Pattern) [56], are extracted by detecting the points of the local features in an image and locally describing the feature for each point. In recent years, image features based on DNNs (Deep Neural Networks) have also been proposed. DELG [59] unifies deep local and global features for Google Landmark Recognition.

Meanwhile, a few similarities between those images that are not general-purpose but specific to cells have been defined. CellSim [60] has been developed as a software of bioinformatics for researchers to calculate the similarity between different cells by the semantic similarity algorithm [61] based on the cell ontology network and cell-specific regulation network in over 2000 different human cell types, e.g.,

Auditory Epithelial Cell; Blood Progenitor Cell; Connective Tissue Cell; Dendritic Cell; Embryo Cell; Epithelial Cell; Epithelial Stem Cell and Muscle Myoblast; Germ Cell; Germ Cell and Spore; Hematopoietic Cell; Keratinocyte Cell; Kidney Cell (part); Kidney Epithelial; Lymphocyte; Macrophage; Marrow Cell; Microfold Cell; Muscle Cell; Myoepithelial Cell; Neurecto-epithelial Cell; Neurogliocyte; Neuron; Neuron Cell; Osteoblast Mesenchymal Stem Cell; Pigment Cell; Secreting Cell; Sensory Epithelial Cell; Somatic Stem Cell; Step Cell (mixed); and Vessel Endothelial,

from FANTOM Ontology [62] and provides the sharing regulation networks of part cells. CellSim can also predict cell types by inputting a list of genes as a query, including more than 250 human normal-tissue-specific cell types and 130 cancer cell types, and provide the prediction results in both tables and spider charts, which can be preserved easily and freely.

The proposed similarity, SimMolCC, in this paper is a graphic similarity between instances of cells by automatically detecting the bio-molecule clusters in an image of a fluorescent cell and describing its global features based on their size/area, fluorescence intensity, ratio of width to height of Bounding Box, and ratio of area to Bounding Box, while CellSim is a semantic similarity between types (i.e., classes in the context of “Object-Orientation”; categories in the context of “Image Categorization”) of cells.

For your information, two kinds of SimCells are not related to this paper: one [63], developed at the Tokyo Institute of Technology, is a processor simulator for multi-core architecture research, and the other [64], developed at the University of Oxford, is a platform for human health – cells made simple.

3. Automatic Detection of Bio-Molecule Clusters in a Fluorescent Cell Image

This section describes in detail novel methods for automatic detection of bio-molecule clusters in a TIRF image of a fluorescent cell.

3.1. Overview

Figure 2 provides an overview of the proposed methods, which have the following input and outputs (as shown in Figure 3), and the following five steps (with fourteen sub-steps):

Input 

is a TIRF image (.tif, unsigned 16-bit grayscale, 512 × 512 [pixels]) of a fluorescent cell.

Outputs 

are bio-molecule clusters in a fluorescent cell, and also their size/area, fluorescence intensity, ratio of area to Bounding Box, and ratio of width to height of Bounding Box. In addition, various histograms (size/area → frequency, etc.) and heatmaps (size/area × intensity → frequency, etc.) can be outputted.

Step 1. 

Segmenting the target cell in an input TIRF image (described in Section 3.2).

Step 1(a). 

Filtering out pixels outside the target cell by an automatically calculated threshold ${\theta }_{\mathrm{step}1\mathrm{a}}$.

Step 1(b). 

Averaging (i.e., filtering out some sort of noise).

Step 2. 

Segmenting and dividing the regions of the target bio-molecule clusters in an input TIRF image of a fluorescent cell (described in Section 3.4).

Step 2(a). 

Filtering out pixels that seem not to be candidates for the target bio-molecule clusters of a fluorescent cell by an automatically calculated threshold ${\theta }_{\mathrm{step}1\mathrm{a}}^{″}$ at Step 1(a).

Step 2(b). 

Laplacian edge extraction with the size of kernel, kernel_size $\in [1,13]$.

Step 2(c). 

Dividing all the regions of the target bio-molecule clusters into each region of bio-molecule cluster.

Step 3. 

Clustering and assigning the regions of the target bio-molecule clusters in an input TIRF image of a fluorescent cell with their ID (described in Section 3.5).

Step 3(a). 

Canny edge extraction for the target cell’s edges and the target bio-molecule clusters’ edges by applying Otsu method [65,66].

Step 3(b). 

Filtering Canny edges out from the target bio-molecule clusters to make them independent.

Step 3(c). 

Clustering and assigning the target bio-molecule clusters with their ID (Identification Data).

Step 3(d). 

Integrating Canny edges filtered out at Step 3(b) back into one of the target bio-molecule clusters.

Step 4. 

Filtering bio-molecule clusters (described in Section 3.6).

Step 4(a). 

Filtering out bio-molecule clusters that do not touch any Canny edges (i.e., any outline of candidates for bio-molecule clusters) in the target cell in an input TIRF image.

Step 4(b). 

Filtering out bio-molecule clusters that touch the Canny edge (i.e., the outline) of the target cell in an input TIRF image.

Step 4(c). 

Filtering out bio-molecule clusters whose size/area is less than 5 pixels.

Step 4(d). 

Filtering bio-molecule clusters based on their fluorescence intensity.

Step 5. 

Calculating the size/area, fluorescence intensity, ratio of area to Bounding Box, and ratio of width to height of Bounding Box of each automatically detected bio-molecule cluster, and also creating various histograms and heatmaps of automatically detected bio-molecule clusters as visualization.

Figure 2. An overview of the proposed method for an input TIRF image of fluorescent cell #6 to automatically detect its bio-molecule clusters by Steps 1 to 4 and to output histograms, heatmaps, and its global feature vector for SimMolCC by Step 5.

Figure 2. An overview of the proposed method for an input TIRF image of fluorescent cell #6 to automatically detect its bio-molecule clusters by Steps 1 to 4 and to output histograms, heatmaps, and its global feature vector for SimMolCC by Step 5.

Figure 3. Comparison of a bio-molecule cluster’s features between the proposed method and Mosaic Particle Tracker 2D/3D [13] for an input TIRF image of fluorescent cell #6.

Figure 3. Comparison of a bio-molecule cluster’s features between the proposed method and Mosaic Particle Tracker 2D/3D [13] for an input TIRF image of fluorescent cell #6.

3.2. Step 1—Cell Segmentation

Step 1 segments the target cell in an input TIRF image by the following two sub-steps as precisely as possible as shown in Figure 4:

Step 1(a). 

First, the histogram of fluorescence intensity of each pixel ∈ 512 × 512 [pixels] in an input TIRF image of a fluorescent cell is calculated as shown in Figure 5, where the number of bins, bins, is set based on the following Sturges’ rule [67]:

$$\begin{array}{c}\hfill \mathrm{Sturges}’ \mathrm{optimal} \mathrm{number} \mathrm{of} \mathrm{bins}=\lceil {log}_{2}N+1\rceil \end{array}$$

(1)

where N means the number of samples for the histogram, e.g., $N=512·512$, at the initial Step 1, and the symbol $\lceil x\rceil$ means “ceiling”, i.e., round the answer x up to the nearest integer. As a result, Sturges’ optimal number of bins is always calculated as 20 at Step 1(a).

Next, peaks are found in the histogram by find_peaks() of signal processing of SciPy [68], and the local minimum between the 1st- and 2nd-highest peaks is also found. For example, in the histogram as shown in Figure 5,

−

the 1st-highest peak’s fluorescence intensity is 202.0.

−

the 2nd-highest peak’s fluorescence intensity ${\theta }_{\mathrm{step}1\mathrm{a}}^{″}$ is 579.7.

−

the fluorescence intensity ${\theta }_{\mathrm{step}1\mathrm{a}}$ whose frequency is the lowest between the 1st- and 2nd-highest peaks is 452.8.

Finally, any pixel of the input TIRF image of a fluorescent cell whose fluorescence intensity is lower than or equal to the above-calculated fluorescence intensity ${\theta }_{\mathrm{step}1\mathrm{a}}$ of the local minimum between the 1st- and 2nd-highest peaks is filtered out as shown in Figure 5.

Step 1(b). 

The filtered TIRF image of a fluorescent cell by the fluorescence intensity ${\theta }_{\mathrm{step}1\mathrm{a}}$ is averaged by the averaging filter, whose size of kernel is set to 19 × 19, and any pixel of the filtered TIRF image of a fluorescent cell whose averaged value is lower than or equal to ${\displaystyle \frac{14·14}{19·19}}\fallingdotseq 0.543$ is filtered out as shown in Figure 4 because some sort of noise, e.g., salt-and-pepper noise, has to be filtered out.

3.3. Step 2—Bio-Molecule Cluster Segmentation

Step 2 segments and divides the regions of the target bio-molecule clusters in an input TIRF image of a fluorescent cell by the following three sub-steps as precisely as possible as shown in Figure 6:

Step 2(a). 

Any pixel of the filtered TIRF image of a fluorescent cell after Step 1(b) whose fluorescence intensity is lower than or equal to the 2nd-highest peak’s fluorescence intensity ${\theta }_{\mathrm{step}1\mathrm{a}}^{″}$ at Step 1(a) is filtered out. The remainder seems to include not-independent candidates for the target bio-molecule clusters of the fluorescent cell.

Step 2(b). 

Laplacian edges are extracted from the filtered TIRF image of a fluorescent cell after Step 1(b) by OpenCV’s Laplacian operator [69], cv2.Laplacian(), which has the size of kernel to be optimized, kernel_size $\in [1,13]$, in this paper.

Step 2(c). 

Laplacian edges are filtered out from the filtered TIRF image of a fluorescent cell after Step 2(a) in order to divide all the regions of candidates for the target bio-molecule clusters of the fluorescent cell into each region of bio-molecule cluster.

Figure 6. Step 2 has three sub-steps, Step 2(a), Step 2(b), and Step 2(c), to segment the regions of bio-molecule clusters in an input TIRF image of fluorescent cell #6 as precisely as possible.

Figure 6. Step 2 has three sub-steps, Step 2(a), Step 2(b), and Step 2(c), to segment the regions of bio-molecule clusters in an input TIRF image of fluorescent cell #6 as precisely as possible.

3.4. Step 3—Bio-Molecule Cluster Clustering

Step 3 divides the regions of the target bio-molecule clusters in an input TIRF image of a fluorescent cell into each region of bio-molecule cluster by the following four sub-steps as precisely as possible as shown in Figure 7:

Step 3(a). 

First, Canny edges are extracted from the filtered TIRF image of a fluorescent cell after Step 1(b) by OpenCV’s Canny [70], cv2.Canny(), which has the first and second thresholds to be optimized. This paper automatically optimizes the two thresholds by applying Otsu method [65,66].

Next, Canny edges are divided into the target cell’s ones or the target bio-molecule clusters’ ones depending on whether or not they touch any pixel outside the target cell, which has already been filtered out and thus whose intensity has already been set to “0 (zero).”

Step 3(b). 

Canny edges are filtered out from the filtered TIRF image of a fluorescent cell after Step 2(c) in order to divide all the regions of candidates for the target bio-molecule clusters of the fluorescent cell into each region of bio-molecule cluster. The remainder seems to include independent candidates for the target bio-molecule clusters of the fluorescent cell.

Step 3(c). 

The regions of the target bio-molecule clusters in the filtered TIRF image of a fluorescent cell after Step 3(b) have “Clustering” applied. As a result, each bio-molecule cluster becomes independent and is assigned the sequential ID (Identification Data); e.g., the number of bio-molecule clusters is calculated as 5507 in Figure 7.

Step 3(d). 

Canny edges filtered out at Step 3(b) are integrated back into one of the target bio-molecule clusters in the filtered TIRF image of a fluorescent cell after Step 3(c). Note that the number of bio-molecule clusters at Step 3(c) and also at Step 3(d), e.g., 5507, seems to be too many. Therefore, the following Step 4 is required.

Figure 7. Step 3 has four sub-steps, Step 3(a), Step 3(b), Step 3(c), and Step 3(d), to divide the regions of bio-molecule clusters in an input TIRF image of fluorescent cell #6 as precisely as possible, and finally each bio-molecule cluster is independent and assigned the sequential ID.

Figure 7. Step 3 has four sub-steps, Step 3(a), Step 3(b), Step 3(c), and Step 3(d), to divide the regions of bio-molecule clusters in an input TIRF image of fluorescent cell #6 as precisely as possible, and finally each bio-molecule cluster is independent and assigned the sequential ID.

3.5. Step 4—Bio-Molecule Cluster Filtering

Step 4 filters bio-molecule clusters by the following four heuristic rules:

Step 4(a). 

“Correct bio-molecule clusters have to have their edge (i.e., outline) in the target cell.” Therefore, Step 4 filters out bio-molecule clusters that do not touch any Canny edges (i.e., any outline of candidates for bio-molecule clusters) in the target cell in an input TIRF image.

Step 4(b). 

“Correct bio-molecule clusters have not to exist in protrusions near the edge (i.e., outline) of the target cell.” Therefore, Step 4 filters out bio-molecule clusters that touch the Canny edge (i.e., the outline) of the target cell in an input TIRF image.

Step 4(c). 

“The size of 1 correct bio-molecule is about 10 nm, observed as 200–300 nm (2D Gaussian, $\sigma =120$–130 nm)” because the TIRF’s resolutions of x- and y-axes are diffraction-limited. Therefore, Step 4 filters out bio-molecule clusters whose area is less than 5 [pixels] in an input TIRF image of a fluorescent cell.

Step 4(d). 

“Correct bio-molecule clusters have to have unusually higher fluorescence intensity in the target cell.” First, the n kinds of sampled histograms of fluorescence intensity of each pixel that has not yet been filtered out and thus whose value has not yet been “0 (zero)” in the filtered TIRF image of a fluorescent cell after Step 4(c) are calculated, where the number of bins, bins, is set based on the Sturges’ optimal number of bins [67] from $+0$ to $+(n-1)$, as shown in Figure 8. Note that the number n of sampled histograms is set to 5 in this paper.

Next, the threshold to filter out bio-molecule clusters that do not have unusually higher fluorescence intensity in the target cell is automatically searched by either of the following two kinds of ways:

1st: 

The threshold flagged as “1st” is set to be the average of the n fluorescence intensities of the bin that first violates “Monotone Decreasing” in each sampled histogram.

3rd: 

The threshold flagged as “3rd” is set to be the average of the n fluorescence intensities of the bin that violates “The difference of frequency (between the bin and the bin followed by it) is not 3rd compared with its pre-difference and its post-difference” in each sampled histogram.

Note that the number of bio-molecule clusters at Step 4(d) (kernel_size $=3$ flagged as “3rd”) is calculated as 237 in Figure 8.

Figure 8. Step 4 has four sub-steps, Step 4(a), Step 4(b), Step 4(c), and Step 4(d), to filter bio-molecule clusters by four kinds of heuristic rules.

Figure 8. Step 4 has four sub-steps, Step 4(a), Step 4(b), Step 4(c), and Step 4(d), to filter bio-molecule clusters by four kinds of heuristic rules.

3.6. Step 5—Visualization

Step 5 can calculate four kinds of features such as the size/area, fluorescence intensity, ratio of area to Bounding Box, and ratio of width to height of Bounding Box of each automatically detected bio-molecule cluster in the target cell of an input TIRF image at each above-mentioned step, and also create various histograms and heatmaps of automatically detected bio-molecule clusters as visualization. For example, Figure 9 shows the four kinds of histograms of the size/area, fluorescence intensity, ratio of area to Bounding Box, and ratio of width to height of Bounding Box of each of the 237 automatically detected bio-molecule clusters in an input TIRF image of a fluorescent cell #6 at Step 4(d) (kernel_size $=3$ flagged as “3rd”), and Figure 10 shows the six kinds of heatmaps between four kinds of features such as the size/area, fluorescence intensity, ratio of area to Bounding Box, and ratio of width to height of Bounding Box of each of the 237 automatically detected bio-molecule clusters in an input TIRF image of a fluorescent cell #6 at Step 4(d) with the size of kernel, kernel_size $=3$, for OpenCV’s Laplacian operator and flagged as “3rd.” These figures could help experts to conduct deeper analyses of bio-molecule clusters in a TIRF image of a fluorescent cell.

4. SimMolCC: Similarity of Automatically Detected Bio-Molecule Clusters between Fluorescent Cells

This section defines a novel similarity of automatically detected bio-molecule clusters between fluorescent cell images, i.e., SimMolCC, as follows:

$$\begin{array}{ccc}\hfill \mathrm{SimMolCC}(im{g}_1,im{g}_2)& :=& \hfill \mathrm{cosine}\text{-}\mathrm{similarity}(\overrightarrow{v_1},\overrightarrow{v_2})\hfill \end{array}$$

(2)

$$\begin{array}{ccc}& :=& \hfill {\displaystyle \frac{\overrightarrow{v_1}·\overrightarrow{v_2}}{\parallel \overrightarrow{v_1}\parallel ·\parallel \overrightarrow{v_2}\parallel }}\hfill \end{array}$$

(3)

where $\overrightarrow{v_1}$, $\overrightarrow{v_2}$ mean each global feature vector extracted from an input TIRF image of a fluorescent cell.

In the following Experiment II, the four kinds of histograms will be adopted as the global feature vector $\overrightarrow{v_i}$ of each input TIRF image of a fluorescent cell. For example,

• area: the histogram of size/area of Figure 9, where $\mathtt{range}=(0,200)$ and $\mathtt{bins}=40$, is converted to the 40-dimensional global feature vector $\overrightarrow{v_i}$ of an input TIRF image of a fluorescent cell #6, $(0,40,25,45,37,23,21,16,4,5,6,2,3,0,1,0,1,1,1,2,1,1,2,0,0,0,\dots )$;
• intensity: the histogram of mean fluorescence intensity of Figure 9, where $\mathtt{range}=(0,4096)$ and $\mathtt{bins}=64$, is converted to the 64-dimensional global feature vector $\overrightarrow{v_i}$ of an input TIRF image of a fluorescent cell #6, (0, …, 0, 97, 65, 24, 19, 11, 10, 2, 3, 2, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, …);
• ratio_area_BB: the histogram of ratio of area to Bounding Box of Figure 9, where $\mathtt{range}=(0.0,1.0)$ and $\mathtt{bins}=50$, is converted to the 50-dimensional global feature vector $\overrightarrow{v_i}$ of an input TIRF image of a fluorescent cell #6, (0, …, 0, 1, 0, 0, 0, 0, 3, 5, 2, 6, 3, 6, 4, 6, 8, 9, 10, 7, 5, 11, 11, 3, 11, 17, 8, 7, 20, 8, 4, 18, 12, 5, 6, 5, 1, 3, 4, 1, 7);
• ratio_width_height: the histogram of ratio of area to Bounding Box of Figure 9, where $\mathtt{range}=(0.0,1.0)$ and $\mathtt{bins}=50$, is converted to the 50-dimensional global feature vector $\overrightarrow{v_i}$ of an input TIRF image of a fluorescent cell #6, (0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 2, 0, 4, 1, 2, 0, 0, 18, 0, 5, 2, 1, 8, 7, 0, 19, 5, 4, 1, 14, 4, 1, 28, 19, 12, 6, 6, 4, 3, 0, 0, 58).

Note that the six kinds of heatmaps and various hybrids with some of the four kinds of histograms and/or some of the six kinds of heatmaps can also be adopted as the global feature vector $\overrightarrow{v_i}$ of each input TIRF image of a fluorescent cell, and note that frequencies of a global feature vector can be converted by the $lo{g}_2\left(\right)$ function, like TF–IDF (Term Frequency–Inverse Document Frequency).

5. Experiments

This section shows the experimental results to validate the two kinds of proposed methods in this paper:

Experiment I 

on automatic detection of bio-molecule clusters in a fluorescent cell image (as described in Section 3).

Experiment II 

on SimMolCC, a similarity of automatically detected bio-molecule clusters between fluorescent cell images (as described in Section 4).

5.1. Datasets

As shown in Figure 11, the dataset, Dataset I, for Experiment I on automatically detected bio-molecule clusters in a fluorescent cell image, has 15 sets of the following data:

• A raw fluorescent cell movie (.tif) consisting of 100 frames (unsigned 16-bit grayscale, 512 × 512 [pixels]).
• An averaged fluorescent cell image (.tif, unsigned 16-bit grayscale, 512 × 512 [pixels]) by Fiji’s Z Projection [71] with “Average Intensity” as the projection type. Note that it is used as an input image to the proposed method for automatic detection of bio-molecule clusters in a fluorescent cell image.
• An averaged fluorescent cell image (.tif, unsigned 24-bit RGB, 512 × 512 [pixels]) with its particles detected by the Mosaic Particle Tracker 2D/3D [13] with the parameters, $\mathtt{radius}=3$ (default), $\mathtt{Cutoff}=0.001$ (default), and Per/Abs (absolute is unchecked and not used. The parameter Per, which means percentile to determine which intense (bright) pixels are accepted as particles, was set to 0.50 (default) or 0.80 resultantly.) optimized manually by the 3rd author. Note that it tends to include noisy particles, e.g., particles outside the target cell and particles in protrusions near the edges of the target cell, and has not yet been able to be adopted as a ground truth for the proposed method for automatic detection of bio-molecule clusters in a fluorescent cell image, and also note that its particles detected by the Mosaic Particle Tracker 2D/3D [13] can be only circular and uniform in size, while the proposed method could automatically detect bio-molecule clusters that are not only circular and not always uniform in size.
• An averaged fluorescent cell image (.tif, unsigned 24-bit RGB, 512 × 512 [pixels]) with its particles filtered manually by the 1st author and checked by the 2nd author. Note that it filtered noisy particles out, e.g., particles outside the target cell and particles in protrusions near the edges of the target cell, as precisely and exhaustively as possible, and has been adopted as a ground truth for the proposed method for automatic detection of bio-molecule clusters in a fluorescent cell image.

Note that plasma membranes of HEK293 cells attached to a coverslip were stained with wheat germ agglutinin lectin conjugated fluorescent dye (CF488 WGA Dye, biotium), and images of the membranes attached to the coverslip were acquired by TIRF (Total Internal Reflection Fluorescence) microscopy.

Figure 11. The averaged image (i.e., an input image for the proposed method), the averaged image with its particles detected by the Mosaic Particle Tracker 2D/3D [13], and the averaged image with its particles filtered manually (i.e., a ground truth for the proposed method) of an input movie (fluorescent cell #6 or #14).

Figure 11. The averaged image (i.e., an input image for the proposed method), the averaged image with its particles detected by the Mosaic Particle Tracker 2D/3D [13], and the averaged image with its particles filtered manually (i.e., a ground truth for the proposed method) of an input movie (fluorescent cell #6 or #14).

The dataset, Dataset II, for Experiment II on SimMolCC, a similarity of automatically detected bio-molecule clusters between fluorescent cell images, has 105 ($={}_{15}C_2$) similarities on bio-molecule clusters between the above-mentioned 15 averaged fluorescent cell images and 15 similarities on bio-molecule clusters between each of the 15 averaged fluorescent cell images and itself; i.e., the latter 15 similarities should be recognized as 100% (perfectly matched) by human subjects. Each similarity of bio-molecule clusters between two averaged fluorescent cell images is 11-grade-evaluated by two of three human subjects: one expert and one candidate for an expert on Cell Physiology at the Faculty of Medicine, Akita University, the former of whom responded “I am very familiar with it and/or an expert.” and the latter of whom responded “I am familiar with it and/or a candidate for an expert.” Meanwhile, the remainder who responded “I am not at all familiar with it.” were filtered out. More specifically, a human subject was randomly offered one of 120 pairs of 15 averaged fluorescent cell images and selected the 11-grade similarity for each pair: from “10: 100% (perfectly similar/matched)” to “0: 0%.” Note that, as a result, two accepted human subjects precisely evaluated “10: 100% (perfectly similar/matched)” for any pair of each of the 15 averaged fluorescent cell images and itself.

5.2. Experiment I

Experiment I shows the experimental results to validate the proposed method for automatic detection of bio-molecule clusters in a fluorescent cell image (as described in Section 3) using Dataset I.

Figure 12 shows the mAP @ IoU and F1-score @ IoU of the proposed method of Step 4(c) and Step 4(d) flagged as “1st” or “3rd” by manually optimizing the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b), and Figure 13 shows each example of automatically detected bio-molecule clusters by the proposed methods of Step 4(c) and Step 4(d) flagged as “1st” or “3rd” of an input image. In addition, Table 2 compares the mAP @ IoU = 0.5 and F1-score @ IoU = 0.5 of the proposed methods of Step 4(c) and Step 4(d) flagged as “1st” or “3rd” for each input of 15 average fluorescent cell images. An analysis of these figures and table provides the following findings:

• The proposed method of Step 4(c) performs not low with respect to F1-score @ IoU, while it performs too low with respect to mAP @ IoU (i.e., precision @ IoU).
• The proposed method of Step 4(d) flagged as “1st” performs the best with respect to mAP @ IoU, while it performs too low with respect to F1-score @ IoU (i.e., recall @ IoU).
• The proposed method of Step 4(d) flagged as “3rd” performs the best with respect to F1-score @ IoU and also performs not low with respect to mAP @ IoU.
• The particles detected by the Mosaic Particle Tracker 2D/3D [13] can be only circular and uniform in size (e.g., $\mathtt{radius}=3$), while the proposed method could automatically detect bio-molecule clusters that are not only circular and not always uniform in size, as shown in Figure 3 and Figure 13.
• F1-score @ IoU of the proposed method is lower than mAP @ IoU. More specifically, the recall @ IoU is worse than the precision @ IoU. It seems to be caused by over-filtering of Step 4(d) and the limitations of Dataset I; e.g., the ground truth is based on the particles detected by the Mosaic Particle Tracker 2D/3D [13], which can be only circular and uniform in size, while the proposed method could automatically detect bio-molecule clusters that are not only circular and not always uniform in size. The future work will make the dataset larger and more ground-true.

Therefore, this paper has concluded that the proposed method of Step 4(d) flagged as “3rd” is the best for automatic detection of bio-molecule clusters in a fluorescent cell image using Dataset I.

Figure 12. The mAP @ IoU and F1-score @ IoU of the proposed methods of Step 4(c) and Step 4(d) flagged as “1st” or “3rd” by manually optimizing the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

Figure 12. The mAP @ IoU and F1-score @ IoU of the proposed methods of Step 4(c) and Step 4(d) flagged as “1st” or “3rd” by manually optimizing the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

Figure 13. The ground truth, Step 4(c), and Step 4(d) flagged as “1st” or “3rd” of an input image (fluorescent cell #6 or #14) for automatic detection of bio-molecule clusters with the size of kernel, $\mathtt{kernel}\_\mathtt{size}=1$, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b). (a) Cell #6; (b) Cell #14.

Figure 13. The ground truth, Step 4(c), and Step 4(d) flagged as “1st” or “3rd” of an input image (fluorescent cell #6 or #14) for automatic detection of bio-molecule clusters with the size of kernel, $\mathtt{kernel}\_\mathtt{size}=1$, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b). (a) Cell #6; (b) Cell #14.

Table 2. The mAP @ IoU = 0.5 and F1-score @ IoU = 0.5 of the proposed methods of Steps 4(c) and 4(d) flagged as “1st” or “3rd” for each input of 15 average fluorescent cell images with the size of kernel, $\mathtt{kernel}\_\mathtt{size}=5$, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

Cell #	Step 4(c)		Step 4(d) w “1st”		Step 4(d) w “3rd”
	mAP	F1-Score	mAP	F1-Score	mAP	F1-Score
Cell #1	0.288	0.186	1.000	0.019	1.000	0.019
Cell #2	0.604	0.284	0.741	0.114	0.698	0.460
Cell #3	0.309	0.140	0.554	0.130	0.554	0.127
Cell #4	0.385	0.166	0.609	0.152	0.491	0.229
Cell #5	0.364	0.261	0.833	0.019	0.506	0.119
Cell #6	0.478	0.135	1.000	0.044	0.584	0.411
Cell #7	0.440	0.177	0.806	0.031	0.747	0.041
Cell #8	0.573	0.256	0.735	0.300	0.641	0.458
Cell #9	0.593	0.216	0.686	0.055	0.695	0.423
Cell #10	0.424	0.192	0.975	0.099	0.921	0.122
Cell #11	0.536	0.283	1.000	0.015	0.639	0.404
Cell #12	0.393	0.233	0.729	0.069	0.729	0.069
Cell #13	0.491	0.337	0.833	0.028	0.618	0.283
Cell #14	0.603	0.300	1.000	0.037	0.899	0.123
Cell #15	0.609	0.415	0.565	0.088	0.698	0.458
Avg. ($\mu$)	0.472	0.239	0.804	0.080	0.695	0.250
SD ($\sigma$)	0.107	0.075	0.157	0.072	0.144	0.165

Table 2. The mAP @ IoU = 0.5 and F1-score @ IoU = 0.5 of the proposed methods of Steps 4(c) and 4(d) flagged as “1st” or “3rd” for each input of 15 average fluorescent cell images with the size of kernel, $\mathtt{kernel}\_\mathtt{size}=5$, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

Cell #	Step 4(c)		Step 4(d) w “1st”		Step 4(d) w “3rd”
	mAP	F1-Score	mAP	F1-Score	mAP	F1-Score
Cell #1	0.288	0.186	1.000	0.019	1.000	0.019
Cell #2	0.604	0.284	0.741	0.114	0.698	0.460
Cell #3	0.309	0.140	0.554	0.130	0.554	0.127
Cell #4	0.385	0.166	0.609	0.152	0.491	0.229
Cell #5	0.364	0.261	0.833	0.019	0.506	0.119
Cell #6	0.478	0.135	1.000	0.044	0.584	0.411
Cell #7	0.440	0.177	0.806	0.031	0.747	0.041
Cell #8	0.573	0.256	0.735	0.300	0.641	0.458
Cell #9	0.593	0.216	0.686	0.055	0.695	0.423
Cell #10	0.424	0.192	0.975	0.099	0.921	0.122
Cell #11	0.536	0.283	1.000	0.015	0.639	0.404
Cell #12	0.393	0.233	0.729	0.069	0.729	0.069
Cell #13	0.491	0.337	0.833	0.028	0.618	0.283
Cell #14	0.603	0.300	1.000	0.037	0.899	0.123
Cell #15	0.609	0.415	0.565	0.088	0.698	0.458
Avg. ($\mu$)	0.472	0.239	0.804	0.080	0.695	0.250
SD ($\sigma$)	0.107	0.075	0.157	0.072	0.144	0.165

Figure 14 shows the dependency of the mAP @ IoU and F1-score @ IoU of the proposed method of Step 4(d) flagged as “3rd” and $n=5$ (set as the default for the number of sampled histograms for Step 4(d) in this paper) on the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b), respectively. An analysis of the figures provides the following findings:

• The dependency of mAP @ IoU on the size of kernel is more stable, while the dependency of F1-score @ IoU on the size of kernel is less stable. More specifically, the dependency of recall @ IoU on the size of kernel is less stable than the dependency of precision @ IoU on the size of kernel. It might be caused by the limitations of Dataset I. The future work will make the dataset larger and more ground-true.
• The curve of mAP over IoU is the best when the size of kernel is set to 1 and the 2nd best when the size of kernel is set to 5, and then, the larger the size of kernel is, the slightly worse the curve of mAP over IoU is.
• The curve of F1-score over IoU is the best when the size of kernel is set to 5 and the 2nd best when the size of kernel is set to 3, and then, the larger the size of kernel is, the worse the curve of F1-score over IoU is.
• Overall, the curves of both mAP over IoU and F1-score over IoU come in a slamming 1st place when the size of kernel is set to 5.

Figure 14. The mAP and F1-score @ IoU of Step 4(d) flagged as “3rd” and $n=5$ depend on the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

Figure 14. The mAP and F1-score @ IoU of Step 4(d) flagged as “3rd” and $n=5$ depend on the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

Figure 15 shows the dependency of the mAP @ IoU and F1-score @ IoU of the proposed methods of Step 4(d) flagged as “3rd” and kernel_size $=5$ (which is manually optimized as the overall finding of the dependency analysis of mAP @ IoU and F1-score @ IoU on the size of kernel) on the number of sampled histograms, n, for Step 4(d), respectively. An analysis of the figures provides the following findings:

• The dependency of mAP @ IoU on the number of sampled histograms is more stable, while the dependency of F1-score @ IoU on the number of sampled histograms is less stable. More specifically, the dependency of recall @ IoU on the number of sampled histograms is less stable than the dependency of precision @ IoU on the number of sampled histograms. This might be caused by the limitations of Dataset I. The future work will make the dataset larger and more ground-true.
• The larger the number of sampled histograms is, the slightly worse the curve of mAP over IoU is. Note that it seems to converge.
• The larger the number of sampled histograms is, the better the curve of F1-score over IoU is. Note that it seems to converge.
• Overall, the curves of both mAP over IoU and F1-score over IoU come in 1st place when the number of sampled histograms is set to 5 as the default in this paper. Note that, the larger the number n of sampled histograms is, the greater the computation time for sampling n kinds of histograms is.

Figure 15. The mAP and F1-score @ IoU of Step 4(d) flagged as “3rd” and kernel_size $=5$ depend on the number of sampled histograms, n, for Step 4(d).

Figure 15. The mAP and F1-score @ IoU of Step 4(d) flagged as “3rd” and kernel_size $=5$ depend on the number of sampled histograms, n, for Step 4(d).

Finally, Figure 16 shows the mAP @ IoU and F1-score @ IoU of the proposed methods of from Step 3(d) and Step 4(c) by manually optimizing the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b). The proposed method of Step 4(c) is superior to its following methods from Step 3(d) to Step 4(b); i.e., Step 4(a) to Step 4(c) as well as Step 4(d) have good effects on automatic detection of bio-molecule clusters in a fluorescent cell image using Dataset I.

Note that the pre-trained models of YOLOv8 [72] on the COCO dataset and ImageNet dataset, which is a state-of-the-art object detection for general purposes, cannot detect any bio-molecule clusters in an input TIRF image of a fluorescent cell. To achieve good performance while avoiding experts’ (i.e., supervisors’) biases, the existing AI technologies based on supervised ML (Machine Learning) or DL (Deep Learning) specific to the practical purpose of this paper need a larger dataset of TIRF images (maybe at least 1000 images) of fluorescent cells and their ground truth of bio-molecule clusters manually annotated by as many experts as possible. This is too expensive and takes too much time. Meanwhile, the proposed method does not need any large dataset for pre-training but only needs heuristics and statistics.

5.3. Experiment II

Experiment II shows the experimental results to validate the proposed SimMolCC, a similarity of automatically detected bio-molecule clusters between fluorescent cells (as described in Section 4) using Dataset II.

It has been finally found that ratio_area_BB at Step 3(d) ($\mathtt{kernel}\_\mathtt{size}=5$) provides the best Pearson Correlation Coefficient with two human subjects’ 11-grade similarity (i.e., ground truth in the Dataset II). Figure 17 compares the Pearson Correlation Coefficient between two human subjects’ 11-grade similarity and the proposed SimMolCC by cosine-similarity between two vectors of input images of a fluorescent cell based on the following 4 kinds features (i.e., histograms) of its automatically detected bio-molecule clusters at Step 3(d), and shows their dependency on the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b):

• area: The histogram of area of each automatically detected bio-molecule cluster, where $\mathtt{range}=(0,200)$ and $\mathtt{bins}=40$.
• intensity: The histogram of mean fluorescence intensity of each automatically detected bio-molecule cluster, where $\mathtt{range}=(0,4096)$ and $\mathtt{bins}=64$.
• ratio_area_BB: the histogram of ratio of area to Bounding Box of each automatically detected bio-molecule cluster, where $\mathtt{range}=(0.0,1.0)$ and $\mathtt{bins}=50$.
• ratio_width_height: the histogram of ratio of width to height or ratio of height to width, whichever is smaller, of Bounding Box of each automatically detected bio-molecule cluster, where $\mathtt{range}=(0.0,1.0)$ and $\mathtt{bins}=50$.

Figure 17 also compares the Pearson Correlation Coefficient between two subjects’ 11-grade similarity and the proposed SimMolCC by cosine-similarity between two vectors of input images of a fluorescent cell based on ratio_area_BB at Step 3(d), Step 4(c), and Step 4(d) (3rd), and shows their dependency on the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

An analysis of the figures has found the following:

• ratio_area_BB at Step 3(d) ($\mathtt{kernel}\_\mathtt{size}=5$) provides the best Pearson Correlation Coefficient with two human subjects’ similarity (i.e., ground truth in the Dataset II) and could help experts to conduct deeper analyses of bio-molecule clusters in a TIRF image of a fluorescent cell as their global features (not local features).
• ratio_area_BB (and ratio_width_height) of our proposed SimMolCC can represent the “shape” of each automatically detected bio-molecule cluster with not a uniform size, while Mosaic Particle Tracker 2D/3D [13], which is one of the most conventional methods for experts, can detect only circular one with a uniform size (e.g., $\mathtt{radius}=3$ of the target particles, meaning that the area is uniformly 29 [pixels]).
• Meanwhile, intensity provides too low Pearson Correlation Coefficient with two human subjects’ similarity, independent of the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b).

Figure 17. The Pearson Correlation Coefficient between two human subjects’ 11-grade similarity and the proposed similarity, SimMolCC, depends on the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b). (a) A comparison between histograms when Step 3(d) is constantly adopted. (b) A comparison between steps when ratio_area_BB is constantly adopted.

Figure 17. The Pearson Correlation Coefficient between two human subjects’ 11-grade similarity and the proposed similarity, SimMolCC, depends on the size of kernel, kernel_size, for OpenCV’s Laplacian operator [69], cv2.Laplacian(), at Step 2(b). (a) A comparison between histograms when Step 3(d) is constantly adopted. (b) A comparison between steps when ratio_area_BB is constantly adopted.

Figure 18 shows the scatter plot of two human subjects’ 11-grade similarity and our proposed SimMolCC by cosine-similarity between two vectors of input images of a fluorescent cell based on ratio_area_BB at Step 3(d) ($\mathtt{kernel}\_\mathtt{size}=5$) for each of 105 ($={}_{15}C_2$) pairs between the above-mentioned 15 averaged fluorescent cell images, and also shows the scatter plot of two human subjects’ similarity and the converted SimMolCC′ from our proposed SimMolCC by the following formula:

$$\begin{array}{c}\hfill {\mathrm{SimMolCC}}^{\prime }(im{g}_1,im{g}_2):={\displaystyle \frac{1}{10}}·(563.18·\mathrm{SimMolCC}(im{g}_1,im{g}_2)-554.67)\end{array}$$

(4)

where $y=563.18·x-554.67$ has been obtained by simple linear regression from SimMolCC (as x) for two human subjects’ 11-grade similarity (as y).

Finally, Figure 19 shows an example result of similarity-based retrieving (ranking) by inputting a TIRF image (fluorescent cell #14) as a query and calculating its SimMolCC’ with the other 14 TIRF images. The ranking based on the converted SimMolCC’ from the proposed SimMolCC by simple linear regression has achieved similar results as the ranking based on two human subjects’ 11-grade similarity.

6. Conclusions

In the field of studies on the “Neural Synapses” in the nervous system, its experts manually (or pseudo-automatically) detect bio-molecule clusters (e.g., of proteins) in many TIRF (Total Internal Reflection Fluorescence) images of a fluorescent cell and analyze their static/dynamic behaviors. This paper has proposed a novel method for the automatic detection of the bio-molecule clusters in a TIRF image of a fluorescent cell and conducted several experiments on its performance, e.g., mAP @ IoU (mean Average Precision @ Intersection over Union) and F1-score @ IoU, as an objective/quantitative means of evaluation. As a result, the best of the proposed methods has achieved 0.695 as its mAP @ IoU = 0.5 and 0.250 as its F1-score @ IoU = 0.5 and would have to be improved, especially with respect to its recall @ IoU. But, the proposed method could automatically detect bio-molecule clusters that are not only circular and not always uniform in size, and can output various histograms and heatmaps for novel deeper analyses of the automatically detected bio-molecule clusters, while the particles detected by the Mosaic Particle Tracker 2D/3D [13], which is one of the most conventional methods for experts, can be only circular and uniform in size.

In addition, this paper has defined and validates a novel similarity of automatically detected bio-molecule clusters between fluorescent cells, i.e., SimMolCC. As a result, the best of the proposed methods has achieved 0.518 (p-value $<0.001$, statistically significant [73,74]) as its Pearson Correlation Coefficient with two human subjects’ 11-grade similarity, which would have to be improved in the future. But, the findings include that the histogram of the ratio of area to Bounding Box, ratio_area_BB, of each automatically detected bio-molecule cluster is superior to the histogram of its intensity as its global features help experts to conduct deeper analyses of the bio-molecule clusters in a TIRF image of a fluorescent cell; i.e., the “shape” of each automatically detected bio-molecule cluster with a non-uniform size plays an important role in novel deeper analyses by experts.

In the near future, the implemented tools with the proposed method will be developed for experts and applied in various studies on “Neural Synapses” for more advances in both Brain Science and Artificial Neural Networks. In addition, the future work includes validating the other definitions of SimMolCC based on the six kinds of heatmaps and also various hybrids with some of the four kinds of histograms, e.g., a hybrid of ratio_area_BB at Step 3(d) with kernel_size $=5$ and ratio_width_height at Step 3(d) with kernel_size $=13$, and/or some of the six kinds of heatmaps, such as the global feature vector $\overrightarrow{v_i}$ of each input TIRF image of a fluorescent cell, with or without converting their frequencies of a global feature vector $\overrightarrow{v_i}$ by the $lo{g}_2\left(\right)$ function, like TF–IDF (Term Frequency–Inverse Document Frequency).