Research

14 pages, 1706 KiB

Open AccessArticle

Critical Dynamics Mediate Learning of New Distributed Memory Representations in Neuronal Networks

by Quinton M. Skilling, Nicolette Ognjanovski, Sara J. Aton and Michal Zochowski

Entropy 2019, 21(11), 1043; https://doi.org/10.3390/e21111043 - 26 Oct 2019

Cited by 5 | Viewed by 2960

We explore the possible role of network dynamics near a critical point in the storage of new information in silico and in vivo, and show that learning and memory may rely on neuronal network features mediated by the vicinity of criticality. Using a [...] Read more.

We explore the possible role of network dynamics near a critical point in the storage of new information in silico and in vivo, and show that learning and memory may rely on neuronal network features mediated by the vicinity of criticality. Using a mean-field, attractor-based model, we show that new information can be consolidated into attractors through state-based learning in a dynamical regime associated with maximal susceptibility at the critical point. Then, we predict that the subsequent consolidation process results in a shift from critical to sub-critical dynamics to fully encapsulate the new information. We go on to corroborate these findings using analysis of rodent hippocampal CA1 activity during contextual fear memory (CFM) consolidation. We show that the dynamical state of the CA1 network is inherently poised near criticality, but the network also undergoes a shift towards sub-critical dynamics due to successful consolidation of the CFM. Based on these findings, we propose that dynamical features associated with criticality may be universally necessary for storing new memories. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

12 pages, 574 KiB

Open AccessArticle

Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

by Giorgio Gosti, Viola Folli, Marco Leonetti and Giancarlo Ruocco

Entropy 2019, 21(8), 726; https://doi.org/10.3390/e21080726 - 25 Jul 2019

Cited by 21 | Viewed by 4568

Abstract

In a neural network, an autapse is a particular kind of synapse that links a neuron onto itself. Autapses are almost always not allowed neither in artificial nor in biological neural networks. Moreover, redundant or similar stored states tend to interact destructively. This [...] Read more.

In a neural network, an autapse is a particular kind of synapse that links a neuron onto itself. Autapses are almost always not allowed neither in artificial nor in biological neural networks. Moreover, redundant or similar stored states tend to interact destructively. This paper shows how autapses together with stable state redundancy can improve the storage capacity of a recurrent neural network. Recent research shows how, in an N-node Hopfield neural network with autapses, the number of stored patterns (P) is not limited to the well known bound

0.14 N

, as it is for networks without autapses. More precisely, it describes how, as the number of stored patterns increases well over the

0.14 N

threshold, for P much greater than N, the retrieval error asymptotically approaches a value below the unit. Consequently, the reduction of retrieval errors allows a number of stored memories, which largely exceeds what was previously considered possible. Unfortunately, soon after, new results showed that, in the thermodynamic limit, given a network with autapses in this high-storage regime, the basin of attraction of the stored memories shrinks to a single state. This means that, for each stable state associated with a stored memory, even a single bit error in the initial pattern would lead the system to a stationary state associated with a different memory state. This thus limits the potential use of this kind of Hopfield network as an associative memory. This paper presents a strategy to overcome this limitation by improving the error correcting characteristics of the Hopfield neural network. The proposed strategy allows us to form what we call an absorbing-neighborhood of state surrounding each stored memory. An absorbing-neighborhood is a set defined by a Hamming distance surrounding a network state, which is an absorbing because, in the long-time limit, states inside it are absorbed by stable states in the set. We show that this strategy allows the network to store an exponential number of memory patterns, each surrounded with an absorbing-neighborhood with an exponentially growing size. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

28 pages, 1733 KiB

Open AccessArticle

Stationary-State Statistics of a Binary Neural Network Model with Quenched Disorder

by Diego Fasoli and Stefano Panzeri

Entropy 2019, 21(7), 630; https://doi.org/10.3390/e21070630 - 26 Jun 2019

Cited by 3 | Viewed by 2817

Abstract

In this paper, we study the statistical properties of the stationary firing-rate states of a neural network model with quenched disorder. The model has arbitrary size, discrete-time evolution equations and binary firing rates, while the topology and the strength of the synaptic connections [...] Read more.

In this paper, we study the statistical properties of the stationary firing-rate states of a neural network model with quenched disorder. The model has arbitrary size, discrete-time evolution equations and binary firing rates, while the topology and the strength of the synaptic connections are randomly generated from known, generally arbitrary, probability distributions. We derived semi-analytical expressions of the occurrence probability of the stationary states and the mean multistability diagram of the model, in terms of the distribution of the synaptic connections and of the external stimuli to the network. Our calculations rely on the probability distribution of the bifurcation points of the stationary states with respect to the external stimuli, calculated in terms of the permanent of special matrices using extreme value theory. While our semi-analytical expressions are exact for any size of the network and for any distribution of the synaptic connections, we focus our study on networks made of several populations, that we term “statistically homogeneous” to indicate that the probability distribution of their connections depends only on the pre- and post-synaptic population indexes, and not on the individual synaptic pair indexes. In this specific case, we calculated analytically the permanent, obtaining a compact formula that outperforms of several orders of magnitude the Balasubramanian-Bax-Franklin-Glynn algorithm. To conclude, by applying the Fisher-Tippett-Gnedenko theorem, we derived asymptotic expressions of the stationary-state statistics of multi-population networks in the large-network-size limit, in terms of the Gumbel (double exponential) distribution. We also provide a Python implementation of our formulas and some examples of the results generated by the code. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

12 pages, 331 KiB

Open AccessArticle

Stability Analysis for Memristor-Based Complex-Valued Neural Networks with Time Delays

by Ping Hou, Jun Hu, Jie Gao and Peican Zhu

Entropy 2019, 21(2), 120; https://doi.org/10.3390/e21020120 - 28 Jan 2019

Cited by 1 | Viewed by 2410

Abstract

In this paper, the problem of stability analysis for memristor-based complex-valued neural networks (MCVNNs) with time-varying delays is investigated extensively. This paper focuses on the exponential stability of the MCVNNs with time-varying delays. By means of the Brouwer’s fixed-point theorem and M-matrix, [...] Read more.

In this paper, the problem of stability analysis for memristor-based complex-valued neural networks (MCVNNs) with time-varying delays is investigated extensively. This paper focuses on the exponential stability of the MCVNNs with time-varying delays. By means of the Brouwer’s fixed-point theorem and M-matrix, the existence, uniqueness, and exponential stability of the equilibrium point for MCVNNs are studied, and several sufficient conditions are obtained. In particular, these results can be applied to general MCVNNs whether the activation functions could be explicitly described by dividing into two parts of the real parts and imaginary parts or not. Two numerical simulation examples are provided to illustrate the effectiveness of the theoretical results. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

20 pages, 874 KiB

Open AccessArticle

Modeling of a Neural System Based on Statistical Mechanics

by Myoung Won Cho and Moo Young Choi

Entropy 2018, 20(11), 848; https://doi.org/10.3390/e20110848 - 05 Nov 2018

Cited by 1 | Viewed by 3102

Abstract

The minimization of a free energy is often regarded as the key principle in understanding how the brain works and how the brain structure forms. In particular, a statistical-mechanics-based neural network model is expected to allow one to interpret many aspects of the [...] Read more.

The minimization of a free energy is often regarded as the key principle in understanding how the brain works and how the brain structure forms. In particular, a statistical-mechanics-based neural network model is expected to allow one to interpret many aspects of the neural firing and learning processes in terms of general concepts and mechanisms in statistical physics. Nevertheless, the definition of the free energy in a neural system is usually an intricate problem without an evident solution. After the pioneering work by Hopfield, several statistical-mechanics-based models have suggested a variety of definition of the free energy or the entropy in a neural system. Among those, the Feynman machine, proposed recently, presents the free energy of a neural system defined via the Feynman path integral formulation with the explicit time variable. In this study, we first give a brief review of the previous relevant models, paying attention to the troublesome problems in them, and examine how the Feynman machine overcomes several vulnerable points in previous models and derives the outcome of the firing or the learning rule in a (biological) neural system as the extremum state in the free energy. Specifically, the model reveals that the biological learning mechanism, called spike-timing-dependent plasticity, relates to the free-energy minimization principle. Basically, computing and learning mechanisms in the Feynman machine base on the exact spike timings of neurons, such as those in a biological neural system. We discuss the consequence of the adoption of an explicit time variable in modeling a neural system and the application of the free-energy minimization principle to understanding the phenomena in the brain. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

33 pages, 3128 KiB

Open AccessFeature PaperArticle

The Capacity for Correlated Semantic Memories in the Cortex

by Vezha Boboeva, Romain Brasselet and Alessandro Treves

Entropy 2018, 20(11), 824; https://doi.org/10.3390/e20110824 - 26 Oct 2018

Cited by 12 | Viewed by 4307

Abstract

A statistical analysis of semantic memory should reflect the complex, multifactorial structure of the relations among its items. Still, a dominant paradigm in the study of semantic memory has been the idea that the mental representation of concepts is structured along a simple [...] Read more.

A statistical analysis of semantic memory should reflect the complex, multifactorial structure of the relations among its items. Still, a dominant paradigm in the study of semantic memory has been the idea that the mental representation of concepts is structured along a simple branching tree spanned by superordinate and subordinate categories. We propose a generative model of item representation with correlations that overcomes the limitations of a tree structure. The items are generated through “factors” that represent semantic features or real-world attributes. The correlation between items has its source in the extent to which items share such factors and the strength of such factors: if many factors are balanced, correlations are overall low; whereas if a few factors dominate, they become strong. Our model allows for correlations that are neither trivial nor hierarchical, but may reproduce the general spectrum of correlations present in a dataset of nouns. We find that such correlations reduce the storage capacity of a Potts network to a limited extent, so that the number of concepts that can be stored and retrieved in a large, human-scale cortical network may still be of order 10⁷, as originally estimated without correlations. When this storage capacity is exceeded, however, retrieval fails completely only for balanced factors; above a critical degree of imbalance, a phase transition leads to a regime where the network still extracts considerable information about the cued item, even if not recovering its detailed representation: partial categorization seems to emerge spontaneously as a consequence of the dominance of particular factors, rather than being imposed ad hoc. We argue this to be a relevant model of semantic memory resilience in Tulving’s remember/know paradigms. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

21 pages, 462 KiB

Open AccessArticle

Statistical Mechanics of On-Line Learning Under Concept Drift

by Michiel Straat, Fthi Abadi, Christina Göpfert, Barbara Hammer and Michael Biehl

Entropy 2018, 20(10), 775; https://doi.org/10.3390/e20100775 - 10 Oct 2018

Cited by 15 | Viewed by 5057

Abstract

We introduce a modeling framework for the investigation of on-line machine learning processes in non-stationary environments. We exemplify the approach in terms of two specific model situations: In the first, we consider the learning of a classification scheme from clustered data by means [...] Read more.

We introduce a modeling framework for the investigation of on-line machine learning processes in non-stationary environments. We exemplify the approach in terms of two specific model situations: In the first, we consider the learning of a classification scheme from clustered data by means of prototype-based Learning Vector Quantization (LVQ). In the second, we study the training of layered neural networks with sigmoidal activations for the purpose of regression. In both cases, the target, i.e., the classification or regression scheme, is considered to change continuously while the system is trained from a stream of labeled data. We extend and apply methods borrowed from statistical physics which have been used frequently for the exact description of training dynamics in stationary environments. Extensions of the approach allow for the computation of typical learning curves in the presence of concept drift in a variety of model situations. First results are presented and discussed for stochastic drift processes in classification and regression problems. They indicate that LVQ is capable of tracking a classification scheme under drift to a non-trivial extent. Furthermore, we show that concept drift can cause the persistence of sub-optimal plateau states in gradient based training of layered neural networks for regression. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

11 pages, 3123 KiB

Open AccessArticle

Modelling of Behavior for Inhibition Corrosion of Bronze Using Artificial Neural Network (ANN)

by D. Elusaí Millán-Ocampo, Arianna Parrales-Bahena, J. Gonzalo González-Rodríguez, Susana Silva-Martínez, Jesús Porcayo-Calderón and J. Alfredo Hernández-Pérez

Entropy 2018, 20(6), 409; https://doi.org/10.3390/e20060409 - 26 May 2018

Cited by 14 | Viewed by 3927

Abstract

In this work, three models based on Artificial Neural Network (ANN) were developed to describe the behavior for the inhibition corrosion of bronze in 3.5% NaCl + 0.1 M Na₂SO₄, using the experimental data of Electrochemical Impedance Spectroscopy (EIS). [...] Read more.

In this work, three models based on Artificial Neural Network (ANN) were developed to describe the behavior for the inhibition corrosion of bronze in 3.5% NaCl + 0.1 M Na₂SO₄, using the experimental data of Electrochemical Impedance Spectroscopy (EIS). The database was divided into training, validation, and test sets randomly. The parameters process used as the inputs of the ANN models were frequency, temperature, and inhibitor concentration. The outputs for each ANN model and the components in the EIS spectrum (Z_re, Z_im, and Z_mod) were predicted. The transfer functions used for the learning process were the hyperbolic tangent sigmoid in the hidden layer and linear in the output layer, while the Levenberg–Marquardt algorithm was applied to determine the optimum values of the weights and biases. The statistical analysis of the results revealed that ANN models for Z_re, Z_im, and Z_mod can successfully predict the inhibition corrosion behavior of bronze in different conditions, where what was considered included variability in temperature, frequency, and inhibitor concentration. In addition, these three input parameters were keys to describe the behavior according to a sensitivity analysis. Full article

(This article belongs to the Special Issue Statistical Mechanics of Neural Networks)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Statistical Mechanics of Neural Networks

Share This Special Issue

Special Issue Editor

Special Issue Information

Keywords

Published Papers (8 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI