Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron, i.e., simple perceptron. It is shown that in the neuron problem, the general formula that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter appears. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and the Green function technique is introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green function of the Fokker-Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them is one similar to the Parisi phase in long-range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared with the prediction of the theory.

Original languageEnglish
Pages (from-to)263-392
Number of pages130
JournalPhysics Reports
Volume342
Issue number4-5
Publication statusPublished - Feb 2001

Fingerprint

neurons
replicas
broken symmetry
Green's functions
storage stability
synapses
self organizing systems
Fokker-Planck equation
correlators
partial differential equations
spin glass
differential equations
free energy
predictions
simulation
interactions

Keywords

  • 07.05.Mh
  • 61.43. - j
  • 75.10Nr
  • 84.35.+i
  • Neural networks
  • Pattern storage
  • Replica symmetry breaking
  • Spin glasses

ASJC Scopus subject areas

  • Physics and Astronomy(all)

Cite this

Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron. / Györgyi, G.

In: Physics Reports, Vol. 342, No. 4-5, 02.2001, p. 263-392.

Research output: Contribution to journalArticle

@article{241435f3eec04931a8805f7d9aa018c0,
title = "Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron",
abstract = "In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron, i.e., simple perceptron. It is shown that in the neuron problem, the general formula that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter appears. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and the Green function technique is introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green function of the Fokker-Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them is one similar to the Parisi phase in long-range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared with the prediction of the theory.",
keywords = "07.05.Mh, 61.43. - j, 75.10Nr, 84.35.+i, Neural networks, Pattern storage, Replica symmetry breaking, Spin glasses",
author = "G. Gy{\"o}rgyi",
year = "2001",
month = "2",
language = "English",
volume = "342",
pages = "263--392",
journal = "Physics Reports",
issn = "0370-1573",
publisher = "Elsevier",
number = "4-5",

}

TY - JOUR

T1 - Techniques of replica symmetry breaking and the storage problem of the McCulloch-Pitts neuron

AU - Györgyi, G.

PY - 2001/2

Y1 - 2001/2

N2 - In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron, i.e., simple perceptron. It is shown that in the neuron problem, the general formula that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter appears. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and the Green function technique is introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green function of the Fokker-Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them is one similar to the Parisi phase in long-range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared with the prediction of the theory.

AB - In this article we review the framework for spontaneous replica symmetry breaking. Subsequently that is applied to the example of the statistical mechanical description of the storage properties of a McCulloch-Pitts neuron, i.e., simple perceptron. It is shown that in the neuron problem, the general formula that is at the core of all problems admitting Parisi's replica symmetry breaking ansatz with a one-component order parameter appears. The details of Parisi's method are reviewed extensively, with regard to the wide range of systems where the method may be applied. Parisi's partial differential equation and related differential equations are discussed, and the Green function technique is introduced for the calculation of replica averages, the key to determining the averages of physical quantities. The Green function of the Fokker-Planck equation due to Sompolinsky turns out to play the role of the statistical mechanical Green function in the graph rules for replica correlators. The subsequently obtained graph rules involve only tree graphs, as appropriate for a mean-field-like model. The lowest order Ward-Takahashi identity is recovered analytically and shown to lead to the Goldstone modes in continuous replica symmetry breaking phases. The need for a replica symmetry breaking theory in the storage problem of the neuron has arisen due to the thermodynamical instability of formerly given solutions. Variational forms for the neuron's free energy are derived in terms of the order parameter function x(q), for different prior distribution of synapses. Analytically in the high temperature limit and numerically in generic cases various phases are identified, among them is one similar to the Parisi phase in long-range interaction spin glasses. Extensive quantities like the error per pattern change slightly with respect to the known unstable solutions, but there is a significant difference in the distribution of non-extensive quantities like the synaptic overlaps and the pattern storage stability parameter. A simulation result is also reviewed and compared with the prediction of the theory.

KW - 07.05.Mh

KW - 61.43. - j

KW - 75.10Nr

KW - 84.35.+i

KW - Neural networks

KW - Pattern storage

KW - Replica symmetry breaking

KW - Spin glasses

UR - http://www.scopus.com/inward/record.url?scp=0035255691&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0035255691&partnerID=8YFLogxK

M3 - Article

VL - 342

SP - 263

EP - 392

JO - Physics Reports

JF - Physics Reports

SN - 0370-1573

IS - 4-5

ER -