50
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      Overcoming Catastrophic Interference in Connectionist Networks Using Gram-Schmidt Orthogonalization

      research-article
      1 , 2 , * , 2 , 3
      PLoS ONE
      Public Library of Science

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Connectionist models of memory storage have been studied for many years, and aim to provide insight into potential mechanisms of memory storage by the brain. A problem faced by these systems is that as the number of items to be stored increases across a finite set of neurons/synapses, the cumulative changes in synaptic weight eventually lead to a sudden and dramatic loss of the stored information (catastrophic interference, CI) as the previous changes in synaptic weight are effectively lost. This effect does not occur in the brain, where information loss is gradual. Various attempts have been made to overcome the effects of CI, but these generally use schemes that impose restrictions on the system or its inputs rather than allowing the system to intrinsically cope with increasing storage demands. We show here that catastrophic interference occurs as a result of interference among patterns that lead to catastrophic effects when the number of patterns stored exceeds a critical limit. However, when Gram-Schmidt orthogonalization is combined with the Hebb-Hopfield model, the model attains the ability to eliminate CI. This approach differs from previous orthogonalisation schemes used in connectionist networks which essentially reflect sparse coding of the input. Here CI is avoided in a network of a fixed size without setting limits on the rate or number of patterns encoded, and without separating encoding and retrieval, thus offering the advantage of allowing associations between incoming and stored patterns.

          PACS Nos.: 87.10.+e, 87.18.Bb, 87.18.Sn, 87.19.La

          Related collections

          Most cited references19

          • Record: found
          • Abstract: found
          • Article: not found

          Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type.

          Q Bi, G Bi, M Poo (1998)
          In cultures of dissociated rat hippocampal neurons, persistent potentiation and depression of glutamatergic synapses were induced by correlated spiking of presynaptic and postsynaptic neurons. The relative timing between the presynaptic and postsynaptic spiking determined the direction and the extent of synaptic changes. Repetitive postsynaptic spiking within a time window of 20 msec after presynaptic activation resulted in long-term potentiation (LTP), whereas postsynaptic spiking within a window of 20 msec before the repetitive presynaptic activation led to long-term depression (LTD). Significant LTP occurred only at synapses with relatively low initial strength, whereas the extent of LTD did not show obvious dependence on the initial synaptic strength. Both LTP and LTD depended on the activation of NMDA receptors and were absent in cases in which the postsynaptic neurons were GABAergic in nature. Blockade of L-type calcium channels with nimodipine abolished the induction of LTD and reduced the extent of LTP. These results underscore the importance of precise spike timing, synaptic strength, and postsynaptic cell type in the activity-induced modification of central synapses and suggest that Hebb's rule may need to incorporate a quantitative consideration of spike timing that reflects the narrow and asymmetric window for the induction of synaptic modification.
            Bookmark
            • Record: found
            • Abstract: found
            • Article: not found

            A theory of cerebellar cortex.

            D. Marr (1969)
            1. A detailed theory of cerebellar cortex is proposed whose consequence is that the cerebellum learns to perform motor skills. Two forms of input-output relation are described, both consistent with the cortical theory. One is suitable for learning movements (actions), and the other for learning to maintain posture and balance (maintenance reflexes).2. It is known that the cells of the inferior olive and the cerebellar Purkinje cells have a special one-to-one relationship induced by the climbing fibre input. For learning actions, it is assumed that:(a) each olivary cell responds to a cerebral instruction for an elemental movement. Any action has a defining representation in terms of elemental movements, and this representation has a neural expression as a sequence of firing patterns in the inferior olive; and(b) in the correct state of the nervous system, a Purkinje cell can initiate the elemental movement to which its corresponding olivary cell responds.3. Whenever an olivary cell fires, it sends an impulse (via the climbing fibre input) to its corresponding Purkinje cell. This Purkinje cell is also exposed (via the mossy fibre input) to information about the context in which its olivary cell fired; and it is shown how, during rehearsal of an action, each Purkinje cell can learn to recognize such contexts. Later, when the action has been learnt, occurrence of the context alone is enough to fire the Purkinje cell, which then causes the next elemental movement. The action thus progresses as it did during rehearsal.4. It is shown that an interpretation of cerebellar cortex as a structure which allows each Purkinje cell to learn a number of contexts is consistent both with the distributions of the various types of cell, and with their known excitatory or inhibitory natures. It is demonstrated that the mossy fibre-granule cell arrangement provides the required pattern discrimination capability.5. The following predictions are made.(a) The synapses from parallel fibres to Purkinje cells are facilitated by the conjunction of presynaptic and climbing fibre (or post-synaptic) activity.(b) No other cerebellar synapses are modifiable.(c) Golgi cells are driven by the greater of the inputs from their upper and lower dendritic fields.6. For learning maintenance reflexes, 2(a) and 2(b) are replaced by2'. Each olivary cell is stimulated by one or more receptors, all of whose activities are usually reduced by the results of stimulating the corresponding Purkinje cell.7. It is shown that if (2') is satisfied, the circuit receptor --> olivary cell --> Purkinje cell --> effector may be regarded as a stabilizing reflex circuit which is activated by learned mossy fibre inputs. This type of reflex has been called a learned conditional reflex, and it is shown how such reflexes can solve problems of maintaining posture and balance.8. 5(a), and either (2) or (2') are essential to the theory: 5(b) and 5(c) are not absolutely essential, and parts of the theory could survive the disproof of either.
              Bookmark
              • Record: found
              • Abstract: found
              • Article: not found

              Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex.

              The development of stimulus selectivity in the primary sensory cortex of higher vertebrates is considered in a general mathematical framework. A synaptic evolution scheme of a new kind is proposed in which incoming patterns rather than converging afferents compete. The change in the efficacy of a given synapse depends not only on instantaneous pre- and postsynaptic activities but also on a slowly varying time-averaged value of the postsynaptic activity. Assuming an appropriate nonlinear form for this dependence, development of selectivity is obtained under quite general conditions on the sensory environment. One does not require nonlinearity of the neuron's integrative power nor does one need to assume any particular form for intracortical circuitry. This is first illustrated in simple cases, e.g., when the environment consists of only two different stimuli presented alternately in a random manner. The following formal statement then holds: the state of the system converges with probability 1 to points of maximum selectivity in the state space. We next consider the problem of early development of orientation selectivity and binocular interaction in primary visual cortex. Giving the environment an appropriate form, we obtain orientation tuning curves and ocular dominance comparable to what is observed in normally reared adult cats or monkeys. Simulations with binocular input and various types of normal or altered environments show good agreement with the relevant experimental data. Experiments are suggested that could test our theory further.
                Bookmark

                Author and article information

                Contributors
                Role: Editor
                Journal
                PLoS One
                PLoS ONE
                plos
                plosone
                PLoS ONE
                Public Library of Science (San Francisco, USA )
                1932-6203
                2014
                2 September 2014
                : 9
                : 9
                : e105619
                Affiliations
                [1 ]School of Physics, University of Hyderabad, Hyderabad, India
                [2 ]Centre for Neural and Cognitive Sciences, University of Hyderabad, Hyderabad, India
                [3 ]Department of Physiology, Development and Neuroscience, University of Cambridge, Cambridge, United Kingdom
                Tokai University, Japan
                Author notes

                Competing Interests: The authors have declared that no competing interests exist.

                Wrote the paper: VS DJP. Carried out the simulations: SS. Plotted the data: SS.

                Article
                PONE-D-13-47449
                10.1371/journal.pone.0105619
                4152133
                25180550
                9a60b445-2f4e-4949-b779-9d1bfffd04ac
                Copyright @ 2014

                This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

                History
                : 12 November 2013
                : 26 July 2014
                Page count
                Pages: 7
                Funding
                The Royal Society; The Leverhulme Foundation (UK); National Initiative of Research in Cognitive Science by the Department of Science and Technology, Government of India. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
                Categories
                Research Article
                Biology and Life Sciences
                Anatomy
                Nervous System
                Computational Biology
                Neuroscience
                Learning and Memory
                Neural Networks
                Physiology
                Nervous System Physiology
                Theoretical Biology
                Computer and Information Sciences
                Information Technology
                Information Storage and Retrieval
                Physical Sciences
                Mathematics
                Applied Mathematics
                Research and Analysis Methods
                Specimen Preparation and Treatment
                Mechanical Treatment of Specimens
                Specimen Disruption
                Electroporation
                Social Sciences

                Uncategorized
                Uncategorized

                Comments

                Comment on this article