Reducing the Dimensionality of Data with Neural Networks

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.

Related collections

Most cited references 8

Record: found
Abstract: found
Article: not found

Neural networks and physical systems with emergent collective computational abilities.

J Hopfield (1982)

Computational properties of use of biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.

0 comments Cited 1807 times – based on 0 reviews      Review now

Record: found
Abstract: found
Article: not found

A fast learning algorithm for deep belief nets.

Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh (2006)

We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.

0 comments Cited 1230 times – based on 0 reviews      Review now

Record: found
Abstract: found
Article: not found

A global geometric framework for nonlinear dimensionality reduction.

Joshua B. Tenenbaum, Vin de Silva, John C. Langford (2000)

Scientists working with large volumes of high-dimensional data, such as global climate patterns, stellar spectra, or human gene distributions, regularly confront the problem of dimensionality reduction: finding meaningful low-dimensional structures hidden in their high-dimensional observations. The human brain confronts the same problem in everyday perception, extracting from its high-dimensional sensory inputs-30,000 auditory nerve fibers or 10(6) optic nerve fibers-a manageably small number of perceptually relevant features. Here we describe an approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set. Unlike classical techniques such as principal component analysis (PCA) and multidimensional scaling (MDS), our approach is capable of discovering the nonlinear degrees of freedom that underlie complex natural observations, such as human handwriting or images of a face under different viewing conditions. In contrast to previous algorithms for nonlinear dimensionality reduction, ours efficiently computes a globally optimal solution, and, for an important class of data manifolds, is guaranteed to converge asymptotically to the true structure.

0 comments Cited 606 times – based on 0 reviews      Review now

All references

Author and article information

Journal

Title: Science

Abbreviated Title: Science

Publisher: American Association for the Advancement of Science (AAAS)

ISSN (Print): 0036-8075

ISSN (Electronic): 1095-9203

Publication date Created: July 28 2006

Publication date (Print): July 28 2006

Volume: 313

Issue: 5786

Pages: 504-507

Affiliations

[1 ]Department of Computer Science, University of Toronto, 6 King's College Road, Toronto, Ontario M5S 3G4, Canada.

Article

DOI: 10.1126/science.1127647

PubMed ID: 16873662

SO-VID: da424a5c-d7b5-4486-8816-ea305b7583ff

History

Data availability:

Comments

Comment on this article

Cited by 3,350

See all cited by

Most referenced authors 92

See all reference authors

- Version 1