The Neural Network Revival

The Road to Artificial Intelligence — Part 9 of 15

WHEN NEURAL NETWORKS CAME BACK

By the late 1980s, AI's commercial comeback was faltering.

Expert systems had entered businesses, specialized Lisp machines had promised dedicated AI power, and Japan's ambitious Fifth Generation project had drawn international attention. But rule-based systems were costly to maintain, specialized hardware faced competition from cheaper general-purpose machines, and confidence began to fade. The downturn that followed is often called the second AI winter.

The field didn't simply retreat, though. Neural-network researchers were developing methods that would later become central to machine learning. Probabilistic, statistical, and reinforcement-learning approaches were gaining visibility. Specialized symbolic systems kept producing remarkable results. Beneath the commercial contraction, AI was changing direction.

Expert systems meet the real world

The expert systems that powered AI's comeback depended on human specialists to express their knowledge in a form computers could use.

XCON, built for Digital Equipment Corporation to help configure computer systems, showed both the value and the burden of that approach. A 1990 study by John Sviokla reported Digital's estimate that eliminating a manufacturing testing step through XCON saved about $15 million over the system's first five years. The study also documented the ongoing effort needed to maintain it.

That burden pointed to a deeper problem. Capturing expert knowledge was hard, and keeping it current was harder. Rules that worked in one context could turn brittle as products, procedures, or exceptions changed. Expert systems were far from useless, but their success often depended on carefully limited domains, sustained maintenance, and a place inside larger organizational processes.

A commercial contraction

The second AI winter is conventionally dated from around 1987 into the early 1990s, though those dates are retrospective rather than sharp. Commercial activity contracted sharply in 1987 and 1988, with weakness continuing into the early 1990s.

Several pressures converged. Expert systems were expensive to build and maintain, and their narrowness made them hard to generalize. Companies also grew less willing to pay premium prices for specialized AI hardware when increasingly capable Unix workstations cost far less.

The Lisp-machine industry was especially exposed. Its systems were built around Lisp and marketed for AI development, but general-purpose workstations increasingly offered adequate performance, broader software support, and lower prices. Symbolics, once a leading Lisp-machine company, filed for Chapter 11 bankruptcy protection in 1993. The bankruptcy came several years after the industry's sharp contraction, reflecting the longer decline of the specialized Lisp-machine market. Contemporary reporting described a sharp reversal from the company's earlier peak. The episode shows the difference between a promising technical platform and a sustainable market.

The decline of Lisp machines had no single cause. Software ecosystems, price, hardware performance, changing business demand, and the growing appeal of general-purpose computing all played a part.

The Fifth Generation challenge

Japan's Fifth Generation Computer Systems project added an international dimension to the period. It sought to advance knowledge-based computing, logic programming, and parallel processing, and its influence reached well beyond Japan.

It produced technical achievements and stimulated research abroad. Later evaluations, however, criticized the limited real-world deployment of its systems and the gap between its research ambitions and commercial results. The project fell short of some of its commercial ambitions, but it contributed to research, computing infrastructure, and international competition in computer science.

Similar programs continued elsewhere. Britain's Alvey initiative was a major five-year national computing effort, and Europe's ESPRIT supported broad information-technology research. Neither was a single expert-system project.

Neural networks continue

As commercial enthusiasm for expert systems cooled, neural-network research carried on.

In 1980, Kunihiko Fukushima described the Neocognitron, a hierarchical neural architecture designed to recognize patterns even when they shifted position. It became an important precursor to later convolutional neural networks and helped establish the idea of hierarchical pattern recognition in neural architectures.

John Hopfield's 1982 paper introduced a major framework for associative memory using recurrent neural networks, in which a partial or noisy pattern could lead the network toward a stored one. It tied neural computation to ideas from physics, particularly energy landscapes and collective behavior.

The 1980s also brought renewed attention to learning in multilayer networks. In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published "Learning representations by back-propagating errors" in Nature. The paper made backpropagation highly influential by showing how networks with hidden layers could learn internal representations from error signals. Backpropagation had important earlier roots, including work by Paul Werbos and David Parker, as well as a longer history of reverse-mode differentiation. But the 1986 paper helped establish a practical, influential method for training multilayer networks.

Other demonstrations showed what learning systems could do without implying human-like understanding. NETtalk, for example, learned to convert strings of written English letters into phonemes that a speech synthesizer could read out. NETtalk demonstrated learned text-to-speech, while remaining a specialized system rather than a model of language comprehension.

Probability and learning become more visible

The downturn coincided with a broader diversification of AI.

Probability had never left the field, but probabilistic reasoning and statistical methods grew more important. Hidden Markov models drove major advances in speech recognition. Decision trees offered practical ways to classify data. Reinforcement learning developed more distinct theoretical and experimental communities.

By the mid-1990s, these approaches were becoming part of a wider movement toward statistical, data-driven machine learning. Support-vector networks, described by Corinna Cortes and Vladimir Vapnik in 1995, offered another influential approach to classification, using geometric margins and nonlinear mappings to separate complex patterns in data. No single method replaced expert systems. Together, though, these approaches reflected a change in what AI research meant. Instead of relying mainly on hand-written rules, researchers increasingly explored systems that could infer patterns from data, optimize behavior, or reason under uncertainty.

Deep Blue and specialized computation

The decade also showed that symbolic methods weren't obsolete.

IBM's Deep Blue defeated Garry Kasparov in a six-game match in 1997, 3½–2½. The system combined large-scale search, evaluation functions, specialized hardware, parallel computation, and human chess knowledge, including 480 custom chess chips.

Its victory demonstrated the power of specialized search and computation, not general-purpose intelligence. Deep Blue was built for chess. Its success showed that carefully designed symbolic search and dedicated computation could stay powerful even as statistical and neural approaches gained attention.

The connection to AI

The second AI winter redirected the field without ending it. The period exposed the limits of systems built on manually encoded knowledge, especially in changing environments.

It also weakened the business case for specialized hardware and exposed the gap between commercial expectations and what many AI systems could reliably deliver.

Meanwhile, research continued along several paths:

• Neural networks learned internal representations.
• Probabilistic models represented uncertainty.
• Statistical methods used growing bodies of data.
• Reinforcement learning studied action and feedback.
• Support-vector methods linked geometry with generalization.
• Specialized machines demonstrated the power of search and hardware design.

Later advances drew on several of these traditions, alongside much larger datasets, more powerful computing, and improved software. The history isn't a straight march toward modern AI. Researchers pursued different approaches, some of which became practical only as computing power, data, and software improved.

Looking ahead

The commercial contraction around expert systems didn't end the search for machine intelligence. During this period, greater attention also turned toward learning, uncertainty, data, and the computing resources needed to make those approaches practical.

The next article follows the convergence of data, GPUs, improved software, and neural-network methods—conditions that would help turn a research revival into the deep-learning era.

SOURCES

• Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. "Learning representations by back-propagating errors." Nature, 323, 533–536, 1986. DOI: 10.1038/323533a0.
• Cortes, Corinna; Vapnik, Vladimir. "Support-vector networks." Machine Learning, 20, 273–297, 1995. DOI: 10.1007/BF00994018.
• Hopfield, John J. "Neural networks and physical systems with emergent collective computational abilities." PNAS, 79(8), 2554–2558, 1982.
• Fukushima, Kunihiko. "Neocognitron." Biological Cybernetics, 36, 193–202, 1980.
• Sviokla, John J. "An Examination of the Impact of Expert Systems on the Firm: The Case of XCON." MIS Quarterly, 14(2), 127–140, 1990. DOI: 10.2307/248770.
• Computer History Museum. "Digital at Work," section on XCON, 1992.
• ICOT/AIST. "Final Evaluation Report on the FGCS Project," 1992–1993.
• Hsu, Feng-Hsiung. "IBM's Deep Blue chess grandmaster chips." IEEE Micro, 1999.
• Parker, David B. "Learning-Logic: Casting the Cortex of the Human Brain in Silicon." MIT technical report TR-47, 1985.
• Johnson, Tim; Hewett, Julian; Guilfoyle, Christine; Jeffcoate, Judith. "Expert systems—the second wave." The Knowledge Engineering Review, 3, 1988.
• Los Angeles Times. "Symbolics Inc. Seeks Chapter 11 Protection," February 2, 1993.
• Oakley, Brian; Owen, Kenneth. Alvey: Britain's Strategic Computing Initiative. MIT Press, 1990.
• European Commission. ESPRIT: European Strategic Programme for Research and Development in Information Technology—1987 Annual Report, 1988.
• Sejnowski, Terrence J.; Rosenberg, Charles R. "Parallel Networks That Learn to Pronounce English Text." Complex Systems, 1, 145–168, 1987.
• Rabiner, Lawrence R. "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition." Proceedings of the IEEE, 77(2), 257–286, 1989. DOI: 10.1109/5.18626.
• Quinlan, J. Ross. "Induction of Decision Trees." Machine Learning, 1, 81–106, 1986. DOI: 10.1007/BF00116251.
• Sutton, Richard S. "Learning to Predict by the Methods of Temporal Differences." Machine Learning, 3, 9–44, 1988.