Machine-Learning Maestro Michael Jordan on the Delusions of Big Data and Other Huge Engineering Efforts

Big-data boondoggles and brain-inspired chips are just two of the things we’re really getting wrong

Photo-Illustration: Randi Klett

The overeager adoption of big data is likely to result in catastrophes of analysis comparable to a national epidemic of collapsing bridges. 
Hardware designers creating chips based on the human brain are engaged in a faith-based undertaking likely to prove a fool’s errand. 
Despite recent claims to the contrary, we are no further along with computer vision than we were with physics when Isaac Newton sat under his apple tree.

Those may sound like the Luddite ravings of a crackpot who breached security at an IEEE conference. In fact, the opinions belong to IEEE Fellow Michael I. Jordan, Pehong Chen Distinguished Professor at the University of California, Berkeley. Jordan is one of the world’s most respected authorities on machine learning and an astute observer of the field. His CV would require its own massive database, and his standing in the field is such that he was chosen to write the introduction to the 2013 National Research Council report “Frontiers in Massive Data Analysis.” San Francisco writer Lee Gomes interviewed him for IEEE Spectrum on 3 October 2014.

Michael Jordan on…

1. Why We Should Stop Using Brain Metaphors When We Talk About Computing
2. Our Foggy Vision About Machine Vision
3. Why Big Data Could Be a Big Fail
4. What He’d Do With US $1 Billion 
5. How Not to Talk About the Singularity 
6. What He Cares About More Than Whether P = NP 
7. What the Turing Test Really Means

1- Why We Should Stop Using Brain Metaphors When We Talk About Computing

IEEE Spectrum: I infer from your writing that you believe there’s a lot of misinformation out there about deep learning, big data, computer vision, and the like.

Michael Jordan: Well, on all academic topics there is a lot of misinformation. The media is trying to do its best to find topics that people are going to read about. Sometimes those go beyond where the achievements actually are. Specifically on the topic of deep learning, it’s largely a rebranding of neural networks, which go back to the 1980s. They actually go back to the 1960s; it seems like every 20 years there is a new wave that involves them. In the current wave, the main success story is the convolutional neural network, but that idea was already present in the previous wave. And one of the problems with both the previous wave, that has unfortunately persisted in the current wave, is that people continue to infer that something involving neuroscience is behind it, and that deep learning is taking advantage of an understanding of how the brain processes information, learns, makes decisions, or copes with large amounts of data. And that is just patently false.

Spectrum: As a member of the media, I take exception to what you just said, because it’s very often the case that academics are desperate for people to write stories about them.

Michael Jordan: Yes, it’s a partnership.

Spectrum: It’s always been my impression that when people in computer science describe how the brain works, they are making horribly reductionist statements that you would never hear from neuroscientists. You called these “cartoon models” of the brain.

Michael Jordan: I wouldn’t want to put labels on people and say that all computer scientists work one way, or all neuroscientists work another way. But it’s true that with neuroscience, it’s going to require decades or even hundreds of years to understand the deep principles. There is progress at the very lowest levels of neuroscience. But for issues of higher cognition—how we perceive, how we remember, how we act—we have no idea on

• how neurons are storing information, 
• how they are computing, 
• what the rules are, 
• what the algorithms are, 
• what the representations are, and 
• the like. 

So we are not yet in an era in which we can be using an understanding of the brain to guide us in the construction of intelligent systems.

Spectrum: In addition to criticizing cartoon models of the brain, you actually go further and criticize the whole idea of “neural realism”—the belief that just because a particular hardware or software system shares some putative characteristic of the brain, it’s going to be more intelligent. What do you think of computer scientists who say, for example, “My system is brainlike because it is massively parallel.”

IBM Chip Processes Data Similar to the Way Your Brain Does

A chip that uses a million digital neurons and 256 million synapses may signal the beginning of a new era of more intelligent computers.

WHY IT MATTERS

Computers that can comprehend messy data such as images could revolutionize what technology can do for us.

New thinking: IBM has built a processor designed using principles at work in your brain.

A new kind of computer chip, unveiled by IBM today, takes design cues from the wrinkled outer layer of the human brain. Though it is no match for a conventional microprocessor at crunching numbers, the chip consumes significantly less power, and is vastly better suited to processing images, sound, and other sensory data.

IBM’s SyNapse chip, as it is called, processes information using a network of just over one million “neurons,” which communicate with one another using electrical spikes—as actual neurons do. The chip uses the same basic components as today’s commercial chips—silicon transistors. But its transistors are configured to mimic the behavior of both neurons and the connections—synapses—between them.

The SyNapse chip breaks with a design known as the Von Neuman architecture that has underpinned computer chips for decades. Although researchers have been experimenting with chips modeled on brains—known as neuromorphic chips—since the late 1980s, until now all have been many times less complex, and not powerful enough to be practical (see “Thinking in Silicon”). Details of the chip were published today in the journal Science.

The new chip is not yet a product, but it is powerful enough to work on real-world problems. In a demonstration at IBM’s Almaden research center, MIT Technology Review saw one recognize cars, people, and bicycles in video of a road intersection. A nearby laptop that had been programed to do the same task processed the footage 100 times slower than real time, and it consumed 100,000 times as much power as the IBM chip. IBM researchers are now experimenting with connecting multiple SyNapse chips together, and they hope to build a supercomputer using thousands.

When data is fed into a SyNapse chip it causes a stream of spikes, and its neurons react with a storm of further spikes. The just over one million neurons on the chip are organized into 4,096 identical blocks of 250, an arrangement inspired by the structure of mammalian brains, which appear to be built out of repeating circuits of 100 to 250 neurons, says Dharmendra Modha, chief scientist for brain-inspired computing at IBM. Programming the chip involves choosing which neurons are connected, and how strongly they influence one another. To recognize cars in video, for example, a programmer would work out the necessary settings on a simulated version of the chip, which would then be transferred over to the real thing.

In recent years, major breakthroughs in image analysis and speech recognition have come from using large, simulated neural networks to work on data (see “Deep Learning”). But those networks require giant clusters of conventional computers. As an example, Google’s famous neural network capable of recognizing cat and human faces required 1,000 computers with 16 processors apiece (see “Self-Taught Software”).

Although the new SyNapse chip has more transistors than most desktop processors, or any chip IBM has ever made, with over five billion, it consumes strikingly little power. When running the traffic video recognition demo, it consumed just 63 milliwatts of power. Server chips with similar numbers of transistors consume tens of watts of power—around 10,000 times more.

The efficiency of conventional computers is limited because they store data and program instructions in a block of memory that’s separate from the processor that carries out instructions. As the processor works through its instructions in a linear sequence, it has to constantly shuttle information back and forth from the memory store—a bottleneck that slows things down and wastes energy.

IBM’s new chip doesn’t have separate memory and processing blocks, because its neurons and synapses intertwine the two functions. And it doesn’t work on data in a linear sequence of operations; individual neurons simply fire when the spikes they receive from other neurons cause them to.

Horst Simon, the deputy director of Lawrence Berkeley National Lab and an expert in supercomputing, says that until now the industry has focused on tinkering with the Von Neuman approach rather than replacing it, for example by using multiple processors in parallel, or using graphics processors to speed up certain types of calculations. The new chip “may be a historic development,” he says. “The very low power consumption and scalability of this architecture are really unique.”

One downside is that IBM’s chip requires an entirely new approach to programming. Although the company announced a suite of tools geared toward writing code for its forthcoming chip last year (see “IBM Scientists Show Blueprints for Brainlike Computing”), even the best programmers find learning to work with the chip bruising, says Modha: “It’s almost always a frustrating experience.” His team is working to create a library of ready-made blocks of code to make the process easier.

Asking the industry to adopt an entirely new kind of chip and way of coding may seem audacious. But IBM may find a receptive audience because it is becoming clear that current computers won’t be able to deliver much more in the way of performance gains. “This chip is coming at the right time,” says Simon.

ORIGINAL: Tech Review

By Tom Simonite

August 7, 2014

IBM's Brain-Inspired Computer Chip Comes from the Future

Illustration: IBM

Brain-inspired computers have tickled the public imagination ever since Arnold Schwarzenegger's character in “Terminator 2: Judgment Day” uttered: “My CPU is a neural net processor; a learning computer.” Today, IBM researchers backed by U.S. military funding unveiled a new computer chip that they say could revolutionize everything from smartphones to smart cars—and perhaps pave the way for neural networks to someday approach the computing capabilities of the human brain.

The IBM neurosynaptic computer chip consists of one million programmable neurons and 256 million programmable synapses conveying signals between the digital neurons. Each of chip’s 4,096 neurosynaptic cores includes the entire computing package—memory, computation, and communication. They all operate in parallel based on “event-driven” computing, similar to the signal spikes and cascades of activity when human brain cells work in concert. Such architecture helps to bypass the bottleneck in traditional computing where program instructions and operation data cannot pass through the same route simultaneously.

“We have not built a brain,” says Dharmendra Modha, chief scientist and founder of IBM’s Cognitive Computing group at IBM Research-Almaden. “But we have come the closest to creating learning function and capturing it in silicon in a scalable way to provide new computing capability that was not possible before.”

Such capability could enable new mobile device applications that emulate the human brain’s capability to swiftly process information about new events or other changes in real-world environments, whether that involves recognizing familiar sounds or a certain face in a moving crowd. IBM envisions its new chips working together with traditional computing devices as hybrid machines—providing an added dose of brain-like intelligence for smart car sensors, cloud computing applications or mobile devices such as smartphones. The chip's architecture was detailed in a new paper published in the 7 August online issue of the journal Science.

Add caption

With a total of 5.4 billion transistors the computer chip, named TrueNorth, is one of the largest CMOS chips ever built. Yet the chip uses just 70 milliwatts while running and has a power density of 20 milliwatts per square centimeter— almost 1/10,000th the power of most modern microprocessors. That brings the new chip's efficiency much closer to the human brain’s astounding power consumption of just 20 watts, or less than the average incandescent light bulb.

“This is literally a supercomputer the size of a postage stamp, light like a feather, and low power like a hearing aid,” Modha says.

One reason IBM was able to minimize power usage is that its chip's computation only triggers when needed. Traditional computer chips have a clock that uses power to trigger and coordinate all the computational processes. But the IBM chip's digital neurons can work together asynchronously when triggered by the signal spikes. IBM also designed its chip to have low power consumption by creating an on-chip network to interconnect all the neurosynaptic cores and building the chip with a low-power process technology used for making mobile devices.

It’s also a supercomputer that can easily scale up in size. IBM designed its computer chip architecture so that it could simply add new neurosynatpic cores within the chip. The chips themselves can be arranged in a repeatable 2-D tile pattern to create bigger machines—IBM has already tested that idea with a 16-chip configuration. That’s the “blueprint of a scalable supercomputer,” Modha says.

Past brain-inspired neural networks have used a combination of both analog and digital to represent the individual neurons. IBM chose to represent the neurons in digital form, which provided several advantages. (At least one other project, SpiNNaker also depends on digital.)

First, the choice allowed IBM engineers to avoid the physical problems of dealing with differences in the manufacturing process or temperature fluctuations. Second, it provided a “one to one equivalence with software and hardware” that allowed the IBM software team to build applications on a simulator even before the physical chip had been designed and tested—applications that ran without problems on the finished chip. Third, the lack of analog circuitry allowed the IBM team to dramatically shrink the size of its circuits. (IBM fabricated its chip using Samsung’s 28-nm process technology—typical for manufacturing chips for mobile devices.)

IBM’s new chip represents the culmination of a decade of Modha’s personal research and almost six years of funding from the U.S. Defense Advanced Research Projects Agency (DARPA). Modha currently heads DARPA’s SyNAPSE project, a global effort that has committed US $53 million to making learning computers since 2008.

Now IBM has built an entire ecosystem around its new chip hardware and software, including a new programming language and a curriculum to teach coders everything they need to know. And the company is reaching out to potential customers, universities, government agencies, and IBM employees to fully explore the commercial applications of its chip technology.

“Our long-term end goal is to build a ‘brain in a box’ with 100 billion synapses consuming 1 kilowatt of power,” Modha says. “In the near future, we’ll be looking at multiple things for empowering smartphones, mobile devices and cloud services with this technology.”

Learn More DARPA SyNAPSE IBM cognitive computing brain-inspired computerslearning computers neural networks neuro-inspired computing

ORIGINAL: IEEE Spectrum

By Jeremy Hsu

Posted 7 Aug 2014 | 18:00 GMT