Detecting the invisible: Software that can see invisible motion

ORIGINAL: H+ Magazine

By: Lochlan Bloom

Published: March 16, 2013

New software from MIT can now reveal details in videos previously hidden to the human eye. The technique known as Eulerian Video Magnification was developed by graduate student Michael Rubinstein, recent alumni Hao-Yu Wu ’12, MNG ’12 and Eugene Shih SM ’01, PhD ’10, and professors William Freeman, Fredo Durand and John Guttag was presented this past summer at SIGGRAPH 2012.

The ground-breaking computer code analyses each frame of a video to determine invisible variations and offers some truly exciting possibilities for machine interaction. It also raises an interesting question – if machines can look back over our recorded lives and pull out previously hidden behaviour will that change the way we relate to our past?

The software in question has been developed by researchers at MIT and works with any existing video footage. By amplifying minute changes in pixel shading the software is able to determine fluctuations over time. As a result it is already able to predict fairly complex factors about humans or animals appearing in a video.

The researchers demonstrated the power of the program by analysing a video of a new born baby and extracting its heart rate. In this case, invisible changes in blood flow to the baby’s face created a hidden measure of its heartbeat. By comparing with data from a heart monitor recorded at the same time as the video they were able to confirm that their readings were correct.

While the researchers are currently touting the medical benefits of such a system – to remotely monitor at risk patients – there are undoubtedly huge implications for Artificial Intelligence and computer interfaces in general. The retrospective aspect of this is ably demonstrated in the below video where the researchers are able to pull Christian Bale’s heartbeat from the recent Batman film.

The idea that a computer can see things which are invisible to a human is not new. With the wide array of sensors and interfaces already on the market today, a machine is able to detect phenomena far and beyond the five meagre human senses. However the ability to go back and reassess existing footage with newly developed software and new technologies is something that has so far been little explored.

Consider recent history. Could a machine detect anything invisible to the human eye by analyzing a video of an assassination? Or a politician’s speech? There is a correlation between blood flow and lying, so a machine can be used as a aid to help humans determine whether to believe a rival in a business or diplomatic negotiation. Or could this type of machine become a standard device for job interviews?

The open-source software released by MIT is already a clear step towards a future where machines are indispensable in uncovering the hidden truths around us and it is only one of many such new techniques. When a computer can predict what someone is feeling more accurately than a human then at what stage do we stop trusting our instincts and rely instead on machines to guide our social interactions?

###

Lochlan Bloom is a writer of fiction and non-fiction. His novella Trade, focused on the collision of technology and the sex industry is out now.

http://lochlanbloom.com

@lochlanbloom

Endnote: The Eulerian Video Magnification (EVM) software can be downloaded and run or run it via a web-based interface . There are also plans for a smartphone app although no timeline has been announced.

Get the code: Matlab (2 MB, v1.1 2013-03-02) – reproduces all the results in the paper (see README.txt for details).

This code is provided for non-commercial research purposes only. By downloading and using the code, you are consenting to be bound by all terms of this software release agreement. Contact the authors if you wish to use the code commercially. This work is patent pending.

Secrets of Human Speech Uncovered

ORIGINAL: UCSF

By Jason Bardi
February 20, 2013

Work at UCSF Shows Brain Exerts Symphony-Like Control of Vocal Tract During the Act of Speaking

A team of researchers at UC San Francisco has uncovered the neurological basis of speech motor control, the complex coordinated activity of tiny brain regions that controls our lips, jaw, tongue and larynx as we speak.

Edward Chang, MD

Described this week in the journal Nature, the work has potential implications for developing computer-brain interfaces for artificial speech communication and for the treatment of speech disorders. It also sheds light on an ability that is unique to humans among living creatures but poorly understood.

“Speaking is so fundamental to who we are as humans – nearly all of us learn to speak,” said senior author Edward Chang, MD, a neurosurgeon at the UCSF Epilepsy Center and a faculty member in the UCSF Center for Integrative Neuroscience. “But it’s probably the most complex motor activity we do.”

The complexity comes from the fact that spoken words require the coordinated efforts of numerous “articulators” in the vocal tract – the lips, tongue, jaw and larynx – but scientists have not understood how the movements of these distinct articulators are precisely coordinated in the brain.

To understand how speech articulation works, Chang and his colleagues recorded electrical activity directly from the brains of three people undergoing brain surgery at UCSF, and used this information to determine the spatial organization of the “speech sensorimotor cortex,” which controls the lips, tongue, jaw, larynx as a person speaks. This gave them a map of which parts of the brain control which parts of the vocal tract.

They then applied a sophisticated new method called “state-space” analysis to observe the complex spatial and temporal patterns of neural activity in the speech sensorimotor cortex that play out as someone speaks. This revealed a surprising sophistication in how the brain's speech sensorimotor cortex works.

They found that this cortical area has a hierarchical and cyclical structure that exerts a split-second, symphony-like control over the tongue, jaw, larynx and lips.

“These properties may reflect cortical strategies to greatly simplify the complex coordination of articulators in fluent speech,” said Kristofer Bouchard, PhD, a postdoctoral fellow in the Chang lab who was the first author on the paper.

In the same way that a symphony relies upon all the players to coordinate their plucks, beats or blows to make music, speaking demands well-timed action of several various brain regions within the speech sensorimotor cortex.

Brain Mapping in Epilepsy Surgery 

The patients involved in the study were all at UCSF undergoing surgery for severe, untreatable epilepsy. Brain surgery is a powerful way to halt epilepsy in its tracks, potentially completely stopping seizures overnight, and its success is directly related to the accuracy with which a medical team can map the brain, identifying the exact pieces of tissue responsible for an individual's seizures and removing them.

The UCSF Comprehensive Epilepsy Center is a leader in the use of advanced intracranial monitoring to map out elusive seizure-causing brain regions. The mapping is done by surgically implanting an electrode array under the skull on the brain’s outer surface or cortex and recording the brain’s activity in order to pinpoint the parts of the brain responsible for disabling seizures. In a second surgery a few weeks later, the electrodes are removed and the unhealthy brain tissue that causes the seizures is removed.

This setting also permits a rare opportunity to ask basic questions about how the human brain works, such as how it controls speaking. The neurological basis of speech motor control has remained unknown until now because scientists cannot study speech mechanisms in animals and because non-invasive imaging methods lack the ability to resolve the very rapid time course of articulator movements, which change in hundredths of seconds.

But surgical brain mapping can record neural activity directly and faster than other noninvasive methods, showing changes in electrical activity on the order of a few milliseconds.

Prior to this work, the majority of what scientists knew about this brain region was based on studies from the 1940’s, which used electrical stimulation of single spots on the brain, causing a twitch in muscles of the face or throat. This approach using focal stimulation, however, could never evoke a meaningful speech sound. 

Chang and colleagues used an entirely different approach to studying the brain activity during natural speaking brain using the implanted electrodes arrays. The patients read from a list of English syllables – like bah, dee, goo. The researchers recorded the electrical activity within their speech-motor cortex and showed how distinct brain patterning accounts for different vowels and consonants in our speech.

“Even though we used English, we found the key patterns observed were ones that linguists have observed in languages around the world – perhaps suggesting universal principles for speaking across all cultures,” said Chang.

The article, “Functional organization of human sensorimotor cortex for speech articulation” is authored by Kristofer E. Bouchard, Nima Mesgarani, Keith Johnson and Edward F. Chang. It appears in the February 20, 2012 issue of the journal Nature. After this date, the article can be accessed at: http://dx.doi.org/10.1038/nature11911

This work was funded by the National Institutes of Health via grant #R00-NS065120, #DP2-OD00862 and #R01-DC012379 and by the Ester A. and Joseph Klingenstein Foundation.