Two actors wrapped in motion sensors circle each other as engineering researchers take note while standing at the perimeter of a USC Viterbi School of Engineering laboratory.
It’s an unusual partnership between the artists and engineers – a union the National Science Foundation expects will achieve more precise methods of modeling human behavior.
The foundation, under its Creative Information Technologies program, has awarded a three-year grant to faculty from USC Viterbi and the USC School of Theatre to study expressive human behavior through improvisation and motion-capture technology.
“The Holy Grail is to be able to build technologies to mimic aspects of human behavior,” said Shri Narayanan, the project’s principal investigator, the Andrew J. Viterbi Professor of Engineering and professor of electrical engineering and computer science.
Armed with such sophisticated technologies, scientists could build technologies to help autistic children, create advanced methods for recognizing human speech and visual behavior, and perhaps even quantify humor.
“The applications are limitless given the fundamental nature of the issue we’re addressing — understanding human behavior,” said Sharon Carnicke, a professor at the USC School of Theatre and Narayanan’s co-investigator on the project.
USC Viterbi dean Yannis C. Yortsos noted that the Narayanan-Carnicke partnership exemplifies the type of multidisciplinary alliance that yields results.
“Solving the world’s complex challenges will require a seamless blending of left-brain and right-brain skills,” Yortsos said. “The future impact of engineering and other disciplines depends on such partnerships.”
In the lab, Narayanan and Carnicke seek to collect digital representations of human emotion and behavior, one bit at a time. Drawing upon acting students, Narayanan and Carnicke have collected hundreds of sequences for analysis and created a database called the USC CreativeIT Database.
“It’s human data,” Narayanan explained. “What can we predict from these measurements? Can we develop a mathematical way of explaining patterns in human behavior?”
On one particular day, Carnicke supervised an exercise with two actors as Narayanan monitored a sophisticated motion-capture (mocap) system, which collects data from tiny sensors embedded in the actors’ black spandex mocap suits.
“The resulting motion-capture images make possible an intensely close analysis of what happens from moment to moment in the rehearsal hall,” Carnicke said. “It exposes the bones of the actors’ interactions.”
Carnicke instructed her actors to improvise within specific constraints. She gave each performer one action verb and one phrase they were allowed to utter to achieve opposing objectives, using an improvisation technique called “active analysis,” invented by Russian actor and director Konstantin Stanislavski.
Aside from these verbal instructions, the sky was the limit on physical interaction and expression. By controlling certain elements, the researchers could record data for other variables. And that’s where the real interest lies, according to Carnicke.
“The words are fixed,” Carnicke said. “Everything else is based on manipulating expressive voice and body.”
Will Thomas reach for Rose’s arm? Will Rose eventually get frustrated and raise her voice? The conflict is easy to see, but how their interaction will play out is the real question.
As the actors moved, the sensors recorded each movement. This carefully engineered data collection and annotation provided a method to quantify aspects of human communication and interaction.
The mocap technology is the same as that used on the sets of films such as Avatar and Shrek. But the purpose is different.
“Perhaps virtual humans or robots can eventually be designed to improvise upon themselves,” Narayanan said. “Not just deciding whether to do it — but how to do it, also.”
In the first year of research, the professors used scenes from Shakespeare and Chekhov to draw out the actors’ improvisation. Next year, the researchers plan to use real-life scenarios, such as how humans behave after waiting for 90 minutes in line at the DMV.
Carnicke noted that her experience working with Narayanan has led to “real discoveries” about how active analysis works for actors.
“Putting right- and left-brain people together for practice-based research has offered avenues of experimentation that would not have come up had I stayed only within my discipline,” Carnicke said.
Yet while the insights are real, Narayanan noted that quantitative understanding of human behavior is only a tool for scientific discovery. Using actors and acting techniques offers a solid baseline.
“But ultimately what matters are not the new insights they lead to about human behavior – but those that can be translated into useful applications,” Narayanan said. “These are ongoing research challenges.”
Potential applications span a number of domains that relate to behavior. They include addiction treatment, cognitive and behavioral therapy, customer care in business settings and global security applications in which socio-cultural behaviors come into play.
Carnicke will present a keynote speech based on the project at the Australian Dramatic Studies Conference in Canberra on June 30.