The USC Institute for Creative Technologies’ SimSensei prototype has been named one of the year’s top 10 most promising digital initiatives by the NetExplo Forum, a partnership with UNESCO to recognize technologies that are shaping the future in education, health and communication.
SimSensei is a virtual human interviewer that can be used to identify signals of depression and other mental health issues. In recognizing ICT’s innovation, NetExplo’s organizers noted SimSensei’s potential as “a state-of-the-art tool that health care providers can use for screening and monitoring patients.”
SimSensei leverages ICT’s advances in developing interactive virtual humans — computer-generated characters that use language, have appropriate gestures, show emotion, and react to verbal and nonverbal stimuli. It also incorporates ICT’s MultiSense technology, which provides real-time tracking and analysis of nonverbal behaviors, including facial expressions, eye gaze, body posture and voice intonation.
From these signals, SimSensei can engage a user in conversation, follow up with appropriate questions based on an individual’s answers and body language, and use this data to infer signs of emotional distress. SimSensei is not designed for therapy or medical diagnosis, but it is intended as a support tool for clinicians and health care providers.
“Think about SimSensei as a diagnostic tool, like a blood sample,” said Louis-Philippe Morency, director of ICT’s MultiComp Lab, who co-leads the project with Skip Rizzo, director for medical virtual reality at ICT. “You send a blood sample to the lab, and you get the result. The people doing the diagnosis are the clinicians, but they use these objective measures to make the diagnosis.”
The SimSensei research and development effort is funded as part of the Defense Advanced Research Project Agency’s Detection and Computational Analysis of Psychological Signals project. This effort aims to address the high numbers of soldiers affected with post-traumatic stress by developing new tools to assess their mental health and enable them to seek timely help.
“Many people suffer in silence because they fear the stigma that may come from seeking help through traditional channels or because they simply don’t know where to turn,” Rizzo said. “Computer‐mediated care offers anonymity and access that may help reach these servicemen and women who need it most and could help support them in deciding to seek help with a live provider.”
Additional ICT projects are using the underlying MultiSense technology to enable richer human-computer interactions. These include an interactive virtual human audience that can give feedback to people looking to improve their public-speaking skills and virtual human role players that can help people prepare for job interviews.