> Sociofeedback
Real-Time Feedback for Monitoring and Facilitating Discussions
Time Line
Ongoing since 2011
Team
JD, Shoko Dauwels, Martin Constable (NTU, ADM), Umer Rasheed, Yasir Tahir, Moe Elgendi (now at University of Alberta) and Sanat Sarda (now at National University Singapore), Nadia Magnetat-Thalmann (NTU, IMI), Daniel Thalmann (NTU, IMI)
Problem
Humans have varying individual characteristics such as personality traits, behavior, emotions etc. When people communicate, all these aspects in different combinations are reflected in their speaking mannerism such as how much an individual speaks, how much he interrupts, his pitch, volume, speaking rate, etc. Speaking mannerisms in turn play a vital role for the meetings to be conducive and efficient. If speaking mannerisms become mutually compatible and aligned, the meetings are more likely to be productive. Hence, it is desirable to receive appropriate feedback on an individual's speaking mannerisms so that necessary modification could be made in an on-going conversation. We aim to develop a system that can provide a real-time feedback on social behavior in conversations, assisting individuals to adjust their speaking mannerisms for each other. Such a system can play a vital role in boosting the effectiveness of job interviews, group discussions, coaching sessions, public speaking etc.
Contribution
Our system provides in real-time feedback about speaking mannerisms, generated from audio and video signals. The system extracts several speech and visual cues from the on-going conversation and these cues are used as features and employed in machine learning algorithms to extract higher level characterisation of the speaking mannerisms such as level of dominance, interest, discord, consistency and mirroring etc. That information is eventually exploited to generate real-time feedback for every participant in the meeting. It can inform the speakers about their speaking mannerisms, and if needed, provide guidelines.
Several platforms are being explored to provide feedback. This includes a Skype application that can provide feedback to callers via messages in Skype. Similarly, an android application is being developed to provide real-time feedback to callers. Also, we are collaborating with ADM (School of Art, Design and Media) to provide retrospective feedback via animations that summarize the salient features of social interactions during the discussion. Apart from that, we are in the phase of implementing our system on Nao robot aiming to develop a robot that can act as social moderator.
Figure: (left) Illustration of turn-taking, interruption, failed interruption and interjection derived from binary speaking status (speaking and non-speaking). Periods of speaking and non-speaking are indicated in black and white respectively ; (right)Retrospective feedback in the form of animations: Continuous turn taking (top) and excessive interruptions (bottom).
References
Sanat Sarda, Martin Constable, Justin Dauwels, Shoko Dauwels (Okutsu), Mohamed Elgendi, Zhou Mengyu, Umer Rasheed, Yasir Tahir, Daniel Thalmann, Nadia Magnenat-Thalmann, Real-Time Feedback System for Monitoring and Facilitating Discussions, 4th International Workshop on Spoken Dialog System IWSDS 2012, November 28-30, 2012, Paris, France. [ PDF ]
Sanat Sarda, Martin Constable, Justin Dauwels, Shoko Dauwels (Okutsu), Mohamed Elgendi, Zhou Mengyu, Umer Rasheed, Yasir Tahir, Daniel Thalmann, Nadia Magnenat-Thalmann, Real-Time Feedback System for Monitoring and Facilitating Discussions, in Natural Interaction with Robots, Knowbots and Smartphones - Putting Spoken Dialog Systems into Practice, pp. 375-387, 2013, Springer. [ PDF ]
Rasheed, U., Tahir, Y., Dauwels, S., Dauwels, J., Thalmann, D., & Magnenat-Thalmann, N., Real-time comprehensive sociometrics for two-person dialogs, In Human Behavior Understanding, Lecture Notes in Computer Science Volume 8212, pp. 196-208, Springer International Publishing, 2013. [ PDF ]
Y. Tahir, U. Rasheed, K. Hui, S. Dauwels, J. Dauwels, D. Thalmann, N. Magnenat Thalmann, NAO Robot as a Social Mediator: A User Study, International Conference on Social Robotics (ICSR2013), Bristol, UK, October 27-29, 2013.
Y. Tahir, Umer Rasheed, Shoko Dauwels, and Justin Dauwels, Perception of humanoid social mediator in two-person dialogs, In Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction (HRI 2014), pp. 300-301, 2014. [ PDF ]