S3P 2015 Seminar

Emotion modeling and affective audio-visual classification, Prof. Sergio Benini, Università degli Studi di Brescia (IT)


The way people enjoy movies, music clips, or home-made videos has dramatically changed in the last years, thanks also to the introduction of new video on-demand services and technologies. A person that feels like watching a movie may rely on the suggestions of his or her group of friends, or on the opinions of a virtual community that shares the same interests. Alternatively, the possibility of tuning automatic recommendation systems according to the emotional state or wishes of users is becoming an active research field. However a direct mapping of audio-visual properties into emotion categories elicited by films is rather difficult, due to the high variability of individual reactions. To reduce the gap between the objective level of video features and the subjective sphere of emotions, a possible proposal is to shift the representation towards the connotative properties of movies: the set of shooting and editing conventions that help in transmitting meaning to the audience. Connotation provides an intermediate representation that exploits the objectivity of audiovisual descriptors to predict the subjective emotional reaction of single users. This is done without the need of registering users’ physiological signals. It is not done by employing other people’s highly variable emotional rates, but by relying on the intersubjectivity of connotative concepts and on the knowledge of user’s reactions to similar stimuli. This approach extends previous work by extracting audiovisual and film grammar descriptors and, driven by users’ rates on connotative properties, creates a shared framework where movie scenes are placed, compared, and recommended according to connotation, reaching the objective of meeting user’s emotional preferences.



Sergio was born in Verona, Italy. He has received his MSc degree in Electronic Engineering (cum laude) at the University of Brescia in 2000 with a thesis which won a prize granted by Italian Academy of Science. Between May 2001 and May 2003 he has been working in Siemens Mobile Communication R&D, on mobile network management projects. He received his Ph.D. degree in Information Engineering from the University of Brescia in 2006, working on video content analysis. During his Ph.D. studies, between September 2003 and September 2004 he has conducted a placement in British Telecom Research, Ipswich, U.K. working in the ``Content & Coding Lab’'. He is currently an Assistant Professor in the Telecommunications group of DII at the University of Brescia, Italy. In 2012 Sergio co-founded Yonder (http://yonderlabs.com/), a spin-off company specialized in Natural Language Processing, Data Science, and Multimedia Content Analysis.