In the Shortcuts project, we are developing methods to automatically and unobtrusively learn the social network structure that arises within a group based on data collected using the sociometer.

We have built the sociometer, a wearable sensor package, for measuring face-to-face interactions between people. We have developed a computational framework for learning the communication structure automatically from the sociometer data. The dynamics of a person’s interactions, and how one person’s dynamics affects the other’s style of interaction are also modeled.

More details are in my thesis and future progress will be available from the project page for 'Creating Dynamic Social Network Models from Sensor Data'

The Influence Model was developed by Chalee Asavathiratham as a generative mechanism to efficiently model the effects of many interacting Markov processes. His work showed how complex phenomena involving interactions between large numbers of chains could be simulated through this simplified model, such as the up/down time for power stations across the US power grid.In his description, all states were observed, and he did not develop a mechanism for learning the parameters of the model.We extend his model by adding the notion of hidden states and observations. We also develop an approximate algorithm for the parameters. Joint work with Brian Clarkson and Sumit Basu. (Technical Report)

The facilitator room project is an attempt to observe, model, and affect the interaction patterns of its users. This involves sensing the user's interaction in the room using computer vision and audition technologies, and then interacting with them using active components in the room (speakers, projectors, etc.) in order to facilitate interactions. Joint work with Brian Clarkson and Sumit Basu. (CVPR '01 Cues in Communication workshop paper)

Bayesian networks are an attractive modeling tool for human sensing, as they combine an intuitive graphical representation with efficient algorithms for inference and learning. Boosting is a general method of improving the performance of a classifier. This work focuses on developing algorithms for boosting Bayesian networks. By boosting the structure and parameters of Bayesian networks can we build better classifiers. Joint work with Jim Rehg and Vladimir Pavlovic. (ICPR '02 Paper) 

A framework for automatic detection, recognition and interpretation of facial expressions for the purpose of understanding the emotional or cognitive states that generate certain expressions. This work focuses on the study and analysis of facial expressions in natural conversations --- starting from data recording to feature extraction and modeling. All the analysis is done for person specific models. To allow the use of person specific models, a multi-modal person recognition system is developed for robust recognition in noisy environments. The study shows that it is very difficult to process and model events from spontaneous and natural interactions. The results show that some expressions are more easily identifiable, such as blinks, nods and head shakes, whereas expressions like a talking mouth and smiles are harder to identify. Data from conversations was recorded under different conditions, ranging from fully natural and unconstrained to having subjects' heads fixed in place. Observations made from comparing natural conversation data with constrained conversation data show that useful expression information can be lost due to imposing constraints on a person's movement. Thus, if automatic expression analysis is to be a useful input modality in different applications, it is necessary to study expressions in a natural and unconstrained environments. (Master's Thesis, ICPR '00 Paper)

The focus of this work is to develop a person identification technique that can recognize and verify people from unconstrained video and audio. We do not expect fully frontal face image or clean speech as our input. Our  algorithm is able detect and compensate for pose variation and changes in the auditory background and also select the most reliable video frame and audio clip to use for recognition. We also use 3D depth information of a human head to detect the presence of an actual person as opposed to an image of that person. Joint work with Brian Clarkson. (AVBPA '99 paper, Cover Feature from IEEE Computer Magazine)

Principal Component Analysis (PCA) decomposes high dimensional data into low dimensional sub-space. This decomposition is used for data compression and also widely used for classification tasks. The focus of this work is to derive discriminative principal components that is best suited for the classification task at hand. Performance of discriminative PCA was compared with regular PCA in classifying various facial features such as male/female, smiling/serious, children/teen/adult/senior, white/african-american/asian/hispanic etc. 

Oren and Poggio propose using wavelet template generated from an overcomplete basis set for  object detection. In this project we implement the algorithm proposed by Oren & Poggio and compare its performance with existing techniques that use skin color, facial symmetry and principal component analysis for face detection. We measure the robustness of both techniques in presence of illumination, scale and rotation. 

This goal of this project is to segment, localize and track eyebrows for recognition of different facial expressions. Color and horizontal edge detection is combined to localize eyebrows. A deformable contour model of the eyebrow is learned and used to generate features points on the eyebrows for expression recognition. Finally hidden Markov models are used to recognize different eyebrow expressions.

In this project, I worked on face and facial feature detection using color and shape information combined with valley energy functions. Once the features have been detected a generic wire-frame model of the head and shoulder is fitted to the first frame of the video sequence. The head movement is tracked by tracking the mesh node points ad warping the mesh triangles based on node triangle vectors. (Senior Thesis)