Data Modeling Strategies for Imbalanced Learning in TRECVID Search Task

Jelena Tesic

Intelligent Information Analysis
IBM T.J. Watson Research Center

Date: Friday, February 23, 2007
Place: Buchanan 1930
Time: 2:00 pm — 3:00 pm


Abstract:

In this talk, we will present a novel approach to the difficult problem of querying video databases using visual topics with few examples. Typically with visual topics the examples are not sufficiently diverse to create robust model of the user's need in descriptor's space. As a result, direct modeling using the provided topic examples as training data is inadequate. Otherwise, systems resort to multiple content-based searches using each example in turn, which typically provides poor results.

We explore the relevance of visual concept modelling and how they help refine the query topics. First, we propose a new technique of leveraging unlabeled data to expand the diversity of the topic examples as well as provide a robust set of positive and negative visual examples that allow direct modeling in the descriptor space by using development set for the modelling task. The proposed method outperforms TRECVID 2006 baseline visual and text fusion by over 18%. Second, we propose a way to leverage underlying semantics contained in the visual query topic examples to improve the search. Moreover, we explore the visual context in fusion with text and visual search baseline and examine how this component can help disambiguate word and visual senses. We apply the proposed methods in the overall video search system, and show how the underlined semantics of the dataset can significantly improve the overall visual search results, and enhance performance of other modalities by at least %15.

JELENA TESIC is a Research Staff Member in the Intelligent Information Analysis Group at IBM T. J. Watson Research Center, Hawthorne, New York. She received the Dipl. Ing. degree from the School of Electrical Engineering, University of Belgrade, Serbia, in 1998, and M.S. and Ph.D. degrees in Electrical and Computer Engineering from University of California, Santa Barbara, in 1999 and 2004, respectively.

Jelena's current research work is in the areas of multimedia management, content and social network analysis, scalable indexing, learning, and semantic mining. She has made several contributions to boost the top performance of the IBM team in 2004, 2005, and 2006 TREC Video Retrieval benchmark organized by NIST. Her prior work includes topic in computer vision, image processing, and pattern recognition applied to various scientific multimedia collections.

Host: Professor B.S. Manjunath, Electrical and Computer Engineering