Talk by Dr Manish Gupta
Title: Multimodal Retrieval for Visual Question Answering, Image Search, and Moment Localization
Time: 29th Oct at 4-5 pm
Venue: EE Committee Room (Block II - 236)
Abstract: In this talk, I will present recent advancements in multimodal retrieval systems, focusing on three distinct and challenging computer vision tasks: visual question answering, image search, and temporal moment localization. I will introduce novel frameworks that leverage diverse input modalities (including text, sketches, and video) to interpret complex user intent and context. First, I will talk about Retrieval-Based Visual Question Answering, where the model must perform both a retrieval step over a pool of images and a reasoning step to generate a free-form answer. Second, I will move to the problem of Composite Sketch+Text Based Image Retrieval, proposing a new form of multimodal image search for objects that are difficult to name but easy to draw, or objects involved in complex, hard-to-sketch interactions. Finally, we will delve into the temporal domain with Video-to-Video Moment Retrieval, which is a form of moment localization where a query video is used to pinpoint a semantically matching event in a longer target video. Collectively, these works showcase the critical role of developing advanced multimodal systems to handle nuanced retrieval and reasoning challenges across diverse and complex user intents expressed via images and videos.
Bio: Manish Gupta is a Principal Applied Researcher at Microsoft. He is also an Adjunct Faculty at IIIT, Hyderabad and a visiting faculty at ISB, Hyderabad. He received his Masters in Computer Science from IIT Bombay in 2007 and his Ph.D. from UIUC in 2013. Before this, he worked at Yahoo! Bangalore for two years. His research interests are in the areas of deep learning, natural language processing, multimodal NLP and neuroscience. He has published more than 200 research papers in reputed refereed journals and conferences. He has also co-authored two books: one on Outlier Detection for Temporal Data and another one on Information Retrieval with Verbose Queries.
- 29 - 29 October,2025
- 04:00 pm - 05:00 pm
- EE Committee Room (Block II - 236)