Senior Research Scientist, NVIDIA
I work on world foundation models and generative AI at NVIDIA — building models that understand and simulate the physical world, alongside generative systems for audio, speech, and music. Most recently I led joint audio-visual generation for Cosmos 3, NVIDIA's open omnimodal world foundation model for physical AI.
I am a Senior Research Scientist in the Cosmos Lab at NVIDIA. My current work spans world foundation models for physical AI, generative audio, speech and singing synthesis, video-to-audio generation, and multimodal understanding. Most recently, I was the tech lead for joint audio-visual generation in Cosmos 3, NVIDIA's open omnimodal world foundation model that jointly reasons over and generates language, image, video, audio, and action.
Before NVIDIA, I was an AI Scientist at Electronic Arts working on expressive speech synthesis. I received my Ph.D. in Music Technology from Georgia Tech, where I worked on weakly supervised methods for identifying musical instruments in audio. I also hold B.Tech. and M.Tech. degrees in Computer Science and Engineering from IIT Kharagpur.
My path into machine learning began with music — I still play guitar, and that interest in understanding and creating sound runs through much of my research.
An open frontier world foundation model that unifies physical reasoning, world generation, and action generation across language, image, video, audio, and action, via a two-tower mixture-of-transformers architecture.
An open audio-language model family for long-form understanding and reasoning over speech, sound, and music.
A benchmark for evaluating models that reason jointly over visual, audio, and textual signals in long, real-world videos.
A flow-based world foundation model for physical AI, enabling controllable world simulation and video generation for downstream training.
A foundational generative audio model for flexible text, audio, and music generation tasks.
A platform of world foundation models, tokenizers, data pipelines, and post-training tools for physical AI.
A singing voice synthesis system for multilingual and multi-style score-based generation with explicit performance controls.
A multilingual and multiaccented text-to-speech system with explicit controls over speaker, accent, language, pitch, and energy.
A survey of music performance analysis across measurement, performer intent, listener perception, and MIR.
An approach to text-to-speech prosody transfer using interpretable pitch and loudness features.
Analyzes failure modes of loss-gradient saliency methods used to interpret audio classification models.
Semi-supervised methods for multi-label audio classification when only part of the labels are available.
Neural models that use score information to assess the quality of student music performances.
Studies attention over time-frequency representations for multi-label instrument recognition.
Detects sampled audio segments that are reused across polyphonic music tracks.
Audio features for automatically assessing the quality of student instrumental performances.
A dataset and preliminary methods for automatically logging instrumental practice sessions.
Patent family covering neural speech-driven video generation from voice, image features, and facial landmarks.
Patent family for expressive text-to-speech generation with controllable speaking style and speaker attributes.
Expressive speech generation methods for interactive game development and accessibility workflows.
Methods for robust expressive speech audio generation in game content creation pipelines.