Vivek Kumar
Vivek Kumar is a Research Director at Google DeepMind, where he leads the Sound Team within Frontier AI. His team's work spans audio understanding, generation, and modification, building the core audio capabilities that let AI hear, speak, compose, and transform sound. Prior to joining Google, he incubated and led the Audio AI research team at Dolby Laboratories.
Research Areas
Authored Publications
Sort By
SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
Yong Cheng
Zhiruo Wang
Yanping Huang
David Ross
Yonatan Bisk
Alex Hauptmann
Lu Jiang
NeurIPS (2023)
Preview abstract
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks. Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over 25%.
View details
