Machine Perception
Research in machine perception tackles the hard problems of understanding images, sounds, music and video. In recent years, our computers have become much better at such tasks, enabling a variety of new applications such as: content-based search in Google Photos and Image Search, natural handwriting interfaces for Android, optical character recognition for Google Drive documents, and recommendation systems that understand music and YouTube videos. Our approach is driven by algorithms that benefit from processing very large, partially-labeled datasets using parallel computing clusters. A good example is our recent work on object recognition using a novel deep convolutional neural network architecture known as Inception that achieves state-of-the-art results on academic benchmarks and allows users to easily search through their large collection of Google Photos. The ability to mine meaningful information from multimedia is broadly applied throughout Google.
Recent Publications
StreamLens: An Open-Source ABR Simulation Environment for Multi-Region Buffering, Seeking, and Prefetching
Bei Zhao
Jinhui Song
Jiayu Zhu
Yih‑Chun Hu
2026
Preview abstract
ABR simulators are useful for studying ABR behavior under different network conditions. SABRE is an open-source simulator for evaluating ABR algorithms with different videos, network traces,
and ABR policies. However, SABRE has several limitations. First, it does not accurately emulate user seeks or their impact on the viewing experience. Second, it provides limited support for nonlinear buffering, where the client buffers non-contiguous video chunks. Furthermore, SABRE does not provide adequate analysis and visualization support for interpreting simulation results. This paper presents StreamLens, an advanced video simulator that supports: 1) pluggable ABR policies, including BOLA, BOLA-E, Dynamic, DynamicDash, and throughput-based policy; 2) configurable user seeks; 3) nonlinear buffering with customizable prefetching policy; 4) browser-based viewer for result visualization, including
stalls, seeks, rebuffering time, QoE, bitrate, buffer, etc. StreamLens allows developers to configure videos, network traces, and seek patterns, and to evaluate ABR performance across different setups, including ABR policies, linear versus nonlinear buffering, and user seek traces. It also provides metrics and visualizations that help developers inspect streaming behavior in a straightforward way. Several predefined scenarios are included as both evaluation examples and configuration templates, covering synthetic seek and prefetch settings, session-replay workflows, generated reports, and regression tests. The software is available publicly at
https://github.com/Slothbetty/StreamLens.
View details
ARM-Orchestrated Dynamic Workload Partitioning for Vision Model Training on FPGA Accelerators
Ram Babu Penugonda
Swaroop Arlagadda
2026
Preview abstract
Silent performance degradation and accelerator contention remain largely unaddressed in contemporary heterogeneous training systems, where Field-Programmable Gate Array (FPGA) offloading decisions are often static, heuristic-driven, or evaluated using idealized assumptions that ignore queueing effects, transfer overheads, and shared accelerator load. This paper introduces a runtime, price-conscious co-execution integration of CPU and FPGA to train deep learning models, where execution choices are dynamically made at the training stage granularity with a discrete latency cost model. The proposed method incorporates the CPU execution latency estimation, FPGA queue depth monitoring, time-varying service rate modeling, background contention, data transfer overhead, and switching hysteresis into a single decision mechanism that is driven by an ARM-based runtime monitor. The model is realistic FPGA queueing model, which is used to simulate shared accelerator contention and avoid optimistic assumptions of performance. Experimental evaluation on a representative convolutional training pipeline demonstrates that the cost-minimizing orchestration case achieves a mean training step time of 0.0213 s (±0.0036) with a throughput proxy of 47.03 steps/s, while maintaining bounded FPGA queue behavior and 99.94% stall avoidance relative to static FPGA execution. Conversely, a different execution setup that has a greater FPGA usage shows a greater queue buildup, an average step time of 0.0330 s, a maximum FPGA queue depth of 77 units, and a high level of congestion exposure, which demonstrates how crucial a queue-aware execution configuration is. These findings underscore the fact that the acceleration capability of the FPGA is not controlled by the raw compute capability but it is controlled by runtime sensitive and contention-sensitive orchestration, creating a principled and reproducible methodology to evaluate heterogeneous training systems in realistic conditions of accelerator sharing.
View details
Rolling Shutter Relative Pose Estimation Made Practical
Daniel Barath
European Conference on Computer Vision (ECCV) (2026)
Preview abstract
Rolling shutter (RS) cameras equip virtually all consumer devices, yet RS-aware relative pose estimation has remained impractical: the state-of-the-art solver requires a minimum of 20 point correspondences, making RANSAC-based robust estimation prohibitively expensive due to the exponential dependence of the iteration count on the sample size. We make RS relative pose estimation practical by introducing affine correspondences (ACs) into the RS two-view geometry. We derive novel \emph{RS-corrected affine constraints} that account for the coupling between point perturbations and the row-dependent essential matrix, providing two equations per correspondence beyond the standard epipolar constraint. Building on these constraints, we develop a linearized algebraic solver that estimates pose and RS motion from only 7 ACs. The solver exploits the physical smallness of RS parameters to linearize the constraints, eliminates the 12 RS unknowns via null-space projection, and solves the remaining degree-20 system via action matrices in 1.2\,ms. On the TUM RS benchmark, our method achieves the best pose and RS parameter accuracy among all tested methods and, uniquely among RS solvers, provides accurate translational velocity estimates -- which are poorly conditioned from point correspondences alone due to a $\vec{v}$-$\vec{t}$ coupling. On the global-shutter EuRoC MAV dataset, the solver achieves comparable accuracy to the standard 5-point algorithm, demonstrating that it generalizes well to the GS setting. Code will be made public.
View details
Preview abstract
High-fidelity removal of eyeglasses from video is a major challenge in facial attribute editing, as the underlying facial geometry is often obscured by complex refractive distortions and view-dependent specular reflections. While large-scale generative priors have shown promise in static image inpainting, they often lack the structural constraints necessary to maintain identity, expression, and pose, leading to visible “identity drift” in both static images and dynamic sequences. In this paper, we propose a novel distillation framework that addresses the stochastic nature of generative priors. Our pipeline first extracts high-fidelity outputs from Gemini, regularizes them via dense geometric landmark constraints to preserve identity, expression, and pose, and finally applies physically-based simulation of lens optics to generate realistic paired training data. This process transfers Gemini’s photo-realistic, multi-view knowledge into a specialized restoration architecture, JFSnet (Joint Feature-Spatial network). JFSnet integrates DINOv2-based semantic features with a convolutional decoder for spatial reconstruction, leveraging equivariance constraints to improve high-frequency detail preservation and stability. Evaluations show that our approach significantly outperforms existing diffusion and GAN-based baselines, achieving the lowest FID scores and ranking highest in user preference studies across visual fidelity, identity preservation, and temporal consistency.
View details
Integrating Level-Design for Editable, Shared Worlds in Diffusion Game Engines
Po Ryan
David Zhang
Amir Hertz
Gordon Wetzstein
Neal Wadhwa
Nataniel Ruiz
2026
Preview abstract
Video world models have shown immense promise for interactive simulation and entertainment, but current systems still struggle with two important aspects of interactivity: user control over the environment for reproducible, editable experiences, and shared inference where players hold influence over a common world. To address these limitations, we introduce an explicit external memory into the system, a persistent state operating independent of the model’s context window, that is continually updated by user actions and queried throughout the generation roll-out. Unlike conventional diffusion game engines that op-
erate as next-frame predictors, our approach decomposes generation into Memory, Observation, and Dynamics modules. This design gives users direct, editable control over environment structure via an editable memory representation, and it naturally extends to real-time multiplayer rollouts with coherent viewpoints and consistent cross-player interactions.
View details
Preview abstract
Latent Diffusion Models (LDM) face three compounding efficiency challenges in practical deployment: i) the temporal latency of iterative sampling; ii) the architectural overhead of heavy backbone parameter counts; and iii) the spatial cost of high-dimensional latent grids. While recent acceleration methods have made substantial progress on temporal distillation and architectural compression, the spatial axis is often inherited from the teacher tokenizer and treated as fixed. In this work, we recast latent resolution as an optimizable efficiency axis and introduce a unified framework that optimizes all three dimensions simultaneously. We introduce a novel strategy of Score-Compatible Tokenizer Distillation (SCTD), which leverages score-matching principles to align a spatially compact latent space with the induced distribution of a frozen, powerful teacher model, distilling the teacher's generative prior into a compressed, lower-dimensional compatible manifold. With flexibility provided by SCTD, we can surrogate a computationally heavy teacher backbone with a lightweight student architecture operating strictly within this new compressed space. Finally, we apply temporal distillation to collapse the sampling trajectory, producing a one-step generator that operates at peak efficiency. Our method yields a student generator outperforming existing single-axis acceleration methods in efficiency and throughput, while maintaining competitive generation quality. With reduced peak memory usage and latency, our method enables resource-constrained deployment and high-volume serving of high-fidelity LDM.
View details
Some of our teams
×
![]()