Human-Computer Interaction and Visualization

HCI researchers at Google have enormous potential to impact the experience of Google users as well as conduct innovative research. Grounded in user behavior understanding and real use, Google’s HCI researchers invent, design, build and trial large-scale interactive systems in the real world. We declare success only when we positively impact our users and user communities, often through new and improved Google products. HCI research has fundamentally contributed to the design of Search, Gmail, Docs, Maps, Chrome, Android, YouTube, serving over a billion daily users. We are engaged in a variety of HCI disciplines such as predictive and intelligent user interface technologies and software, mobile and ubiquitous computing, social and collaborative computing, interactive visualization and visual analytics. Many projects heavily incorporate machine learning with HCI, and current projects include predictive user interfaces; recommenders for content, apps, and activities; smart input and prediction of text on mobile devices; user engagement analytics; user interface development tools; and interactive visualization of complex data.

Recent Publications

Preview abstract Every year an estimated 200,000 people go missing in the UK alone. Missing persons investigations involve challenging time-critical sensemaking tasks based on fragmented data sources. This paper describes a mixed-methods participatory study evaluating data science and AI-driven techniques (summarisation, fact extraction, and data visualisation) for supporting these investigations as part of a human-centered workflow. A series of human-AI interfaces were iteratively designed and tested with search officers and domain experts at Police Scotland. Based on findings, we describe: (1) user and information needs for missing persons investigations; (2) Insights on the benefits and challenges of applying LLM-based techniques in high-risk contexts; and (3) lessons for integrating AI for information and sensemaking tasks in policing more broadly. We highlight that in high-stakes contexts, where accuracy and context-sensitivity are paramount, AI techniques must be balanced with other approaches and designed in close partnership with end-users. View details
Preview abstract Human-Computer Interaction research and design pedagogy rely on idealized process models, such as the Double Diamond, to describe how user experiences are designed. These models assume an orderly, linear design process that, while easy to understand, fails to capture the iterative and collaborative reality of professional practice. A few qualitative studies have successfully captured this complexity -- still, they often suffer from retrospective narrative smoothing and lack systemic scale. To understand how design unfolds in real products, we analyzed historical snapshots of 102 Figma files from a multi-national technology company and investigated the true trajectories of the design process at scale. Our analysis reveals that while the established process models might be applicable, the operational details are highly non-linear. Rather than a straight line from ideation toward completion, design advances are repeatedly reset to the ideation stage as feedback is received. We argue that by treating design files as operational telemetry, the industry can move beyond abstract frameworks to build practices and collaborative tools that support the non-linear realities of modern product development. View details
Preview abstract The promise of tailored agent behavior is undermined by a critical explainability challenge: it is difficult to assess how closely and consistently the agent follows user-defined rules. As Large Language Models (LLMs) transition from static assistants to autonomous agents, developers have pioneered markdown-based rule files (e.g., GEMINI.md, CIDER_AGENT.md) to steer agent behavior and mitigate a "organizational context gap" that emerges when general-purpose models lack the "organizational context" necessary for contextually relevant results. This paper presents a qualitative study of 12 Google software developers (n=12) to investigate the authoring and efficacy of these agent rules. Our findings reveal that while rules are intended as technical steering mechanisms, they function as a "Black Box" of validation, where 12/12 participants rely on anecdotal "vibe checks" due to a profound lack of formal evaluation and explainability frameworks. We identify this opacity as a systemic Attribution Gap, which prevents developers from discerning whether a successful outcome was the result of deliberate logic or "pure luck." Paradoxically, these files serve a dual role as "Living Documentation," bridging technical instruction for AI with sociotechnical onboarding for humans. We argue for a transition toward library-level governance and rigorous traceability to transform agent customization from an ad-hoc craft into a human-centered science by revealing the internal "seams" of rule interpretation. View details
Mentorship, Interrupted: How AI Disrupts the GSoC Pipeline and Why Mentors Matter More
Ruizhe Xing
Mary Radomile
Zixuan Feng
Stephanie Taylor
Anita Sarma
2026
Preview abstract Generative AI (GenAI) can ease software contribution while making contributors and their work harder to assess. We investigate how this tension affects the sustainability of the Google Summer of Code (GSoC) mentoring pipeline and how stakeholders respond, through a sequential qualitative study of mentor mailing-list discussions and interviews with 15 mentors and organization administrators. Participants reported that GenAI helps with onboarding, coordination, and initial review but weakens evidence of mentees’ capability, understanding, and ownership, adding verification and governance work that strains newcomer learning and mentors’ willingness to continue. Drawing on reported strategies that distribute responsibility across mentees, mentors, organizations, and GSoC, we propose a GenAI Mentoring and Governance Decision Framework of five acceptable-use stances. Sustaining OSS mentoring requires protecting newcomer development and mentor capacity together, with governance adapted to local conditions. View details
“I want to be pushed, I want to grow”: Enabling social workers to design evaluations of LLM augmentation in their work
Anna Kawakami
Chloe Qianhui Zhao
Fernando Diaz
Haiyi Zhu
Kenneth Holstein
(2026), 1486–1487 (to appear)
Preview abstract Workers are increasingly asked to adopt AI systems to assist their work, yet are rarely given a voice in defining what meaningful AI augmentation should look like or how to evaluate for it. In this paper, we propose worker-driven AI measurement—a bottom-up approach to AI evaluation where workers collaboratively shape decisions about which tasks AI should augment, what “successful” augmentation looks like, and how it should be measured. We explore how to support this through a case study with 19 workers from a local school social work organization. Through a series of eight workshops, workers iteratively develop their own measurement goals for AI evaluation, systematize these goals, and thendesign a benchmark to capture how effectively an LLM can “challenge” them to reflect on their own assumptions and biases in the context of their day-to-day work. Workers collaboratively design and refine an LLM-as-a-judge rubric based on their professional and lived expertise. In validations of the worker-created benchmark, we find that there is strong agreement between worker and LLM judge ratings and that the resulting benchmark can differentiate performance across six state-of-the-art LLMs. Based on our case study, we discuss opportunities for future work to support worker-driven AI measurement as a complementary approach to existing topdown AI evaluation approaches. View details
Preview abstract The field of Human-Computer Interaction is approaching a critical inflection point, moving beyond the era of static, deterministic systems into a new age of self-evolving systems. We introduce the concept of Adaptive generative interfaces that move beyond static artifacts to autonomously expand their own feature sets at runtime. Rather than relying on fixed layouts, these systems utilize generative methods to morph and grow in real-time based on a user’s immediate intent. The system operates through three core mechanisms: Directed synthesis (generating new features from direct commands), Inferred synthesis (generating new features for unmet needs via inferred commands), and Real-time adaptation (dynamically restructuring the interface's visual and functional properties at runtime). To empirically validate this paradigm, we executed a within-subject (repeated measures) comparative study (N=72) utilizing 'Penny,' a digital banking prototype. The experimental design employed a counterbalanced Latin Square approach to mitigate order effects, such as learning bias and fatigue, while comparing Deterministic interfaces baseline against an Adaptive generative interfaces. Participant performance was verified through objective screen-capture evidence, with perceived usability quantified using the industry-standard System Usability Scale (SUS). The results demonstrated a profound shift in user experience: the Adaptive generative version achieved a System Usability Scale (SUS) score of 84.38 ('Excellent'), significantly outperforming the Deterministic version’s score of 53.96 ('Poor'). With a statistically significant mean difference of 30.42 points (p < 0.0001) and a large effect size (d=1.04), these findings confirm that reducing 'navigation tax' through adaptive generative interfaces directly correlates with a substantial increase in perceived usability. We conclude that deterministic interfaces are no longer sufficient to manage the complexity of modern workflows. The future of software lies not in a fixed set of pre-shipped features, but in dynamic capability sets that grow, adapt, and restructure themselves in real-time to meet the specific intent of the user. This paradigm shift necessitates a fundamental transformation in product development, requiring designers to transcend traditional, linear workflows and evolve into 'System Builders'—architects of the design principles and rules that facilitate this new age of self-evolving software. View details
×