Giving the agent a hand: Google's AgentHands prototype explains with virtual hands
Google Research's XR prototype, published at CHI 2026, adds speech-synchronized virtual hands to a conversational agent: from printer repair to recipes, it points its way through physical tasks step by step.
The oldest pain of voice assistants is describing space. A sentence like "the second button on the left panel" asks the listener to build a map in their head, and that map is often built wrong. Google Research's AgentHands prototype is a concrete answer to that gap: it gives a conversational AI agent virtual hands, rendered in extended reality (XR), that move in sync with what it says. When the agent says "this button", it now actually points.
The technical design is elegant: through a meta-instruction, the large language model embeds gesture events into its verbal response, aligned to specific words; each event carries the gesture type and its parameters. At runtime a parser turns those events into time-stamped hand poses, and an animation system renders expressive hands synchronized with speech. The gestures fall into three families: deictic (pointing), iconic (conveying shape and size) and expressive (carrying emotion).
The shared examples are everyday ones: pointing at the printer's parameter panel, showing which drawer would fit next to the window, flagging danger with a hand when warning that the nozzle is hot, and a high-five when the job is done. The work was published at CHI 2026, human-computer interaction's most respected venue; Google frames it as a glimpse of spatial computing's future.
The bigger picture: this week we covered agents writing code, messaging each other and being set up for free on a phone. AgentHands is the moment the same agent reaches into the physical world through a headset: the assistant that answered on a screen becomes the craftsman gesturing at the workbench.
