What's new
There are more than 200 sign languages in the world, and for roughly 70 million deaf and hard-of-hearing people, they serve as the primary means of communication. Until recently, artificial intelligence was barely used to translate these languages in everyday products. Google DeepMind decided to fix this: the new SL2T (sign-language-to-text) model translates sign language into text and is already being integrated into familiar apps.
Users will be able to "dictate" with gestures instead of typing on a keyboard. The feature is rolling out in Gboard and the Live Transcribe captioning app on Pixel 11 smartphones — the first pairing will be American Sign Language (ASL) and English. More languages and devices are promised later.
What does this mean in practice? You can search the web with gestures, compose messages or documents, and even talk to the Gemini assistant. In Live Transcribe, you'll be able to respond with gestures right in the middle of a conversation, without switching to text chat. According to testers, this kind of communication is faster and more natural than typing in English. For the deaf community, this is more than just convenience: sign language is the foundation of cultural identity, and being able to use it in the digital world means a great deal.
Why it's hard
Sign languages are not "English on the hands" — they are independent natural languages with their own grammar and vocabulary. That's why sign-to-text translation can't be reduced to matching each gesture to a word one by one. Meaning is conveyed simultaneously through hand, body, and head movements plus facial expressions, and the model has to track all of this with high accuracy in real time.
Early attempts like "sign language gloves" were fundamentally limited precisely because they treated signs as a code for ordinary words. Real machine translation requires understanding the structure of the language, including spatial constructions and non-manual markers. Modern models learn from large datasets: SL2T used more than 100,000 hours of video across over 50 sign languages, with roughly a quarter dedicated to ASL. Joint training across different languages and dialects helps find common patterns and yields better results than separate models for each language.
How the translation works
Instead of processing the video stream, the model uses a compact representation: it tracks the coordinates of key body points and turns that sequence of coordinates into text. This solves two problems at once — it reduces computational load and protects privacy. The original video is deleted immediately, and only geometric coordinates are sent to the server for translation. Point tracking is handled by the built-in MediaPipe Holistic model.
Translation happens directly, without intermediate annotations (glosses). This approach makes it possible to convey aspects like non-manual markers and spatial constructions, and avoids artificial vocabulary constraints. Quality scales with data volume, as benchmarks confirm: on the FLEURS-ASL (sd-test) test, the model achieves a zero-shot score of 70 BLEURT, far ahead of all previous systems.
Practical improvements matter too. The developers minimized streaming latency, taught the model not to hallucinate on non-sign input, accounted for left-handed users (about 10% of them), and improved recognition of one-handed signing when the other hand is holding a smartphone.
Privacy and the future
This is the first time AI for sign languages has moved out of the lab and into mass consumer products. SL2T technology is already available on some devices, and more languages and platforms are expected down the road. For the deaf and hard-of-hearing community, this opens the path to full digital inclusion: now you can communicate freely in your native language using an ordinary smartphone.
The future of such systems lies in more natural and faster interaction between sign and spoken languages. The next generations of models will be able not only to translate individual phrases but also to sustain full conversations, preserving the nuances of emotion and style. And that will change not just technology, but the everyday lives of millions of people.
