Stony Brook University researchers develop GlanceWriter, a hands-free typing tool for people traveling without a keyboard and individuals with motor disabilities.
Have you ever noticed how your iPad camera follows you when you’re FaceTiming a friend? Now imagine that camera studying your eye movements to help you type words as you look at a keyboard.
The idea of typing using gestures is not new. In May of 2017, Abdul Basit from Pakistan set the world record for typing, “The razor-toothed piranhas of the genera Serrasalmus and Pygocentrus are the most ferocious freshwater fish in the world. In reality they seldom attack a human,” in only 17.5 seconds, and he used gesture typing — sliding his fingers to connect letters, instead of hitting each individual key. A lot has changed since then.
For the past few years, researchers have been trying to develop ways to type without using any hand gestures, so people don’t always have to carry their keyboards around, and so those with motor disabilities may also text their loved ones with ease.
A group of AI scientists at Stony Brook University recently developed GlanceWriter, an AI model that allows users to enter words by glancing over letters one by one, without causing severe eye fatigue.
A person looking at a keyboard to type
Unlike previous eye-gaze text entry techniques, they do not need a person to dwell too much on one key, or require them to consciously indicate the starting and ending letter of each word. GlanceWriter uses probability to determine which word in the dictionary matches what the eye gaze indicates, and combines the AI language models to determine if these words fit next to each other in the resulting sentence.
Wenzhe Cui, the paper’s lead author, said “To validate the effectiveness of our method, we performed two experiments.” The first one involved fourteen users between the ages of 24 and 32, all familiar with the QWERTY layout, and 10 of whom wore glasses. The two methods being compared — EyeSwipe and GlanceWriter — followed a participant’s gaze and showed them five options for each word, just like regular keyboards. They could select the intended option, or use their gaze to hit backspace and correct the word.
EyeSwipe (Pink) vs GlanceWriter (Green)
Rui Liu, a Ph.D. student at SBU, added, “In another experiment, we compared GlanceWriter with Tobii Communicator 5, a commercially available product, which represents the status quo of gaze-based input technology.” The team gathered 12 people for this experiment (7 of whom wore glasses) and had them interact with GlanceWriter and Tobii on Macbook Pros.
Tobii (Blue) vs GlanceWriter (Green)
“The results are promising,” said Zhi Li, another Ph.D. student at Stony Brook University’s Department of Computer Science. GlanceWriter is feasible and easy to familiarize yourself with. It improves the text entry performance over EyeSwipe, with a speed increase from 6.49 to 10.89 words per minute and an error rate reduction from 6.85% to 2.71%.
When compared with Communicator 5, GlanceWriter improves the word-level text entry speed from 7.41 to 9.54 words per minute, and reduces the error rate from 16.32% to 12.89%!
“This is a big step in making communication more accessible,” said IV Ramakrishnan, Professor and Associate Dean for Research at Stony Brook University. “Individuals with severe motion impairments also need to use computers, but they may not be able to control a mouse or work with a keyboard. If the motion impairment resulted from advanced neurological disease, they may also not be able to use a speech-recognition system as an input interface. A camera-based AI model that doesn’t strain their eyes and provides faster typing capabilities offers great potential.”
There are several projects that extend gesture typing to other devices, like tilt-based gesture typing, where the user draws a gesture by tilting the device, and others like mid-air gesture input, gesture input on a watch or on a ring. While researchers continue to expand their work in each of these fields, the team at Stony Brook plans to focus on their next step — incorporating out-of-vocabulary words that cannot be found in a dictionary, to increase GlanceWriter’s speed and accuracy.
SBU Professor Fusheng Wang, working at the intersection of Biomedical Informatics and Computer Science, said, “We also need to better understand the need of people with motion impairments, and plan to include them as participants in future research.”
Communications Assistant
Ankita Nagpal