How often do you wonder where your day went, if it was as productive as you wanted it to be, and if not, what disrupted your schedule? For over a decade, people have been using wearable devices, like a smartwatch, to count how many steps they took, measure their heart rate, and track their sleep patterns. But what if you could also check how much time you spent cooking dinner, ironing your clothes, or playing the guitar?

According to Yifeng Huang, a Ph.D. candidate at Stony Brook University, the biggest challenge faced by wearable technology is not in making it intelligent and more conversational, but lies at a more fundamental level. The devices we wear today are only capable of counting the actions they have been trained to recognize and measure. “This specialization restricts their adaptability,” Yifeng adds. “Consequently, relying on action-specific counters proves inadequate and unscalable in managing the wide range of possible action categories encountered in the real world.”

The practice of using examples to teach a machine how to count a variety of actions isn’t novel. However, when it comes to wearable technology, most approaches involve some form of physical training. In order to help scale the technology, Yifeng collaborated with researchers from VinAI and The Posts & Telecommunications Institute of Technology to work on a project titled ‘Count What You Want.’ They started by categorizing countable actions under various domains, including Physical Exercise, Daily Routines, Chores, Factory Activities, Kitchen Activities, Rehabilitation Training, and Instrumental Exercise. These activities were then measured by using a novel framework that allows users to provide exemplars of the actions they want to count, by vocalizing predefined sounds — “one,” “two,” and “three” — as they perform the relevant gesture.

This chart represents 1502 entries of wearable-device data from 37 subjects across seven broad categories: kitchen activities, household chores, physical exercises, factory activities, daily routines, instrument-involved activities, and rehabilitation training

This approach, which is innovative and distinct from existing works in various fields, was validated using experimental evaluations, demonstrating that the method yields low counting errors, even for novel actions performed by people not encountered in the training data.

Professor Shubham Jain of Stony Brook’s Computer Science Department, who specializes in wearable technology, comments, “Counting is an important problem in wearable systems that aim to detect events in real-world settings. This work presents a simple yet intuitive approach to solving it and has applications in many domains, particularly fitness and rehabilitation."

Professor Minh Hoai Nguyen adds, “Our experiments show the viability of this method in counting instances of actions that were not part of the training data. The average discrepancy between the predicted count and the ground truth value is 7.47, significantly lower than the errors of previously used methods. This technology can not only be used to track our fitness and healthcare goals but also to monitor any activities captured by wearable devices.” The applications, he believes, can tremendously change the way we collaborate with machines.”

Communications Assistant
Ankita Nagpal