Hand analysis has played a significant role in Computer Vision applications. For instance, there are machine learning algorithms that can identify sign language hand gestures and assist people who are deaf. While many Computer Vision works study hands and human bodies separately, they seldom capture them together. 

Professor Minh Hoai Nguyen from the Stony Brook Department of Computer Science and PhD student Supreeth Narasimhaswamy conducted research titled “Whose Hands are These? Hand Detection and Hand-Body Association in the Wild.” The work addresses the hand-body association task, which detects hands and simultaneously localizes the correct person associated with each hand in an image. 

“This is one of the few existing works which estimates the entire human body location based on the hand location,” says Narasimhaswamy.

Hand-Body association is useful for action recognition and scene understanding, especially when multiple people are present. It has sign language recognition, virtual reality, and safety applications. 

“Let’s say you have people working in a factory. They likely have to use different tools or machines and could be working with dangerous devices. You can have a monitoring system where you observe the hands to detect ongoing activity. If they’re operating the device dangerously, you can detect the corresponding person and potentially alert them,” says Narasimhaswamy.

The method is built upon MaskRCNN, a two-stage object detector with an additional novel Hand-Body Association Network. The network can detect and identify corresponding hands and bodies together. The network consists of two modules. The first module is the Overlap Estimation Module, which “uses the visual features of hands and bodies to estimate if they can overlap.” The second module, the Positional Density network, “uses hand features to estimate a density over possible body locations for each detected hand.”  

This research also contributes a dataset of “around 20,000 images with bounding box annotations for more than 57,000 hands and 63,000 body instances.” 

Thanks to contributions from researchers like Nguyen and Narasimhaswamy, artificial intelligence is on its way to developing into systems that make working environments safer for humans. 

“I would say this was a fun project to work on,” says Narasimhaswamy.

-Sara Giarnieri, Communications Assistant