A lot of things are changing due to pandemic and technology is the number one channel to help address the remoteness that this event brought to us. My country, Philippines, is one of the heavily affected countries and I could imagine that even after getting the right numbers in terms of vaccination, people will still be cautious about it. Things like filling in paper forms with a shared pen in public places will be a thing of the past, rather we would have to think about frictionless signups. This applies the same in devices where shared phones or tablets are used for registration.
Since then Artificial Intelligence (AI) technology has been consistent in helping this kind of situation by providing analysis on large sets of data and presenting them in a consumable format. Been seeing in social media face mask detection software and got interested on what’s available on the web to easily enable developers to come up with such. That’s when I came across with Handtrack.js — which allows developers to detect multiple hand poses from an image, in any orientation, in the browser. It uses an object detection neural network model, optimised to run in Javascript via Tensorflow.js.
With the passion I have in a low-code platform OutSystems, I tried integrating it in a driving scenario where I used Mario Kart to simulate that and here’s the output below…
I think there are more use cases where we could use this library. I can imagine a use case like signing on air just like how Iron man navigates inside his command center.
I had a chance to create a Handtrack.js Forge component for developers to easily consume this library on the apps. You can download this component now.
It produces a reusable block that can be consumed from UI which will show instantly a camera and will throw events based predictions on what Handtrack library was able to capture.
Part of the output is the Class parameter which detects 6 Hand Poses + Face — open, closed, pinch, point, face, pointtip, pinchtip. Score is tied up to it that shows the confidence level of the prediction.
I’ll be happy to see what other use cases you have in mind. Happy hand tracking!
John Salamat
Technical Lead & OutSystems MVP
Connect with John on LinkedIn
This article was originally published on medium.com.
