With the nature of work shifting and remaining online, video chat communication for most of us has just become part of our daily lives.

Consequently, researchers have identified psychological issues like ‘Zoom fatigue’ due to several reasons, some examples being:

  • An excessive amount of close-up eye contact with multiple people at once
  • Facing their own on-screen reflection for long periods
  • Reduced mobility through a fixed field of view, limiting their usual nonverbal cues.

That’s where the Solarflare Launch team jumped in to test out some intuitive object hand interactions within video calls. Why? To allow people to trigger responses with less disruption, higher engagement, and reduce their overall cognitive load.

How We hacked IT

Using Google’s AI, MediaPipe, our apprentice developed a hand gesture recognition system that would track the user’s x and y landmark coordinates through their webcam and trigger intuitive emoji reactions.

By firstly expanding upon the MediaPipe JavaScript base code, points on each joint of the hand acted as landmarks relative to the HTML canvas. Through being able to determine the measurement between multiple landmarks on the hand, he was able to write conditional statements that would allow the user’s hand gesture to be recognised with a high degree of accuracy every frame/second.

In the final step, once the gesture was determined, a string was returned of the emoji, with a separate script written to handle the falling animation and follow the user’s hand position.

The result

With our designer creating a branded set of bespoke emojis we tested out the augmented reality add-on with the wider team. Take a look a the outcome below.

As platforms, companies, and brands continue to explore hybrid working as well as their digital twins, we’re faced with that recurring question of what will be that communication sweet spot.

To try out the filter for yourself, take a look at this page on your desktop!


Now it's your turn

Give yourself a little wave, or even a rock on sign.

Why not try…