TensorflowJS Demo

What Is This?

This is a demo of machine learning in the browser. It's based on Tensorflow JS and a pre-trained image recognition model called MobileNet. Using a webcam, we can extend the MobileNet model to recognize four new gestures or poses. After training, the model recognizes your pose and triggers the corresponding action.

How Do I Get Started?

Think of a pose for each of the blue buttons. This can be something as simple as leaning into or away from the webcam. Strike your pose and hold one of the buttons down until you've captured 10-30 images (the more training examples, the more accurate the model). Congratulations! You've just mapped a gesture to an action. Similarily, make a gesture and capture images for the rest of the buttons. When you're done you should be able to trigger the four actions just by making a simple gesture.

About the Confidence Score

The orange meter next to the buttons shows the confidence score or the degree of certainty. This is the computer saying for example: "I'm about 70% certain you just made the play gesture". Sometimes the camera sees elements from more than one of your training examples though. (This is why multiple confidence meters may move at once.) In this project, only gestures with an 80% confidence score or higher actually trigger the corresponding action.

Why ML in the Browser?

Ease of use, speed, and privacy - these are the primary advantages. Web projects are easier to deploy and access than mobile apps. The browser is free and compatible with both Windows and Mac. Training the model in the browser, using a technique called transfer learning, is much faster than training in the cloud. Once the session ends, your images are "dumped" - maintaining privacy.

What Is This?

How Do I Get Started?

About the Confidence Score

Why ML in the Browser?

Resources