How It Works: Computer Vision & Interaction

This project combines Artificial Intelligence with Creative Coding to turn your hand into a physical controller for the digital world.

1. What is p5.js?

Think of p5.js as a digital sketchbook. Unlike a normal program that runs once and stops, a p5.js sketch works like a flipbook animation. It relies on two main "loops":

• setup(): This runs exactly once when the page loads. It creates the canvas and wakes up the camera.
• draw(): This runs roughly 60 times every second. It clears the screen and redraws the video, the hand skeleton, and the ball so fast that it looks like smooth motion.

2. The AI "Brain" (HandPose)

We use a library called ml5.js, which provides a pre-trained AI model named HandPose. This model has been "trained" by looking at thousands of photos of hands. It doesn't see "fingers"—it sees a grid of pixels and predicts the X and Y coordinates of 21 specific joints (keypoints).

3. The Math of the "Pinch"

How does the computer know you are grabbing the ball? It uses Geometry. Every frame, we check the distance between two specific points:

• Point 4: The tip of the Thumb.
• Point 8: The tip of the Index Finger.

We use the Distance Formula: $d = \sqrt{(x_2-x_1)^2 + (y_2-y_1)^2}$. If that distance is small (less than 40 pixels), the code triggers a "pinch" event.

4. Spatial Coordinates

In computer graphics, the screen is a grid. However, webcams are usually "mirrored." To make the interaction feel natural, we flip the video and the tracking data. This ensures that when you move your hand to the right, the ball on the screen moves to the right too.