How to Build Interactive Motion Apps Using SimpleOpenNI Interactive motion apps have changed how we interact with software, moving beyond keyboards to natural human gestures. SimpleOpenNI is a powerful wrapper for Processing that brings the capabilities of depth sensors like the Xbox 360 Kinect directly into your creative coding workflow. This guide will walk you through setting up SimpleOpenNI and building your first motion-controlled application. Understanding the Technology Stack
Building motion apps requires a combination of hardware and software working in harmony. SimpleOpenNI acts as the bridge between your depth sensor and your development environment.
Processing: An open-source graphical library and IDE designed for the visual arts and creative coding.
OpenNI: An open-source framework that provides a standard API for 3D sensors and body tracking.
SimpleOpenNI: A library created specifically for Processing that simplifies OpenNI commands into accessible code.
Hardware: A compatible depth camera, most commonly the original Microsoft Xbox 360 Kinect (Model 1414). Setting Up Your Development Environment
Before writing code, you must install the correct library versions, as modern software updates can sometimes break legacy depth-sensing frameworks.
Install Processing: Download and install Processing version 2.2.1 or 3.x (Processing 2.2.1 offers the highest stability for legacy SimpleOpenNI releases).
Install Hardware Drivers: Connect your Kinect to your computer. On Windows, you may need the Kinect for Windows SDK v1.8 or OpenNI drivers. On macOS, use Homebrew to install the required libusb libraries.
Add the SimpleOpenNI Library: Open Processing, navigate to Sketch > Import Library > Add Library…, search for SimpleOpenNI, and click install. Alternatively, manually download the library files and drop them into your “libraries” folder inside your Processing sketchbook directory. Initializing the Depth Stream
The first step in any SimpleOpenNI sketch is initializing the camera object and starting the data tracking loops. The following baseline code opens the camera and displays a live depth map.
import SimpleOpenNI.; SimpleOpenNI context; void setup() { size(640, 480); context = new SimpleOpenNI(this); if (context.isInit() == false) { println(“Can’t init SimpleOpenNI, check your camera connection!”); exit(); return; } // Enable depth map generation context.enableDepth(); // Enable user skeleton tracking context.enableUser(); } void draw() { context.update(); image(context.depthImage(), 0, 0); } Use code with caution. Tracking the Human Skeleton
SimpleOpenNI excels at identifying human forms and mapping them to a digital skeleton. Once a user stands in front of the camera, the library automatically attempts to calibrate and track their joint coordinates. Calibration and User Events
To track a skeleton, you must implement callback functions that trigger when a new user is detected or lost.
void onNewUser(SimpleOpenNI curContext, int userId) { println(“onNewUser - userId: ” + userId); curContext.startTrackingSkeleton(userId); } void onLostUser(SimpleOpenNI curContext, int userId) { println(“onLostUser - userId: ” + userId); } Use code with caution. Drawing the Skeleton Joints
Once tracking is active, you can request the 3D coordinates of specific joints, convert them to 2D screen space, and use them to draw visual elements.
void drawSkeleton(int userId) { // Create a vector to store joint coordinates PVector jointPos = new PVector(); // Get the 3D position of the user’s right hand context.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_HAND, jointPos); // Convert 3D space to 2D screen coordinates PVector screenPos = new PVector(); context.convertRealWorldToProjective(jointPos, screenPos); // Draw an interactive circle on the hand fill(255, 0, 0); ellipse(screenPos.x, screenPos.y, 20, 20); } Use code with caution.
To see the skeleton, loop through all tracked users inside your main draw() function and call your custom skeleton drawing function:
IntVector userList = new IntVector(); context.getUsers(userList); for (int i = 0; i < userList.size(); i++) { int userId = userList.get(i); if (context.isTrackingSkeleton(userId)) { drawSkeleton(userId); } } Use code with caution. Building a Simple Motion Interaction
Now that you can track the right hand, you can turn this setup into an interactive application. Let’s create a simple air-painting app where your hand acts as a digital paintbrush.
Create a Canvas Background: Draw a persistent background layer that does not clear on every frame.
Track Hand Movement: Store the current hand position and the previous frame’s hand position.
Draw Lines: Use Processing’s line() function to connect the coordinates, allowing the user to paint in mid-air.
import SimpleOpenNI.; SimpleOpenNI context; PGraphics canvas; PVector prevHandPos = new PVector(); void setup() { size(640, 480); context = new SimpleOpenNI(this); context.enableDepth(); context.enableUser(); canvas = createGraphics(width, height); canvas.beginDraw(); canvas.background(255); canvas.endDraw(); } void draw() { context.update(); image(canvas, 0, 0); IntVector userList = new IntVector(); context.getUsers(userList); for (int i = 0; i < userList.size(); i++) { int userId = userList.get(i); if (context.isTrackingSkeleton(userId)) { PVector hand3D = new PVector(); PVector hand2D = new PVector(); context.getJointPositionSkeleton(userId, SimpleOpenNI.SKEL_RIGHT_HAND, hand3D); context.convertRealWorldToProjective(hand3D, hand2D); // Paint on canvas if hand position is valid if(prevHandPos.x != 0 && prevHandPos.y != 0) { canvas.beginDraw(); canvas.stroke(0, 102, 153); canvas.strokeWeight(5); canvas.line(prevHandPos.x, prevHandPos.y, hand2D.x, hand2D.y); canvas.endDraw(); } prevHandPos.set(hand2D); } } } void onNewUser(SimpleOpenNI curContext, int userId) { curContext.startTrackingSkeleton(userId); } Use code with caution. Tips for Optimizing Motion Apps
Lighting Conditions: Depth sensors use infrared light. Avoid direct sunlight or reflective surfaces in the camera’s field of view to prevent tracking glitches.
Distance: Position users between 1.5 to 3.5 meters away from the sensor for optimal skeleton detection.
Frame Rate: Limit heavy graphic rendering inside the main loop to keep your application running at a smooth 30 frames per second, matching the hardware output of the sensor. If you want to expand this sketch, tell me: What depth camera hardware model are you using? What operating system and Processing version are installed?
What specific interaction (e.g., gesture trigger, games, avatar mirrors) do you want to build next?
I can provide the targeted updates and code snippets for your exact project needs.
Leave a Reply