Kinect
Introduction
A pixel is the smallest unit of an image.In a black and white image, the pixel will be some shade of gray between 100% black and 100% white. In a color image, pixels are described in terms of RGB values. In all images, a pixel is described by a number that represents its intensity. The larger the number the more memory the pixel needs. A colored image needs more memory than a grayscale image.
The image captured by the Kinect is 640 pixels by 480 pixels (or 307,200 pixels). Each pixel's number represents both its grayscale intensity and its distance in space. You can use these numbers to either interpret the data as an image or to calculate the distance of the object that the pixel is a part of.
In the captured frame, each pixel can describe the distance between the object and the Kinect. The brightest part of the image (closest to 255) is closest to the Kinect, while the darkest part of the image (closer to 0),is the farther away from the camera.
Lesson
Read Chapter 2 of Greg Borenstein's Making Things SeeSetting up SimpleOpenNI
Simple SimpleOpenNI
Images and Arrays
Getting Interactive
Setting up SimpleOpenNI
SimpleOpenNI is a library that provides access to all of the data from the Kinect, as well as a series of tools and helpers.
- Install PrimeSense's OpenNI in your applications folder. This application communicates with the Kinect in order to access and process the data
MAC
Download OpenNI_NITE_Installer-OSX-0.xx.zip and Max Rheiner's SimpleOpenNI
- On the MAC you should see the following:
- install.sh
- LGPL.txt
- Readme.txt
- data
- Open Terminal (Command_CTRL+U)
- Change directory to the downloaded folder (type cd and a space, then drag the folder into the terminal window, then press RETURN)
- Run the installer with the command:
Type your password when prompted.
sudo ./install.sh
Windows
Installing OpenNI requires a number of pieces of supporting software that all have to be installed for things to work. A company called ZigFu provides a oneclick installer that puts all the pieces in place for you.- Download the ZigFu DevBundle (or download all the files from here.
If you’ve installed the Microsoft Kinect SDK, you need to uninstall it. The OpenNI software does not work with the Microsoft Kinect drivers.
- On the MAC you should see the following:
- Install the SimpleOpenNI library
in your Processing>Libraries folder
- Quit and restart Processing if it was open
- Plug in the Kinect and make sure it is facing you.
- Test out the install by running one of the built-in examples that ships with
SimpleOpenNI. Inside of OpenNI, find DepthImage and double-click it.
This is a basic sketch. Hit
the play button in the top left corner of the Processing window to run the sketch.(Slow start
up is normal when using SimpleOpenNI.)
- If the sketch fails, you might see thisIf you see that your Kinect is not plugged in or not fully powered. Make sure your Kinect is connected to a power outlet and plugged into your computer’s USB port. If you’re working on a laptop also make sure that your laptop is plugged into power. Some laptops will provide inadequate power to their USB ports when running off a battery.
Invalid memory access of location 0x8 eip=0xb48d8fda
Simple Kinect Sketch
The color of each pixel in the depth image represents the distance of that pixel in the scene.
Objective:When you print on the image, Processing will print out information on individual pixels
- Connect your Kinect
- Open Processing and create a new sketch
- Paste in the following:
import SimpleOpenNI.*; SimpleOpenNI kinect; void setup(){ // double the width to display two images side by side size(640*2, 480); kinect = new SimpleOpenNI(this); kinect.enableDepth(); kinect.enableRGB(); } void draw(){ kinect.update(); PImage depthImage = kinect.depthImage(); PImage rgbImage = kinect.rgbImage(); image(depthImage, 0, 0); image(rgbImage, 640, 0); } void mousePressed(){ color c=get(mouseX, mouseY); println("r: "+red(c)+"g: "+green(c)+"b: "+blue(c)); }
- Run the sketch. You might find that some of the values are surprising. This is because when you look at the world you see colors as relative to the environment. But this is
not how a digital camera sees them.To test this out, look for the whitest part of a color image. It if were 100% white, you should get 255 or each R, G, B values
- If you click on te depth image, you will get brightness data (shades of gray). But how do you translate brightness to distance?
The kinect can detect distances between 20 inches and 25 feet with millimeter precision.
You can figure out that a brightness value of 96 is 37% between 0 and 255, or about 14.5 feet away. Chances are this distance is not correct. The issue is that an 8bit pixel is manageable to work with, but is not as accurate as it can be. The
kinect can be more accurate, but the processing time would also get extended.
But if you use the more precise piel sparingly, you should be able to get accurate distances without slowing your application down too much.
Images and Arrays
While we think of images as columns and rows, an image is actually described with a 1-dimensional array.
Accessing the array of pixels directly allows you to run your program faster, and allows you to work with higher resolution data, as well as enabling you to work with more than one pixel at a time.
- Open Photoshop by clicking on the icon on the dock
- Create a new 8-bit image by pressing Command+N
- Create a new layer (Command+Shift+N)
- Select Edit>Fill and fill image with 50% gray
- Save the file as Photoshop Raw
- This file is stored as a 1 dimensional array, which means every pixel is assigned a number in sequential order.
1 2 3 4 5 6 7 8 - Open Audacity.
- Click on Project>Import Raw Data
- You should see a straight line because the image is uniform (there is no change).
- Back in Photoshop, add some noise
- Resave and re-open in Audacity.
- Now back in Photoshop add splotches of black and white.
- Resave and reopen in Audacity.
- Audacity plays the image file from top left to bottom right, reading each pixel in sequential order. The frequency is the width of the image.
The first pixel is 0, 0.
- Even though graphics programs stores image data in a one-dimensional array, you want to be able the think about the data in terms of two-dimensions.
Each pixel can be described with this formula:int pixel=x+(y*width_of_image)
Getting Interactive
Tracking the nearest object
If you create a sketch that tracks an object closest to the kinect (let's say your hand), you could have your hand control shapes or pixels.Finding the closest pixel is like finding a minimum or maximum value. You store the first value of the array as your target, you iterate through the array and compare each subsequent value to the target and update the target as necessary.
import SimpleOpenNI.*; SimpleOpenNI kinect; int closestValue; int closestX; int closestY; void setup(){ // double the width to display two images side by side size(640, 480); kinect = new SimpleOpenNI(this); kinect.enableDepth(); } void draw(){ //set closestValue to the farthest possible value (8000) ____________________________ kinect.update(); //create an array of ints to hold the depth values ____________________________ //create a for loop with y as the variable. //start at 0 and continue as long as y is less than the //height of the image, increment by 1 ____________________________ //create a for loop with x as the variable. //start at 0 and continue as long as x is less than //the width of the image, increment by 1 for (int x=0;x<640;x++){ //create a variable to hold the pixel //remember that the formula is //int pixel=x+(y*the_width_of_the_image) ____________________________ //set the currentDepth to the current iteration in the depth value array ____________________________ //compare if the current depth is greater than 0 and less than //the closestValue ____________________________ //update the closest value ____________________________ //update the closestX ____________________________ //update the closestY ____________________________ } } } PImage depthImage = kinect.depthImage(); image(depthImage, 0, 0); //pick a fill color ____________________________ //draw a circle at the closestX and closestY ____________________________ }
Modify the previous sketch so that instead of an ellipse, you draw a line.
- You'll need:
Two new variables of type float:- lastX
- lastY
- Set the background to a color (Do this in the setup())
- Don't display the depth image (comment the appropriate line out)
- Set the variable that was holding the pixel , to hold the mirrored pixel, or translated a distance
from the right edge:
int reversedX = 640-x-1; //int pixel=x+(y*width_of_image) int pixel=_____________;
- Only set the current X and Ys if the the current depth is less than the closest depth and the current depth is between 2 and 5 feet (610 and 1525)
- Perform a linear interpolation (lerp).
lerp() takes three arguments,
the first two are the two numbers to interpolate between and the last is a float
that specifies how much to move between the two points (with 1.0 being all the way
to the second point and 0.0 being none).
Here's how lerp() works:
In your sketch set interpolatedX and interpolatedY:
float x = lerp(-20.0, 60.0, 0.0); // Assign -20.0 to x float y = lerp(-20.0, 60.0, 0.5); // Assign 20.0 to y float z = lerp(-20.0, 60.0, 1.0); // Assign 60.0 to z
and do the same for the yfloat interpolatedX = lerp(your_last_x, your_closest_x, 0.3f);
- Instead of fill(), set the stroke() to a color.
- Set the width of the line (strokeWeight())
- Use the line() function. This should take the last point and the interpolated point.
- update the last variables (lastX and lastY) to hold the interpolated X and Y values
- Add a way to clear the screen.
Assignment 1:Thresholding, or grabbing a slice
Create a Processing Sketch where you analyze the depth image and if the pixels are within a range, color the pixels blue, otherwise don't display the pixels.- You'll need an instance variable of type PImage. Name it myImage
- Create instance variables of type int that will hold your min and max values. Set them to 0 and 750 respectively.
- Set your PImage to a new PImage with the appropriate dimensions.
- Inside the draw() method loop through all the depth pixels, setting the pixels within the threshold range to the color blue and everything else to black.
yourPImage.pixels[i]=0x0000FFFF; yourPImage.pixels[i]=0;
- You'll need to update the pixels of your PImage:
yourPImage.updatePixels();
- Draw the image to the screen.