Picky, Part I

In this post, I'm trying out color picking. In the interest of keeping these posts to a more manageable size, I am planning to just color pick the red GLKit cube. My next post will build on the work in this one and handle the blue 2.0 cube.

The basic idea behind color picking is once the user has tapped somewhere, render the objects in your scene into the back buffer using a unique color for each object, then ask OpenGL to give you the color of the pixel where the user tapped. This color tells you what object was tapped. As long as you don't present the scene to the user, the user never sees the weird colors. This is fast and efficient, since you only render the scene an extra time whenever the user taps, but generally is good only for per-object or per-triangle picking. Getting the exact point on a triangle that the user clicked on will probably require a different technique.

Step One: Create a new OpenGL Game project

For instructions how to do this, refer to my texture post. Once you get it set up, you can run it and see two cubes. The red cube is rendered using GLKit. The blue cube is rendered using OpenGL ES 2.0 shaders. In this blog I color pick the red cube.

Step Two: Set up a tap gesture recognizer and action function (event handler)

Refer to my gesture post for step by step instructions.

Step Three: Read the pixel the user tapped on

Add code to the event handler so that it looks like the following:

- (IBAction)tapThat:(id)sender {

GLubyte pixel[4]; // output array for the red, green, blue, and alpha pixel components

// get the point the user tapped on

CGPoint tapPoint = [sender locationInView:self.view];

// read the pixel at the tapped location. We use the screen height to convert

// between iOS screen coordinates, which is (0,0) at the upper left, and OpenGL

// screen coordinates, which is (0,0) at the LOWER left.

int height = [self.view bounds].size.height;

glReadPixels(tapPoint.x,height - tapPoint.y,1,1,GL_RGBA,GL_UNSIGNED_BYTE,&pixel);

// log the results.

NSLog(@"%u, %u, %u, %u",pixel[0],(int)pixel[1],pixel[2],pixel[3]);

}

This just outputs the color of the pixel that the user tapped on to the log. Go ahead and run the program in the simulator and start clicking around on the simulated screen. If you hooked everything up right you should see something like this:

2012-03-02 16:16:51.505 Picky[5480:10103] 84, 84, 210, 210

2012-03-02 16:16:51.912 Picky[5480:10103] 81, 81, 203, 203

2012-03-02 16:16:52.104 Picky[5480:10103] 89, 89, 224, 224

2012-03-02 16:16:52.279 Picky[5480:10103] 94, 94, 236, 236

2012-03-02 16:16:52.831 Picky[5480:10103] 202, 87, 87, 255

2012-03-02 16:16:53.004 Picky[5480:10103] 194, 83, 83, 255

2012-03-02 16:16:53.507 Picky[5480:10103] 147, 65, 65, 255

2012-03-02 16:16:54.001 Picky[5480:10103] 185, 80, 80, 255

2012-03-02 16:16:54.184 Picky[5480:10103] 192, 83, 83, 255

2012-03-02 16:16:54.734 Picky[5480:10103] 77, 37, 37, 255

2012-03-02 16:16:54.932 Picky[5480:10103] 109, 49, 49, 255

2012-03-02 16:16:55.140 Picky[5480:10103] 138, 61, 61, 255

2012-03-02 16:16:56.104 Picky[5480:10103] 100, 100, 250, 250

The four comma-separated numbers at the end of each line are the red, green, blue, and alpha values of the pixel the user clicked on. Note that the data type that openGL returns is unsigned bytes, which go from 0 to 255. You generally send colors to OpenGL as floating point numbers between 0 and 1, so be aware that you may have to convert by dividing or multiplying by 255.

Step Four: Extract the render code

Of course, this function isn't very useful in and of itself. We need to add the code to render our object in a different color. To do this, We're going to pull out the render code into a separate function. We could just copy and paste the code from the view function to the tap action function, but that would be fragile, dangerous code. If the render functions aren't exactly the same except for the colors of the objects, we run the risk of rendering the objects in different locations when we are picking them. The user would click on a cube, but the action function would render it somewhere else, and our program wouldn't register the tap.

So create a new function declaration in the interface section of your ViewController.m file:

- (void)renderGL;

Then create the function in the implementation section, cut and paste the render code from the view function, and call the new render function inside the view function. When you are done, the view function and render function should look like this:

- (void)glkView:(GLKView *)view drawInRect:(CGRect)rect

{

[self renderGL];

}

- (void)renderGL

{

glClearColor(0.65f, 0.65f, 0.65f, 1.0f);

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

glBindVertexArrayOES(_vertexArray);

// Render the object with GLKit

[self.effect prepareToDraw];

glDrawArrays(GL_TRIANGLES, 0, 36);

// Render the object again with ES2

glUseProgram(_program);

glUniformMatrix4fv(uniforms[UNIFORM_MODELVIEWPROJECTION_MATRIX], 1, 0, _modelViewProjectionMatrix.m);

glUniformMatrix3fv(uniforms[UNIFORM_NORMAL_MATRIX], 1, 0, _normalMatrix.m);

glDrawArrays(GL_TRIANGLES, 0, 36);

}

As a sanity check, go ahead and run your program in the simulator to make sure it still renders correctly.

Step Five: Render with a different color on tap

Now we need to call our render function from within the tap action function, but using a different color to draw our object. Add code to your tap action function so that it looks like this:

- (IBAction)tapThat:(id)sender {

GLubyte pixel[4]; // output array for the red, green, blue, and alpha pixel components

// turn off lighting and turn on constant color for picking

self.effect.useConstantColor = GL_TRUE;

self.effect.light0.enabled = GL_FALSE;

self.effect.constantColor = GLKVector4Make(0.0f, 0.0f, 1.0f, 1.0f);

// render the scene

[self renderGL];

// set lighting and the constant color usage back to the way they were

self.effect.useConstantColor = GL_FALSE;

self.effect.light0.enabled = GL_TRUE;

// get the point the user tapped on

CGPoint tapPoint = [sender locationInView:self.view];

// read the pixel at the tapped location. We use the screen height to convert

// between iOS screen coordinates, which is (0,0) at the upper left, and OpenGL

// screen coordinates, which is (0,0) at the LOWER left.

int height = [self.view bounds].size.height;

glReadPixels(tapPoint.x,height - tapPoint.y,1,1,GL_RGBA,GL_UNSIGNED_BYTE,&pixel);

// log the results.

NSLog(@"%u, %u, %u, %u",pixel[0],(int)pixel[1],pixel[2],pixel[3]);

}

In the code we added, we turn off lighting and turn on a constant, bright blue color. Then we render as normal. Afterwards, we turn the light back on and the constant color back off.

If you run this and click on the red cube, you will see numbers representing bright blue in the console:

2012-03-02 17:01:56.106 Picky[5746:10103] 0, 0, 255, 255

As an interesting exercise, you can comment out the two lines setting the OpenGL state back to the way it was and run the simulator. When you click on the screen, you'll see the object turn bright blue.

//self.effect.useConstantColor = GL_FALSE;

//self.effect.light0.enabled = GL_TRUE;

Step Six: Add a visual reaction

Now we just need to react to the user clicking on the cube. For this example, I arbitrarily decided to show the click by making the cube smaller. Presumably you will have your own application logic.

I started by adding a scale value to the view controller .m file

float _scale;

Next, I initialized it to 1 in the setupGL function.

_scale = 1.0f;

Next, I added a new scale transformation to the GLKit cube in the update function. Now it looks like this:

- (void)update

{

float aspect = fabsf(self.view.bounds.size.width / self.view.bounds.size.height);

GLKMatrix4 projectionMatrix = GLKMatrix4MakePerspective(GLKMathDegreesToRadians(65.0f), aspect, 0.1f, 100.0f);

self.effect.transform.projectionMatrix = projectionMatrix;

GLKMatrix4 baseModelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, -4.0f);

baseModelViewMatrix = GLKMatrix4Rotate(baseModelViewMatrix, _rotation, 0.0f, 1.0f, 0.0f);

// Compute the model view matrix for the object rendered with GLKit

GLKMatrix4 modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, -1.5f);

modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, _rotation, 1.0f, 1.0f, 1.0f);

modelViewMatrix = GLKMatrix4Scale(modelViewMatrix, _scale, _scale, _scale);

modelViewMatrix = GLKMatrix4Multiply(baseModelViewMatrix, modelViewMatrix);

self.effect.transform.modelviewMatrix = modelViewMatrix;

// Compute the model view matrix for the object rendered with ES2

modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, 1.5f);

modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, _rotation, 1.0f, 1.0f, 1.0f);

modelViewMatrix = GLKMatrix4Multiply(baseModelViewMatrix, modelViewMatrix);

_normalMatrix = GLKMatrix3InvertAndTranspose(GLKMatrix4GetMatrix3(modelViewMatrix), NULL);

_modelViewProjectionMatrix = GLKMatrix4Multiply(projectionMatrix, modelViewMatrix);

_rotation += self.timeSinceLastUpdate * 0.5f;

}

Finally, I added the logic that checks to see if we clicked on our cube and change the scale. Now the tap action function looks like this:

- (IBAction)tapThat:(id)sender {

GLubyte pixel[4]; // output array for the red, green, blue, and alpha pixel components

// turn off lighting and turn on constant color for picking

self.effect.useConstantColor = GL_TRUE;

self.effect.light0.enabled = GL_FALSE;

self.effect.constantColor = GLKVector4Make(0.0f, 0.0f, 1.0f, 1.0f);

// render the scene

[self renderGL];

// set lighting and the constant color usage back to the way they were

self.effect.useConstantColor = GL_FALSE;

self.effect.light0.enabled = GL_TRUE;

// get the point the user tapped on

CGPoint tapPoint = [sender locationInView:self.view];

// read the pixel at the tapped location. We use the screen height to convert

// between iOS screen coordinates, which is (0,0) at the upper left, and OpenGL

// screen coordinates, which is (0,0) at the LOWER left.

int height = [self.view bounds].size.height;

glReadPixels(tapPoint.x,height - tapPoint.y,1,1,GL_RGBA,GL_UNSIGNED_BYTE,&pixel);

// did we click on our cube? If so, make it smaller

if (pixel[0] == 0 && pixel[1] == 0 && pixel[2] == 255 && pixel[3] == 255)

_scale = _scale / 1.5f;

// log the results.

NSLog(@"%u, %u, %u, %u",pixel[0],(int)pixel[1],pixel[2],pixel[3]);

}

Go ahead and run the program in the simulator. If you click on the background or the blue cube, nothing happens. Clicking on the red cube, however, makes it shrink!

I chose this particular method of setting the color of our object because it was the simplest way I could think of that would demonstrate the technique. More complex scenes would require a more complex method. To implement per-triangle color picking, you could add a per-vertex color attribute to the vertex array, then use that color in your test. As I said above, my next post will add color picking to the blue 2.0 cube.

And as always, this is code I wrote while trying to figure this stuff out. It is probably not suitable for use in any production environment, and I strongly advise against using it.

PS -- I may be the only blogger on the internet who did not know how to automatically apply formatting to my source code in my blog, but just in case anyone else out there is as derp as I am, I found that if you copy your source from Xcode to TextEdit, then save as html, TextEdit will apply the color and font styles for you. Then you can just paste the html into your blog editor. Whatever you do, don't click back to compose mode once you have pasted your HTML. Oh, I also had to change the 'Apple-tab-span' style to 'Apple-converted-space' so it actually matched the style in the rest of the document.