Showing posts with label Gestures. Show all posts
Showing posts with label Gestures. Show all posts

Friday, March 2, 2012

Picky, Part II

This is a continuation of my last post, where I set up color picking for the red GLKit cube.  In this post I'm setting up color picking for the blue 2.0 shader cube.  I'm starting off right where I left off, with the changes I made in my last post already made.


I'm using the exact same strategy as last time.  Namely, re-rendering the scene off screen in response to a user tap, but with lighting turned off and a unique color set.  The implementation is a little more involved since we have to communicate with the shader, but on the other hand, we can use a lot of the same work that we have already done.


Just as a reminder, when we left off, we had the red cube hooked up to the color picker so it got smaller whenever it was clicked on.




Step One:  Modify the vertex shader


The vertex shader cannot have a hard coded diffuse light color anymore, since we have to change it based on whether we're rendering it for the screen or for picking.  We also need to be able to turn lighting on or off.  We can accomplish both of those tasks by setting up uniform variables.  Modify the Shader.vsh file so it looks like this:


attribute vec4 position;

attribute vec3 normal;


varying lowp vec4 colorVarying;


uniform mat4 modelViewProjectionMatrix;

uniform mat3 normalMatrix;

uniform vec4 color;

uniform bool lit;


void main()

{

    if (lit)

    {

        vec3 eyeNormal = normalize(normalMatrix * normal);

        vec3 lightPosition = vec3(0.0, 0.0, 1.0);

    

        float nDotVP = max(0.0, dot(eyeNormal, normalize(lightPosition)));

                 

        colorVarying = color * nDotVP;

    }

    else

    {

        colorVarying = color;

    }

    

    gl_Position = modelViewProjectionMatrix * position;

}


The first thing we had to do was declare the uniform variables color and lit.  The uniform keyword simply means these values remain the same for all the vertices passed to the shader during the function call.


Inside the main function, we wrapped the existing lighting code inside an if statement.  I'd really rather not have control logic inside my shader, but for this example, maybe it's ok.  Notice how the vec4 diffuseColor declaration went away.  Now we're using the uniform color variable in its place.  Finally, if lit is set to false, we simply assign colorVarying equal to color.



Step Two:  Connect the uniforms in the view controller code


Now that we have the vertex shader set up, we need to pass it correct values.  This involves creating and configuring the two uniform variables in our view controller.


Start by adding two new members to the uniform enum near the top of the ViewController.m file.  When you're done, it should look like this:


enum

{

    UNIFORM_MODELVIEWPROJECTION_MATRIX,

    UNIFORM_NORMAL_MATRIX,

    UNIFORM_COLOR,

    UNIFORM_LIT,

    NUM_UNIFORMS

};


Next, declare two new variables in the ViewController interface section.  I put mine right below the _normalMatrix declaration.


    GLKVector4 _color;

    GLboolean _lit;


Next, initialize the two variables you just declared in the setupGL function.  I put mine at the end after my _scale initialization from the last post.  I also initialized the color to the same diffuse color value that used to be hard coded in the vertex shader.


    _color = GLKVector4Make(0.4f, 0.4f, 1.0f, 1.0f);

    _lit = true;


Now, in the loadShaders function, add the code to get the locations of the two new uniforms we added:


    uniforms[UNIFORM_COLOR] = glGetUniformLocation(_program, "color");

    uniforms[UNIFORM_LIT] = glGetUniformLocation(_program, "lit");


This code goes right after the two glGetUniformLocation calls that already exist near the end of the function.   We are simply getting the uniform IDs from OpenGL and storing them in our uniforms array for later use.


Finally, in the renderGL function we wrote in the last post, add the following code right below the two glUniformMatrix* calls that are already there.


    glUniform4fv(uniforms[UNIFORM_COLOR], 1, _color.v);

    glUniform1i(uniforms[UNIFORM_LIT], _lit);


This is the code that actually assigns the values we put in _color and _lit with the uniform variables we declare and use in the vertex shader.


As a sanity check, it's a good idea to run your program in the simulator at this point.  It should behave the same as it did at the beginning of this post.



Step Three:  Picking


Now that we've added the ability to change the blue cube's color, we need to take advantage of it.


Go down to the action handler we created in the last post.  Add the following code right before the call to [self renderGL]:


    // set the shader uniform color and disable lighting

    GLKVector4 oldColor = _color;

    _color = GLKVector4Make(1.0f, 0.0f, 0.0f, 1.0f);

    _lit = false;


Conceptually, we're doing the exact same thing we did we the red cube: we're changing the color and we're turning off lighting.  The only difference here is we need to store the old color so we can change it back when we're done.


If you run the program now in the simulator you should see the blue cube turn bright red when you click on the screen:



Now let's go ahead and change the color and lighting state back to the way it was.  You can add this code right after the [self renderGL] call:


    // change the shader color and lighting back to normal

    _color = oldColor;

    _lit = true;


Run the program in the simulator again.  The blue cube should remain visually unchanged, but the console output should indicate a bright red color when you click on the blue cube:


2012-03-02 22:43:57.792 Picky[7308:10103] 255, 0, 0, 255



Step Four:  Reaction!


Just as before, I set the blue cube up to get smaller when you click on it.  To see it happen in your program, just follow the steps from the last post.


First, create a new variable right next to your _scale variable from last time:


    float _shadeScale;


Next, initialize the variable to 1 in the setupGL function:


_shadeScale = 1.0f;


Next, add another scaling transformation to the update function.  This one goes right after the GLKMatrix4Rotate function call in the ES2 model view matrix section:


modelViewMatrix = GLKMatrix4Scale(modelViewMatrix, _shadeScale, _shadeScale, _shadeScale);


Finally, in your tap action function, add an else if statement to your previous if statement to check for the bright red color and modify the _shadeScale variable:


    else if (pixel[0] == 255 && pixel[1] == 0 && pixel[2] == 0 && pixel[3] == 255)

        _shadeScale = _shadeScale / 1.5f;


Now run the program in the simulator.  Clicking on either cube causes that cube to shrink.



Step Five:  Troubleshooting


Using shader uniform variables is a multistep process, which means it's easy to get something wrong.  In this section I purposely introduce various bugs to repo some common error states.


In the vertex shader file, I replaced all instances of the color variable with _color.  This compiled and ran with no shader compile errors in the log, but since the ViewController.m file still refers to it as "color" in this line in loadShaders, we never actually succeed in passing the color to the shader, and so the cube appears black.


uniforms[UNIFORM_COLOR] = glGetUniformLocation(_program, "color");


When I stepped through this line of code in the debugger, I saw that the glGetUniformLocation call returned -1, indicating an error.


When I commented out the declaration of the color variable in the Shader.vsh file, but left references to it, I got the following messages in the error log:


//uniform vec4 color;


2012-03-07 20:18:43.563 Picky[568:10103] Shader compile log:

ERROR: 0:28: Use of undeclared identifier 'color'

ERROR: 0:32: Use of undeclared identifier 'color'

2012-03-07 20:18:43.570 Picky[568:10103] Failed to compile vertex shader


With this compile error, the blue cube does not show up at all.  The loadShaders function returns when it detects a shader compile error, and does not even get to the set of glGetUniformLocation calls.


Another way to get a black cube is to mess up the call to glUniform4fv in the renderGL function.  Here's how it's supposed to look:


glUniform4fv(uniforms[UNIFORM_COLOR], 1, _color.v);


The v at the end of the function name indicates that we are passing a vector.  The f indicates that the vector contains float values.  The 4 indicates that it is 4 dimensional.  Using 3 instead of 4 results in a black cube.  


The first parameter is the id we use to refer to the shader uniform.  We got this id from OpenGL when we loaded the shaders.  If we pass in an incorrect id, we get a black cube. 


The second parameter is the number of vectors we are sending.  It is not the number of bytes or the dimensions in the vector, so '1' is the correct value here.  Any other value gets you a black cube.  


Calling glGetError() after any one of these incorrect function calls returns error code 1282, which means invalid operation.


Conclusion


As I said in my previous post, color picking is fine for such a simple scene, but for more complex scenes, other algorithms such as ray-casting may be more effective.


As always, please let me know if you find any mistakes in my code, or if you know of a better or easier way to solve this problem.  Also, use this code at your own risk.

Picky, Part I

In this post, I'm trying out color picking. In the interest of keeping these posts to a more manageable size, I am planning to just color pick the red GLKit cube. My next post will build on the work in this one and handle the blue 2.0 cube.


The basic idea behind color picking is once the user has tapped somewhere, render the objects in your scene into the back buffer using a unique color for each object, then ask OpenGL to give you the color of the pixel where the user tapped. This color tells you what object was tapped. As long as you don't present the scene to the user, the user never sees the weird colors. This is fast and efficient, since you only render the scene an extra time whenever the user taps, but generally is good only for per-object or per-triangle picking. Getting the exact point on a triangle that the user clicked on will probably require a different technique.



Step One: Create a new OpenGL Game project


For instructions how to do this, refer to my texture post. Once you get it set up, you can run it and see two cubes. The red cube is rendered using GLKit. The blue cube is rendered using OpenGL ES 2.0 shaders. In this blog I color pick the red cube.




Step Two: Set up a tap gesture recognizer and action function (event handler)


Refer to my gesture post for step by step instructions.



Step Three: Read the pixel the user tapped on


Add code to the event handler so that it looks like the following:


- (IBAction)tapThat:(id)sender {

GLubyte pixel[4]; // output array for the red, green, blue, and alpha pixel components


// get the point the user tapped on

CGPoint tapPoint = [sender locationInView:self.view];

// read the pixel at the tapped location. We use the screen height to convert

// between iOS screen coordinates, which is (0,0) at the upper left, and OpenGL

// screen coordinates, which is (0,0) at the LOWER left.

int height = [self.view bounds].size.height;

glReadPixels(tapPoint.x,height - tapPoint.y,1,1,GL_RGBA,GL_UNSIGNED_BYTE,&pixel);

// log the results.

NSLog(@"%u, %u, %u, %u",pixel[0],(int)pixel[1],pixel[2],pixel[3]);

}


This just outputs the color of the pixel that the user tapped on to the log. Go ahead and run the program in the simulator and start clicking around on the simulated screen. If you hooked everything up right you should see something like this:


2012-03-02 16:16:51.505 Picky[5480:10103] 84, 84, 210, 210

2012-03-02 16:16:51.912 Picky[5480:10103] 81, 81, 203, 203

2012-03-02 16:16:52.104 Picky[5480:10103] 89, 89, 224, 224

2012-03-02 16:16:52.279 Picky[5480:10103] 94, 94, 236, 236

2012-03-02 16:16:52.831 Picky[5480:10103] 202, 87, 87, 255

2012-03-02 16:16:53.004 Picky[5480:10103] 194, 83, 83, 255

2012-03-02 16:16:53.507 Picky[5480:10103] 147, 65, 65, 255

2012-03-02 16:16:54.001 Picky[5480:10103] 185, 80, 80, 255

2012-03-02 16:16:54.184 Picky[5480:10103] 192, 83, 83, 255

2012-03-02 16:16:54.734 Picky[5480:10103] 77, 37, 37, 255

2012-03-02 16:16:54.932 Picky[5480:10103] 109, 49, 49, 255

2012-03-02 16:16:55.140 Picky[5480:10103] 138, 61, 61, 255

2012-03-02 16:16:56.104 Picky[5480:10103] 100, 100, 250, 250


The four comma-separated numbers at the end of each line are the red, green, blue, and alpha values of the pixel the user clicked on. Note that the data type that openGL returns is unsigned bytes, which go from 0 to 255. You generally send colors to OpenGL as floating point numbers between 0 and 1, so be aware that you may have to convert by dividing or multiplying by 255.



Step Four: Extract the render code


Of course, this function isn't very useful in and of itself. We need to add the code to render our object in a different color. To do this, We're going to pull out the render code into a separate function. We could just copy and paste the code from the view function to the tap action function, but that would be fragile, dangerous code. If the render functions aren't exactly the same except for the colors of the objects, we run the risk of rendering the objects in different locations when we are picking them. The user would click on a cube, but the action function would render it somewhere else, and our program wouldn't register the tap.


So create a new function declaration in the interface section of your ViewController.m file:


- (void)renderGL;


Then create the function in the implementation section, cut and paste the render code from the view function, and call the new render function inside the view function. When you are done, the view function and render function should look like this:


- (void)glkView:(GLKView *)view drawInRect:(CGRect)rect

{

[self renderGL];

}


- (void)renderGL

{

glClearColor(0.65f, 0.65f, 0.65f, 1.0f);

glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

glBindVertexArrayOES(_vertexArray);

// Render the object with GLKit

[self.effect prepareToDraw];

glDrawArrays(GL_TRIANGLES, 0, 36);

// Render the object again with ES2

glUseProgram(_program);

glUniformMatrix4fv(uniforms[UNIFORM_MODELVIEWPROJECTION_MATRIX], 1, 0, _modelViewProjectionMatrix.m);

glUniformMatrix3fv(uniforms[UNIFORM_NORMAL_MATRIX], 1, 0, _normalMatrix.m);

glDrawArrays(GL_TRIANGLES, 0, 36);

}


As a sanity check, go ahead and run your program in the simulator to make sure it still renders correctly.



Step Five: Render with a different color on tap


Now we need to call our render function from within the tap action function, but using a different color to draw our object. Add code to your tap action function so that it looks like this:


- (IBAction)tapThat:(id)sender {

GLubyte pixel[4]; // output array for the red, green, blue, and alpha pixel components


// turn off lighting and turn on constant color for picking

self.effect.useConstantColor = GL_TRUE;

self.effect.light0.enabled = GL_FALSE;

self.effect.constantColor = GLKVector4Make(0.0f, 0.0f, 1.0f, 1.0f);

// render the scene

[self renderGL];

// set lighting and the constant color usage back to the way they were

self.effect.useConstantColor = GL_FALSE;

self.effect.light0.enabled = GL_TRUE;

// get the point the user tapped on

CGPoint tapPoint = [sender locationInView:self.view];

// read the pixel at the tapped location. We use the screen height to convert

// between iOS screen coordinates, which is (0,0) at the upper left, and OpenGL

// screen coordinates, which is (0,0) at the LOWER left.

int height = [self.view bounds].size.height;

glReadPixels(tapPoint.x,height - tapPoint.y,1,1,GL_RGBA,GL_UNSIGNED_BYTE,&pixel);

// log the results.

NSLog(@"%u, %u, %u, %u",pixel[0],(int)pixel[1],pixel[2],pixel[3]);

}


In the code we added, we turn off lighting and turn on a constant, bright blue color. Then we render as normal. Afterwards, we turn the light back on and the constant color back off.


If you run this and click on the red cube, you will see numbers representing bright blue in the console:


2012-03-02 17:01:56.106 Picky[5746:10103] 0, 0, 255, 255


As an interesting exercise, you can comment out the two lines setting the OpenGL state back to the way it was and run the simulator. When you click on the screen, you'll see the object turn bright blue.


//self.effect.useConstantColor = GL_FALSE;

//self.effect.light0.enabled = GL_TRUE;




Step Six: Add a visual reaction


Now we just need to react to the user clicking on the cube. For this example, I arbitrarily decided to show the click by making the cube smaller. Presumably you will have your own application logic.


I started by adding a scale value to the view controller .m file


float _scale;


Next, I initialized it to 1 in the setupGL function.


_scale = 1.0f;


Next, I added a new scale transformation to the GLKit cube in the update function. Now it looks like this:


- (void)update

{

float aspect = fabsf(self.view.bounds.size.width / self.view.bounds.size.height);

GLKMatrix4 projectionMatrix = GLKMatrix4MakePerspective(GLKMathDegreesToRadians(65.0f), aspect, 0.1f, 100.0f);

self.effect.transform.projectionMatrix = projectionMatrix;

GLKMatrix4 baseModelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, -4.0f);

baseModelViewMatrix = GLKMatrix4Rotate(baseModelViewMatrix, _rotation, 0.0f, 1.0f, 0.0f);

// Compute the model view matrix for the object rendered with GLKit

GLKMatrix4 modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, -1.5f);

modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, _rotation, 1.0f, 1.0f, 1.0f);

modelViewMatrix = GLKMatrix4Scale(modelViewMatrix, _scale, _scale, _scale);

modelViewMatrix = GLKMatrix4Multiply(baseModelViewMatrix, modelViewMatrix);

self.effect.transform.modelviewMatrix = modelViewMatrix;

// Compute the model view matrix for the object rendered with ES2

modelViewMatrix = GLKMatrix4MakeTranslation(0.0f, 0.0f, 1.5f);

modelViewMatrix = GLKMatrix4Rotate(modelViewMatrix, _rotation, 1.0f, 1.0f, 1.0f);

modelViewMatrix = GLKMatrix4Multiply(baseModelViewMatrix, modelViewMatrix);

_normalMatrix = GLKMatrix3InvertAndTranspose(GLKMatrix4GetMatrix3(modelViewMatrix), NULL);

_modelViewProjectionMatrix = GLKMatrix4Multiply(projectionMatrix, modelViewMatrix);

_rotation += self.timeSinceLastUpdate * 0.5f;

}


Finally, I added the logic that checks to see if we clicked on our cube and change the scale. Now the tap action function looks like this:


- (IBAction)tapThat:(id)sender {

GLubyte pixel[4]; // output array for the red, green, blue, and alpha pixel components


// turn off lighting and turn on constant color for picking

self.effect.useConstantColor = GL_TRUE;

self.effect.light0.enabled = GL_FALSE;

self.effect.constantColor = GLKVector4Make(0.0f, 0.0f, 1.0f, 1.0f);

// render the scene

[self renderGL];

// set lighting and the constant color usage back to the way they were

self.effect.useConstantColor = GL_FALSE;

self.effect.light0.enabled = GL_TRUE;

// get the point the user tapped on

CGPoint tapPoint = [sender locationInView:self.view];

// read the pixel at the tapped location. We use the screen height to convert

// between iOS screen coordinates, which is (0,0) at the upper left, and OpenGL

// screen coordinates, which is (0,0) at the LOWER left.

int height = [self.view bounds].size.height;

glReadPixels(tapPoint.x,height - tapPoint.y,1,1,GL_RGBA,GL_UNSIGNED_BYTE,&pixel);

// did we click on our cube? If so, make it smaller

if (pixel[0] == 0 && pixel[1] == 0 && pixel[2] == 255 && pixel[3] == 255)

_scale = _scale / 1.5f;

// log the results.

NSLog(@"%u, %u, %u, %u",pixel[0],(int)pixel[1],pixel[2],pixel[3]);

}


Go ahead and run the program in the simulator. If you click on the background or the blue cube, nothing happens. Clicking on the red cube, however, makes it shrink!



I chose this particular method of setting the color of our object because it was the simplest way I could think of that would demonstrate the technique. More complex scenes would require a more complex method. To implement per-triangle color picking, you could add a per-vertex color attribute to the vertex array, then use that color in your test. As I said above, my next post will add color picking to the blue 2.0 cube.


And as always, this is code I wrote while trying to figure this stuff out. It is probably not suitable for use in any production environment, and I strongly advise against using it.


PS -- I may be the only blogger on the internet who did not know how to automatically apply formatting to my source code in my blog, but just in case anyone else out there is as derp as I am, I found that if you copy your source from Xcode to TextEdit, then save as html, TextEdit will apply the color and font styles for you. Then you can just paste the html into your blog editor. Whatever you do, don't click back to compose mode once you have pasted your HTML. Oh, I also had to change the 'Apple-tab-span' style to 'Apple-converted-space' so it actually matched the style in the rest of the document.


Monday, February 20, 2012

Rude Gestures

In this post I'm changing the default OpenGL Game project to rotate based on user gestures instead of automatically.  This is actually super easy to do once you know how, but if you're like me and you don't know much about Objective C or Xcode, there are a couple of annoying gotchas.  I'm assuming you are running a Mac with Lion and already have downloaded Xcode and the iOS software development kit.  If not you can find both in the Mac app store.

To create the project, open Xcode, and choose File->New->New Project... from the Xcode menu bar.  Choose iOS Application on the upper left, and the OpenGL Game option in the middle of the window, then press the Next button.


On the next screen, name your project whatever you want.  I have no idea how to use an Xcode storyboard, so I leave that un-checked, but I like the idea of automatic reference counting.


You should end with a new OpenGL project with the project file opened on the screen.  I recommend taking a snapshot of your project now, so if you mess anything up you can revert.  Go to File->Create Snapshot... in the menu bar and follow the instructions.


You can run the project (Command-R) and see two cubes rotating around each other.  One is rendered using GLKit, which among other things is Apple's re-implementation of the OpenGL ES 1.x fixed functionality render pipeline, and the other is rendered using the OpenGL ES 2.0 vertex and fragment shaders.


Now to add gesture support we need to go to the interface builder and edit the nib file.  The Project navigator should be opened on the left side of the screen, showing a tree view of the files and folders that make up your project.  If it's not opened, you can show it by pressing Command-1 on the keyboard.  Once it's opened select the file with the .xib extension by single clicking on it.  You will see a big blank view.  Find the tap gesture recognizer in the object library in the lower right of the window.  You can filter for it by typing "gesture" in the little text box at the very bottom of the window.


Now drag the tap gesture recognizer from the object library to the blank view in the middle of the screen.  It's possible to drag the recognizer other places, such as the list of objects to the right of the view, but those are bad, wrong places that will cause the tap gesture recognizer to silently fail.  I say again, drag the tap gesture recognizer to the blank white area in the middle of your window, and nowhere else.  Once you drag it over, it will appear in the list of objects in the nib file.  

Now we need to add an action to our view controller.  Activate the assistant editor by pressing option-command-enter on your keyboard, or by clicking the little tuxedo button in the upper right of the window.  The view controller .h file should appear in a new pane on the right side of the window.


Now right click and drag from the tap gesture recognizer in the list of objects in the nib file (NOT from the object library) to the area in the view controller .h file between @interface and @end.  In the little context dialog box that pops up, select an Action connection and name it something meaningful.  I changed the type to UIPanGestureRecognizer, but I'm not sure it makes a difference.


Once you hit the Connect button you will see a new IBAction declaration, which is basically analogous to an event handler.  


The skeleton of its implementation was also automatically added to the view controller .m file, and that's the file we are going to edit next.

Get rid of the assistant editor by pressing command-enter on your keyboard or by going to the View->Standard Editor->Show Standard Editor menu option in the menu bar.  Then select the view controller .m file in the project navigator to the left of your window.  The first thing we want to do is comment out the last line in the update function which increments the rotation value.  The update function is called by the framework before every frame and it's this line that causes the cubes to automatically rotate around each other.


Now we need to update the rotation value ourselves, in the new pan gesture event handler that Xcode added for us at the bottom of the view controller file.  Add these two lines to the empty function:

    CGPoint translation = [sender translationInView:self.view];
    _rotation = translation.x / 100.0f;

The first line gets the amount that the user panned in both the x and y directions.  The second line sets our rotation value equal to the amount the user slide his finger back or forth.  We divide by 100 because the rotation value is in radians.  2*pi radians (a little over 6) is equivalent to a full 360 degree rotation, so if every frame the user drags their finger 4 or 5 pixels the cubes will rotate almost all the way around.  We want to slow that down so it looks better and is easier to control.


 Now run the project again.  When the simulator comes up the cubes will be stationary until you click and drag your mouse or trackpad across the screen.  Then you make the cubes rotate back and forth.


This post is very much the chronicle of me fumbling around in the dark with Xcode and iOS development. I'm writing this in the hopes of helping other people avoid the same pitfalls and blind alleys I stumbled on.  If you find any mistakes or know of a safer, saner, or more interesting way to do this, please let me know.

PS - One of the many cool things I learned how to do while writing this was how to take screenshots on the Mac.  Pressing Command-Shift-4 allows you to select an area of your screen to which will automatically appear as an image on your desktop.  Pressing Command-Shift-4, then the space bar will allow you to click on a window to take a screenshot of the entire window.