The use of incorrect parameters in gluPerspective may lead to incorrect results while trying to read scene coordinates via gluUnproject.
After several nights spent trying to figure out the math leading coordinate transformations, I’ve come across the final (by now) solution. The problem is simple. Given local window coordinates x,y, calculate the scene coordinates x,y,z using a projection matrix ad libitum.
First of all, some say you need to enable DEPTH_TEST by using:
glEnable(GL_DEPTH_TEST);
glDepthRange(0,1);
The second instruction maps window z coordinate to the range 0,1 (anyway it should be the deafault).This means that 0 refers to the near plane, while 1 is the far plane. Before drawing you should take care to clean the DEPTH_BUFFER, like this:
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT );
Now you should be able to retrieve the correct value for each scene coordinate:
glReadPixels( point.x, point.y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &z );
gluUnProject((GLdouble) x, (GLdouble) y, (GLdouble)z, _modelMatrix,_projMatrix,_viewport,&(drawingPoint.x),&(drawingPoint.y),&(drawingPoint.z));
This is all correct but it’s not enough. I was trying to set up an orthogonal perspective by using gluPerspective and gluLookAt and I didn’t succeed in getting the correct scene coordinates. There are two different problems. The first is about the choice of the camera position and the eye direction. If the ratio between these two values is incorrect you’ll find yourself messing up things much more than you should.
Let’s suppose I want to look at the x-y plane of coordinate zPlane from top to bottom. I define the followings values:
lower=(zPlane==0)?zPlane-10:(zPlane-2*(zPlane<0 ? -zPlane : zPlane));
upper=(zPlane==0)?zPlane+10:(zPlane+2*(zPlane<0 ? -zPlane : zPlane));
This way they can’t be zero and they’ll be twice the absolute value of zPlane (- or +).
Then I set up the camera like this:
gluLookAt(0,0,upper,0,0,lower,0,1,0);
If I want a bottom-top view it’s sufficient to swap upper and lower. No problem either for other planes. Obviously, before doing this I’ve set up my perspective:
gluPerspective(aperture,angle, near, far);
The real problem is the correct choice of near and far. I define them this way:
near=upper-zPlane-0.00001;
far=upper-zPlane+0.00001;
Obviously in a bottom-top view, you should use lower (the z coordinate of the camera, anyway). I’ve found that using a lower tolerance leads to incorrect values returned by gluUnproject. But this could be only a hardware specific issue.