I'm using simple constraints on the rotation angles for the eyes because that's quick to implement, but that will result in abrupt stops when the eye reaches its rotation limits. A better way would be to determine a Field of Vision (FOV) for the eye, and if the target escapes the boundaries of this FOV, the target can be adjusted according to the desired animation behaviour: it could clamp to the sides, so that the eye is still trained to it, or the eye could simply reset and look at something else.
This way the motion-smoothing acceleration/deceleration of eye-movement will remain, while nicely limiting the eye rotation.
If the eye FOV is defined as a viewing frustum similar to that of the camera, the eye can have its own Eye Space, with the frustum defined as a box. The target point can be converted to Eye Space where collision detection can be performed. This is advantageous due to it being faster to calculate collisions with an axis-aligned bounding box (AABB) than an arbitrarily-oriented trapezoid.
"Eye Space" is confusing because it is reminiscent of "Camera Space", in which the frustrum is defined as a trapezoid. The AABB comes after the Camera Space vertices undergo the perspective divide.
Therefore the target should be converted to "Eye View Space", and then given a perspective divide so it can be clipped with the eye's viewing frustum in "Eye Clip Space".
Seeing if this works with a quick test. If not, I don't have time to make it work so I'll just limit the angles.
Eye viewing range limiter outline:
Generate frustum for eye when eye is at origin and create perspective matrix.
Multiply target point by inverse of eye rotation+translation matrix (EyeViewSpace)
Multiply target point by perspective matrix (EyeClipSpace).
Detect clipping (target point coords should be within [0, 1]).
If target point is inside frustum: calculate lookAt quaternion as normal.
- If target point is outside frustum:
- Find closest line from point to the axis in the viewing direction.
- See where point hits the frustum. This is the new target point (in EyeClipSpace)
- Multiply new target point with the inverse of the perspective matrix to find the target point in EyeViewSpace, then multiply it by the eye's rotation+translation matrix to return it to WorldSpace for lookAt quaternion as normal.
Target's a bit dodgy - it doesn't follow the cursor directly. Must include a couple of rotations of the eyeball in the eye transform matrix, but don't actually apply them to the physical eye.