Материал: [2.1] 3D Imaging, Analysis and Applications-Springer-Verlag London (2012)

where xi,j is the image of world point Xj in image i and xˆ i,j is the predicted projection of the same world point according to Eq. (2.7) (using estimated intrinsic and extrinsic camera parameters) followed by radial distortion according to Eq. (2.6).

The vector p contains all of the free parameters within the planar projection (homography) function plus two radial distortion parameters k1 and k2 as described in Sect. 2.3.3. Initial estimates of these radial distortion parameters can be set to zero. LM iteratively updates all parameters according to the equation:

pk+1 = pk + δpk

δpk = − JT J + λJ diag JT J −1JT e,

where J is the Jacobian matrix containing the first derivatives of the residual e with respect to each of the camera parameters.

Thus computation of the Jacobian is central to LM minimization. This can be done either numerically or with a custom routine, if analytical expressions for the Jacobian entries are known. In the numerical approach, each parameter is incremented and the function to be minimized (the least squares error function in this case) is computed and divided by the increment, which should be the maximum of 10−6 and 10−4 × |pi |, where pi is some current parameter value [21]. In the case of providing a custom Jacobian function, the expressions are long and complicated in the case of camera calibration, and so the use of a symbolic mathematics package can help reduce human error in constructing the partial differentials.

Note that there are LM implementations available on many platforms, for example in MATLAB’s optimization toolbox, or the C/C++ levmar package. A detailed discussion of iterative estimation methods including LM is given in Appendix 6 of Hartley and Zisserman’s book [21].

2.4.4 Calibration of a Stereo Rig

It is common practice to choose the optical center of one camera to be the origin of a stereo camera’s 3D coordinate system. (The midpoint of the stereo baseline, which connects the two optical centers is also occasionally used.) Then, the relative rigid location of cameras, [R, t], within this frame, along with both sets of intrinsic parameters, is required to generate a pair of projection matrices and hence a pair of 3D rays from corresponding image points that intersect at their common scene point.

The previous two subsections show how we can calculate the intrinsic parameters for any single camera. If we have a stereo pair, which is our primary interest, then we would compute a pair of intrinsic parameter matrices, one for the left camera and one for the right. In most cases, the two cameras are the same model and hence we would expect the two intrinsic parameter matrices to be very similar.

Also, we note that, for each chessboard position, two sets of extrinsic parameters, [R, t], are generated, one for the left camera’s position relative to the calibration plane and one for the right. Clearly, each left-right pair of extrinsic parameters

should have approximately9 the same relationship, which is due to the fixed rigid rotation and translation of one camera relative to another in the stereo rig.

Once two sets of intrinsic parameters and one set of extrinsic parameters encoding the relative rigid pose of one camera relative to another has been computed, the results are often refined in a global stereo optimization procedure, again using the Levenberg-Marquardt approach. To reduce n sets of relative extrinsic parameters to one set, we could choose the set associated with the closest calibration plane or compute some form of robust average.

All parameter estimates, both intrinsic and extrinsic, can be improved if the LM optimization is now performed over a minimal set of parameters, since the extrinsic parameters are reduced from 12 (two rotations and two translations) to 6 (one rotation and one translation) per calibration grid location. This approach ensures global rigidity of the stereo rig going from left to right camera. An implementation of global stereo optimization to refine stereo camera parameters is given in the Caltech camera calibration toolbox for MATLAB [9].

2.5 Two-View Geometry

3D reconstruction from an image pair must solve two problems: the correspondence problem and the reconstruction problem.

•Correspondence problem. For a point x in the left image, which is the corresponding point x in the right image, where x and x are images of the same physical scene point X?

•Reconstruction problem. Given two corresponding points x and x , how do we compute the 3D coordinates of scene point X?

Of these problems the correspondence problem is significantly more difficult as it is a search problem whereas, for a stereo camera of known calibration, reconstruction to recover the 3D measurements is a simple geometric mechanism. Since we have sets of three unique points, (x, x , X), this mechanism is called triangulation (not to be confused with surface mesh triangulation, described in Chap. 4).

This section is designed to give the reader a good general grounding in two-view geometry and estimation of the key two-view geometric relations that can be useful even when extrinsic or intrinsic camera calibration information is not available.10 As long as the concept of epipolar geometry is well understood, the remaining main sections of this chapter can be followed easily.

9‘Approximately’, because of noise in the imaged corner positions supplied to the calibration process.

10Extrinsic parameters are always not known in a structure from motion problem, they are part of what we are trying to solve for. Intrinsic parameters may or may not be known, depending on the application.

Fig. 2.7 (a) The epipolar geometry establishes the relationship between the two camera views. (b) The epipolar planes rotate around the baseline and all epipolar lines intersect at the epipole

may prefer some camera vergence to improve the field-of-view overlap between the two cameras and, in this case, the images need to be warped so that the epipolar lines become horizontal again. This rectification process is discussed later in the chapter.

The epipolar constraint can be represented algebraically by a 3 × 3 matrix called the fundamental matrix (F), when we are dealing with raw pixel coordinates, and by the essential matrix (E) when the intrinsic parameters of the cameras are known and we are dealing with metrically expressed coordinates (e.g. millimeters) in the image plane.

2.5.2 Essential and Fundamental Matrices

Both the essential and fundamental matrices derive from a simple co-planarity constraint. For simplicity it is best to look at the epipolar relation using the essential matrix first and then adapt it using the camera intrinsic parameters to obtain a relation for pixel-based image coordinates, which involves the fundamental matrix.

Referring to Fig. 2.8, we have a world point X that projects to points xc and xc in the image planes. These image plane points are expressed as 3-vectors, so that they are effectively the 3D positions of the imaged points expressed metrically in their own camera frame, hence the subscript c. (Note also that they can be regarded as normalized homogeneous image coordinates, with the scale set to the focal length, f , although any non-zero scale would suffice.) We know that the three vectors Cxc , C xc and t are co-planar, so we can choose one of the two camera frames to express this co-planarity, using the scalar triple product. If we choose the right frame (primed), then we must rotate vector Cxc using rotation matrix R, to give:

xcT (t × Rxc ) = 0.

Expressing the cross product with t by the multiplication with the skew-symmetric matrix [t]x , we have:

xc [t]x Rxc = 0,

Fig. 2.8 The essential matrix E = [t]x R encodes the epipolar geometry. It is used to relate the correspondences xc and xc between two images, when these image locations are expressed in metric units. If pixel-based coordinates are used (for example, if intrinsic camera parameters are unknown) epipolar geometry is encoded by the fundamental matrix F

Thus the essential matrix encapsulates only extrinsic parameters, namely, the rotation and translation associated with the relative pose of the two cameras. The implication of this is that, in applications where R and t have not been computed in a calibration procedure, they may be recoverable from an estimate of E, which will be discussed further in Sect. 2.8.2 in the context of structure from motion.

In many practical situations, we also need to deal with uncalibrated cameras where the intrinsic parameters are unknown (i.e. the mapping between metric image coordinates and raw pixel values is unknown). The shifting and scaling operations required for this conversion can be encapsulated in matrices K and K , as follows:

where K and K are the 3 × 3 matrices containing the intrinsic camera parameters for the two cameras. Inserting these relations into Eq. (2.16) gives:

x T K−T EK−1x = 0

x T Fx = 0

thus

F = K−T EK−1 = K−T [t]x RK−1