Материал: [2.1] 3D Imaging, Analysis and Applications-Springer-Verlag London (2012)
2 Passive 3D Imaging |
59 |
and we can see that the fundamental matrix encapsulates both intrinsic and extrinsic parameters. The interpretation of the epipolar constraint given by the fundamental matrix, is that, if points x and x correspond, then x must lie on the epipolar line given by l = Fx and therefore the dot product between x and Fx is zero.
Some key properties of the fundamental matrix are summarized below:
•If F is the fundamental matrix between camera P and camera P , then FT is the fundamental matrix between camera P and camera P.
•F is a projective mapping taking a point to a line. If l and l are corresponding (i.e. conjugate) epipolar lines, then any point x on l maps to the same line l . Hence,
there is no inverse mapping (zero determinant, rank 2).
•F has seven degrees of freedom. While a 3 × 3 homogeneous matrix has eight independent ratios, there is also an additional constraint that the determinant of F is zero (F is rank 2), which further removes one degree of freedom.
•For any point x in the first image, the corresponding epipolar line in the second image is l = Fx. Similarly, l = FT x represents the epipolar line in the first image corresponding to x in the second image.
•The epipoles are determined as the left and right nullspaces of the fundamen-
tal matrix. This is evident, since each epipole is on every epipolar line in their respective image. This is written as e T l = e T Fx = 0 x, hence e T F = 0. Similarly lT e = x T Fe = 0 x , hence Fe = 0.
•The SVD (Singular Value Decomposition) of F is given as F = U diag(σ1, σ2, 0) VT where U = [u1, u2, e ], V = [v1, v2, e]. Thus finding the column in V that corresponds to the zero singular value gives a simple method of computation of the
epipoles from the fundamental matrix.
•For cameras with some vergence (epipoles not at infinity) to give camera projection matrices: P = K[I|0] and P = K [R|t], then we have: F = K −T [t]×RK−1 = [K t]×K RK−1 = K −T RKT [KRT t]× [21].
2.5.3 The Fundamental Matrix for Pure Translation
If the two identical cameras (K = K ) are separated by a pure translation (R = I), the fundamental matrix has a simple form, which can be shown to be [21]:
F = [Kt]x = e x = |
|
0 |
−ez |
ey |
|
|
ez |
0 |
−ex |
. |
|||
|
|
−ey |
ex |
0 |
|
|
In this case, the epipoles are at the same location in both images. If the translation is parallel to the image plane, the epipoles are at infinity with ez = ez = 0 and the epipolar lines are parallel in both images. When discussing rectilinear stereo rigs and rectification later, we will be particularly interested in the case when the translation is parallel to the camera’s x-axis, in which case the epipolar lines are
60 |
S. Se and N. Pears |
parallel and horizontal and thus correspond to image scan (raster) lines. In this case e = [1, 0, 0]T and the fundamental matrix is:
|
|
0 |
0 |
0 |
|
F |
= |
0 |
0 |
−1 |
|
|
0 |
1 |
0 |
and hence the relationship between corresponding points x and x is given by x T Fx = 0 which reduces to y = y .
2.5.4 Computation of the Fundamental Matrix
As the fundamental matrix is expressed in terms of corresponding image points, F can be computed from image correspondences alone. No camera calibration information is needed and pixel coordinates are used directly. Note that there are degenerate cases in the estimation of F. These occur in two common and well-known instances: (i) when the relative pose between the two views can be described by a pure rotation and (ii) when the scene is planar. For now we consider scenarios where
such degeneracies do not occur and we return to them later.
By expanding x T Fx = 0 where x = [x, y, 1]T and x = [x , y , 1]T and
F |
f11 |
f12 |
f13 |
|
f21 |
f22 |
f23 |
|
|
|
= f31 |
f32 |
f33 |
we obtain:
x xf11 + x yf12 + x f13 + y xf21 + y yf22 + y f23 + xf31 + yf32 + f33 = 0.
As each feature correspondence provides one equation, for n correspondences, we get the following set of linear equations:
x1x1 |
x1y1 |
x1 |
y1x1 |
y1y1 |
y1 |
x1 |
y1 |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
. |
xnxn |
xnyn |
xn |
ynxn |
ynyn |
yn |
xn |
yn |
or more compactly,
|
|
f11 |
|
|
|
|
f12 |
|
|
|
|
f13 |
|
|
|
|
|
|
|
1 |
|
f21 |
|
|
. |
|
|
0 (2.17) |
|
. |
|
|
|
|
. |
|
f22 |
|
|
|
|
= |
|
|
|
|
|
|
|
1f23
f31
f32 f33
Af = 0,
where A is termed the data matrix and f is the vector of unknown elements of F.
2 Passive 3D Imaging |
61 |
The eight-point algorithm12 [27] can be used as a very simple method to solve for F linearly using eight correspondences. As this is a homogeneous set of equations, f can only be determined up to a scale factor. With eight correspondences, Eq. (2.17) can be solved by linear methods, where the solution is the nullspace of A. (This can be found from the column in V that corresponds to the zero singular value in D in the singular value decomposition A = UDVT .) However, a solution with a minimal set of correspondences is often inaccurate, particularly if the correspondences are not well spread over the images, or they may not provide enough strong constraints if some of them are near-collinear or co-planar. It is preferable to use more than eight correspondences, then the least squares solution for f is given by the singular vector corresponding to the smallest singular value of A.
Note that this approach is similar to that for determining the homography matrix, discussed earlier in Sect. 2.4.1. As with that approach, it is essential to normalize the pixel coordinates of each image before applying SVD [19, 21], using a mean-
centering translation and a scaling so that the RMS distance of the points to the
√
origin is 2. When using homogeneous coordinates, this normalization can be applied using matrix operators N, N , such that new normalized image coordinates are
given as xn = Nx, xn = N x .
In general the solution for Fn (the subscript n now denotes that we have based the estimate on normalized image coordinates) will not have zero determinant (its rank will be 3 and not 2), which means that the epipolar lines will not intersect at a single point. In order to enforce this, we can apply SVD a second time, this time to the initially estimated fundamental matrix so that Fn = UDVT . We then set the smallest singular value (in the third row and third column of D) to zero to produce matrix D and update the estimate of the fundamental matrix as Fn = UD VT .
Of course, the estimate of Fn maps points to epipolar lines in the normalized image space. If we wish to search for correspondences within the original image space, we need to de-normalize the fundamental matrix estimate as F = N T FnN.
Typically, there are many correspondences between a pair of images, including mostly inliers but also some outliers. This is inevitable, since matching is a local search and ambiguous matches exist, which will be discussed further in Sect. 2.7. Various robust methods for estimating the fundamental matrix, which address the highly corrupting effect of outliers, are compared in [55]. In order to compute F from these correspondences automatically, a common method is to use a robust statistics technique called Random Sample Consensus (RANSAC) [16], which we now outline:
1.Extract features in both images, for example, from a corner detector [18].
2.Perform feature matching between images (usually over a local area neighborhood) to obtain a set of potential matches or putative correspondences.
3.Repeat the following steps N times:
12There are several other approaches, such as the seven-point algorithm.
62
Table 2.1 Number of samples required to get at least one good sample with 99 % probability for various sample size s and outlier fraction ε
S. Se and N. Pears
Sample size s ε = 10 % ε = 20 % ε = 30 % ε = 40 % ε = 50 %
4 |
5 |
9 |
17 |
34 |
72 |
5 |
6 |
12 |
26 |
57 |
146 |
6 |
7 |
16 |
37 |
97 |
293 |
7 |
8 |
20 |
54 |
163 |
588 |
8 |
9 |
26 |
78 |
272 |
1177 |
•Select eight putative correspondences randomly.
•Compute F using these eight points, as described above.
•Find the number of inliers13 that support F.
4.Find the F with the highest number of inliers (largest support) among the N trials.
5.Use this F to look for additional matches outside the search range used for the original set of putative correspondences.
6.Re-compute a least squares estimate of F using all inliers.
Note that re-computing F in the final step may change the set of inliers, as the epipolar lines are adjusted. Thus, a possible refinement is to iterate computation of a linear least squares estimate of F and its inliers, until a stable set of inliers is achieved or some maximum number of iterations is reached. The refinement achieved is often considered to be not worth the additional computational expense if processing time is considered important or if the estimate of F is to be used as the starting point for more advanced iterative non-linear refinement techniques, described later.
In the RANSAC approach, N is the number of trials (putative F computations) needed to get at least one good sample with a high probability (e.g. 99 %). How large should N be? The probability p of getting a good sample is given by:
p = 1 − 1 − (1 − ε)s N ,
where ε is the fraction of outliers (incorrect feature correspondences) and s is the number of correspondences selected for each trial. The above equation can be rearranged as:
N |
|
log(1 − p) |
. |
(2.18) |
||||
|
|
|||||||
|
= log(1 |
− |
(1 |
− |
ε)s ) |
|
||
|
|
|
|
|||||
The number of samples required for various sample size and outlier fraction based on Eq. (2.18) are shown in Table 2.1. It can be seen that the number of samples gets higher as the outlier fraction increases.
By repeatedly selecting a group of correspondences, the inlier support would be high for a correct hypothesis in which all the correspondences within the sample
13An inlier is a putative correspondence that lies within some threshold of its expected position predicted by F. In other words image points must lie within a threshold from their epipolar lines generated by F.
2 Passive 3D Imaging |
63 |
size, s, are correct. This allows the robust removal of outliers and the computation of F using inliers only. As the fraction of outliers may not be known in advance, an adaptive RANSAC method can be used where the number of outliers at each iteration is used to re-compute the total number of iterations required.
As the fundamental matrix has only seven degrees of freedom, a minimum of seven correspondences are required to compute F. When there are only seven correspondences, det(F) = 0 constraint also needs to be imposed, resulting in a cubic equation to solve and hence may produce up to three solutions and all three must be tested for support. The advantage of using seven correspondences is that fewer trials are required to achieve the same probability of getting a good sample, as illustrated in Table 2.1.
Fundamental matrix refinement techniques are often based on the LevenbergMarquardt algorithm, such that some non-linear cost function is minimized. For example a geometric cost function can be formulated as the sum of the squared distances between image points and the epipolar lines generated from their associated corresponding points and the estimate of F. This is averaged over both points in a correspondence and over all corresponding points (i.e. all those that agree with the estimate of F). The minimization can be expressed as:
F min |
1 |
N |
d x |
i |
|
F |
|
2 |
d |
|
FT |
xi |
2 |
, |
|
|
|
, |
xi |
|
xi , |
|
|||||||||
= |
F |
|
|
|
|
+ |
|
|
|
||||||
|
N i=1 |
|
|
|
|
|
|
|
|
|
|||||
where d(x, l) is the distance of a point x to a line l, expressed in pixels. For more details of this and other non-linear refinement schemes, the reader is referred to [21].
2.5.5 Two Views Separated by a Pure Rotation
If two views are separated by a pure rotation around the camera center, the baseline is zero, the epipolar plane is not defined and a useful fundamental matrix cannot be computed. In this case, the back-projected rays from each camera cannot form a triangulation to compute depth. This lack of depth information is intuitive because, under rotation, all points in the same direction move across the image in the same way, regardless of their depth. Furthermore, if the translation magnitude is small, the epipolar geometry is close to this degeneracy and computation of the fundamental matrix will be highly unstable.
In order to model the geometry of correspondences between two rotated views, a homography, described by a 3 × 3 matrix H, should be estimated instead. As described earlier, a homography is a projective transformation (projectivity) that maps points to points and lines to lines. For two identical cameras (K = K ), the scene-to-image projections are:
x = K[I|0]X, x = K[R|0]X |
|
hence |
|
x = KRK−1x = Hx. |
(2.19) |