Understanding Reprojection Error
In camera calibration, the reprojection error (RPE) is of central importance. What exactly is it and how can we interpret it? This article aims to shed some light on this important, and sometimes slightly misunderstood topic.
What is the reprojection error?
Formally, a reprojection error is a:
2d vector of the difference between measured and projected point
Below you can see how Calibrator visualises both the measured (green cross) and the projected (red circle) point. Because the individual error tends to be much smaller than a single pixel, Calibrator draws the reprojection error as a 10 x scaled version using a red line.
Let call the measured point by $p_{ij}$. Here $i$ denotes the point's unique id on the calibration board and $j$ the pose/image number. In the camera calibration setting, its' coordinates are found using a feature detection algorithm - e.g. chessboard detector, circle grid detector, etc. The detector's job is to find the calibration target in the image. This step is usually followed by sub-pixel optimisation where the surrounding image values are used to actually determine the feature's position with accuracy well below 1px.
The projected point $\breve{\vec{p}}_{ij}$ is obtained by taking the $i$-th point nominal calibration board coordinates, applying the $j$-th pose's extrinsic transformation, and lastly applying the camera's projection mapping. The extrinsic transformation is a 3D roto-translation which related the calibration target's local coordinate frame in the specific pose/shot to the camera's coordinate frame. The camera's projection mapping depends on which (mathematical) camera model is being used. Most common camera models perform perspective projection combined with non-linear lens distortion.
The objective function being minimised in camera calibration is:
$$\sum_i{\sum_j{||\vec{p}_{ij} - \breve{\vec{p}}_{ij}||^2}} \quad ,$$
- that is the sum of squares of the norm/length of all re-projection errors is minimised. In other words, the calibration process adjusts all extrinsics and camera parameters such that the points project as close to where they were measured as possible.
Most commonly, the root-mean-square of all reprojection error norms is reported by calibration software, and called simply the reprojection error.
$$\textrm{RPE}_{\textrm{RMS}} = \sqrt{1/N \cdot \sum_i{\sum_j{||\vec{p}_{ij} - \breve{\vec{p}}_{ij}||^2}}}$$
Beware - the mean or median of reprojection error are also sometimes reported, and nomenclature is not $100\%$ consistent in the literature.
$$\textrm{RPE}_{\textrm{mean}} = 1/N \cdot \sum_i{\sum_j{||\vec{p}_{ij} - \breve{\vec{p}}_{ij}||}}$$
$$\textrm{RPE}_{\textrm{median}} = \textrm{median}(||\vec{p}_{ij} -\breve{\vec{p}}_{ij}||)$$
Especially in the case of severe outliers, these quantities could differ quite significantly. In conclusion - technically the reprojection error is a single vector quantity, but in most settings a statistic (RMS) of all reprojection errors is meant with reprojection error.
What are the Error Sources?
Why does the RPE never become exactly zero? There is a few sources which contribute to this error that cannot be fully eliminated.
First is feature noise - aka non-perfect detection of the saddle points or circles on a calibration target. It can be further broken down into image noise (comprised of dark current, shot noise and quantisation noise) and detector error/bias.
Secondly, the mathematical camera model might not be flexible enough to accurately model the complex optical behaviour of the lens and sensor.
Thirdly, the calibration target which normally is assumed perfect, might not be. For planar boards, stiffness, rigidity and temperature stability are crucial, and if not good enough will lead to higher RPE.
Lastly, the initial guess used for calibration may not have been in the vicinity of the global minimum, which leads to a non-optimal solution.
Error sources can be unbiased and stochastic, or biased, which could lead the calibration process off-track. The important question is whether the errors are correlated with the camera parameters or not.
As an example, while image noise does contribute to RPE, its stochastic nature would leads to a zero net-effect on the solution if enough images and feature coordinates are used.
A calibration target printed in a normal laser printer could easily have a bias with checker being non-square, etc. This effect will contribute to RPE and most likely bias the found solution.
What does the RPE tell us?
Now that we have clearly defined the reprojection error and analysed its' sources, we can understand what meaning it really bears.
First of all, note that the units of RPE is pixels. Hence, with higher resolution camera sensor and smaller pixels, we'd automatically expect higher RPE. Multiplying the RPE by pixel size of the given sensor (usually few micrometers), you get a somewhat comparable quantity which using trigonometry can be used to derive the equivalent error at the working distance.
As such, the RPE gives an indication of how accurately we are able to detect features and how accurately we project 3D coordinates.
However, the RPE of the calibration images used is a training error - it was the lowest number achieved on the given data. As common in statistics, we risk overfitting by focusing on the RPE alone. As an example, using very few calibration images, covering only a small part of the image, and with many calibration parameters active (flexible camera model), we most likely achieve a very low RPE. Closer analysis of the covariance structure allows us to estimate the much more conducive test error.
With Calibrator, we can analyse the projection and triangulation errors, and hence quantify our calibration with statistics that are much more meaningful and comparable than the RPE, and tell us how accurate we can expect the given camera setup to be in real-world units.