Lenses and imaging systems generally fall into a few major categories in terms of the geometry of the images they produce - rectilinear, fisheye (equidistant or orthographic - see here ), scanning cameras and various kinds of curved mirror systems etc. A realworld lens or camera system, except for a pinhole camera, will vary more or less in its actual image geometry from its ideal type. A nominally rectilinear lens on a 35mm film camera will always, for example, have measurable distortions. These are apparent visually as "barrel" or "pincushion" distortion in some cases but some variations from ideal rectilinear imaging will always be present. It is possible to measure the distortions of a lens quite accurately with various methods - a slide with a grid marked on it can, for example, be placed in the film aperture, a light shone through it and the projected image measured. Once the distortions of a lens are known the image can be processed with image warping software to remove the measured distortions. The process of measurement of a lens's distortion factors is known as lens calibration and the process of removing these distortions through image processing is called image rectification.

A fisheye lens (the usual equidistant type say) will produce what appear to be distorted images to the casual observer but these distortions are mostly inherent to the basic geometry of this lens type. As well as the normal visual distortion of any fisheye lens there will be a variation from the ideal fisheye geometry. As with other imaging geometry types it is possible to measure this variation from ideal form and remove it with an image processing step.

Lens calibration and rectification is often done for photographic applications where measurements need to be made from images - forensic scenes, aerial photography for mapping, forestry tree cover surveys etc - and image rectification also has some potential everday photographic uses eg. the removal of "barrel" and "pincushion" distortions from images taken with nominally rectilinear lenses. Panoramic and mosaic stitching operations will often benefit from or require lens calibration and image rectification steps too.

Helmut Dersch's Panorama Tools includes a feature, the "Correct" tool, which can be used to produce image rectification of images from various lens and camera types including rectilinear, fisheye and scanning cameras. A common way of specifying lens calibration data is in terms of "Radial distortion" factors. The Radial Distortion formula is an expression which relates the actual locations of pixels in an image with their theoretical locations in an ideal lens image and it can be used to rectify an image. This expression does a good job with many real world images with only four terms or factors. These radial distortion factors, a,b,c and d can be derived from a calibration process and entered into the Correct tool to rectify a given image.

Actual measurement of the radial distortion factors (the "Correct" factors) of a lens or camera system with Panorama Tools is also possible. A precise grid can be photographed for instance and from a single image the Optimizer tool in Panorama Tools can be used to calculate the Correct factors. More conveniently a sequence of two or three images of a regular realworld scene can be used instead and the Optimizer will calculate the Correct factors from the locations of a series of common features in the image series. A very good kind of subject matter for this kind of work is a high rise inner city scene where the buildings extend way up and down in the visual field from the standpoint of the camera position eg. from an elevated roadway or from halfway up a tall building.

This estimation of the Correct factors should only need to be done once for a given lens or camera system and thereafter the factors can be applied with the Correct tool to any image (in vertical or horizontal format) from that lens or camera system without modification.

The use of the Optimizer in Panorama Tools to calculate lens calibration data from an image sequence and the use of this data to produce more accurate image geometry with the Correct tool is here illustrated with a fullframe fisheye lens example (Olympus Zuiko 16mm) but similar image sequences can be used to calibrate other lens types and correct their images.

In other photography contexts the Optimiser in Panorama Tools can be used to deduce many characteristics (roll, tilt and yaw values for instance) of image sequences but for lens calibration purposes it is desirable to keep as many variables as constant as possible so that the target variables - the a, b , c and d Correct factors and the lens Field of View (FOV) can be calculated more precisely and unambiguously.

My scanner (Nikon) does not scan images in precise registration and it does not show the entire image area so for panorama stitching purposes I scan so that the left and top of the actual image area is visible in the scan. I crop the images tightly to the left hand end and top of frame and then expand the images with the Canvas tool in Photoshop so that the final images are the same size as if I was able to scan the entire image area.

Next I make copies of my three images and mark distinct features with the arrow tool in Photoshop (the line tool with arrowheads selected). Note this has nothing to do with the barcode pointers that Helmut Dersch provides as an automatic point reading device. My arrows are just visual reminders. The images now look like this:

The information is given to the Optimiser in the form of a script. This is a simple text file which can be made in Notepad on Win. Save it as something like myscript1.txt  For my three images my first script looked like this:

p w1500

i w1926 h1296 f3 v135 r0.0 p0.0 y0.0 a0.01 b0.01 c0.01
i w1926 h1296 f3  r0.0 p0.0 y45.0 v=0 a=0 b=0 c=0
i w1926 h1296 f3  r0.0 p0.0 y90.0 v=0 a=0 b=0 c=0

v v0 a0 b0 c0
 

c n0 N1 x1076 y87 X528 Y71
c n0 N1 x1078 y815 X420 Y831
c n0 N1 x996 y1127 X415 Y1165
c n0 N1 x1696 y1257 X1179 Y1196
c n0 N1 x1625 y658 X963 Y664
c n0 N1 x1530 y116 X963 Y160
c n1 N2 x1179 y1196 X624 Y1204
c n1 N2 x963 y664 X320 Y666
c n1 N2 x963 y160 X411 Y128
c n1 N2 x1740 y1070 X1167 Y1024
c n1 N2 x1768 y696 X1139 Y693
c n1 N2 x1595 y98 X1056 Y147

You can read about the meaning of these script lines in the "align.pdf" and the "stitch.pdf" Acrobat documents in the Panorama Tools documentation but here are some notes:

The first line means that the Optimiser - if it was to build a panorama(p) from these images - it is to make it 1500 pixels wide(w) and it will be of PSphere type (default)

The next three lines contain the image(i) data you are giving the Optimizer. You have three images hence three lines. Reading the first line this means:
this image has a width of 1926 pixels, a height of 1296 pixels, it has a horizontal field of view (v) of, I guess, about 135degrees.
Camera roll(r) is (I hope) zero, pitch is (I hope) zero, yaw as this is the first imageof the sequence is zero too and as I have no idea what the correction values are I give them arbitrary values of 0.01 each. (Note that is seems necessary for these initial guesses for the Correct settings to be non-zero.)

The final two lines of the image line set contain the elements v=0 a=0 b=0 c=0.
These tell the Optimiser that you are using the same lens for each of the three shots and that hence FOV(v) and a, b and c are the same for each. These lines also tell the Optimiser that the yaw values are 45 and 90 degrees respectively for these images.

The next line ie. v v0 a0 b0 c0 tells the Optimizer what variables(v) you want it to 'optimize" or calculate ie. in this case the FOV(v) and the a, b  c values.

The next twelve lines contain the locations of the corresponding points in your two pairs of images ie. the left and centre images and the centre and right image. The left image is image 0, centre is 1 and right is 2. The first image in a pair is lower case the second upper case. Thus the first line of the corresponding point data set means: this is a control point(c) and the first image (n0) has feature at location (x1076 y87) which is identical to the feature at location (X538 Y71) in the second image (N1). Since there are six features marked as being in common with the left(0) and centre(1) images there are six lines for their pairs of corresponding points. Ditto for the centre(1) and right(2) images.

So save your script file and close it (PT writes to the text file and it should be closed while the Optimizer is running or the Adjust tool is using a script). To run the Optimizer open an image (any image, the Optimiser needs a dummy image to be open) and go:

Filter
Panorama Tools
Adjust

The Create Panorama window opens Check "Use Script" and "Run Optimizer" - other settings will have no effect.

Use "Browse" to find your saved script text file

Click OK and in a few seconds the text editor will open with your now modified text file - it will have appended at the bottom of
your original lines a bunch of new lines. The Optimiser has calculated your required values (FOV, a b, c) and made extra lines
which can be used by the Adjust tool to build a panorama from the three images. In this case we dont want to build a panorama - we just want to get the FOV and a, b and c values from the panorama-building script it has produced - so we just read
them off from the script. The script above produces the following version on running the Optimizer:

 p w1500

i w1926 h1296 f3 v135 r0.0 p0.0 y0.0 a0.01 b0.01 c0.01
i w1926 h1296 f3  r0.0 p0.0 y45.0 v=0 a=0 b=0 c=0
i w1926 h1296 f3  r0.0 p0.0 y90.0 v=0 a=0 b=0 c=0

v v0 a0 b0 c0
 

c n0 N1 x1076 y87 X528 Y71
c n0 N1 x1078 y815 X420 Y831
c n0 N1 x996 y1127 X415 Y1165
c n0 N1 x1696 y1257 X1179 Y1196
c n0 N1 x1625 y658 X963 Y664
c n0 N1 x1530 y116 X963 Y160
c n1 N2 x1179 y1196 X624 Y1204
c n1 N2 x963 y664 X320 Y666
c n1 N2 x963 y160 X411 Y128
c n1 N2 x1740 y1070 X1167 Y1024
c n1 N2 x1768 y696 X1139 Y693
c n1 N2 x1595 y98 X1056 Y147
*

# ====================================================================
# Output  generated by Panorama Tools
# 504 function evalutations
# number of calls to fcn has reached or exceeded 200*(n+1)
# Parameters for Each Input Image:
# (*) - optimized         (p) - preset

# Image No 0:
# Yaw: 0 deg (p) Pitch: 0 deg (p)
# Roll: 0 deg (p) HFov: 138.315 deg (*)
# Polynomial Coefficients: a   0.016170 (*); b   -0.109805 (*); c   0.028142 (*)
# Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)
# 4th polynomial coefficient: 1.06549
# Command for Panorama Creation:
o f3 r0 p0 y0 v138.315 a0.016170 b-0.109805 c0.028142 -buf

# Image No 1:
# Yaw: 45 deg (p) Pitch: 0 deg (p)
# Roll: 0 deg (p) HFov: 138.315 deg (*)
# Polynomial Coefficients: a   0.016170 (*); b   -0.109805 (*); c   0.028142 (*)
# Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)
# 4th polynomial coefficient: 1.06549
# Command for Panorama Creation:
o f3 r0 p0 y45 v138.315 a0.016170 b-0.109805 c0.028142 +buf -buf

# Image No 2:
# Yaw: 90 deg (p) Pitch: 0 deg (p)
# Roll: 0 deg (p) HFov: 138.315 deg (*)
# Polynomial Coefficients: a   0.016170 (*); b   -0.109805 (*); c   0.028142 (*)
# Horizontal Shift: 0.000000 (p)   Vertical Shift:  0.000000 (p)
# 4th polynomial coefficient: 1.06549
# Command for Panorama Creation:
o f3 r0 p0 y90 v138.315 a0.016170 b-0.109805 c0.028142 +buf
 

# ====================================================================
# Control Points: Distance between desired and fitted Position (in Pixels)

# Control Point No 0:  1.79853
# Control Point No 1:  3.00563
# Control Point No 2:  4.50907
# Control Point No 3:  3.09462
# Control Point No 4:  3.4298
# Control Point No 5:  3.00576
# Control Point No 6:  1.08559
# Control Point No 7:  2.96635
# Control Point No 8:  4.76336
# Control Point No 9:  1.11215
# Control Point No 10:  1.55243
# Control Point No 11:  3.21218
 
 
 
 Thus the values we want are HFov = 138.315 a = 0.0162 b=-0.110 c= 0.028 (d=1.065) (nb a+b+c+d=1)

We can check the visual effect of these values on a single image by using the Correct and Remap tool. Thus we could open the central image say and go:
Filter
Panorama Tools
Correct

Fill in the a, b, c and d values (same for each color)
go OK and we get this image:

The vertical straight lines on the buildings are now straight pretty much throughout the whole image area. This is obviously a requirement for successful stitching of a series of images of lots of tall buildings. If the Remap tool is used alone without previous application of the Correct tool it would be seen that there is residual distortion in the straight lines of the buildings which would make stitching more difficult.

 The script which the Optimizer produced could now be used to build the panorama (you close the script then open the first image, go Adjust, check "Insert", check "Use Script" go OK. Then open the next image, go Adjust, click OK and ditto for the third image) but there would be some errors as the script that the use of the Optimizer produced was reporting feature mismatches as large as 4.76 pixels. So if a good panorama from the three images was required and it was desired to use the Optimizer-produced script  to build it automatically then further applications of the Optimizer would be necessary.

We could, for instance get the Optimizer to calculate the actual pitch, roll and yaw of each shot. Doing this will reduce the errors to less than a pixel. Then the execution of the script in panorama building mode will produce a visually perfect panorama.

If refined roll, pitch and yaw values are available from additional Optimizer applications they can be inserted into a script and used to produce a more refined estimation of the Correct values and the FOV. These refined FOV and (particularly) Correct values might be different from the ones first arrived at but the actual visual effect on the image will likely be much the same.

Peter Murphy 1999