GoPro Lens Distortion Removal
The Fisheye lens that GoPro uses provides a great field of view, however it also distorts
the image. In this project we will remove the distortion by calibrating the camera using Python and OpenCV.
Every camera, with the exception of high end models, imparts some form of distortion on an image. The image on the right was taken from a GoPro Hero 3. As you can see, objects which are supposed to be straight (red lines), like the door frame and cabinets are curved in the image. This is mostly due to the shape of the lens and is typically called radial distortion. The fisheye lens used in GoPro cameras causes increased distortion as you move away from the center of the image. There is a second form of distortion called translational distortion which derives from the fact that the lens is typically not perfectly centered above and parallel to the imaging sensor. For more a in-depth explanation of the camera geometry and related math you can check out these books: Multiple View Geometry in Computer Vision or Learning OpenCV.
Due to the distortion imparted on the image, the geometry, shape and scale of objects in the image are modified compared to the real-world. If we want to take any quantitative measurements from the image we must first remove the distortion so that the image exemplifies the real-world. There are several on-line resources for camera calibration including a great Matlab toolbox and a Python/OpenCV tutorial, which this program is based on. While this project focuses on calibrating GoPro cameras, any camera can be calibrated with some minor adjustments to the code.
This script has been tested on a GoPro Hero 2 and GoPro Hero 3 black.
Calibrating a camera system relies on collecting images of a calibration pattern of known dimensions. This script will collect images of this pattern and compare the dimensions of the pattern in the image to the real life dimensions. This will allow us to model the distortion in the image throughout the field of view and calculate the distortion parameters of the camera. We will then undistort the images or videos based on these values.
For this script we will use a checkboard pattern which can be downloaded above. I typically print out this pattern on a standard 8.5" x 11" sheet of paper and tape it to a piece of
plexiglass. Anything rigid will work, we just don't want the calibration pattern to deform. Next, we need to create a video of the calibration pattern. The script will play back
this video and you will have the ability to save images of the pattern to use in the calibration. With a GoPro this can be a touch difficult without two people and LCD backpack because you
cannot see what you are filming. However, I typically do it with just myself. To record the video make sure the camera is on a stable platform so it remains still. While
recording place the calibration pattern a minimum of ~2 feet away from the camera and move the pattern around the field of view. Since I normally cannot see the video, I slowly move the
pattern right and left, up and down to position it in many different locations. Move the pattern slowly, any motion blur will reduce the accuracy of the calibration. You want to be
able to pull video frames with the pattern in many different positions around the field of view of the camera. Make sure to put the pattern in at least 20 unique positions, try to get the
periphery since this is where the distort is most pronounced. Feel free to move the pattern forward and back, rotation of the pattern is not a problem. The image below shows a mosaic
of images used for calibration.
In this section we will begin the calibration. First open the script and check the calibration parameter sections.
Here you need to direct the script to the calibration video file, named 'filename'. If the file is in the same folder as the script then the name will suffice, if not then you need to add the file directory (i.e. 'C:\Video\GoProVideo.MP4'). Next check the rest of the parameters and change as needed. Typically I use 20 calibration images. If you are using the checkerboard provided then the board_w and board_h should be all set. Check the dimensions of the checkerboard squares and adjust the board_dim value. Lastly change the image_size if your recording at a different resolution. If you change the resolution or the FOV settings on the camera, it will change the distortion model and should be recalibrated.
Once the script is set-up, run the program. The video will begin to play. Press the spacebar to save the video frames for calibration. The video will run until either the video ends or the number of calibration images listed above are collected. You can abort the program by hitting the esc button.
Once the previous step is completed the script will load the calibration images and attempt to locate the checkerboard corners. If the corners are found the program will return as True and display an image similar to that on the right. Check the images to ensure that there is good identification of the corners. When going through the calibration process, it is sometimes useful to try it multiple times to learn what works best. The more you do it the more you learn what images the program likes and what images will be poor quality or rejected.
Once the image has been check the esc button to move to the next image. Once all the images have been analyzed the script will run the calibration function. Below is a sample of the
For the camera calibration there are two important datasets, the intrinsic matrix and the distortion coefficients. The intrinsic matrix in a 3x3 matrix which contains information about the
focal length (position 0,0 and 1,1 in the matrix) and the principal point (position 2,0 and 2,1). The principal point is the point on the image which is directly below the center of the
lens. This value should ideally be in the center of the image but is typically slightly off center. Check the values to ensure that the numbers are not way off. These values
should be roughly half the resolution on the horizontal and vertical. The next data that you will see are the distortion coefficients. These values are the parameters which will go
into the distortion model. Both of these data sets are saved into a *.npz numpy file, which can be imported for future programs.
Lastly the program will calculate the total reprojection error. The closer this value is to zero the better. I usually like values under 0.1. Play around with the calibration,
see what works and what improves it.
After the calibration results, the script will then reload the calibration images and remove the distortion. Press the esc button to move through the images. This is another validation step to make sure that the calibration model is accurate. If the images don't look right then the calibration model may not be accurate and the camera should be recalibrated.
Due to the fisheye distortion in the GoPro's, the pixels around the periphery are more spread out then they should be. The undistortion method takes these pixels and moves them closer to the center of the image. Missing pixels tend to occur around the corners because the distortion is extreme and there is no information outside the video frame to fill in these areas. The standard method in OpenCV crops the image so there is no missing pixels. You will notice a loss of information around the edges. In the new OpenCV 3 version there is a cropping parameter at the top of the script (line 29). If this value is set to 0 then the program will crop out all the black pixels. This will result in some information loss around the periphery. A value of 1 will utilize all the available pixels. This will result in black areas where there is no information from the original image, but it is useful if there is important information near the edges and corners. I found the this value needs to be tweaked to something in the middle ~0.5 to optimize for each situation.
Once the camera has been calibrated the script below can be used to undistort any video collected with that camera. Remember if you change the resolution, FOV or environment (i.e. underwater) then this will effect the calibration.
To run the script, update the video file name and the calibration file name (if changed). If the files are all in the same folder as the script then the name will suffice, if the files are
located elsewhere then the file directory information is required.
Run the script and your done. Due to the limitation in OpenCV, the script currently strips out the audio and save the files in AVI format.