GoPro Lens Distortion Removal

The Fisheye lens that GoPro uses provides a great field of view, however it also distorts the image.  In this project we will remove the distortion by calibrating the camera using Python and OpenCV.


Overview

A image taken with a GoPro Hero 3 Black.  Notice the effects of the "barrel" distortion causing straight lines to appear curved.
A image taken with a GoPro Hero 3 Black. Notice the effects of the "barrel" distortion causing straight lines to appear curved.

Every camera, with the exception of high end models, imparts some form of distortion on an image.  The image on the right was taken from a GoPro Hero 3.  As  you can see, objects which are supposed to be straight (red lines), like the door frame and cabinets are curved in the image.  This is mostly due to the shape of the lens and is typically called radial distortion.  The fisheye lens used in GoPro cameras causes increased distortion as you move away from the center of the image.  There is a second form of distortion called translational distortion which derives from the fact that the lens is typically not perfectly centered above and parallel to the imaging sensor.  For more a in-depth explanation of the camera geometry and related math you can check out these books: Multiple View Geometry in Computer Vision or Learning OpenCV

 

Due to the distortion imparted on the image, the geometry, shape and scale of objects in the image are modified compared to the real-world.  If we want to take any quantitative measurements from the image we must first remove the distortion so that the image exemplifies the real-world.  There are several on-line resources for camera calibration including a great Matlab toolbox and a Python/OpenCV tutorial, which this program is based on.  While this project focuses on calibrating GoPro cameras, any camera can be calibrated with some minor adjustments to the code.

 

This script has been tested on a GoPro Hero 2 and GoPro Hero 3 black.


Software Used


Project Files

Download
Checkerboard calibration image
calib-checkerboard.png
Portable Network Image Format 34.9 KB
Download
GoPro Distortion Calibration Script
GoPro_Calibration.zip
Compressed Archive in ZIP Format 3.4 KB
Download
GoPro Distortion Calibration Script
OpenCV 3.0.0 version
- Includes cropping parameter
GoPro_Calibration_3.0.zip
Compressed Archive in ZIP Format 3.7 KB

Step 1:  Collecting calibration images


Calibrating a camera system relies on collecting images of a calibration pattern of known dimensions.  This script will collect images of this pattern and compare the dimensions of the pattern in the image to the real life dimensions.  This will allow us to model the distortion in the image throughout the field of view and calculate the distortion parameters of the camera. We will then undistort the images or videos based on these values.

 

For this script we will use a checkboard pattern which can be downloaded above.  I typically print out this pattern on a standard 8.5" x 11" sheet of paper and tape it to a piece of plexiglass.  Anything rigid will work, we just don't want the calibration pattern to deform.  Next, we need to create a video of the calibration pattern.  The script will play back this video and you will have the ability to save images of the pattern to use in the calibration.  With a GoPro this can be a touch difficult without two people and LCD backpack because you cannot see what you are filming.  However, I typically do it with just myself.  To record the video make sure the camera is on a stable platform so it remains still.  While recording place the calibration pattern a minimum of ~2 feet away from the camera and move the pattern around the field of view.  Since I normally cannot see the video, I slowly move the pattern right and left, up and down to position it in many different locations.  Move the pattern slowly, any motion blur will reduce the accuracy of the calibration.  You want to be able to pull video frames with the pattern in many different positions around the field of view of the camera.  Make sure to put the pattern in at least 20 unique positions, try to get the periphery since this is where the distort is most pronounced.  Feel free to move the pattern forward and back, rotation of the pattern is not a problem.  The image below shows a mosaic of images used for calibration.

 

This image is a mosaic of video frames taken from a calibration video.
Mosaic of images taken from a calibration video.

Step 2: Video Import and Image Selection


In this section we will begin the calibration.  First open the script and check the calibration parameter sections.

Here you need to direct the script to the calibration video file, named 'filename'.  If the file is in the same folder as the script then the name will suffice, if not then you need to add the file directory (i.e. 'C:\Video\GoProVideo.MP4').  Next check the rest of the parameters and change as needed.  Typically I use 20 calibration images.  If you are using the checkerboard provided then the board_w and board_h should be all set.  Check the dimensions of the checkerboard squares and adjust the board_dim value.  Lastly change the image_size if your recording at a different resolution.  If you change the resolution or the FOV settings on the camera, it will change the distortion model and should be recalibrated. 

 

Once the script is set-up, run the program.  The video will begin to play.  Press the spacebar to save the video frames for calibration.  The video will run until either the video ends or the number of calibration images listed above are collected.  You can abort the program by hitting the esc button.


Step 3: Calibration


Once the previous step is completed the script will load the calibration images and attempt to locate the checkerboard corners.  If the corners are found the program will return as True and display an image similar to that on the right.  Check the images to ensure that there is good identification of the corners.  When going through the calibration process, it is sometimes useful to try it multiple times to learn what works best.  The more you do it the more you learn what images the program likes and what images will be poor quality or rejected. 

 

Once the image has been check the esc button to move to the next image.  Once all the images have been analyzed the script will run the calibration function.  Below is a sample of the output.

For the camera calibration there are two important datasets, the intrinsic matrix and the distortion coefficients.  The intrinsic matrix in a 3x3 matrix which contains information about the focal length (position 0,0 and 1,1 in the matrix) and the principal point (position 2,0 and 2,1).  The principal point is the point on the image which is directly below the center of the lens.  This value should ideally be in the center of the image but is typically slightly off center.  Check the values to ensure that the numbers are not way off.  These values should be roughly half the resolution on the horizontal and vertical.  The next data that you will see are the distortion coefficients.  These values are the parameters which will go into the distortion model.  Both of these data sets are saved into a *.npz numpy file, which can be imported for future programs.


Lastly the program will calculate the total reprojection error.  The closer this value is to zero the better.  I usually like values under 0.1.  Play around with the calibration, see what works and what improves it.


Step 4: Undistorting


After the calibration results, the script will then reload the calibration images and remove the distortion.  Press the esc button to move through the images.  This is another validation step to make sure that the calibration model is accurate.  If the images don't look right then the calibration model may not be accurate and the camera should be recalibrated. 

 

Due to the fisheye distortion in the GoPro's, the pixels around the periphery are more spread out then they should be.  The undistortion method takes these pixels and moves them closer to the center of the image.  Missing pixels tend to occur around the corners because the distortion is extreme and there is no information outside the video frame to fill in these areas.  The standard method in OpenCV crops the image so there is no missing pixels.  You will notice a loss of information around the edges.  In the new OpenCV 3 version there is a cropping parameter at the top of the script (line 29).  If this value is set to 0 then the program will crop out all the black pixels.  This will result in some information loss around the periphery.  A value of 1 will utilize all the available pixels.  This will result in black areas where there is no information from the original image, but it is useful if there is important information near the edges and corners.  I found the this value needs to be tweaked to something in the middle ~0.5 to optimize for each situation.

 

Once the camera has been calibrated the script below can be used to undistort any video collected with that camera.  Remember if you change the resolution, FOV or environment (i.e. underwater) then this will effect the calibration.


Download
Distortion Removal Script
Undistort.zip
Compressed Archive in ZIP Format 1.3 KB
Download
Distortion Removal Script
OpenCV 3.0.0
- Includes cropping parameter
Undistort_3.0.zip
Compressed Archive in ZIP Format 1.4 KB

To run the script, update the video file name and the calibration file name (if changed).  If the files are all in the same folder as the script then the name will suffice, if the files are located elsewhere then the file directory information is required.

Run the script and your done. Due to the limitation in OpenCV, the script currently strips out the audio and save the files in AVI format.


If you found this information helpful, if it worked, if it didn't work, leave a comment below.

Write a comment

Comments: 27
  • #1

    zhenhou (Sunday, 22 March 2015 06:05)

    hi, thx for sharing ur great work! i tried to use your code on raspberry pi but i just cannot sample the video,it just straightly skip to next step and report error. could u help me with it?

  • #2

    theeminentcodfish (Wednesday, 25 March 2015 10:05)

    zhenhou, can you be a little bit more specific? What step was skipped? What was the error reported?

  • #3

    tim.b (Friday, 10 April 2015 08:07)

    Hello,
    Thank you very much for the useful information.
    Are you willing to share your calibration results of Gopro 3 at different resolution?

  • #4

    Esther (Monday, 11 May 2015 08:31)

    Hi,
    I am a PhD student in gear technology, and wish to thank you for this nice tutorial.
    I am intending to use stereo recordings of gillnets from 2 GoPro cameras. Have you happen to try this too ? If so, would you have any advice on the appropriate distance between the two cameras ?
    Thanks!
    Best regards.

  • #5

    theeminentcodfish (Friday, 15 May 2015 10:15)

    Hello, Esther.
    I am glad that the website is helping. This script was the precursor to the a full blown stereoscopic system with GoPro's. I haven't worked out all the bugs but when I do I will post the code. Currently I am having accuracy issues, which I think stems from the severe distortion removal. However I have built a system using two webcams which worked well and was very accurate in initial testing. I had the exact same question that you had, what was the appropriate distance? I ran some tests and found out that it depends on the environment. I tested a system with distances ranging from 6"-36". When the cameras are close together you get good overlap in the two images close to the camera which allows you to measure objects which are close accurately. As the objects move farther away the disparity becomes smaller and the inaccuracies increase. The opposite is true when the cameras are far apart. At this position you cannot get the object in both images when they are close to the camera, however you get better accuracy farther from the camera. If I had to compromise I would probably say around 12" or 30 cm. However this information was collected in air, underwater will change these values but I am uncertain how much. I haven't tested that yet.

  • #6

    Xixuan (Sunday, 28 June 2015 09:16)

    It is very helpful. Thanks!

  • #7

    Yusuf (Monday, 10 August 2015 13:58)

    Hi

    After the images have been captured, I keep getting 'false' and then an error. What could be causing this? How can I resolve this?

    Thank you

  • #8

    Matin (Monday, 26 October 2015 16:22)

    Hi,

    I tried it but I got the following error:

    Starting camera calibration....
    Step 1: Image Collection
    We will playback the calibration video. Press the spacebar to save
    calibration images.

    We will collect 20 calibration images.

    All the calibration images are collected.
    ------------------------------------------------------------------------
    Step 2: Calibration
    We will analyze the images take and calibrate the camera.
    Press the esc button to close the image windows as they appear.

    Loading... Calibration_Image1.png
    OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /media/Data/Code/opencv/modules/imgproc/src/color.cpp, line 3256
    Traceback (most recent call last):
    File "GoPro_calib_web.py", line 224, in <module>
    ImageProcessing(n_boards, board_w, board_h, board_dim)
    File "GoPro_calib_web.py", line 135, in ImageProcessing
    grey_image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
    cv2.error: /media/Data/Code/opencv/modules/imgproc/src/color.cpp:3256: error: (-215) scn == 3 || scn == 4 in function cvtColor

    In fact, the video did not open in order for me to collect images.

    What could be the problem?

  • #9

    Tamas (Tuesday, 09 February 2016 04:26)

    I had the exact same problem and the cause was an invalid image path (I used a relative path).

  • #10

    SolarFlare (Thursday, 28 April 2016 12:48)

    So out of curiosity, based on the camera sensor and lens you are using, how well does the data from your calibration matrix correspond to the actual focal length of the camera setup? For example, in your case Fx = 857 pixels and Fy = 877 pixels. What would you get if you converted back to mm and compared this with the focal length of the lens you are using, as stated on the specifications.

  • #11

    emilio.cremades@gmail.com (Tuesday, 10 May 2016 06:37)

    Great job! I have no gopro camera, i have a sj4000 and the algorithm works perfectly. I'm using python 3.4 and i must to correct some lines, but its works perfectly. I will use it to a real time capture conversion.

    (to Matin) The cause of your error is that you don't capture any image. Probably you are using python 3 and when you press spacebar it isn't detected and no image is saved. I solve this error
    76 k = cv2.waitKey(int(FrameDuration)) & 0xFF

  • #12

    Izwan (Saturday, 01 October 2016 09:43)

    Hi,

    what mean by board_dim ? size of the board? or the size of the square?

  • #13

    Chris (Wednesday, 05 October 2016 10:43)

    Hi Izwan. Sorry for the confusion. Board_dim is the dimension of the individual squares. I have it in cm but it could be in any unit.

  • #14

    JB (Friday, 28 October 2016 08:48)

    Dear Chris,

    I am having troubles using your script. The camera's i want to calibrate are the DJI Phantom 4 and the Gopro Hero4. I do not use video, only (4000 by 3000) photo's.

    I deleted the part of the script where it captures the images from the video. No matter what I try, every image turn out to be 'False' :(. The images are taken from a 27 inch monitor with a 10 by 5 checkerboard displayed.

    The weird thing is that when I get a random (much smaller) image from the internet of a checkerboard, it does work.

    What do you think I am doing wrong?

  • #15

    Chris (Friday, 28 October 2016 13:08)

    Hey JB,
    Not quite sure what is happening but let's see if we can't troubleshoot a couple things. First and foremost are the images actually getting in? Can you run a simple image import and display script. Something like:
    import cv2
    image = cv2.imread('image.png')
    cv2.imshow("Test", image)

    If that works, are you using OpenCV 2 or 3. The commands for OpenCV 3 are a bit different. Enough to cause problems. Did you change the checkerboard parameters at the top of the script (board_w and board_h)? This is something I should have been more clear on. These parameters are actually the number of inside intersections, so a 10x5 checkerboard will be board_w = 9 and board_h = 4.

    Next, try different positions of the checkerboard. It's actually a bit nuanced, the board should be pretty close to the camera, but sometimes if it's really close (i.e. taking up the whole screen) that can cause problems. Alternatively, if it's to far away then it'll work but the calibration won't be quite as good.

    Test these things out. Let me know what works or doesn't work.

    Chris

  • #16

    JB (Monday, 31 October 2016 05:09)

    Dear Chris, thanks for your response.

    I have tried different settings and distances from the board. I do not know which version of opencv I am using. However, I just tried an A3 printed checkerboard, and now it does seem to work. Only, the file has so many pixels that I cannot even see the colored detected corners. Also, when it tries to un-distort the image, I just get a really badly distorted image (much worse than the original image). Is the 4000x3000 resolution maybe a problem?

  • #17

    Chris (Monday, 31 October 2016 12:27)

    Hey JB,

    On your other checkerboard, are the squares square, or are they rectangular? That would effect the code. Right now the "actual" points are assumed to be square to one another.

    I don't believe that the 4000x3000 resolution should be a problem. I haven't tested that size but I've tested other sizes smaller then that. Just make sure that you changed the image_size parameter with the script. Additionally, if you can't see the whole images, you can down scale the resolution just prior to showing it.

    resized = cv2.resize(image, (800,640), interpolation = cv2.INTER_AREA)

    Another question, how badly distorted are the images that your working with. For badly fisheyed images, like wide-angle GoPro videos, I've had some problems getting the best calibration. Cropping the images might help.

    Chris

  • #18

    JB (Tuesday, 01 November 2016 10:34)

    Dear Chris,

    Thanks for the suggestions, after scaling the image before the imshow, I have got it working now. I find it really weird because it shouldnt affect the K matrix, right?

    When I use the found distortion and camera matrix to undistort gopro images, in the center it works fairly well. However on the edges it starts to go south and the whole image looks like a sphere. Even though it will dispose part of my measurement, I think I will just cut the middle part out.

    Thanks for your help!

  • #19

    Nima (Sunday, 20 November 2016 13:11)

    My error is :
    Loading... Calibration_Image1.png
    OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /media/Data/Code/opencv/modules/imgproc/src/color.cpp, line 3256
    Traceback (most recent call last):
    File "GoPro_calib_web.py", line 224, in <module>
    ImageProcessing(n_boards, board_w, board_h, board_dim)
    File "GoPro_calib_web.py", line 135, in ImageProcessing
    grey_image = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
    cv2.error: /media/Data/Code/opencv/modules/imgproc/src/color.cpp:3256: error: (-215) scn == 3 || scn == 4 in function cvtColor
    please guide me!

  • #20

    Aaron (Tuesday, 10 January 2017 06:14)

    Nima,
    As another user mentioned above (and I have experienced myself), that error seems to be appear when you are not loading the video correctly. There's a few reasons this could happen. First, check your filename. if the video you're trying to load is not in the same directory as the script itself, make sure to include the full path ('C:/blah/blah2/video.mp4' on Windows or '/home/blah/Desktop/video.mp4' on Linux, for example). If that appear correct, then try to run it through OpenCV yourself outside of the script - the program may be obscuring an error with OpenCV. For example, if you're in Linux, open the terminal, start Python, and try

    import cv2
    filename = '/home/blah/video.mp4'
    vid = cv2.VideoCapture(filename)

    ...and you might see an error start there. In my case, my videos were saved as 59.4 FPS, and I had to use avconv to up it to a full 60 FPS (which apparently OpenCV requires).

    Third, I had issues with the .get statements in the script. In his script, he uses the full name of the properties he is fetching, but that for whatever reason does not work on my Ubuntu system. Try replacing the following variables as prescribed:

    FPS = video.get(5)
    width = video.get(3)
    height = video.get(4)
    total_frames = video.get(7)

    These are all in the same block of code, within 7 or so lines of each other, in the definition of ImageCollect.

    Hopefully that helps any other stray wanderers passing through here. Now I just need to figure out why it can't detect the squares in my video....

    -Aaron

  • #21

    Rice (Tuesday, 27 June 2017 11:29)

    Hi, may i ask if this will be able to work with live videos? For example, if i were to use a gopro/action camera for stationary filming and I am viewing it live within a distance on like a mobile phone and doesn't want it to look distorted. Will it be able to work?

  • #22

    Chris (Tuesday, 27 June 2017 12:05)

    Hello Rice,

    Yes this could be done, but it might not be easy. If you can gain access to the video frames, then they can be processed as they come in. This would be easy for a webcam since OpenCV provides an easy interface with most USB webcams. However if you are using a GoPro, then this would be more difficult due to the challenge of gaining access to the incoming frames. As far as I know, the GoPro App doesn't allow that. It could be done but you would have to write a bunch of code to intercept the wi-fi signal, decode it and then process it. I know the YI 4K action camera has an opensource GitHub site which would provide access but I believe it's written in java. Basically, if you can get the frames into python then it's easy, but getting it into python might be challenging.

  • #23

    Chris (Tuesday, 27 June 2017 12:12)

    Rice,

    Another follow up. To access the GoPro live stream you can check out this python package:
    http://goprohero.readthedocs.io/en/latest/

    Chris

  • #24

    Adam (Friday, 11 August 2017 08:14)

    Hi there,
    I have calibrated my camera in Matlab, and now have the raw values which I want to transfer into openCV using your undistort script. I noticed in your code you are loading an npz file (which I just became familiar with but having some trouble). What do you suggest would be the easiest way to load my calibration values? Creating my own npz file to match with your code? Or is there another filepath I can use i.e. how should I should I go about setting up my calibration values to make this work?
    Thank you in advance

  • #25

    Chris (Friday, 11 August 2017 10:02)

    Hey Adam,
    The npz file is just an easy way to save and reload data in python, you don't need to try to recreate it. You have a couple options. I believe, though I don't remember because I haven't used Matlab recently, that Matlab will save the camera parameters in a *.m or *.mat file. There are ways to bring that into python, but you'd have to google it. Alternatively, you could just hard code the camera parameters into the script. The Undistort method requires two parameters, the distortion coefficients and the intrinsic matrix. These are simple arrays. You can manual insert them using numpy as follows:

    #Import numpy
    import numpy as np

    distCoeff = np.array([[-2.5761420e-01, 8.7708999e-02, -2.5697083e-04, -5.9339089e 04, -1.5219491e-02]])
    intrinsic_matrix = np.array([[ 857.48296979, 0. , 968.06224829],
    [ 0. , 876.71824265, 556.37145899],
    [ 0. , 0. , 1. ]])

    Obviously you numbers would be different. Hopefully this works for you.
    Good luck,
    Chris

  • #26

    Adam (Sunday, 13 August 2017 19:23)

    Hi Chris,
    Thanks for your help, that makes a lot of sense I will plug in my calibration values and give it a shot. I also wanted to ask your opinion on how effective the Undistort function was when you applied it to the GoPro? I am developing a video analysis system for monitoring sprint performance - where a major parameter involves obtaining a wide field of view. We are thinking of purchasing an action camera to test its performance but also are weary of the amount of radial distortion it will produce. Any advice will be appreciated :)
    Adam

  • #27

    Aaron Zettler-Mann (Thursday, 17 August 2017 15:48)

    I'm having some trouble running the code for calibrating GoPro video. The code appears to run fine at step 1, returning the "print" messages as though everything was working correctly. However, it never actually opens and plays the video, meaning I can't select the calibration images. The software continues to run through step 2, until it tries to load Calibration_Image1.png, which it can't find.
    I'm using the complete file directory information so I know it's looking for the video in the correct place. Thanks for any help you can offer.
    Aaron