Zed-ros-wrapper: Camera Calibration Parameters and Rectification

[aseyfi]

Hello,

I am trying to use my ZED camera with an algorithm that accepts rectified images and also needs the camera calibration parameters to do some depth related processing. I am running zed.launch file. Are the topics "left/image_rect_color" and "right/image_rect_color" rectified images? I used ROS stereo camera calibration node and the parameters that I get for VGA are different from the ones that I downloaded from the Stereolab's database. For example my calibration is giving me

[narrow_stereo/left]
camera matrix
741.613868 0.000000 325.273949
0.000000 736.184966 253.435234
0.000000 0.000000 1.000000

and my downloaded calibration file says

[LEFT_CAM_VGA]
fx=699.417
fy=699.417
cx=297.493
cy=243.168

Would somebody please help me to understand if I am doing something wrong in the calibration process or actually calibration parameters change after rectification and my calibration may be correct.

Thank You,
Ahmad

[adujardin]

Hi,

The images outputting from the ZED SDK are already rectified. So you can ignore the calibration process, it's already taking care of.

However if you want to recalibrate you can use our ZED Calibration tool (easiest solution) or other software (openCV, ROS stereo camera calibration node...) but in the later, you will need to provide unrectified images which are currently not accessible using the ZED SDK. So you will need to code an other ROS node using VideoCapture or something similar.

[chuong]

Hi Aduljadin,

I would like to process images saved from ZED Explorer. I found camera parameters from SN####.conf file. How is stereo rotation-translation matrix computed from RX, RZ, CV, and Baseline?

[aseyfi]

I just found out the calibration file that I downloaded manually from this link "http://calib.stereolabs.com/?SN=NNNN" (I got the same result with 0000NNNN) which NNNN is my camera's serial number is totally different than the one in settings folder. Here are the parameters for VGA from the downloaded file

[LEFT_CAM_VGA]
fx=699.417
fy=699.417
cx=297.493
cy=243.168
k1=-0.171585
k2=0.0253922

[RIGHT_CAM_VGA]
fx=700.648
fy=700.648
cx=350.405
cy=259.796
k1=-0.173575
k2=0.0264623

and here are the parameters from the file in the settings folder

[LEFT_CAM_VGA]
cx=323.747
cy=188.584
fx=349.707
fy=349.707
k1=-0.171585
k2=0.0253922

[RIGHT_CAM_VGA]
cx=350.203
cy=196.898
fx=350.325
fy=350.325
k1=-0.173575
k2=0.0264623

Would somebody please explain why the two files are different?

[adujardin]

Hi @aseyfi

The difference is due to the firmware. The new firmware (1142) introduced the wide VGA. However, your camera has been factory calibrated with the previous firmware (922, 4:3 VGA).
The setting file download / conversion is handled by the ZED Explorer (with command line option --dc or GUI) but not yet with https://calib.stereolabs.com since we can't determine the firmware version in use.

So the parameters from your settings folder (the second) are the correct ones.

[adujardin]

Hi @chuong,

The rotation (in rotation vector form, convertible into matrix with Rodrigues formula) are the following :
Rx = RX (in rad),
Ry = CV,
Rz = RZ.
The translations are set to 0 except of course for Tx :
Tx = - baseline,
Ty = 0,
Tz = 0.

However please note that these values are refined by the autocalibration process to achieve higher accuracy with the ZED SDK.

[chuong]

Hi @adujardin ,

Thanks for the info. Is stereo rotation matrix equal to Rx*Ry*Rz or Rz*Ry*Rx?

I calculated stereo translation vector and rotation matrix for HD resolution as:

T = np.array([-BaseLine, 0, 0])
Rz, _ = cv2.Rodrigues(np.array([0, 0, RZ_HD]))
Ry, _ = cv2.Rodrigues(np.array([0, CV_HD, 0]))
Rx, _ = cv2.Rodrigues(np.array([RX_HD, 0, 0]))
R = np.dot(Rz, np.dot(Ry, Rx)) # Rz*Ry*Rx

I tried bothRz*Ry*Rx and Rx*Ry*Rz for R, but epipolar lines on stereo rectified images are off about 5 pixels between same feature points on stereo images. I got the parameters after recalibrate the camera with ZED Calibration tool. Are the parameters in SN####.conf file as good as stereo calibration using OpenCV with a normal chessboard?

How can I get refined camera parameters from autocalibration process?

[chuong]

Sorry. I made a mistake by undistorting images before stereo rectification. It works correctly now.
It seems either Rz_Ry_Rx or Rx_Ry_Rz work fine for R.

[aseyfi]

Thank you @adujardin . I am using Matlab to open ZED as a regular webcam. The only possible resolution is 2560x720 which is actually two 1280x720 images for the left and right cameras. I would like to process the images in VGA, exactly the same size as ZED ROS wrapper is publishing (672x376). Should I just resize the whole 1280x720 image to 672x376? Are there any other steps involved?

[P-yver]

hi @aseyfi ,
take a look at the Stereolabs Blog, at the end of the page there is a snippet explaining how to grab the ZED images.

The line:
zed.Resolution = zed.AvailableResolutions{1};
let you choose the resolution from the list 'AvailableResolutions' displayed by
zed = webcam('ZED')

[aseyfi]

Thank You @P-yver . In that example we can see "AvailableResolutions: {1x4 cell}" which means there are four different resolution options and and they are using the first one (zed.Resolution = zed.AvailableResolutions{1};). In my case, I have only one option (AvailableResolutions: {'2560x720'}). I am using Mac and there is no ZED SDK for Mac. Do I need to install the SDK if I need to have access to all of the possible resolutions?

[aseyfi]

Hi, I am using the "zed_ros_wrapper" to publish HD images which they are undistorted and rectified (output of ZED SDK). I fixed the camera facing to a scene and saved one frame from the laft camera and one from the right. then I selected one point in the scene. Assuming it is located on [col_l, row_l] and [col_r, row_r] in the left and right images respectively, what are the calculations based on [col_l, row_l], [col_r, row_r] and the data in the calibration file to find 3D location of that point in the world?

[P-yver]

Hi @aseyfi ,

Based on our convention you have :
disparity = col_r - col_l // should be negative
then the 3D point (X Y Z):
Z = (fx_l * Baseline) / (-disparity)
X = (col_l - cx_l) * Z / fx_l
Y = (row_l - cy_l) * Z / fy_l

Have you fixed your problem with the desired resolution on Mac ? I have not yet tested it (we currently don't support Mac).

[aseyfi]

Thank you @P-yver . No, actually I installed Matlab on Ubuntu 14.04 and I get this output. The available resolution is different, but I still have only one option

cam=webcam
cam =
webcam with properties:
Name: 'ZED'
Resolution: '4416x1242'
AvailableResolutions: {'4416x1242'}

based on this instructions (https://www.stereolabs.com/blog/index.php/2015/07/03/tutorial-3-zed-and-matlab/) I am supposed to see this output:

cam =
webcam with properties:
Name: 'ZED'
Resolution: '2560x720'
AvailableResolutions: {1x4 cell}
WhiteBalanceMode: 'auto'
Sharpness: 4
Saturation: 5
Hue: 0
Gain: 4
WhiteBalance: 4600
Contrast: 4
Brightness: 4
Exposure: 2
ExposureMode: 'auto'

EDIT: My Matlab version is R2016a

[aseyfi]

@P-yver , I used the formulas you provided to estimate 3D location of two points in the world. I fixed my camera about 90cm away from a wall and marked two points with 80cm distance on the wall. Then I used zed_ros_wrapper to capture WVGA images from the left and right cameras. Using Matlab, I clicked on the points on the left and right images (as accurate as I could) and found the [row, col] values of each point. Then I used the provided formulas to calculate (X,Y,Z) of each point and calculated the distance between the two points afterwards. From my calculations, I estimated the distance between the two pint 77.5cm. I carried out a similar test for two other points with different distances from the camera too. The actual distance between two points was 6.1m and I estimated it to be 4.86m (1.24m error). Is this the accuracy I should expect from ZED?
I have a question about rectification too: We know that after un-distorting and rectifying the images, they will warped and we have to crop a square out of the warped images. Depending on the rectangle that we select, the camera matrix would be different. Does ZED crop/scale the rectified images in a way that the camera matrix remains exactly as it was estimated during calibration?

[P-yver]

Hi @aseyfi ,
The camera parameters given by the .conf file (or shown by the ZED Explorer) are the initial values. They are changed by the auto-calibration step performed by the SDK during the initialization. To retrieve the parameters corresponding (modified by the rectification) you have to get those parameters with 'left_cam_info_topic'.

The disparity is not linear (it decreases rapidly with the distance) and the sub pixel accuracy is required to achieve realistic measures, click on images is not appropriate to benchmark the process.

The ZED Depth Viewer display the depth when you click on the disparity/depth/left image as well as the sample ZED with OpenCV. Why did you use the ros wrapper to capture images ?

[aseyfi]

Hi @P-yver , Thank you very much for your comment, it was really helpful and I was able to get the rectified image's camera matrix. I was initially using zed_ros_wrapper to publish data for another software in ROS. Then I was trying to go through the steps and understanding stereo vision theory, so I was just more comfortable using this package compared to OpenCV.

[aseyfi]

@P-yver Are the snapshots captured by ZED Explorer raw images?

[P-yver]

Hi @aseyfi ,
Yes, the snapshots captured by ZED Explorer are raw. There is no processing on ZED Explorer, only visualization. The images are therefore unrectified.

[aseyfi]

Thank you @P-yver . In you previous comment you mentioned auto-calibration:

"They are changed by the auto-calibration step performed by the SDK during the initialization."

Would you please let me know what the auto-calibration is? Can it change the camera matrix of the rectified images every time it is being initialized?

Also, I am still having problem to access all of the possible resolutions of ZED in Matlab R2016a and Ubuntu 14.04, and the only resolution that I can have access is '4416x1242'. Does ZED support Matlab R2016a on Ubuntu 14.04 and do you see the same issue?

Thank you,

[aseyfi]

Hello, I printed out the data published on "/right/camera_info" using "rostopic echo" command and here is part of the published data

height: 376
width: 672
distortion_model: plumb_bob
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [337.1528015136719, 0.0, 349.25897216796875, 0.0, 337.1528015136719, 197.14279174804688, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [337.1528015136719, 0.0, 349.25897216796875, -0.04045833647251129, 0.0, 337.1528015136719, 197.14279174804688, 0.0, 0.0, 0.0, 1.0, 0.0]

I think the fourth element of P is representing fx * baseline. Using the baseline=0.12m and fx=337.15, I expect the number to be (337.15*0.12)=40.46, but the published number is 0.04045. Why is the published number scaled?

[arkrobo]

Hi, I installed zed sdk v2.0.0 & cuda 8 (Laptop dell precision 7710 - nvidia graphic card quadro M5000M, windows 8.1 pro). I have updated zed camera firmware to zed_fw_v1142_spi through ZED Explorer.

When run Depth viewer I am get "Unable to open the ZED. Error Code: calibration Filte not available" message.

I downloaded SN1947.conf file from "https://www.stereolabs.com/developers/calib/"pasted "C:UsersElgiAppDataRoamingStereolabssettings" folder. Also, I generated conf file using ZED Calibration. But, Still the same error persist.

Can you suggest how to overcome this?

[S-Shiozawa]

Hi, arkrobo. I had same problem.

"https://www.stereolabs.com/developers/calib/" may be out of date .
You should try self calibration with ..../zed/tools/ZED Calibration.
Before calibrate, remove or rename the old SNxxxx.conf.

Then new calibration file is made in setting/NNNN.conf and solved in my case.
Thanks.

[axing0805091]

@arkrobo I had same problem too. I think there is something unexpected happened when I run /ZED/tools/ZED Depth Viewer. After trying ../tools/ZED Calibration, I can run every .exe in ..ZED/sample/bin. But if I run ../tools/ZED Depth Viewer and change the resolution into VGA@100, the app will throw an error. _And it will download SNxxxx.conf automatically and replace the SNxxxx.conf generated by ../ZED Calibration_, which will lead to every .exe in ZED/sample/bin not working, and throw the "Unable to open the ZED. Error Code: calibration Filte not available".
So please check if the SNxxxx.conf file has been changed automatically if you meet this error. My English is not very good, hope can be helpful.

[arkrobo]

Thanks S- Shiozawa & axing0805091,

The generated SNxxxx.conf file is overwritten by online conf file. fixed the issue by replacing the generated file.

[alarajesh]

Hi,

I need to find camera matrix (3x3) and distort coefficient (1x5) inorder to find pose estatimation using opencv of known pattern. But in SNxxxx.conf file I can find only fx, fy, cx, cy, k1 & k2.

   Camera Matrix 3x3:

   |        fx          0             cx     |
   |         0          fy            cy     |
   |         0          0             1     |

   fx : focal length in direction of camera's x axis 
   fy : focal length in direction of camera's y axis 
   cx,cy  : Principal point in image


   Distortion Coefficient (1x5):
          [k1    k2     p1     p2     k3]

   k1,k2,k3 : Radial distortion coefficients
   p1, p2 : Tangential distortion coefficinets
   k3 - set to zero (only applicable for fish-eyed lens)

1. How can I get distortion coefficient p1, p2?

2. If we do camera calibration using opencv "calibrateCamera(obj_pts, image_pts, Size, camera_matrix, distoriton_coeff, rot_temp, trans_temp, CV_CALIB_FIX_K3);" we can get camera matrix and distortion coefficients.

Can we do this calibration with the images we get from zed.retrieveImage - left image (this Image is rectified) ? If i use opencv videocapture to grab ZED camera image I am getting image of size 376x1344. How to get images of size 2K, FHD & HD for calibration?

Will the camera matrix determined with rectified image be same with SNxxxx.conf file?

[pavloblindnology]

Hello @adujardin,

The rotation (in rotation vector form, convertible into matrix with Rodrigues formula) are the following :
Rx = RX (in rad),
Ry = CV,
Rz = RZ.

Can you please clarify which of the following options is true:
1) Rotational matrix = Rodrigues([Rx, Ry, Rz]))
or
2) Rotational matrix = Rodrigues([Rx, 0, 0]))Rodrigues([0, Ry, 0]))Rodrigues([0, 0, Rz]))
or
3) Rotational matrix = Rodrigues([0, 0, Rz]))Rodrigues([0, Ry, 0]))Rodrigues([Rx, 0, 0]))

Thanks.

[adujardin]

Hi @pavloblindnology,

The first one

1. Rotational matrix = Rodrigues([Rx, Ry, Rz]).

So, for instance something like this using OpenCV Rodrigues :
cv::Rodrigues([Rx, Ry, Rz] /*in*/, RotMatrix3x3 /*out*/)

[casvr]

Hi there, followed your conversation, a question i have, i am trying to calibrate a DSLR lens to the left eye (cam) from the ZED per custom OCV script in order to displace me 4K video later to achieve a 6DOF in VR. the depth map is aligned with the left camera from the ZED and i have managed to rig the ZED above the physical lens and got a pretty decent match per hand, basically fine tuning (warping and un-distorting) my live video feed from the DSLR in Nuke but there is sure a much better way to do this, at least optically more precise. Can someone point me in the right direction? Thank you in advance, Mark

[jamestodo]

Hi @adujardin ,
The rotation matrix called Rotm calculated by
cv::Rodrigues([Rx, Ry, Rz] /*in*/, RotMatrix3x3 /*out*/).

I want to know if the Rotm is the rotation from left camera to right camera or from right camera to left camera.

And is the camera coordinate system in right-handed,y-down coordinate system.

Thanks.

[ravikt]

@adujardin How can we make the ZED camera use calibration parameters obtained by ROS ?

[yaminiMathur]

Can we get the cx and cy (principal points of the camera) using the ZED Camera API? Basically, by calling some function on left_cam or right_cam?

[Myzhar]

@yaminiMathur please do not post questions on closed issues.
Create a new issue instead