I have a stereo image pair and the respective intrinsics and extrinsics of both cameras. With this information, I can calculate the fundamental Matrix between the two cameras (let's call it F).

I can also use this information to rectify both images. In the rectified images I then find features, match them, and approximate the fundamental matrix. This approximated fundamental matrix, F_rect, is of course a completely different one from the F calculated above.

How can I get the "true" rectified fundamental matrix in OpenCV, so I can compare the approximated to the true one? My code looks something like this:

void rectify(cv::Mat& image1, cv::Mat& image2,
            cv::Mat& RT1, cv::Mat& RT2,
            cv::Mat& K1, cv::Mat& K2,
            cv::Mat& dst_img1, cv::Mat& dst_img2,
            cv::Mat* P1_new = nullptr, cv::Mat* P2_new = nullptr) {

    cv::Mat RT(3,4, CV_64FC1);

    LtoR(RT1, RT2, RT);
    cv::Mat distortion(1,4, CV_64FC1);
    cv::Mat R1, R2, P1, P2, Q;

    cv::stereoRectify(K1, distortion, K2, distortion, image1.size(),
                        RT(cv::Rect(0,0,3,3)), RT(cv::Rect(3,0,1,3)), R1, R2, P1, P2, Q,
                        /*flags=*/0, /*alpha=*/1);
    cv::Mat map1x, map1y, map2x, map2y;
    cv::initUndistortRectifyMap(K1, distortion, R1, P1, image1.size(), CV_32FC1, map1x, map1y);
    cv::initUndistortRectifyMap(K2, distortion, R2, P2, image2.size(), CV_32FC1, map2x, map2y);

    cv::remap(image1, dst_img1, map1x, map1y, cv::INTER_LINEAR);
    cv::remap(image2, dst_img2, map2x, map2y, cv::INTER_LINEAR);

    if (P1_new != nullptr) {
        *P1_new = P1.clone();
    if (P2_new != nullptr) {
        *P2_new = P2.clone();

stereoRectify() returns the rotation matrices to rotate the cameras into the rectified coordinate systems and the new projection matrices.

initUndistortRectifyMap returns a transformation, so that the images are rectified after this transformation.

I think I would need the rectifying homography (maybe?), these functions don't return it. Any other way?

TL;DR I rectify a stereo image pair. I have the true fundamental matrix before rectification, and an approximated fundamental after rectification. How do I get the true fundamental after rectification?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy