4
$\begingroup$

This is a bit of a long shot, but I was hoping somebody might have some insight (not sure of a better forum than here but open to suggestions). I have implemented the optical flow algorithm from the paper An Improved Algorithm for TV-L1 Optical Flow. I've tried to stick to exactly the same parameters as the article explains (they have quite detailed implementation notes), and yet I can't reproduce their results.

From my analysis (I'll show images in a moment), it appears that the algorithm works well with small images, so in the coarse-to-fine pyramid you can see the flow is accurately calculated for small-scale version of each image. Then when the images are upscaled beyond a certain point the optimisation seems to converge to some poor local minimum. To confirm this suspicion I reran the algorithm with a scaling factor of 0.9 between pyramid levels (instead of the 0.5 the article uses) and the results are much improved - it seems with a small enough up/downscaling factor we avoid the poor local minima. Here are the flows (Middlebury backyard scene, with scaling factors of 0.5, 0.7, 0.9 in that order):

0.5 0.7 0.9

These are the iteration-by-iteration sequences of each scaling factor (here we see that even 0.5 works well on small images):

0.5: http://youtu.be/EHTO7lJeMrA
0.7: http://youtu.be/-PlTU3VioWg
0.9: http://youtu.be/lK5EP865u0E

I've fiddled with all the other parameters of the algorithm, and they don't have any major impact on this phenomenon. I can upload my Matlab code if anyone is interested. So my question is:

Does anyone have an alternate explanation for this phenomenon? The fact that I can't reproduce their results is troubling me greatly.

Improve this question
$\endgroup$
5
  • $\begingroup$ Have you tried contacting the authors? Also, they have some code available on their website (gpu4vision.icg.tugraz.at/index.php?content=downloads.php) -- in particular, take a look at the Matlab toolbox. $\endgroup$
    – Christian Clason
    Commented May 9, 2014 at 13:23
  • $\begingroup$ @ChristianClason, I haven't tried contacting the authors, I assume they'll shrug me off thinking I've made a mistake in my implementation. Maybe it's worth a try. I'll look at their toolbox more closely, but their code looks difficult to read from first glance. Will report back if I make any progress. Thanks. $\endgroup$
    – bjorne
    Commented May 9, 2014 at 14:26
  • $\begingroup$ Don't assume that without trying :) People are usually happy that someone took enough interest in their work to try to reproduce it. (Start with the most junior person involved that is still in academia.) The key is to a) be polite (without being obsequious) -- in particular, don't give the impression that you are criticizing their work; b) show your effort (the more you have invested in the question, the more they will likely invest in the answer); c) be make your question as concrete and binary as possible (the easier it is to answer, the quicker or more likely it will be answered). $\endgroup$
    – Christian Clason
    Commented May 9, 2014 at 15:04
  • $\begingroup$ (That was more a general remark; your question here was already well formulated in this regard.) $\endgroup$
    – Christian Clason
    Commented May 9, 2014 at 15:06
  • $\begingroup$ Thank you for coming back to answer your own question. As you proposed in your question, it would be nice and helpful to share your code with the community. $\endgroup$
    – maxhaz
    Commented May 14, 2014 at 5:23

2 Answers 2

Reset to default
3
$\begingroup$

It turns out I wasn't multiplying the dual variable by 4 on upsampling which kept it out of sync with the primal variable (because the upsampling adds zero rows/cols and blurs so to maintain value integrity we need to multiply by the ratio of new zero entires to old entries). So trivial and yet so frustrating.

Improve this answer
$\endgroup$
2
$\begingroup$

It looks to me like you might not be giving as much weight to the regularization (TV and/or L1) as they are. That would give the kind of result you're seeing. A good rule of thumb is that the maximum optical flow in each level should be about 1 (in the units of the current scale).

From reading their article, it seems they're rescaling their image to the [-1,1] interval. This is equivalent to using another regularization weight, so maybe you missed this step? I hope it helps you anyway.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your suggestion. I am in fact rescaling the images to the [-1,1] interval. Moreover, I've tried varying weights for the regulation term and while it does have a similar effect to the case above, it isn't the culprit because it doesn't solve the problem - it "smoothes" out the flow by weighing the regulation more and ends up changing the shapes. I'll post some examples when I can, maybe tomorrow. Although excellent insight nonetheless. $\endgroup$
    – bjorne
    Commented May 10, 2014 at 1:31
  • $\begingroup$ Here are some examples that I have on hand, where lambda (the fidelity weight, not the regulation weight i.e. bigger lambda <==> less importance on the regulation, so we are looking for a smaller lambda than what we used above) is 5, 10 and 15 instead of their value of 50: 5: postimg.org/image/gvssnjt93 10: postimg.org/image/mqfgf1qvt 15: postimg.org/image/mjw236p3t With 5 we see a more homogeneous flow, but it has all the detail cut out. As we increase lambda we get some detail back, but then we lose the homogeneity. Excellent idea, but not the solution. $\endgroup$
    – bjorne
    Commented May 10, 2014 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.