This is a synthetic test case, in the real application, everything is rather blurred. The rectangle has to cover as much as possible of the black region, while covering as little as possible of the grey region while the white region is ignored. I don't always see all corners, it could e.g. be possible that I just have the upper half of the rectangle and a much larger white area.
I've so far implemented a straight forward gradient descent to search in the 3d space (x,y, rotation) and optimize for the difference of black and gray pixels within the current pose of the rectangle. This approach is not very fast so I'd like to ask for alternatives.