it is all about valley detection in image processing. I would like to find, for a given pixel, direction for higher second order derivative. I am not quite sure what discrete mask/filter I can use to compute directional derivative, along xx, yx, xy, yy. What should I understand with 'direction of higher second order derivative'? is it one of the four xx, yx, xy, yy with maximum filter value? In the end, I am to found loci of extremal height for the signal, in direction along which second order derivative is of greatest magnitude. How to do so? Could you hint at relevant doc?
You are looking for the directions of the eigenvectors of the Hessian matrix (i.e., the matrix of second derivatives) that correspond to the largest and smallest eigenvalues. At a local minimum, any function can be developed into a Taylor series $$ f(x) \approx f(x_0) + \frac 12 (x-x_0)^T H (x-x_0) = m_2(x) $$ where $H=\nabla^2 f(x_0)$ and where $m_2(x)$ is this local approximation. The directions where $m_2$ curves up the quickest are the directions that correspond to the eigenvectors of the largest eigenvalue of $H$. The directions where $m_2$ grows the slowest (the "valley floor") correspond to the direction of the eigenvector with the smallest eigenvalue of $H$.