Given the fundamental idea of the SPH interpolant with an arbitrary function $A$ (which could be the velocity $\textbf{v}$):
$A(r)=\int_{-\infty}^{\infty} A(r')\delta(r-r')dr'$,
the convolution of an arbitrary $A(r)$ with the Dirac delta $\delta$. Obviously, is not applicable in numerical simulations due to two reasons:
- $\delta$ has infinitesimally small radius, and infinite height
- The integral can be evaluated only analytically.
To solve both issues, first we change $\delta$ to a mollifier function (kernel) to achieve a finite (or sometimes infinitely large) influence radius:
$A(r)\approx\int_{\Omega} A(r')W(r-r',h)dr'$,
where $h$ is analogous to the standard deviation in case of Gaussian function and $\Omega$ is the domain of the kernel. $W(r-r',h)$ is usually a compact bell shaped function of unit volume. Secondly, change the integral to the summation form:
$\langle A(r_i)\rangle = \sum_{j}{A(r_j)\frac{m_j}{\rho_j}W(r_i-r_j,h)}$.
Now $r_i$ is corresponding to the $i^{th}$ discrete point often referred to as particle. This is can be considered as a discrete convolution.
Now apply the continuum convolution to the derivative of A:
$\nabla_{r} A(r)=\int_{\Omega} \nabla_{r'} A(r')W(r-r',h)dr'$,
where $\nabla_{r}$ denotes the derivative at $r$.
Using the rules of differentiation, it also can be written as:
$\nabla_{r} A(r)\approx\int_{\Omega} \nabla_{r'} \big[A(r')W(r-r',h)\big]dr'-\int_{\Omega}A(r')\nabla_{r'} W(r-r',h)dr'$.
Transform the first term in the RHS from volume integral to surface integral using the Gauss-Ostrogradsky theorem:
$\nabla A(r)\approx\int_{\partial\Omega}A(r')W(r-r',h)dS-\int_{\Omega}A(r')\nabla_{r'} W(r-r',h)dr'$.
Since $W(r-r',h)$ is compact and has a zero value at its boundary, the first term on RHS is now zero, so:
$\nabla A(r)\approx-\int_{\Omega}A(r')\nabla_{r'} W(r-r',h)dr'$,
where we can use the identity
$\nabla_{r'}W(r-r',h)=-\nabla_{r}W(r-r',h)$,
to obtain the discrete form
$\langle \nabla A_i\rangle = \sum_{j}{A_j\frac{m_j}{\rho_j}\nabla_i W(r_i-r_j,h)}$.
Unfortunately, this gives a poor estimation on the derivative, because it is extremely sensitive to the particle distribution. To overcome this problem, you can apply the discrete convolution to the formula you've shown:
$\langle\nabla A_i\rangle=\frac{\langle \nabla(\rho_i A_i)\rangle-A_i\langle\nabla \rho_i\rangle}{\rho_i} = \frac{1}{\rho_i}\sum_{j}{(\rho_jA_j)\frac{m_j}{\rho_j}\nabla_i W(r_i-r_j,h)}-\frac{1}{\rho_i}\sum_{j}{(\rho_jA_i)\frac{m_j}{\rho_j}\nabla_i W(r_i-r_j,h)}$,
which is finally:
$\langle\nabla A_i\rangle=\frac{1}{\rho_i}\sum_{j}{(A_j-A_i)m_j \nabla_i W(r_i-r_j,h)}$.
It is important, that this is only one of the several possible forms of differential operators in SPH. They have different consistency and conservation properties forming a trade-off. Which to use? It depends on the equation you want to solve.
Nevertheless, you are free to change the product to use the obtained operator for curl:
$\langle\nabla\times \textbf{v}_i\rangle=\frac{1}{\rho_i}\sum_{j}{(\textbf{v}_j-\textbf{v}_i)\times\nabla_i W(r_i-r_j,h)m_j}$