Ask yourself the following:
First, how does the integration by parts affect the solvability of the problem, and the space of solutions?
Second, for which space of functions can you build a series of subspaces (the ansatz functions) that you can implement?
Let us regard the Poisson problem $u'' = f$ for $f \in L^2$, say, on $[0,1]$, with homogenous Dirichlet boundary conditions. By integration, the left and the right side of the equation can be regarded as bounded functionals on $L^2$, say for $\phi \in L^2$ we have
$\phi \mapsto \int u'' \phi dx$ and $\phi \mapsto \int f \phi dx$
Since any function in $L^2$ can be $L^2$-approximated by smooth functions with compact support, both integral functionals are completely known if you only know the values for all test functions. But with the test functions, you can perform integration by parts, and transform the left-hand side to the functional
$\phi \mapsto -\int u' \phi' dx$
Read this as: "I take a test function $\phi$, compute its differential, and integrate it with -u' over [0,1], and return you the result." But that functional is not defined and bounded on $L^2$, since you cannot take the differential of an arbitrary $L^2$ function. They may look extremely strange in general.
Still we observe, that this functional can be extended to the Sobolev space $H^1$, and it is even a bounded functional on $H_0^1$. That means, given $\phi \in H_0^1$, you can roughly estimate the value of $\int -u' \phi' dx$ by a multiple of the $H_0^1$-norm of $\phi'$. And, furthermore, the functional $\phi \mapsto \int f \phi dx$ is, of course, not only defined and bounded on $L^2$, but also defined and bounded on $H_0^1$.
Now you can, e.g., apply the Lax-Milgram lemma, as it is presented in any PDE-book. A finite element book which describes it as well, only with functional analysis, is e.g. the classic by Ciarlet, or the rather new book by Braess.
The Lax-Milgram lemma gives PDE-people a nice tool for pure analysis, but they employ much stranger tools as well for their purpose. Still, these tools are also relevant for numerical analysits, because you can in fact build a discretization for these spaces.
For example, in order to have discrete subspace of $H_0^1$, just take the hat functions. They do not have jumps and are piecewise differentiable. Their differential is a piecewise constant vector field. This construction works in $d=1,2,3,...$, which is fine, but can you come up with an ansatz space whose functions not only have a gradient (that is nice,i.e., square-integrable), but also whose gradients have in turn a divergence? (again, square-integrable). That is pretty hard in general.
So the reason in general how you build weak formulations is that you want to apply the Lax-Milgram lemma, and to have a formulation such that the functions can in fact be implemented. ( For the record, neither is Lax-Milgram the last word in that context, nor are $H_0^1$ ansatz spaces the last word in discretization, see, e.g., Discontinuous Galerkin methods. )
For the case of mixed boundary conditions, the natural test space may differ from your your search space (in the analytic setting), but I have no idea how to describe that without referring to distribution theory, so I stop here. I hope this is helpful.
