How to choose the cutoff radius so that the atoms do not interact with its periodic image? Especially when simulating macromolecules (proteins).
The question as stated does make sense in the context of molecular dynamics simulations but is very broad.
I recommend that you should check the manual of the MD package your are using about the image convention used. I will cite here for the cut off radius from the GROMACS manual, which is a very mature and well maintained open source MD package:
[...] GROMACS always uses a cut-off radius for the Lennard-Jones interactions and sometimes for the Coulomb interactions as well. The “minimum-image convention” used by GROMACS requires that only one image of each particle in the periodic boundary conditions is considered for a pair interaction, so the cut-off radius cannot exceed half the box size. That is still pretty big for large systems, and trouble is only expected for systems containing charged particles. But then truly bad things can happen, like accumulation of charges at the cut-off boundary or very wrong energies! For such systems, you should consider using one of the implemented long-range electrostatic algorithms, such as particle-mesh Ewald.[...]
Also, there is nothing special about large molecules such as proteins. So picking half the shortest box vector is fine. However performance wise this is not optimal, since a smaller cutoff lead to less pairwise interaction potential calculations. So the trick is to choose the cutoff such that the error you introduce is small while you calculate the minimum number of pairwise interactions.
There are a lot more details about cutoffs and minimum image convention in the manual, which is likely an overkill to cite here.