The minimum image convention (MIC), see for example a short note of W. Smith, is often used in molecular dynamics or monte carlo simulations of periodic systems with an orthorhombic unit cell. For this special case, it is rather trivial to implement the MIC correctly. How does one apply the MIC to systems with a more general triclinic unit cell? The naive idea to simply generalize the algorithm for orthorhomic cells to triclinic cells, does not seem to work.
I could find this document, which provides some clues, such as the importance of the reduced basis of the lattice vectors, but it does not contain enough details to make a computer implementation.
I will implement the answers below in a test code to check which algorithms work (under given assumptions). The Python test program can be found here: https://gist.github.com/3566972. It tests the lammps implementation and one naive attempt of mine. Both fail. If one finds a bug in this test program, please get in touch.
P.S. For those who get worried about MD or MC codes that use a naive algorithm for triclinic cells: please read the note written by W. Smith. It explains a naive algorithm that works as long as the distance cutoff is shorter than half the shortest distance between the opposite faces of the unit cell. For a given cutoff, one may always construct a triclinic supercell for which naive algorithms will work fine.