If $A$ is full-rank, there is only one solution to $Ax=b$, and it is either integer or not.

If $A$ is not full-rank, this problem becomes starts to look like a search for least common multiples in the null space or integer programming.

This isn't something that a general numerical linear algebra solver like those in Octave will be able to do on its own. If your problem is large, you could try entering this as a feasibility problem into an integer programming solver like lpsolve or COIN-CBC. If $A$ is not so big or nearly full-rank, then you might try an eigenvalue decomposition and searching in the null space for the nearest set of integers.

If $A$ is full-rank, there is only one solution to $Ax=b$, and it is either integer or not.

If $A$ is not full-rank, this problem starts to look like a search for least common multiples in the null space or integer programming.

This isn't something that a general numerical linear algebra solver like those in Octave will be able to do on its own. If your problem is large, you could try entering this as a feasibility problem into an integer programming solver like lpsolve or COIN-CBC. If $A$ is not so big or nearly full-rank, then you might try an eigenvalue decomposition and searching in the null space for the nearest set of integers.