I known that this is not a definitive answer, but it can give an idea how to move with CUDA. At this stage is difficult to give advice because for this is necessary a detailed know about the actually code
As already write in comments, in general, is better not invert the matrix, but solve the linear system (for detail about why see this question).
There is the possibility that CUDA can help you , but before you should consider some aspects.
In general the real bottleneck in a CUDA application are the memory transfer, sometimes is better to recalculate things respect than move them, you must consider the memory available and in other case the memory transfer use more time respect the use of cpu.
Another point to focus is the presence of
if condition along the flow you like to parallelize, in this case the flows is stopped and the two different branches are execute in sequence with a big degradation of the performance.
What precision do I need? This is important for the choose of the hardware, and can depend also by the algorithm that you use.
With this in mind you can start study what parts of your algorithm have got the best advantage from a parallelization. For example is better assign to each CUDA Kernel a linear system or is better parallelize only some task?
Maybe an idea, not really radical respect your code, is to parallelize the solution of the linear system (= invert the matrix). Another, if you use feval, is try to use the parallel feval in matlab.
Other possibility can be to consider variants of Newton's method to obtain speed up, this is not only related with the use of GPU.
- CUDA library for solver in cuda.
- pdf with an example in CUDA for a particular case of Netwon's method
- Article pdf with a variant of Newton's method with CUDA. Parallel interval newton method on CUDA by Philip-Daniel Beck, Marco Nehmeier