I have large arrays of data organized so that it can be processed efficiently using array processing libraries. However, there are times when I only need to process slices of the arrays where a slice might be an arbitrary, non-contiguous subset defined by a select set of index values.
result = someArrayProcess(objects) result = result(chosenSet)
might be replaced by
result = someArrayProcess(objects(chosenSet),t)
Is there a way I can get some confidence whether the second example is more or less efficient without actually running tests? What would I be looking for in the array packages that might be available to ascertain whether they handle this sort of slicing efficiently (without running baseline tests)? Up to this point, I have used array slicing as a convenient way to develop readable and manageable code, but now I want to take it to production, where efficiency will be an issue.
As I write this, I'm threading my way through some of the similar questions. I see that striding is a way to efficiently access non-contiguous array elements. I assume that some array packages would leverage methods like this and some wouldn't. What packages maximize the use of methods like this?
This would probably be coded in C++. I am interested in how array packages in this and other languages (including Python) treat slicing with arbitrary subsets.
I'm looking into making some architectural revisions that leverage array processing (and eventually moving it into GPU processing). The objective is to provide architectural simplification (for easier design management) using array expressions rather than element-level processing.
Since this would be a radical transition from the current approach, I will have to address concerns about efficiency trade-offs. The current solutions sacrifice exactness in performance to obtain efficiency, while being overwhelmingly difficult to manage due to their convoluted elements. They are now outdated, considering the advances that have been made in processing hardware since they were originally developed.
I prefer design simplification and performance accuracy over efficiency, but real-time requirements require that I consider efficiency, too.