Summary: Is there an efficient out-of-place GPU tensor transpose operation that scales as $O(n)$ for tensors with $n$ total elements, regardless of the rank $d$? The naive algorithm costs $O(dn)$, since it requires $O(d)$ worth of index manipulation per entry.
Here is the current implementation of high rank tensor transpose in TensorFlow:
It does a flat 1-D loop, deconstructs the index using $O(d)$ divisions by strides, and reconstructs the transposed index. Is there any index trickery / precomputation that would avoid the $O(d)$ cost per entry?