Depending on how high order you go you run into a lot of frustrating difficulties. The derivatives of order $N$ polynomials grow as $N^2$ (see e.g. Markov's inequality to make this more precise), and these large gradients (mostly near boundaries of your elements) often lead to spuriously large eigenvalues in discretizations, making you have to resolve very fine scale oscillations that probably aren't physically meaningful to begin with. So efficient implementation in the high order regime isn't as straightforward, despite the higher formal order of convergence.
Also, simply calculating the discrete derivative operator is difficult. A lot of literature from the 80s and early 90s put considerable effort into stable calculations of high order differencing matrices. The operator can be highly non-normal, leading to a lot of numerical garbage that is hard to predict.
I think a lot of these issues have been well addressed by now, but that just means doing a bit more work to make high order worthwhile.