Most of the time, when I see someone reporting the average timing of a certain algorithm on a computer in a computational mathematics paper, they do something like this:
- Run the operation $n$ times (e.g., $n=100$ times);
- Discard the lowest and highest $m$ times (e.g., $m=10$ outliers);
- Compute the average of the remaining $n-2m$ times.
I can understand why dropping the highest results make sense: these are likely cases in which the computer switched threads into doing something else. I cannot understand why dropping the lowest results makes sense.
Why should I guard against an outlier being faster than the rest of the iterations? If anything, intuitively I would say that it is the minimum of the $n$ results that tells me how much time my computer really needs to run that algorithm. All other instances are cases when the computer switched into something else, or pipelines and multithreading were not used optimally.
The only case in which I can imagine this making sense is when the results reported are very close to the precision of the clock, but in this case I would say that the correct approach is repeatedly measuring the time for a sequence $k$ of runs at the same time (e.g., $k=100$) and then discarding outliers on those numbers. But in a case where outliers due to multitasking are the main concern, I do not see the advantages of this procedure.