In the vast majority of cases, improvements in algorithms make a bigger difference than improvement in optimization. Algorithms are also more portable than low-level optimizations. My advice is to follow general best practices with respect to memory layout for cache reuse, avoiding excessive copies or communication, treating the file system in a sane way, and making floating point kernels have sufficient granularity for vectorization. Sometimes this is enough to reach an acceptably high fraction of "peak" (for this operation).
Always sketch a performance model for operations that you think are important (or that you discover are important by profiling). Then you can use the performance model to estimate what a highly tuned implementation could deliver. If you decide that speedup is worth it (relative to the other things you could be doing), then make the optimization.
Perhaps the hardest challenge is designing high-level, important (in the sense that a lot of code will depend on these choices) interfaces and data structures so that you can optimize later without needing to change the API. In contrast to specific optimizations and general guidelines, I don't know how to teach this except through experience. Working with performance-sensitive open source software helps. As with any API decision, it's important to understand the problem space.