Very few scientific software developers understand good principles of design, so I apologize if this answer is a bit long-winded. From a software engineering perspective, the goal of the scientific software developer is to design a solution that satisfies a set of constraints that are often conflicting.
Here are some typical examples of these constraints, as might be applied to the design of your sparse matrix library:
- Completed in one month
- Runs correctly on your laptop and several workstations
- Runs efficiently
Scientists are gradually paying more attention to some other common requirements from software engineering:
- Documentation (User guide, tutorial, code commenting)
- Maintainability (version control, testing, modular design)
- Reusability (modular design, "flexibility")
You might need more or less of one of these requirements. If you're trying to win a Gordon Bell prize for performance, then even fractions of a percent are relevant, and few of the judges will be evaluating the quality of your code (so long as you can convince them it's right). If you're trying to justify running this code on a shared resource such as a cluster or a supercomputer, frequently you have to defend claims about the performance of your code, but these are rarely very stringent. If you're trying to publish a paper in a journal describing the performance gains of your approach, then you need to legitimately be faster than your competitors, and 20% performance is a trade-off I would gladly make for better maintainability and reusability.
Coming back to your question, "good design", granted enough development time, should never sacrifice performance. If the objective is to make code that runs as fast as possible, then the code should be designed around those constraints. You might use techniques such as code generation, inline assembly, or take advantage of highly-tuned libraries to help you solve your problem.
But what if you don't have enough development time? What's good enough? Well, it depends, and nobody is going to be able to give you a good answer to this question without more context.
FWIW: If you really are interested in writing high-performance sparse matrix kernels, you should be comparing against an optimized PETSc install and working with their team if you are beating them, they'd be happy to incorporate tuned kernels to the library.