There are some differences, however they aren't necessarily in hardware or specs. Note that this is all information I have gained from forums or news releases, so take it all with a grain of salt.
The first is the "scalability and reliability" (source). The K20 was designed to sit in a cluster system and run at full tilt 24/7. The Titan is more designed for gaming, so it will run at this duty cycle, but it may suffer long term lifetime issues if used this way.
The drivers are also different, however I am not sure of the major differences. The difference in focus of the cards' design likely leads to relatively small performance gains for the Tesla cards on this front.
"Some Tesla-exclusive features include:
- NVIDIA GPUDirect RDMA for InfiniBand performance
- Hyper-Q for MPI (Hyper-Q for CUDA Streams is supported on GeForce GTX TITAN)
- ECC protection for all internal and external registers and memories
- Supported tools for GPU and cluster management, such as Bright Computing, Ganglia." (source)
This points to the fact that the main differnce is their scalability. If you are looking to run on a desktop in your office, it would be hard to argue against a Titan over the K20 for the price difference. If you need the extra performance of multiple K20's, find yourself a HPC center and buy time with their servers.
Edit:
After looking a bit more into ECC, I am updating this answer to point out the implications of having it on the K20 and not on the Titan. The following information is a paraphrase of info found here.
ECC is error checking on the DRAM and registers for the GPU. Soft errors are when a bit is incorrectly transfered/stored. The faster and closer together the circuits, the higher the probility of a soft error. If you are solving a set of coupled ODE's or solving a linear system, a single number being off by one bit could significantly change the results in a non-reproducable way. Most standard RAM and caches in the CPU are error checked for these errors using ECC.
GPU's on the other hand, do not, in general, have ECC even though their memory bus are much faster than those on the CPU. This is because if a pixel on the screen is off by a bit for one frame, the quality of the program is not diminished. These errors also don't propogate. Therefore a lot of chip real estate (and cost) can be saved by skipping this feature. This extra complexity likely causes a large portion of the extra cost of the Tesla line.