Here is a slightly different view: Windows is not gone. On the contrary, it's making its way in the HPC again these days. Microsoft has done excellent work with their cluster tools in Windows 2008 HPC Server. Their MPI implementation (based on MPICH2) is as capable as its free sibling and the InfiniBand stack on Windows is quite performant. There is also a great choice of commercial software that is only available on Windows and is now gaining distributed computing capabilities.
That said, I would strongly disagree with anyone that can suggest "the best" HPC platform. There is no "one size fits all" platform and it depends on variety of factors. Most machines in Top 500 come with Linux or other Unix-like OS/kernel but that doesn't mean that you have to stick with Un*x for life. Big machines are all alike - learn to use MPI and one batch system and you can use them all (there is a caveat though - efficient usage of the hardware is highly platform dependent but there are some general rules).
For the real HPC user the platform is just a tool that executes his or her code in production mode. It's the development environment that counts. You can develop with MPI and/or OpenMP on any major OS, be it Windows or Mac, or Linux, or Solaris, or even some obscure OS and then run your code on the Linux supercomputer in your computing centre (or in the cloud) if you adhere to some language standards and don't use platform specific APIs. There are several people in our department that use extensively Visual Studio to develop code that runs on our enormous Linux cluser just because they don't find the development tools on Linux advanced or convenient enough. It's up to you to find the most convenient environment that suits your style and your work.
If you are building your own system then probably the price will be a serious factor and the choice of OS is almost predetermined to be Linux. It might also be your platform if your work depends on software available for Linux only. Besides, Intel provide their development tools for that OS for free if you are going to use them for personal stuff only. Linux also comes with its default GNU compiler collection and large selection of software and libraries for almost anything.
If you would like to run Windows software and you could afford the extra costs, then Windows 2008 HPC Server is a nice option. It integrates the HPC envrionment with ActiveDirectory and makes it really easy for the system to be centrally administered. Intel provides great tools for HPC development on Windows but they come at cost, even for private usage.
Since Intel development tools are available on both platforms and since they support the same variety of programming languages, it is quite easy to migrate code between the two OSes. There are also other vendors like Portland Group that provide multi-platform high performant compilers and scientific libraries.
As for the connectivity - in HPC nowadays it is either InfiniBand or multigigabit Ethernet (e.g. 10 GigE). Both provide low latency and high bandwidths which are a necessity for fast message passing. InfiniBand is faster but costs more.