I am new to computer science and I was wondering whether half precision is supported by modern architecture in the same way as single or double precision is. I thought the 2008 revision of IEEE-754 standard introduced both quadruple and half precisions.
Intel support for IEEE float16 storage format
Intel supports IEEE half as a storage type in processors since Ivy Bridge (2013). Storage type means you can get a memory/cache capacity/bandwidth advantage but the compute is done with single precision after converting to and from the IEEE half precision format.
Intel support for BFloat16
Intel has announced support for BF16 in Cooper Lake and Sapphire Rapids.
https://software.intel.com/content/dam/develop/public/us/en/documents/architecture-instruction-set-extensions-programming-reference.pdf (the June 2020 update 319433-040 describes AMX BF16)
I work for Intel. I’m citing official sources and will not comment on rumors etc.
It is good to be curious about the relative merits of IEEE FP16 vs BF16. There is a lot of analysis of this topic, e.g. https://nhigham.com/2018/12/03/half-precision-arithmetic-fp16-versus-bfloat16/.
Non-Intel Hardware Support
The following is information on other processors. Please verify with the vendors as necessary.
http://on-demand.gputechconf.com/gtc/2017/presentation/s7676-piotr-luszcek-half-precision-bencharking-for-hpc.pdf lists the following hardware support:
- AMD - MI5, MI8, MI25
- ARM - NEON VFP FP16 in V8.2-A
- NVIDIA - Pascal and Volta
NVIDIA Ampere has FP16 support as well (https://devblogs.nvidia.com/nvidia-ampere-architecture-in-depth/).
In my opinion, not very uniformly. Low precision arithmetic seems to have gained some traction in machine learning, but there's varying definitions for what people mean by low precision. There's the IEEE-754 half (10 bit mantissa, 5 bit exponent, 1 bit sign) but also bfloat16 (7 bit mantissa, 8 bit exponent, 1 bit sign) which favors dynamic range over precision, and a variety of other formats (NVidia's 19-bit TensorFloat, AMD's fp24, maybe more?). Most of this stuff is running on special purpose GPGPU-type hardware.
In contrast, float and double have generally agreed-upon meaning, as IEEE-754 compliant, 32-bit (23/8/1) and 64-bit (52/11/1) representations.
The accepted answer provides an overview. I'll add a few more details about support in NVIDIA processors. The support I'm describing here is 16 bit, IEEE 754 compliant, floating point arithmetic support, including add, multiply, multiply-add, and conversions to/from other formats.
Maxwell (circa 2015)
The earliest IEEE 754 FP16 ("binary16" or "half precision") support came in cc (compute capability) 5.3 devices which were in the Maxwell generation, but this compute capability was implemented only in the Tegra TX1 processor (SoC, e.g. Jetson).
Pascal (circa 2016)
Pascal family members have either "full rate" (cc 6.0, 6.2) or "low rate" (cc 6.1) FP16 throughput. cc6.2 was again a Tegra family product, TX2. cc 6.0 and 6.1 found use in a variety of processors in various product families such as GeForce, Quadro, and Tesla. "full rate" here refers to a rate that is equivalent to twice the IEEE 754 FP32 ("binary32" or "single precision") rate for the processor in question, when operations were done using a half2 data type (two half quantities handled in the same register and instruction).
Volta, Turing (2017, 2018)
Ampere (May, 2020)
The recently announced Ampere architecture A100 GPU also supports FP16 in a fashion similar to Volta and Turing, and introduces additional capability for a TF32 datatype, which is a format where the mantissa is the same size (number of bits) as a FP16 mantissa, and the exponent is the same size as a FP32 exponent. Bfloat16 capability was also announced in Ampere.
Apart from the Ampere architecture processor recently announced, support and throughputs for 16-bit floating point operations (and other operations) across compute capabilities (including architectures) can be found in table 3 of the CUDA programming guide. The throughputs are per clock, per multiprocessor, so need to be scaled accordingly for the GPU type and specifications. These throughputs are not for TensorCore operations, and the peak throughputs are generally only applicable when processing on half2 datatypes (two half quantities packed together in a single 32-bit word).
You can find out if your hardware supports half-precision via:
$ lscpu | grep Flags | grep f16c Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d
f16c instruction is documented here.