fp16

FP16 is a half-precision floating-point format that uses 16 bits (10-bit mantissa, 5-bit exponent) compared to 32-bit FP32. It halves memory usage and increases throughput on GPUs with tensor core support, but has a narrower dynamic range that can cause overflow or underflow during training.

Mixed precision training typically keeps a master copy of weights in FP32 while performing forward and backward passes in FP16.