New standard to make AI processing more efficient

In pursuit of faster and more efficient AI system development, Intel, Arm and Nvidia today published a draft specification for what they refer to as a common interchange format for AI. While voluntary, the proposed “8-bit floating point (FP8)” standard, they say, has the potential to accelerate AI development by optimizing hardware memory usage and work for both AI training (i.e., engineering AI systems) and inference (running the systems).

When developing an AI system, data scientists are faced with key engineering choices beyond simply collecting data to train the system. One is selecting a format to represent the weights of the system — weights being the factors learned from the training data that influence the system’s predictions. Weights are what enable a system like GPT-3 to generate whole paragraphs from a sentence-long prompt, for example, or DALL-E 2 to create photorealistic portraits from a caption.

Common formats include half-precision floating point, or FP16, which uses 16 bits to represent the weights of the system, and single precision (FP32), which uses 32 bits. Half-precision and lower reduce the amount of memory required to train and run an AI system while speeding up computations and even reducing bandwidth and power usage. But they sacrifice some accuracy to achieve those gains; after all, 16 bits is less to work with than 32.