Home Technology peripherals AI Microsoft releases new mixed precision training framework FP8: 64% faster than BF16 and 42% less memory usage

Microsoft releases new mixed precision training framework FP8: 64% faster than BF16 and 42% less memory usage

Nov 10, 2023 pm 02:17 PM
ai Microsoft

微软发布新的混合精度训练框架 FP8:速度超过 BF16 64%,内存占用减少 42%

November 10 news, large language models (LLM) are rising rapidly, showing bright prospects in language generation and understanding, and their influence goes beyond the language field and extends to logic and mathematics. , physics and other fields.

微软发布新的混合精度训练框架 FP8:速度超过 BF16 64%,内存占用减少 42%

However, if you want to unlock these "extraordinary energy", you need to pay a high price. For example, training a 540B model requires 6144 TPUv4 chips of Project PaLM; and training a 175B model GPT-3, requires thousands of Petaflop/s-day.

A good solution is to train with low precision, which can increase processing speed and reduce memory usage and communication costs. The current mainstream training systems include Megatron-LM, MetaSeq and Colossal-AI, which use FP16/BF16 mixed precision or FP32 full precision by default to train large language models

Although these accuracy levels are necessary for large language models Indispensable, but they are computationally expensive.

If you use FP8 low precision, you can increase speed by 2 times, reduce memory costs by 50% to 75%, and save communication costs.

Currently only the Nvidia Transformer Engine is compatible with the FP8 framework, primarily leveraging this precision for GEMM (general matrix multiplication) calculations while maintaining master weights and gradients at FP16 or FP32 high precision.

To address this challenge, a team of researchers from Microsoft Azure and Microsoft Research introduced an efficient FP8 mixed-precision framework tailored for large language model training.

微软发布新的混合精度训练框架 FP8:速度超过 BF16 64%,内存占用减少 42%

Microsoft has introduced three optimization stages to leverage FP8 for distributed and mixed precision training. As these levels progress, increased integration of FP8 becomes evident, suggesting a greater impact on the LLM training process.

In addition, in order to overcome problems such as data overflow or underflow, Microsoft researchers proposed two key methods: automatic sampling and precise decoupling. The former involves components that are not sensitive to accuracy, reducing accuracy, and dynamically adjusting the Tensor sampling factor to Make sure gradient values ​​stay within the FP8 representation range. This prevents underflow and overflow events during communication, ensuring a smoother training process.

Microsoft has tested memory usage reduced by 27% to 42% and weight gradient communication overhead significantly reduced by 63% to 65% compared to the widely adopted BF16 mixed precision method. Runs 64% faster than widely adopted BF16 frameworks such as Megatron-LM and 17% faster than the Nvidia Transformer Engine.

微软发布新的混合精度训练框架 FP8:速度超过 BF16 64%,内存占用减少 42%

When training the GPT-175B model, the hybrid FP8 precision framework saves 21% of memory on the H100 GPU platform, and compared to TE (Transformer Engine), the training time is reduced 17%.

This site attaches the GitHub address and paper address: https://www.php.cn/link/7b3564b05f78b6739d06a2ea3187f5ca

The above is the detailed content of Microsoft releases new mixed precision training framework FP8: 64% faster than BF16 and 42% less memory usage. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

What are the top ten virtual currency trading apps? The latest digital currency exchange rankings What are the top ten virtual currency trading apps? The latest digital currency exchange rankings Apr 28, 2025 pm 08:03 PM

The top ten digital currency exchanges such as Binance, OKX, gate.io have improved their systems, efficient diversified transactions and strict security measures.

Bitcoin price today Bitcoin price today Apr 28, 2025 pm 07:39 PM

Bitcoin’s price fluctuations today are affected by many factors such as macroeconomics, policies, and market sentiment. Investors need to pay attention to technical and fundamental analysis to make informed decisions.

Which of the top ten currency trading platforms in the world are the latest version of the top ten currency trading platforms Which of the top ten currency trading platforms in the world are the latest version of the top ten currency trading platforms Apr 28, 2025 pm 08:09 PM

The top ten cryptocurrency trading platforms in the world include Binance, OKX, Gate.io, Coinbase, Kraken, Huobi Global, Bitfinex, Bittrex, KuCoin and Poloniex, all of which provide a variety of trading methods and powerful security measures.

How to use the chrono library in C? How to use the chrono library in C? Apr 28, 2025 pm 10:18 PM

Using the chrono library in C can allow you to control time and time intervals more accurately. Let's explore the charm of this library. C's chrono library is part of the standard library, which provides a modern way to deal with time and time intervals. For programmers who have suffered from time.h and ctime, chrono is undoubtedly a boon. It not only improves the readability and maintainability of the code, but also provides higher accuracy and flexibility. Let's start with the basics. The chrono library mainly includes the following key components: std::chrono::system_clock: represents the system clock, used to obtain the current time. std::chron

Recommended reliable digital currency trading platforms. Top 10 digital currency exchanges in the world. 2025 Recommended reliable digital currency trading platforms. Top 10 digital currency exchanges in the world. 2025 Apr 28, 2025 pm 04:30 PM

Recommended reliable digital currency trading platforms: 1. OKX, 2. Binance, 3. Coinbase, 4. Kraken, 5. Huobi, 6. KuCoin, 7. Bitfinex, 8. Gemini, 9. Bitstamp, 10. Poloniex, these platforms are known for their security, user experience and diverse functions, suitable for users at different levels of digital currency transactions

Decryption Gate.io Strategy Upgrade: How to Redefine Crypto Asset Management in MeMebox 2.0? Decryption Gate.io Strategy Upgrade: How to Redefine Crypto Asset Management in MeMebox 2.0? Apr 28, 2025 pm 03:33 PM

MeMebox 2.0 redefines crypto asset management through innovative architecture and performance breakthroughs. 1) It solves three major pain points: asset silos, income decay and paradox of security and convenience. 2) Through intelligent asset hubs, dynamic risk management and return enhancement engines, cross-chain transfer speed, average yield rate and security incident response speed are improved. 3) Provide users with asset visualization, policy automation and governance integration, realizing user value reconstruction. 4) Through ecological collaboration and compliance innovation, the overall effectiveness of the platform has been enhanced. 5) In the future, smart contract insurance pools, forecast market integration and AI-driven asset allocation will be launched to continue to lead the development of the industry.

How much is Bitcoin worth How much is Bitcoin worth Apr 28, 2025 pm 07:42 PM

Bitcoin’s price ranges from $20,000 to $30,000. 1. Bitcoin’s price has fluctuated dramatically since 2009, reaching nearly $20,000 in 2017 and nearly $60,000 in 2021. 2. Prices are affected by factors such as market demand, supply, and macroeconomic environment. 3. Get real-time prices through exchanges, mobile apps and websites. 4. Bitcoin price is highly volatile, driven by market sentiment and external factors. 5. It has a certain relationship with traditional financial markets and is affected by global stock markets, the strength of the US dollar, etc. 6. The long-term trend is bullish, but risks need to be assessed with caution.

Which of the top ten currency trading platforms in the world are among the top ten currency trading platforms in 2025 Which of the top ten currency trading platforms in the world are among the top ten currency trading platforms in 2025 Apr 28, 2025 pm 08:12 PM

The top ten cryptocurrency exchanges in the world in 2025 include Binance, OKX, Gate.io, Coinbase, Kraken, Huobi, Bitfinex, KuCoin, Bittrex and Poloniex, all of which are known for their high trading volume and security.

See all articles