• bitcoinBitcoin(BTC)$65,447.000.43%
  • ethereumEthereum(ETH)$1,834.620.64%
  • tetherTether(USDT)$1.00-0.02%
  • binancecoinBNB(BNB)$594.620.04%
  • usd-coinUSDC(USDC)$1.000.00%
  • rippleXRP(XRP)$0.96-0.38%
  • solanaSolana(SOL)$72.12-0.04%
  • tronTRON(TRX)$0.3054280.12%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.050.00%
  • HyperliquidHyperliquid(HYPE)$70.684.32%
  • zcashZcash(ZEC)$912.890.76%
  • moneroMonero(XMR)$501.591.41%
  • dogecoinDogecoin(DOGE)$0.0595780.79%
  • USDSUSDS(USDS)$1.000.02%
  • leo-tokenLEO Token(LEO)$9.16-0.83%
  • whitebitWhiteBIT Coin(WBT)$63.230.31%
  • chainlinkChainlink(LINK)$9.20-0.78%
  • cardanoCardano(ADA)$0.153981-0.11%
  • stellarStellar(XLM)$0.138153-1.58%
  • Ethena USDeEthena USDe(USDE)$1.00-0.01%
  • daiDai(DAI)$1.00-0.03%
  • RainRain(RAIN)$0.00607975.19%
  • bitcoin-cashBitcoin Cash(BCH)$210.240.48%
  • nearNEAR Protocol(NEAR)$3.200.20%
  • litecoinLitecoin(LTC)$44.90-1.07%
  • CantonCanton(CC)$0.083110-0.37%
  • avalanche-2Avalanche(AVAX)$7.20-1.34%
  • uniswapUniswap(UNI)$5.190.41%
  • tether-goldTether Gold(XAUT)$3,843.190.15%
  • the-open-networkGram (prev. Toncoin)(GRAM)$1.05-0.41%
  • suiSui(SUI)$0.73-0.93%
  • USD1USD1(USD1)$1.000.00%
  • Ripple USDRipple USD(RLUSD)$1.000.00%
  • Global DollarGlobal Dollar(USDG)$1.000.01%
  • hedera-hashgraphHedera(HBAR)$0.057800-2.28%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.00%
  • paypal-usdPayPal USD(PYUSD)$1.000.02%
  • shiba-inuShiba Inu(SHIB)$0.000004-1.35%
  • quant-networkQuant(QNT)$155.24-0.02%
  • crypto-com-chainCronos(CRO)$0.0434470.77%
  • BittensorBittensor(TAO)$192.31-0.59%
  • Circle USYCCircle USYC(USYC)$1.140.00%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.12-0.25%
  • MemeCoreMemeCore(M)$0.851.37%
  • okbOKB(OKB)$85.97-2.27%
  • pax-goldPAX Gold(PAXG)$3,845.190.10%
  • aaveAave(AAVE)$108.46-0.69%
  • pepePepe(PEPE)$0.0000041.16%
  • Falcon USDFalcon USD(USDF)$1.00-0.01%
  • United StablesUnited Stables(U)$1.00-0.02%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Yandex Introduces YaFSDP: An Open-Source AI Tool that Promises to Revolutionize LLM Training by Cutting GPU Usage by 20%

June 14, 2024
in AI & Technology
Reading Time: 6 mins read
A A
Yandex Introduces YaFSDP: An Open-Source AI Tool that Promises to Revolutionize LLM Training by Cutting GPU Usage by 20%
ShareShareShareShareShare

Developing large language models requires substantial investments in time and GPU resources, translating directly into high costs. The larger the model, the more pronounced these challenges become. 

Recently, Yandex has introduced a new solution: YaFSDP, an open-source tool that promises to revolutionize LLM training by significantly reducing GPU resource consumption and training time. In a pre-training scenario involving a model with 70 billion parameters, using YaFSDP can save the resources of approximately 150 GPUs. This translates to potential monthly savings of roughly $0.5 to $1.5 million, depending on the virtual GPU provider or platform.

Yandex has made YaFSDP publicly available on GitHub. ML engineers can leverage this tool to enhance the efficiency of their LLM training processes. By open-sourcing YaFSDP, Yandex aims to foster innovation and collaboration in the AI community, enabling developers to train models faster and cost-effectively.

The Challenge of Distributed LLM Training

Training LLMs across multiple GPUs involves complex operations that lead to inefficiencies and high memory consumption. One of the main issues is the need to send and receive massive amounts of data between GPUs. For instance, in a typical all_reduce operation, twice the amount of gradient data as there are network parameters must be communicated. In the case of a Llama 70B model, this means transferring 280 GB of data per iteration.

Furthermore, weights, gradients, and optimizer states are duplicated across GPUs, leading to an enormous memory load. The Llama 70B model and the Adam optimizer require over 1 TB of memory, far exceeding the typical 80 GB memory capacity of most GPUs. This redundancy severely slows down the training process and often makes it impractical to fit even moderately sized models into GPU memory.

Introducing YaFSDP

Yandex’s YaFSDP offers a highly effective solution to these challenges. By focusing on optimizing memory consumption and eliminating communication bottlenecks, YaFSDP enhances the efficiency of LLM training. It works by sharding layers instead of individual parameters, maintaining efficient communications and avoiding redundant operations. Additionally, YaFSDP pre-allocates buffers for all required data, ensuring that the Torch allocator does not introduce inefficiencies.

YaFSDP operates by utilizing two buffers for intermediate weights and gradients, with odd layers using one buffer and even layers using the other. 

The weights from different layers are stored in the same memory. If the layers have the same structure, they will always be identical. It is crucial to ensure that when you need layer X, the buffer contains the weights for layer X. All parameters will be stored in the corresponding memory chunk within the buffer.

Memory Consumption

During training, the primary memory consumers are weights, gradients, optimizer states, buffers, and activations. YaFSDP significantly reduces memory consumption by optimizing how these elements are stored and accessed.

  • Weights, Gradients, and Optimizer States: These depend on the number of processes, and their memory consumption tends to approach zero as the number of processes increases. By sharding these components across GPUs, YaFSDP minimizes duplication and thus reduces memory usage.
  • Buffers consume a constant amount of memory and store intermediate values during computations.
  • Activations depend on the model size and the number of tokens processed per GPU.

Activation Checkpointing

Activation checkpointing is a technique that stores only necessary activations during the forward pass and recomputes them during the backward pass. This reduces the memory footprint significantly, as only essential data is stored. For example, in training a Llama 2 70B model with a batch size of 8192 tokens, activation storage can be reduced from over 110 GB to just 5 GB. However, this approach introduces additional computational overhead, which YaFSDP allows to avoid by not using the activation checkpointing for some layers which is possible due to memory optimization.

Communication Optimization

YaFSDP improves GPU communication efficiency by ensuring that data is transferred only when necessary and by overlapping communication with computation. It utilizes CUDA streams to manage concurrent computations and communications effectively. 

The tool uses two streams: a computation stream and a communication stream. Events synchronize these streams, ensuring that operations are executed in the correct order without introducing deadlocks.

The forward pass on the third layer doesn’t start until the all_gather operation is completed (condition 1). Likewise, the all_gather operation on the third layer won’t begin until the forward pass on the first layer that uses the same buffer is completed (condition 2). Since there are no cycles in this scheme, deadlock is impossible.

Experimental Results and Performance Gains

The implementation of YaFSDP has shown remarkable improvements in training efficiency. In a pre-training scenario with a model having 70 billion parameters, YaFSDP was able to save the resources of approximately 150 GPUs. This translates into significant monthly cost savings, ranging from $0.5 to $1.5 million, depending on the virtual GPU provider or platform.

YaFSDP reduces training time by up to 26% compared to existing methods like FSDP and optimizes memory usage, making it possible to train larger models more efficiently. 

YaFSDP represents a significant advancement in LLM training. Addressing the critical challenges of memory consumption and communication inefficiencies enables faster and more efficient training of large language models. 


Check out the GitHub Page. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 44k+ ML SubReddit


YOU MAY ALSO LIKE

Cerebras Releases Tungsten for High-Level Wafer-Scale Programming – Unite.AI

California Sues OpenAI After That Hugging Face Rogue Hacking Incident

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.


🐝 Join the Fastest Growing AI Research Newsletter Read by Researchers from Google + NVIDIA + Meta + Stanford + MIT + Microsoft and many others…


Credit: Source link

ShareTweetSendSharePin

Related Posts

Cerebras Releases Tungsten for High-Level Wafer-Scale Programming – Unite.AI
AI & Technology

Cerebras Releases Tungsten for High-Level Wafer-Scale Programming – Unite.AI

November 16, 2026
California Sues OpenAI After That Hugging Face Rogue Hacking Incident
AI & Technology

California Sues OpenAI After That Hugging Face Rogue Hacking Incident

November 16, 2026
League Launches Healthcare AI Agents With Safety Controls – Unite.AI
AI & Technology

League Launches Healthcare AI Agents With Safety Controls – Unite.AI

November 16, 2026
You Won’t Find Nothing Phones At US Carriers, But Here’s How You Can Still Buy Them
AI & Technology

You Won’t Find Nothing Phones At US Carriers, But Here’s How You Can Still Buy Them

November 16, 2026
Next Post
Watch: Bull on the loose! Animal runs on N.J. Transit tracks

Watch: Bull on the loose! Animal runs on N.J. Transit tracks

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Bessent Gives Nvidia 10 Days to Address Export-Control Questions – Unite.AI

Bessent Gives Nvidia 10 Days to Address Export-Control Questions – Unite.AI

November 15, 2026
Miami Heat unveil updated Dwyane Wade statue

Miami Heat unveil updated Dwyane Wade statue

November 13, 2026
RUM Group Inc. (RUM) Q3 2026 Earnings Call Transcript

RUM Group Inc. (RUM) Q3 2026 Earnings Call Transcript

November 11, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!