• bitcoinBitcoin(BTC)$58,938.00-4.88%
  • ethereumEthereum(ETH)$1,617.34-7.83%
  • tetherTether(USDT)$1.00-0.08%
  • binancecoinBNB(BNB)$539.20-7.12%
  • usd-coinUSDC(USDC)$1.00-0.04%
  • rippleXRP(XRP)$0.91-5.20%
  • solanaSolana(SOL)$69.18-5.33%
  • tronTRON(TRX)$0.301907-2.64%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.02-0.61%
  • HyperliquidHyperliquid(HYPE)$61.25-8.39%
  • zcashZcash(ZEC)$755.82-15.12%
  • moneroMonero(XMR)$454.96-8.27%
  • dogecoinDogecoin(DOGE)$0.054447-6.28%
  • leo-tokenLEO Token(LEO)$9.15-0.78%
  • whitebitWhiteBIT Coin(WBT)$56.38-6.21%
  • chainlinkChainlink(LINK)$8.58-4.52%
  • USDSUSDS(USDS)$1.000.03%
  • cardanoCardano(ADA)$0.143000-5.72%
  • Ethena USDeEthena USDe(USDE)$1.00-0.05%
  • stellarStellar(XLM)$0.127846-3.67%
  • daiDai(DAI)$1.000.01%
  • bitcoin-cashBitcoin Cash(BCH)$193.91-5.84%
  • nearNEAR Protocol(NEAR)$2.86-10.51%
  • litecoinLitecoin(LTC)$40.54-3.69%
  • CantonCanton(CC)$0.074996-6.52%
  • avalanche-2Avalanche(AVAX)$6.54-6.54%
  • USD1USD1(USD1)$1.00-0.08%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • tether-goldTether Gold(XAUT)$3,529.86-2.48%
  • the-open-networkGram (prev. Toncoin)(GRAM)$0.98-3.66%
  • uniswapUniswap(UNI)$4.46-9.31%
  • suiSui(SUI)$0.65-7.79%
  • RainRain(RAIN)$0.003719-14.81%
  • Global DollarGlobal Dollar(USDG)$1.000.00%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.05%
  • paypal-usdPayPal USD(PYUSD)$1.00-0.01%
  • hedera-hashgraphHedera(HBAR)$0.051555-4.48%
  • BittensorBittensor(TAO)$181.16-5.62%
  • shiba-inuShiba Inu(SHIB)$0.000004-5.41%
  • crypto-com-chainCronos(CRO)$0.039830-3.44%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.120.05%
  • quant-networkQuant(QNT)$139.92-6.64%
  • Circle USYCCircle USYC(USYC)$1.140.05%
  • MemeCoreMemeCore(M)$0.863.64%
  • Falcon USDFalcon USD(USDF)$1.00-0.09%
  • okbOKB(OKB)$78.87-8.13%
  • United StablesUnited Stables(U)$1.00-0.05%
  • pax-goldPAX Gold(PAXG)$3,531.19-2.38%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
  • aaveAave(AAVE)$97.88-6.50%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

This AI Paper from CMU and Apple Unveils WRAP: A Game-Changer for Pre-training Language Models with Synthetic Data

February 5, 2024
in AI & Technology
Reading Time: 4 mins read
A A
This AI Paper from CMU and Apple Unveils WRAP: A Game-Changer for Pre-training Language Models with Synthetic Data
ShareShareShareShareShare

Large Language Models (LLMs) have gathered a massive amount of attention and popularity among the Artificial Intelligence (AI) community in recent months. These models have demonstrated great capabilities in tasks including text summarization, question answering, code completion, content generation, etc. 

LLMs are frequently trained on inadequate web-scraped data. Most of the time, this data is loud, unstructured, and not necessarily expressed clearly. Following the existing scaling principles, which indicate that as the size of the model increases, computational power and data quantity should also increase proportionately, comes as a challenge.

There are two main limitations. Firstly, there is the significant computational cost and time involved in pre-training. Secondly, there is the impending problem of the scarcity of high-quality data available on the Internet. In recent research, a team of researchers from Apple and Carnegie Mellon University has addressed these issues by introducing the idea of Web Rephrase Augmented Pre-training (WRAP). 

WRAP is an innovative method that makes use of an already-existing, instruction-tuned LLM. This LLM is used to paraphrase online pages into particular styles, including mimicking the tone of Wikipedia or converting text into an answer-question format. The main goal of WRAP is to improve LLMs’ pre-training by adding both genuine and artificially rephrased data. 

The primary features of WRAP are as follows:

  1. Pre-training Efficiency: Applying WRAP to the noisy C4 dataset considerably speeds up pre-training, around three times faster. This effectiveness is critical in reducing the high expenses and time commitment usually related to LLM training.
  1. Enhancement of Model Performance: WRAP makes the model perform better when run within the same computational budget. Using different subsets of the Pile, a large-scale dataset used for training and assessing LLMs reduces ambiguity by more than 10%. It improves zero-shot question-answer accuracy by over 2% for 13 different activities.
  1. Rephrasing Web Documents: WRAP uses a medium-sized LLM to paraphrase documents from the web into several styles. This method is different from creating new data because it improves already-existing content while preserving the original information’s quality and diversity.

There are two main benefits to the synthetic data produced by WRAP. Firstly, it includes a range of styles that reflect the diversity of languages used in applications farther down the line. With this diversity, the LLM is better prepared for a wider variety of real-world events. Secondly, the synthetic data rephrased is of a higher quality than the raw web-scraped data. This quality enhancement results from language that is more ordered and cohesive, as this promotes more efficient model learning.

In conclusion, WRAP is a big advancement in the field of LLM pre-training. Through the use of superior-quality, different-style synthetic data, WRAP not only expedites the training process but also improves the overall performance of LLMs. Given the abundance of low-quality web data and the resource-intensive nature of classic LLM training approaches, this approach presents a possible way forward. 


Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel


YOU MAY ALSO LIKE

CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI

How To Stream GTA 6 On Your iPhone, iPad Or Mac

Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner.


🎯 [FREE AI WEBINAR] ‘Using ANN for Vector Search at Speed & Scale (Demo on AWS)’ (Feb 5, 2024)


Credit: Source link

ShareTweetSendSharePin

Related Posts

CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI
AI & Technology

CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI

November 20, 2026
How To Stream GTA 6 On Your iPhone, iPad Or Mac
AI & Technology

How To Stream GTA 6 On Your iPhone, iPad Or Mac

November 20, 2026
This GTA 6 Fast Travel Trick Comes At A Price
AI & Technology

This GTA 6 Fast Travel Trick Comes At A Price

November 20, 2026
RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Model Trained With Reinforcement Learning Only
AI & Technology

RadixArk Ships DaisyRose-27B, a 27B Agentic Coding Model Trained With Reinforcement Learning Only

November 20, 2026
Next Post
Wildfire burns on Spanish island amid European heatwave

Wildfire burns on Spanish island amid European heatwave

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Which stocks are worth holding into year-end? Victoria Fernandez goes rapid-fire on her top 5 picks.

Which stocks are worth holding into year-end? Victoria Fernandez goes rapid-fire on her top 5 picks.

November 18, 2026
Owlet, Inc. 2026 Q3 – Results – Earnings Call Presentation (NYSE:OWLT) 2026-11-13

Owlet, Inc. 2026 Q3 – Results – Earnings Call Presentation (NYSE:OWLT) 2026-11-13

November 14, 2026
Authorities warn about risk from chargers after fires

Authorities warn about risk from chargers after fires

November 14, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!