• bitcoinBitcoin(BTC)$63,744.00-1.71%
  • ethereumEthereum(ETH)$1,789.73-1.49%
  • tetherTether(USDT)$1.00-0.01%
  • binancecoinBNB(BNB)$587.74-0.41%
  • usd-coinUSDC(USDC)$1.000.00%
  • rippleXRP(XRP)$0.960.44%
  • solanaSolana(SOL)$72.591.96%
  • tronTRON(TRX)$0.3070420.35%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.02-3.60%
  • HyperliquidHyperliquid(HYPE)$67.970.66%
  • zcashZcash(ZEC)$923.293.53%
  • moneroMonero(XMR)$509.292.45%
  • dogecoinDogecoin(DOGE)$0.058793-0.89%
  • leo-tokenLEO Token(LEO)$9.301.41%
  • USDSUSDS(USDS)$1.000.01%
  • whitebitWhiteBIT Coin(WBT)$61.61-1.65%
  • chainlinkChainlink(LINK)$9.09-1.11%
  • cardanoCardano(ADA)$0.152312-0.68%
  • stellarStellar(XLM)$0.137667-0.01%
  • Ethena USDeEthena USDe(USDE)$1.00-0.01%
  • daiDai(DAI)$1.00-0.01%
  • bitcoin-cashBitcoin Cash(BCH)$207.34-0.30%
  • nearNEAR Protocol(NEAR)$3.170.29%
  • litecoinLitecoin(LTC)$44.33-1.17%
  • RainRain(RAIN)$0.004545-31.73%
  • CantonCanton(CC)$0.080080-2.69%
  • uniswapUniswap(UNI)$5.232.43%
  • avalanche-2Avalanche(AVAX)$7.07-1.16%
  • tether-goldTether Gold(XAUT)$3,839.20-0.26%
  • the-open-networkGram (prev. Toncoin)(GRAM)$1.050.72%
  • suiSui(SUI)$0.72-0.50%
  • USD1USD1(USD1)$1.000.03%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • Global DollarGlobal Dollar(USDG)$1.000.01%
  • hedera-hashgraphHedera(HBAR)$0.056341-2.05%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.15-0.05%
  • paypal-usdPayPal USD(PYUSD)$1.000.01%
  • crypto-com-chainCronos(CRO)$0.0435981.14%
  • shiba-inuShiba Inu(SHIB)$0.000004-1.39%
  • BittensorBittensor(TAO)$190.140.43%
  • quant-networkQuant(QNT)$151.86-2.69%
  • MemeCoreMemeCore(M)$0.884.52%
  • Circle USYCCircle USYC(USYC)$1.140.01%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.120.12%
  • okbOKB(OKB)$85.590.48%
  • pax-goldPAX Gold(PAXG)$3,843.30-0.19%
  • aaveAave(AAVE)$110.122.76%
  • Falcon USDFalcon USD(USDF)$1.00-0.04%
  • United StablesUnited Stables(U)$1.000.04%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Researchers from UC Berkeley and SJTU China Introduce the Concept of a ‘Rephrased Sample’ for Rethinking Benchmark and Contamination for Language Models

November 22, 2023
in AI & Technology
Reading Time: 4 mins read
A A
Researchers from UC Berkeley and SJTU China Introduce the Concept of a ‘Rephrased Sample’ for Rethinking Benchmark and Contamination for Language Models
ShareShareShareShareShare

Large language models are becoming increasingly complex, making evaluation more difficult. The community has produced many benchmarks in a relatively short amount of time, but benchmark scores do not always correspond to actual performance. Some evidence suggests that many popular benchmarks may have tainted datasets used for fine-tuning and pre-training.

Despite widespread agreement that it’s an important issue, pinpointing the source of pollution has been difficult. Both n-gram overlap and embedding similarity search are widely employed. String matching is used extensively by state-of-the-art innovations like GPT-4, PaLM, and Llama for N-gram overlap contamination detection. However, its precision is somewhat low. An embedding similarity search looks at the embeddings of previously trained models (like BERT) to discover related and maybe polluted cases. However, finding the sweet spot between recall and precision when deciding on a similarity level might be difficult. In addition, there is a developing trend in model training that uses synthetic data generated by LLMs (e.g., GPT-4), where contamination may be even more difficult to identify using string matching.

To examine decontamination methods, a new study by UC Berkeley and Shanghai Jiao Tong University introduces the concept of a “rephrased sample,” which has the same semantics as the original sample but is hard to identify by existing contamination tests. LLMs generate rephrased samples by translating and paraphrasing test samples into another language. The researchers demonstrate that if such paraphrased examples are utilized for training, the resulting model is highly susceptible to overfitting and can achieve extremely high performance on test benchmarks. A finely calibrated 13B Llama model can match GPT -4’s performance across all benchmarks while remaining unnoticed by n-gram overlap as contamination. This behavior is observed in widely used benchmarks like MMLU, GSM-8k, and HumanEval. As a result, the ability to identify rephrased samples is crucial. 

The researchers explain the flaws in conventional decontamination techniques and suggest a novel LLM-based approach. To determine if any top-k samples are too similar to the test instance, they first apply an embedding similarity search to find the most similar models to the test sample in question. The results demonstrate the superiority of their suggested LLM decontaminator over conventional techniques. They test their decontaminator on a variety of popular datasets that are used for fine-tuning and preliminary training. It’s also found that GPT-3.5’s synthetic dataset, CodeAlpaca, has a sizable amount of rephrased samples from HumanEval (12.8% to be exact). This hints at a potential for contamination during training with LLM-created fake data. 

The researchers advise the community to establish more thorough decontamination procedures for evaluating LLMs using public benchmarks. They hope to create new, one-time tests, like Codeforces and Kaggle competitions, for the fair evaluation of LLMs to overcome these fundamental issues.


Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 33k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..


YOU MAY ALSO LIKE

Cerebras Puts Disaggregated Inference Into Production With GPUs – Unite.AI

iCloud And Other Apple Services Are Down In An Apparent Outage

Dhanshree Shenwai is a Computer Science Engineer and has a good experience in FinTech companies covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is enthusiastic about exploring new technologies and advancements in today’s evolving world making everyone’s life easy.


🔥 Join The AI Startup Newsletter To Learn About Latest AI Startups

Credit: Source link

ShareTweetSendSharePin

Related Posts

Cerebras Puts Disaggregated Inference Into Production With GPUs – Unite.AI
AI & Technology

Cerebras Puts Disaggregated Inference Into Production With GPUs – Unite.AI

November 17, 2026
iCloud And Other Apple Services Are Down In An Apparent Outage
AI & Technology

iCloud And Other Apple Services Are Down In An Apparent Outage

November 17, 2026
You.com Launches Extract API for Document Retrieval and PDF Parsing – Unite.AI
AI & Technology

You.com Launches Extract API for Document Retrieval and PDF Parsing – Unite.AI

November 17, 2026
NVIDIA Establishes Export Control Committee Following Jensen Huang Resignation
AI & Technology

NVIDIA Establishes Export Control Committee Following Jensen Huang Resignation

November 17, 2026
Next Post
Gaza hit by multiple airstrikes as Israel intensifies attacks

Gaza hit by multiple airstrikes as Israel intensifies attacks

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Real vs. Nominal Interest Rates – Subtract Inflation Before You Save or Borrow

Real vs. Nominal Interest Rates – Subtract Inflation Before You Save or Borrow

November 13, 2026
Anthropic unveils its public IPO filing for first time

Anthropic unveils its public IPO filing for first time

November 16, 2026
Google stock’s historically bad day raises fears of AI bubble popping

Google stock’s historically bad day raises fears of AI bubble popping

November 14, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!