• bitcoinBitcoin(BTC)$62,059.00-0.61%
  • ethereumEthereum(ETH)$1,755.92-0.57%
  • tetherTether(USDT)$1.000.02%
  • binancecoinBNB(BNB)$580.220.30%
  • usd-coinUSDC(USDC)$1.000.01%
  • rippleXRP(XRP)$0.96-0.19%
  • solanaSolana(SOL)$73.140.18%
  • tronTRON(TRX)$0.3103380.53%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.031.42%
  • HyperliquidHyperliquid(HYPE)$66.970.87%
  • zcashZcash(ZEC)$889.96-1.80%
  • moneroMonero(XMR)$492.03-1.97%
  • dogecoinDogecoin(DOGE)$0.058163-0.35%
  • leo-tokenLEO Token(LEO)$9.22-0.57%
  • whitebitWhiteBIT Coin(WBT)$60.19-0.61%
  • chainlinkChainlink(LINK)$8.98-0.41%
  • USDSUSDS(USDS)$1.00-0.01%
  • cardanoCardano(ADA)$0.1514871.01%
  • Ethena USDeEthena USDe(USDE)$1.000.02%
  • stellarStellar(XLM)$0.132595-3.07%
  • daiDai(DAI)$1.000.03%
  • nearNEAR Protocol(NEAR)$3.19-1.98%
  • bitcoin-cashBitcoin Cash(BCH)$205.79-0.29%
  • litecoinLitecoin(LTC)$42.23-3.45%
  • CantonCanton(CC)$0.080276-1.45%
  • avalanche-2Avalanche(AVAX)$6.980.49%
  • RainRain(RAIN)$0.004366-2.62%
  • uniswapUniswap(UNI)$4.94-3.87%
  • tether-goldTether Gold(XAUT)$3,624.30-2.62%
  • suiSui(SUI)$0.71-0.89%
  • the-open-networkGram (prev. Toncoin)(GRAM)$1.020.59%
  • USD1USD1(USD1)$1.000.05%
  • Ripple USDRipple USD(RLUSD)$1.00-0.02%
  • Global DollarGlobal Dollar(USDG)$1.000.00%
  • hedera-hashgraphHedera(HBAR)$0.054153-2.36%
  • paypal-usdPayPal USD(PYUSD)$1.000.02%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.15-0.06%
  • BittensorBittensor(TAO)$191.780.81%
  • shiba-inuShiba Inu(SHIB)$0.0000040.42%
  • quant-networkQuant(QNT)$150.03-0.04%
  • crypto-com-chainCronos(CRO)$0.041218-3.47%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.12-0.06%
  • Circle USYCCircle USYC(USYC)$1.140.01%
  • MemeCoreMemeCore(M)$0.83-5.59%
  • okbOKB(OKB)$85.930.09%
  • Falcon USDFalcon USD(USDF)$1.000.04%
  • United StablesUnited Stables(U)$1.000.02%
  • pax-goldPAX Gold(PAXG)$3,621.60-2.66%
  • aaveAave(AAVE)$104.89-2.81%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

This AI Research Evaluates the Correctness and Faithfulness of Instruction-Following Models For Their Ability To Perform Question-Answering

August 5, 2023
in AI & Technology
Reading Time: 4 mins read
A A
This AI Research Evaluates the Correctness and Faithfulness of Instruction-Following Models For Their Ability To Perform Question-Answering
ShareShareShareShareShare

Recently introduced Large Language Models (LLMs) have taken the Artificial Intelligence (AI) community by storm. These models have been able to successfully imitate human beings by using super-good Natural Language Processing (NLP), Natural Language Generation (NLG) and Natural Language Understanding (NLU). LLMs have become famous for imitating humans for having realistic conversations and are capable of answering simple and complex questions, content generation, code completion, machine translation, and text summarization. The goal of NLP is to make it possible for computer systems to comprehend and react to commands given in natural language, enabling people to engage with them in a more natural and flexible way, the best example of which is the instruction following models.

These models are trained using LLMs, supervised examples, or other types of supervision, and exposure to thousands of tasks written as natural language instructions. In recent research, a team from Mila Quebec AI Institute, McGill University, and Facebook CIFAR AI Chair has researched evaluating the performance of instruction-following models for their ability to perform question-answering (QA) on a given set of text passages. These models can answer questions when provided with a prompt describing the task, the question, and relevant text passages retrieved by a retriever, and the responses produced by these models are known to be natural and informative, which helps build users’ trust and engagement. 

These models can respond to user queries naturally and fluently by only adding retrieved documents and instructions to their input. However, this extra verbosity makes it difficult for conventional QA evaluation metrics like exact match (EM) and F1 score to effectively quantify model performance. This is due to the possibility that the model’s response may include more details that the reference answer omits while still being accurate. The team has provided two criteria for measuring instruction-following models in retrieval-augmented quality assurance (QA) in order to overcome this problem.

  1. Regarding information necessity, accuracy: This dimension evaluates how well the model satisfies the informational requirements of a user. It is concerned with whether the generated response includes pertinent information, even if it goes beyond what is mentioned directly in the reference answer.
  1. Fidelity in relation to information provided: This dimension assesses how well the model grounds answers in the knowledge presented. A true model should refrain from responding when irrelevant information is presented, in addition to giving precise answers when it is accessible.

The authors have evaluated several recent instruction-following models on three diverse QA datasets: Natural Questions for open-domain QA, HotpotQA for multi-hop QA, and TopiOCQA for conversational QA. They analyzed 900 model responses manually and compared the results with different automatic metrics for accuracy and faithfulness. Their research has suggested that recall, which measures the percentage of tokens from the reference answer that are also present in the model response, correlates more strongly with correctness than lexical overlap metrics like EM or F1 score. Compared to other token-overlap metrics for faithfulness, K-Precision, which is the percentage of model answer tokens that exist in the knowledge snippet, has a stronger correlation with human judgments.

In conclusion, this study seeks to advance a more thorough assessment of instruction-following models for QA tasks, taking into account both their advantages and disadvantages. The team has promoted additional advancement in this area by making their code and data accessible on their GitHub repository


Check out the Paper, GitHub, and Tweet. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 27k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.


YOU MAY ALSO LIKE

SpaceX Successfully Catches Starship Rocket After Launch

Nvidia’s Defining Week: CEO Search, Earnings and China

Tanya Malhotra is a final year undergrad from the University of Petroleum & Energy Studies, Dehradun, pursuing BTech in Computer Science Engineering with a specialization in Artificial Intelligence and Machine Learning.
She is a Data Science enthusiast with good analytical and critical thinking, along with an ardent interest in acquiring new skills, leading groups, and managing work in an organized manner.


🔥 Use SQL to predict the future (Sponsored)


Credit: Source link

ShareTweetSendSharePin

Related Posts

SpaceX Successfully Catches Starship Rocket After Launch
AI & Technology

SpaceX Successfully Catches Starship Rocket After Launch

November 19, 2026
Nvidia’s Defining Week: CEO Search, Earnings and China
AI & Technology

Nvidia’s Defining Week: CEO Search, Earnings and China

November 19, 2026
Nvidia Bull: AI Growth Story Remains Intact
AI & Technology

Nvidia Bull: AI Growth Story Remains Intact

November 19, 2026
Nvidia Fallout Adds New Tension to US-China AI Race
AI & Technology

Nvidia Fallout Adds New Tension to US-China AI Race

November 19, 2026
Next Post
‘Bloomberg Technology’ Full Show (6/19/2019)

'Bloomberg Technology' Full Show (6/19/2019)

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Mitsubishi UFJ Financial Group, Inc. 2027 Q2 – Results – Earnings Call Presentation (NYSE:MUFG) 2026-11-18

Mitsubishi UFJ Financial Group, Inc. 2027 Q2 – Results – Earnings Call Presentation (NYSE:MUFG) 2026-11-18

November 18, 2026
A Simple Home Filing System – Sort Papers by How Long You Keep Them

A Simple Home Filing System – Sort Papers by How Long You Keep Them

November 16, 2026
One-on-one with Senate candidate Ashley Hinson in Iowa

One-on-one with Senate candidate Ashley Hinson in Iowa

November 19, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!