• bitcoinBitcoin(BTC)$58,095.00-5.62%
  • ethereumEthereum(ETH)$1,650.02-5.16%
  • tetherTether(USDT)$1.00-0.04%
  • binancecoinBNB(BNB)$529.57-6.81%
  • usd-coinUSDC(USDC)$1.00-0.02%
  • rippleXRP(XRP)$0.90-5.02%
  • solanaSolana(SOL)$68.69-3.88%
  • tronTRON(TRX)$0.302828-1.14%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.031.68%
  • HyperliquidHyperliquid(HYPE)$59.29-6.89%
  • zcashZcash(ZEC)$762.88-6.52%
  • moneroMonero(XMR)$473.903.00%
  • dogecoinDogecoin(DOGE)$0.053675-5.25%
  • leo-tokenLEO Token(LEO)$9.13-0.92%
  • whitebitWhiteBIT Coin(WBT)$56.13-5.85%
  • chainlinkChainlink(LINK)$8.44-3.58%
  • USDSUSDS(USDS)$1.000.03%
  • cardanoCardano(ADA)$0.140205-4.40%
  • Ethena USDeEthena USDe(USDE)$1.00-0.04%
  • stellarStellar(XLM)$0.125531-4.57%
  • daiDai(DAI)$1.000.04%
  • bitcoin-cashBitcoin Cash(BCH)$191.83-3.23%
  • nearNEAR Protocol(NEAR)$2.78-6.21%
  • litecoinLitecoin(LTC)$38.97-5.67%
  • CantonCanton(CC)$0.074255-7.07%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • avalanche-2Avalanche(AVAX)$6.43-5.99%
  • USD1USD1(USD1)$1.000.03%
  • tether-goldTether Gold(XAUT)$3,519.74-1.30%
  • uniswapUniswap(UNI)$4.49-6.09%
  • the-open-networkGram (prev. Toncoin)(GRAM)$0.94-7.17%
  • Global DollarGlobal Dollar(USDG)$1.000.01%
  • suiSui(SUI)$0.63-8.44%
  • RainRain(RAIN)$0.003621-0.33%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.01%
  • paypal-usdPayPal USD(PYUSD)$1.000.01%
  • hedera-hashgraphHedera(HBAR)$0.050208-5.86%
  • BittensorBittensor(TAO)$180.70-2.53%
  • crypto-com-chainCronos(CRO)$0.040243-1.65%
  • shiba-inuShiba Inu(SHIB)$0.000004-4.17%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.120.11%
  • Circle USYCCircle USYC(USYC)$1.140.05%
  • quant-networkQuant(QNT)$134.32-5.54%
  • MemeCoreMemeCore(M)$0.77-11.86%
  • Falcon USDFalcon USD(USDF)$1.00-0.04%
  • United StablesUnited Stables(U)$1.00-0.03%
  • okbOKB(OKB)$77.70-4.22%
  • pax-goldPAX Gold(PAXG)$3,524.30-1.15%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
  • aaveAave(AAVE)$99.43-2.52%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Meta AI introduces SPIRIT-LM: A Foundation Multimodal Language Model that Freely Mixes Text and Speech

February 16, 2024
in AI & Technology
Reading Time: 5 mins read
A A
Meta AI introduces SPIRIT-LM: A Foundation Multimodal Language Model that Freely Mixes Text and Speech
ShareShareShareShareShare

Prompting Large Language Models (LLMs) has emerged as a standard practice in Natural Language Processing (NLP) following the introduction of GPT-3. The scaling of language models to billions of parameters using extensive datasets contributes significantly to achieving broad language understanding and generation capabilities. Moreover, large-scale language models exhibit the ability to tackle novel tasks by leveraging a few examples via in-context, few-shot learning.

Speech Language Models (SpeechLMs), which are language models trained directly on speech, have been introduced by researchers, marking the beginning of an active area of research. Recent studies have contributed to advancing this field.

SPIRIT-LM is introduced as a foundational multimodal language model that seamlessly integrates text and speech. The model builds upon a pre-trained text language model and expands its capabilities to incorporate speech by continual training on a combination of text and speech data. Text and speech sequences are merged into a unified token set and trained using a word-level interleaving approach with a curated speech-text parallel corpus.

SPIRIT-LM is available in two variants: a BASE version employing speech semantic units and an EXPRESSIVE version that incorporates pitch and style units to model expressivity alongside semantic units. Both versions encode text using the subword BPE tokens. The resultant model demonstrates a fusion of semantic comprehension from text models and expressive qualities from speech models.

a. The architecture of SPIRIT-LM involves a language model trained through next-token prediction. Tokens are generated either from speech or text via an encoder and then reconstructed back to their original modality using a decoder. Training of SPIRIT-LM models encompasses a combination of text-only sequences, speech-only sequences, and interleaved speech-text sequences.

b. The scheme for interleaving speech and text involves encoding speech into tokens (depicted in pink) using clusterized speech units such as Hubert, Pitch, or Style tokens and text (depicted in blue) using BPE. Special tokens ([TEXT] for text and [SPEECH] for speech) are used to mark the respective modality. During training, a switch between modalities occurs randomly at word boundaries within aligned speech-text corpora. Speech tokens are deduplicated and then interleaved with text tokens at the boundary where the modality changes.

c. Expressive speech tokens are introduced for SPIRIT-LM-EXPRESSIVE. Pitch tokens and style tokens are interleaved after deduplication.

Their contributions are as follows:

  • They introduce SPIRIT-LM, a unified language model capable of generating both speech and text. SPIRIT-LM is developed by continuously pretraining LLAMA 2 with interleaved speech and text data.
  • Similar to text-based Language Models (LLMs), they observe that SPIRIT-LM can adeptly learn new tasks in a few-shot learning setting across text, speech, and crossmodal tasks (i.e., speech-to-text and text-to-speech).
  • To assess the expressive capabilities of generative models, they introduce the SPEECHTEXT SENTIMENT PRESERVATION benchmark (STSP). This benchmark evaluates how effectively generative models maintain the sentiment of prompts within and across modalities for both spoken and written expressions.
  • Finally, they propose an expressive variant of SPIRIT-LM, named SPIRIT-LM-EXPRESSIVE. Through the use of STSP, they demonstrate that SPIRIT-LM is the first language model capable of preserving the sentiment of both text and speech prompts within and across modalities.

As advancements in Large Language Models (LLMs) and Speech Language Models (SpeechLMs) continue, along with innovative approaches to prompt creation and model design, there’s great potential to improve natural language understanding systems. These advancements could profoundly impact many areas, such as conversational agents, virtual assistants, language translation, and accessibility tools. Ultimately, they could lead to more lifelike interactions between humans and machines.


Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel


YOU MAY ALSO LIKE

IIT Kharagpur Secures $5M From Google to Expand Campus AI Work – Unite.AI

Baseten Extends Decision Models to 128K Context and Image Input – Unite.AI

Janhavi Lande, is an Engineering Physics graduate from IIT Guwahati, class of 2023. She is an upcoming data scientist and has been working in the world of ml/ai research for the past two years. She is most fascinated by this ever changing world and its constant demand of humans to keep up with it. In her pastime she enjoys traveling, reading and writing poems.


🚀 LLMWare Launches SLIMs: Small Specialized Function-Calling Models for Multi-Step Automation [Check out all the models]


Credit: Source link

ShareTweetSendSharePin

Related Posts

IIT Kharagpur Secures M From Google to Expand Campus AI Work – Unite.AI
AI & Technology

IIT Kharagpur Secures $5M From Google to Expand Campus AI Work – Unite.AI

November 21, 2026
Baseten Extends Decision Models to 128K Context and Image Input – Unite.AI
AI & Technology

Baseten Extends Decision Models to 128K Context and Image Input – Unite.AI

November 21, 2026
Baseten Releases Bison-1 and Groundhog-1, Topping Decision Model Index
AI & Technology

Baseten Releases Bison-1 and Groundhog-1, Topping Decision Model Index

November 21, 2026
AWS Quick Adds Chat-Built Flows and Multi-Agent Orchestration – Unite.AI
AI & Technology

AWS Quick Adds Chat-Built Flows and Multi-Agent Orchestration – Unite.AI

November 21, 2026
Next Post
Russian TV shows Putin mobbed by residents of Derbent

Russian TV shows Putin mobbed by residents of Derbent

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Current with Christine Romans – Oct. 22 | NBC News NOW

Current with Christine Romans – Oct. 22 | NBC News NOW

November 18, 2026
Dating at Costco? Wholesaler attracts shoppers seeking social interactions

Dating at Costco? Wholesaler attracts shoppers seeking social interactions

November 19, 2026
Moth’s Tessa Encodes and Recovers 1,568 Values on a 120-Qubit Chip – Unite.AI

Moth’s Tessa Encodes and Recovers 1,568 Values on a 120-Qubit Chip – Unite.AI

November 16, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!