• bitcoinBitcoin(BTC)$62,877.00-2.98%
  • ethereumEthereum(ETH)$1,773.50-2.70%
  • tetherTether(USDT)$1.00-0.03%
  • binancecoinBNB(BNB)$581.14-2.59%
  • usd-coinUSDC(USDC)$1.000.00%
  • rippleXRP(XRP)$0.97-1.48%
  • solanaSolana(SOL)$73.41-2.14%
  • tronTRON(TRX)$0.3094490.50%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.010.25%
  • HyperliquidHyperliquid(HYPE)$66.84-4.34%
  • zcashZcash(ZEC)$910.20-5.51%
  • moneroMonero(XMR)$504.93-0.82%
  • dogecoinDogecoin(DOGE)$0.058648-2.31%
  • leo-tokenLEO Token(LEO)$9.29-0.06%
  • USDSUSDS(USDS)$1.000.00%
  • whitebitWhiteBIT Coin(WBT)$60.93-2.74%
  • chainlinkChainlink(LINK)$9.06-2.61%
  • cardanoCardano(ADA)$0.151101-2.55%
  • stellarStellar(XLM)$0.137144-3.10%
  • Ethena USDeEthena USDe(USDE)$1.00-0.02%
  • daiDai(DAI)$1.000.00%
  • nearNEAR Protocol(NEAR)$3.29-2.87%
  • bitcoin-cashBitcoin Cash(BCH)$208.02-1.92%
  • litecoinLitecoin(LTC)$43.94-3.02%
  • CantonCanton(CC)$0.081798-0.49%
  • RainRain(RAIN)$0.004445-2.68%
  • uniswapUniswap(UNI)$5.17-2.54%
  • avalanche-2Avalanche(AVAX)$7.01-3.80%
  • tether-goldTether Gold(XAUT)$3,730.72-2.62%
  • suiSui(SUI)$0.72-2.58%
  • the-open-networkGram (prev. Toncoin)(GRAM)$1.02-5.40%
  • USD1USD1(USD1)$1.00-0.03%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • Global DollarGlobal Dollar(USDG)$1.000.00%
  • hedera-hashgraphHedera(HBAR)$0.055807-4.15%
  • paypal-usdPayPal USD(PYUSD)$1.000.00%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.10%
  • BittensorBittensor(TAO)$191.80-3.01%
  • shiba-inuShiba Inu(SHIB)$0.000004-2.32%
  • crypto-com-chainCronos(CRO)$0.042864-3.17%
  • quant-networkQuant(QNT)$151.66-2.71%
  • MemeCoreMemeCore(M)$0.890.23%
  • Circle USYCCircle USYC(USYC)$1.140.01%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.12-0.20%
  • okbOKB(OKB)$86.51-0.49%
  • Falcon USDFalcon USD(USDF)$1.00-0.07%
  • aaveAave(AAVE)$108.90-2.97%
  • pax-goldPAX Gold(PAXG)$3,730.85-2.73%
  • United StablesUnited Stables(U)$1.00-0.05%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

This Paper from NYU and Google Explains How Joint Speech-Text Encoders Overcome Sequence-Length Mismatch in Cross-Modal Representations

August 18, 2023
in AI & Technology
Reading Time: 4 mins read
A A
This Paper from NYU and Google Explains How Joint Speech-Text Encoders Overcome Sequence-Length Mismatch in Cross-Modal Representations
ShareShareShareShareShare

It is becoming increasingly apparent that very big models trained on massive unsupervised corpora in a single modality can achieve remarkable results. This has been proved both in the audio domain, where a single model has been shown to adapt to a surprise wide array of acoustic tasks and in the text domain, where language models have attained exceptional zero-shot capabilities. Similar achievements have prompted the inquiry into how to employ similar techniques for situations combining two modalities, which have traditionally relied on manually paired data.

One interesting approach is to train a big encoder on both modalities so that either one can be presented as an unpaired example and the encoder will learn to map the two to similar places in representation space. Achievable and capable of state-of-the-art performance on numerous picture and text comprehension tasks using a single model, such a representation has been demonstrated to be feasible in the image/text-domain.

New research by the New York University and Google investigates whether the performance gains found with the explicit alignments may be achieved by applying consistency regularization to the implicit alignments learned that in upsampling systems. They achieve this by developing a method, motivated by dynamic time warping, that optimally aligns the encoder’s representation of a speech and text example. In the absence of an explicit alignment model, the team demonstrate that the optimum alignment is not just acquired during training but also improves as one progresses through the network’s layers. 

To facilitate pretraining on unpaired voice and text data, there has been a recent trend toward models with a joint speech and text encoder in the field of speech recognition. The lengthier sequence used to represent speech offers a unique difficulty for speech recognition since it involves two sequence modalities. Because of this, comparing an encoder’s speech representation to its text representation frame-by-frame becomes a more difficult process, even though both modalities are represented in the same embedding space.

Finally, the work demonstrates that, in a monolingual and multilingual setting, significant WER improvements can be achieved against strong, semi-supervised baselines without any learned alignment model by modifying the criteria of the consistency regularization to encourage consistency under some alignment rather than a direct frame-wise comparison. Based on their findings, it appears that tolerating misalignment is all that’s needed to enforce consistency in cross-modal representations.


Check out the Paper. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 28k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.


YOU MAY ALSO LIKE

Waymo Pauses Service In San Francisco After Robotaxis Get Stuck On Flooded Streets

Google Launches Gemini 4 Neon for Coding, Knowledge Work and Science – Unite.AI

Dhanshree Shenwai is a Computer Science Engineer and has a good experience in FinTech companies covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is enthusiastic about exploring new technologies and advancements in today’s evolving world making everyone’s life easy.


🔥 Use SQL to predict the future (Sponsored)

Credit: Source link

ShareTweetSendSharePin

Related Posts

Waymo Pauses Service In San Francisco After Robotaxis Get Stuck On Flooded Streets
AI & Technology

Waymo Pauses Service In San Francisco After Robotaxis Get Stuck On Flooded Streets

November 18, 2026
Google Launches Gemini 4 Neon for Coding, Knowledge Work and Science – Unite.AI
AI & Technology

Google Launches Gemini 4 Neon for Coding, Knowledge Work and Science – Unite.AI

November 18, 2026
Rockstar Just Settled A Major Question About GTA 6’s Controversial FPS Decision
AI & Technology

Rockstar Just Settled A Major Question About GTA 6’s Controversial FPS Decision

November 18, 2026
MIT Microrobot Flies Without Motion Capture Using Onboard Sensors – Unite.AI
AI & Technology

MIT Microrobot Flies Without Motion Capture Using Onboard Sensors – Unite.AI

November 18, 2026
Next Post
Levchin Says Crypto Not a Good Currency But He Likes the Tech

Levchin Says Crypto Not a Good Currency But He Likes the Tech

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Costco shoppers panic-buy water, generators after freak outages

Costco shoppers panic-buy water, generators after freak outages

November 17, 2026
Can your ‘digital twin’ help to predict and prevent sickness?

Can your ‘digital twin’ help to predict and prevent sickness?

November 17, 2026
Trump Had Urged Nvidia’s Jensen Huang to Fight for His Job – The New York Times

Trump Had Urged Nvidia’s Jensen Huang to Fight for His Job – The New York Times

November 13, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!