• bitcoinBitcoin(BTC)$60,609.00-1.89%
  • ethereumEthereum(ETH)$1,704.32-2.59%
  • tetherTether(USDT)$1.00-0.05%
  • binancecoinBNB(BNB)$548.78-3.55%
  • usd-coinUSDC(USDC)$1.00-0.02%
  • rippleXRP(XRP)$0.93-1.80%
  • solanaSolana(SOL)$70.74-0.69%
  • tronTRON(TRX)$0.3064380.36%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.031.76%
  • HyperliquidHyperliquid(HYPE)$61.29-2.75%
  • zcashZcash(ZEC)$799.69-1.98%
  • moneroMonero(XMR)$474.772.52%
  • dogecoinDogecoin(DOGE)$0.055326-2.14%
  • leo-tokenLEO Token(LEO)$9.12-0.99%
  • whitebitWhiteBIT Coin(WBT)$58.18-2.87%
  • chainlinkChainlink(LINK)$8.66-1.36%
  • USDSUSDS(USDS)$1.000.04%
  • cardanoCardano(ADA)$0.143915-1.68%
  • Ethena USDeEthena USDe(USDE)$1.00-0.05%
  • stellarStellar(XLM)$0.129269-1.73%
  • daiDai(DAI)$1.000.01%
  • bitcoin-cashBitcoin Cash(BCH)$199.05-0.48%
  • nearNEAR Protocol(NEAR)$2.91-1.65%
  • litecoinLitecoin(LTC)$40.19-3.30%
  • CantonCanton(CC)$0.076585-3.40%
  • avalanche-2Avalanche(AVAX)$6.67-2.19%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • tether-goldTether Gold(XAUT)$3,550.13-0.29%
  • USD1USD1(USD1)$1.000.07%
  • uniswapUniswap(UNI)$4.71-0.30%
  • the-open-networkGram (prev. Toncoin)(GRAM)$0.96-5.30%
  • suiSui(SUI)$0.65-4.64%
  • Global DollarGlobal Dollar(USDG)$1.000.02%
  • RainRain(RAIN)$0.003534-2.20%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.15-0.01%
  • paypal-usdPayPal USD(PYUSD)$1.000.03%
  • hedera-hashgraphHedera(HBAR)$0.051863-2.01%
  • BittensorBittensor(TAO)$188.051.37%
  • shiba-inuShiba Inu(SHIB)$0.000004-1.38%
  • crypto-com-chainCronos(CRO)$0.0411120.32%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.12-0.22%
  • quant-networkQuant(QNT)$140.54-0.12%
  • Circle USYCCircle USYC(USYC)$1.140.00%
  • MemeCoreMemeCore(M)$0.79-8.58%
  • Falcon USDFalcon USD(USDF)$1.00-0.04%
  • okbOKB(OKB)$79.69-2.96%
  • United StablesUnited Stables(U)$1.000.01%
  • pax-goldPAX Gold(PAXG)$3,553.26-0.19%
  • aaveAave(AAVE)$101.10-0.67%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Nomic AI Releases the First Fully Open-Source Long Context Text Embedding Model that Surpasses OpenAI Ada-002 Performance on Various Benchmarks

February 17, 2024
in AI & Technology
Reading Time: 4 mins read
A A
Nomic AI Releases the First Fully Open-Source Long Context Text Embedding Model that Surpasses OpenAI Ada-002 Performance on Various Benchmarks
ShareShareShareShareShare

In the evolving landscape of natural language processing (NLP), the ability to grasp and process extensive textual contexts is paramount. Recent advancements, as highlighted by Lewis et al. (2021), Izacard et al. (2022), and Ram et al. (2023), have significantly propelled the capabilities of language models, particularly through the development of text embeddings. These embeddings serve as the backbone for a plethora of applications, including retrieval-augmented generation for large language models (LLMs) and semantic search. They transform sentences or documents into low-dimensional vectors, capturing the essence of semantic information, which in turn facilitates tasks like clustering, classification, and information retrieval.

However, a glaring limitation has been the context length that these models can handle. The majority of widely recognized open-source models on the MTEB benchmark, such as E5 by Wang et al. (2022), GTE by Li et al. (2023), and BGE by Xiao et al. (2023), are confined to a context length of 512 tokens. This restriction undermines their utility in scenarios where understanding the broader document context is crucial. In contrast, models capable of surpassing a context length of 2048, like Voyage-lite-01-instruct by Voyage (2023) and text-embedding-ada-002 by Neelakantan et al. (2022), remain behind closed doors.

Amid this backdrop, the introduction of nomicembed-text-v1 marks a significant milestone. This model is not only open-source but also boasts an impressive sequence length of 8192, outperforming its predecessors in both short and long-context evaluations. What sets it apart is its comprehensive approach, merging the strengths of open weights, open data, and a 137M parameter design under an Apache-2 license, ensuring accessibility and transparency.

The journey to achieving such a feat involved meticulous stages of data preparation and model training. Initially, a Masked Language Modeling Pretraining phase utilized resources like BooksCorpus and a Wikipedia dump from 2023, employing the bert-base-uncased tokenizer to create data chunks suited for long-context training. This was followed by Unsupervised Contrastive Pretraining, leveraging a vast collection of 470 million pairs across diverse datasets to refine the model’s understanding through consistency filtering and selective embedding. 

The architecture of nomicembed-text-v1 reflects a thoughtful adaptation of BERT to accommodate the extended sequence length. Innovations such as rotary positional embeddings, SwiGLU activation, and the integration of Flash Attention highlight a strategic overhaul to enhance performance and efficiency. The model’s training regimen, characterized by a 30% masking rate and optimized settings, further underscores the rigorous effort to achieve optimal results.

When subjected to the rigors of benchmarks like GLUE, MTEB, and specialized long-context assessments, nomicembed-text-v1 demonstrated exceptional prowess. Notably, its performance in the JinaAI Long Context Benchmark and the LoCo Benchmark underscores its superiority in handling extensive texts, an area where many predecessors faltered.

Yet, the journey of nomicembed-text-v1 extends beyond mere performance metrics. Its development process, emphasizing end-to-end auditability and the potential for replication, sets a new standard for transparency and openness in the AI community. By releasing the model weights, codebase, and a curated training dataset, the team behind nomicembed-text-v1 invites ongoing innovation and scrutiny.

In conclusion, nomicembed-text-v1 emerges not just as a technological breakthrough but as a beacon for the open-source movement in AI. It dismantles barriers to entry in the domain of long-context text embeddings, promising a future where the depth of understanding matches the breadth of human discourse. 


Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 37k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel


YOU MAY ALSO LIKE

Why Sony’s New Digital Walkmans Cost Thousands Of Dollars

OpenAI Ships GPT-Live-1.1 for Voice Agents, Cuts Idle Cost to $0.02/Min

Vineet Kumar is a consulting intern at MarktechPost. He is currently pursuing his BS from the Indian Institute of Technology(IIT), Kanpur. He is a Machine Learning enthusiast. He is passionate about research and the latest advancements in Deep Learning, Computer Vision, and related fields.


🚀 LLMWare Launches SLIMs: Small Specialized Function-Calling Models for Multi-Step Automation [Check out all the models]


Credit: Source link

ShareTweetSendSharePin

Related Posts

Why Sony’s New Digital Walkmans Cost Thousands Of Dollars
AI & Technology

Why Sony’s New Digital Walkmans Cost Thousands Of Dollars

November 21, 2026
OpenAI Ships GPT-Live-1.1 for Voice Agents, Cuts Idle Cost to alt=
AI & Technology

OpenAI Ships GPT-Live-1.1 for Voice Agents, Cuts Idle Cost to $0.02/Min

November 21, 2026
IIT Kharagpur Secures M From Google to Expand Campus AI Work – Unite.AI
AI & Technology

IIT Kharagpur Secures $5M From Google to Expand Campus AI Work – Unite.AI

November 21, 2026
Baseten Extends Decision Models to 128K Context and Image Input – Unite.AI
AI & Technology

Baseten Extends Decision Models to 128K Context and Image Input – Unite.AI

November 21, 2026
Next Post
Long COVID cases are highest in these US states, Census data shows

Long COVID cases are highest in these US states, Census data shows

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
3D Printing Entire Buildings is So Wild To Me

3D Printing Entire Buildings is So Wild To Me

November 16, 2026
SpaceX Successfully Catches Starship Rocket After Launch

SpaceX Successfully Catches Starship Rocket After Launch

November 19, 2026
Commerce Department moves to crack down on Chinese AI chip loophole amid Nvidia scandal

Commerce Department moves to crack down on Chinese AI chip loophole amid Nvidia scandal

November 16, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!