• bitcoinBitcoin(BTC)$58,897.00-5.49%
  • ethereumEthereum(ETH)$1,618.84-8.05%
  • tetherTether(USDT)$1.00-0.07%
  • binancecoinBNB(BNB)$539.13-6.95%
  • usd-coinUSDC(USDC)$1.00-0.04%
  • rippleXRP(XRP)$0.91-6.29%
  • solanaSolana(SOL)$68.64-6.05%
  • tronTRON(TRX)$0.301281-2.96%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.030.90%
  • HyperliquidHyperliquid(HYPE)$60.35-8.87%
  • zcashZcash(ZEC)$746.83-16.06%
  • dogecoinDogecoin(DOGE)$0.054296-6.74%
  • moneroMonero(XMR)$451.57-9.73%
  • leo-tokenLEO Token(LEO)$9.15-0.66%
  • whitebitWhiteBIT Coin(WBT)$56.36-6.74%
  • chainlinkChainlink(LINK)$8.53-5.30%
  • USDSUSDS(USDS)$1.000.02%
  • cardanoCardano(ADA)$0.142780-6.09%
  • Ethena USDeEthena USDe(USDE)$1.00-0.03%
  • stellarStellar(XLM)$0.127190-5.17%
  • daiDai(DAI)$1.000.04%
  • bitcoin-cashBitcoin Cash(BCH)$193.34-6.70%
  • nearNEAR Protocol(NEAR)$2.79-12.27%
  • litecoinLitecoin(LTC)$40.47-4.32%
  • CantonCanton(CC)$0.074606-6.80%
  • avalanche-2Avalanche(AVAX)$6.48-7.57%
  • USD1USD1(USD1)$1.00-0.07%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • tether-goldTether Gold(XAUT)$3,530.00-1.37%
  • the-open-networkGram (prev. Toncoin)(GRAM)$0.96-5.14%
  • uniswapUniswap(UNI)$4.47-9.48%
  • RainRain(RAIN)$0.003766-13.86%
  • suiSui(SUI)$0.64-9.55%
  • Global DollarGlobal Dollar(USDG)$1.000.02%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.00%
  • paypal-usdPayPal USD(PYUSD)$1.000.02%
  • hedera-hashgraphHedera(HBAR)$0.051099-5.57%
  • BittensorBittensor(TAO)$179.42-6.63%
  • shiba-inuShiba Inu(SHIB)$0.000004-5.60%
  • crypto-com-chainCronos(CRO)$0.039563-4.57%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.120.06%
  • quant-networkQuant(QNT)$138.72-6.36%
  • Circle USYCCircle USYC(USYC)$1.140.05%
  • MemeCoreMemeCore(M)$0.850.18%
  • Falcon USDFalcon USD(USDF)$1.000.00%
  • okbOKB(OKB)$78.51-8.30%
  • United StablesUnited Stables(U)$1.00-0.10%
  • pax-goldPAX Gold(PAXG)$3,530.71-1.29%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
  • aaveAave(AAVE)$97.66-6.68%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Zyphra Open-Sources BlackMamba: A Novel Architecture that Combines the Mamba SSM with MoE to Obtain the Benefits of Both

February 6, 2024
in AI & Technology
Reading Time: 4 mins read
A A
Zyphra Open-Sources BlackMamba: A Novel Architecture that Combines the Mamba SSM with MoE to Obtain the Benefits of Both
ShareShareShareShareShare

Processing extensive sequences of linguistic data has been a significant hurdle, with traditional transformer models often buckling under the weight of computational and memory demands. This limitation is primarily due to the quadratic complexity of the attention mechanisms these models rely on, which scales poorly as sequence length increases. The introduction of State Space Models (SSMs) and mixture-of-experts (MoE) models offered a glimpse into potential solutions, with the former providing a way to linearize computational complexity and the latter reducing the computational overhead of training and inference, albeit at the cost of increased memory requirements.

The BlackMamba model by researchers from Zyphra emerges as a sophisticated fusion of SSMs and MoEs designed to leverage each other’s strengths. The architecture of BlackMamba stands out for its innovative combination of attention-free Mamba blocks and routed MLPs. This configuration streamlines the model’s efficiency and enhances its performance across various language tasks. This hybrid model is particularly adept at processing long data sequences, which has traditionally posed significant challenges for existing NLP models.

The methodology behind BlackMamba by alternating between Mamba blocks, which eschew traditional attention mechanisms for a more streamlined approach, and MoE blocks, which selectively engage different expert components of the model depending on the input, BlackMamba achieves a remarkable balance of efficiency and effectiveness. This balance is crucial for scaling up NLP models to handle human language’s vast and varied nuances without incurring prohibitive computational costs.

The performance of BlackMamba has been rigorously evaluated against current benchmarks, revealing its superior capability in handling long sequences with greater efficiency and reducing the training FLOPs required to achieve comparable or superior performance to dense transformer models. BlackMamba exhibits impressive performance metrics across multiple benchmarks, outpacing SSM and MoE models in various tasks. Such achievements underscore the model’s potential to significantly advance the field of NLP, offering a more scalable and cost-effective solution for processing and understanding human language.

The release of BlackMamba as open-source represents a commendable commitment to transparency and collaboration in scientific research. By making the model and its training details publicly available, the research team at Zyphra encourages further exploration, experimentation, and innovation within the AI community. This open-source approach facilitates the widespread adoption and adaptation of BlackMamba and sets a precedent for future developments in the field.

In conclusion, the introduction of BlackMamba by Zyphra researchers marks a significant milestone in the evolution of language models, characterized by:

  • This is a novel integration of state-space models and mixture-of-experts architectures, offering a blueprint for future advancements in natural language processing.
  • An innovative methodology that balances computational efficiency with performance, enabling the processing of long sequences without prohibitive costs.
  • It has demonstrated superior performance metrics across multiple benchmarks, highlighting the model’s effectiveness and efficiency.
  • The open-source release of the model promotes transparency, collaboration, and further innovation within the AI community.

Check out the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our Telegram Channel


YOU MAY ALSO LIKE

Why You Shouldn’t Depend On GTA 6 To Auto-Save

Anysphere Weighs Moving Cursor Away From Chinese Open-Weight Models – Unite.AI

Hello, My name is Adnan Hassan. I am a consulting intern at Marktechpost and soon to be a management trainee at American Express. I am currently pursuing a dual degree at the Indian Institute of Technology, Kharagpur. I am passionate about technology and want to create new products that make a difference.


🎯 [FREE AI WEBINAR] ‘Inventory Management Using Object/Image Detection’ (Feb 7, 2024)


Credit: Source link

ShareTweetSendSharePin

Related Posts

Why You Shouldn’t Depend On GTA 6 To Auto-Save
AI & Technology

Why You Shouldn’t Depend On GTA 6 To Auto-Save

November 20, 2026
Anysphere Weighs Moving Cursor Away From Chinese Open-Weight Models – Unite.AI
AI & Technology

Anysphere Weighs Moving Cursor Away From Chinese Open-Weight Models – Unite.AI

November 20, 2026
CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI
AI & Technology

CrowdStrike Reportedly in Talks to Acquire AI Security Partner Zscaler – Unite.AI

November 20, 2026
How To Stream GTA 6 On Your iPhone, iPad Or Mac
AI & Technology

How To Stream GTA 6 On Your iPhone, iPad Or Mac

November 20, 2026
Next Post
Why McDonald’s is charging  for a Big Mac meal; there’s no relief in sight

Why McDonald's is charging $18 for a Big Mac meal; there's no relief in sight

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
‘Manosphere’ that helped Trump to power seems to be moving away from him

‘Manosphere’ that helped Trump to power seems to be moving away from him

November 15, 2026
Why Microsoft Is Ditching A Few Of Its 365 Companion Apps In 2026

Why Microsoft Is Ditching A Few Of Its 365 Companion Apps In 2026

November 16, 2026
Inside look: Wonder Studios collaborate with filmmakers to create AI movies

Inside look: Wonder Studios collaborate with filmmakers to create AI movies

November 17, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!