• bitcoinBitcoin(BTC)$64,874.00-0.03%
  • ethereumEthereum(ETH)$1,821.950.07%
  • tetherTether(USDT)$1.00-0.02%
  • binancecoinBNB(BNB)$596.110.87%
  • usd-coinUSDC(USDC)$1.00-0.01%
  • rippleXRP(XRP)$0.982.91%
  • solanaSolana(SOL)$75.104.98%
  • tronTRON(TRX)$0.3081211.02%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.01-3.69%
  • HyperliquidHyperliquid(HYPE)$69.86-0.06%
  • zcashZcash(ZEC)$965.296.56%
  • moneroMonero(XMR)$508.953.25%
  • dogecoinDogecoin(DOGE)$0.0600471.94%
  • leo-tokenLEO Token(LEO)$9.301.53%
  • USDSUSDS(USDS)$1.000.02%
  • whitebitWhiteBIT Coin(WBT)$62.68-0.04%
  • chainlinkChainlink(LINK)$9.311.97%
  • cardanoCardano(ADA)$0.1552261.46%
  • stellarStellar(XLM)$0.1412722.58%
  • Ethena USDeEthena USDe(USDE)$1.00-0.01%
  • nearNEAR Protocol(NEAR)$3.397.28%
  • daiDai(DAI)$1.000.01%
  • bitcoin-cashBitcoin Cash(BCH)$212.061.76%
  • litecoinLitecoin(LTC)$45.301.90%
  • CantonCanton(CC)$0.082118-0.23%
  • RainRain(RAIN)$0.004580-23.87%
  • avalanche-2Avalanche(AVAX)$7.282.41%
  • uniswapUniswap(UNI)$5.312.78%
  • tether-goldTether Gold(XAUT)$3,837.20-0.27%
  • the-open-networkGram (prev. Toncoin)(GRAM)$1.082.38%
  • suiSui(SUI)$0.742.53%
  • USD1USD1(USD1)$1.000.03%
  • Ripple USDRipple USD(RLUSD)$1.000.01%
  • Global DollarGlobal Dollar(USDG)$1.000.00%
  • hedera-hashgraphHedera(HBAR)$0.0582591.60%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.150.01%
  • paypal-usdPayPal USD(PYUSD)$1.000.00%
  • BittensorBittensor(TAO)$197.222.88%
  • crypto-com-chainCronos(CRO)$0.0442932.04%
  • shiba-inuShiba Inu(SHIB)$0.0000040.66%
  • quant-networkQuant(QNT)$156.121.47%
  • MemeCoreMemeCore(M)$0.894.54%
  • Circle USYCCircle USYC(USYC)$1.140.01%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.12-0.04%
  • okbOKB(OKB)$86.911.49%
  • aaveAave(AAVE)$112.383.65%
  • pax-goldPAX Gold(PAXG)$3,841.16-0.20%
  • Falcon USDFalcon USD(USDF)$1.000.09%
  • United StablesUnited Stables(U)$1.00-0.02%
  • pepePepe(PEPE)$0.000004-2.26%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

AWS Adds Managed NVIDIA Dynamo to SageMaker HyperPod Inference – Unite.AI

November 17, 2026
in AI & Technology
Reading Time: 3 mins read
A A
AWS Adds Managed NVIDIA Dynamo to SageMaker HyperPod Inference – Unite.AI
ShareShareShareShareShare

Amazon Web Services announced on November 17, 2026, that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for managed multi-node and disaggregated model serving, alongside benchmark results showing 2.05 times the highest qualifying aggregated output-token throughput per GPU under AWS’s latency targets.

YOU MAY ALSO LIKE

CFTC Tells Kalshi To Stop Letting People Bet On Trump’s Speeches

Calif Uses AI to Find Java4Shell as OpenJDK and Oracle Ship Fixes – Unite.AI

The addition addresses two constraints in serving large language models: fitting model weights and the reusable key-value (KV) cache into GPU memory, and keeping long prompt processing from slowing token generation when both share GPUs. AWS illustrates the memory pressure with GLM-5.3, whose 8-bit floating-point (FP8) weights alone exceed the 640 GB of GPU memory on one ml.p5.48xlarge instance before KV cache and runtime overhead are included.

HyperPod supplies the managed infrastructure around the framework: Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, health monitoring for nodes, GPUs and EFA, and node replacement. HyperPod manages that replacement while the orchestration components recreate affected workers.

Disaggregated Serving and KV-Aware Routing

Dynamo is an open-source distributed inference framework that works with vLLM, SGLang and TensorRT-LLM. Its disaggregated topology uses a frontend to route requests to prefill workers, which process prompts, and decode workers, which generate tokens. Those pools can scale and be tuned independently, but both load the full model weights; KV cache moves between them over EFA.

Aggregated serving runs both stages in the same worker. In either topology, workers can run on one instance or span multiple instances using tensor and pipeline parallelism to divide the model across GPUs. Dynamo’s KV-aware router considers reusable prefix cache and active worker load. AWS says prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness helps avoid overloading individual workers.

Benchmarks With Latency Constraints

The Qwen3-235B-A22B-FP8 comparison used Dynamo and TensorRT-LLM on two p5en.48xlarge instances, totaling 16 H200 GPUs. AIPerf generated requests containing 4,000 input tokens and 200 output tokens. AWS retained only measured points with 90th-percentile time to first token below one second and median generation speed of at least 50 tokens per second per user.

AWS reported that disaggregated serving reached 207 output tokens per second per GPU at concurrency 80, compared with 101 at concurrency 32 for the highest qualifying aggregated result. The comparison therefore applies to latency-constrained throughput at different concurrency levels, rather than an unrestricted throughput maximum for either topology.

A separate Qwen3.5-397B-A17B-FP8 test used Dynamo and vLLM to compare eight H200 GPUs on one p5en.48xlarge instance with four GPUs on each of two instances, keeping the model, parallelism and workload configuration the same. AWS reported 126.4 output tokens per second per GPU for the two-node setup, or 99.1% of single-node throughput, at concurrency 32 with high prefix reuse. The model fits on one instance and would not normally be split in practice, so the test served as a controlled baseline for the throughput effect of crossing a node boundary.

Managed Configuration and Resource Planning

Users have two configuration paths. Existing Dynamo users can bring a DynamoGraphDeployment manifest describing the serving graph and manage storage, networking, pod placement, endpoints, TLS, registration and autoscaling themselves. The InferenceEndpointConfig path uses dynamoSpec for managed endpoint setup and Kubernetes Event-driven Autoscaling (KEDA) for either serving topology. It defaults to vLLM, provides defaults for KV-aware routing and replica counts, and supports an OpenAI-compatible interface through SageMaker endpoints.

The add-on installs Dynamo’s custom resource definitions and serving and scheduling components, while AWS Deep Learning Containers provide the runtime. On the managed path, the HyperPod Inference Operator stages weights from Amazon S3 or FSx and generates the DynamoGraphDeployment. Grove coordinates related pod groups, while KAI Scheduler uses gang scheduling so each multi-node worker’s pods are placed together. Adopting a supported Dynamo release requires updating the add-on.

AWS’s GLM-5.3 example makes the GPU-allocation choice concrete. Its aggregated configuration runs one model-worker replica across two ml.p5.48xlarge instances with 16-way tensor parallelism and two frontend replicas. The disaggregated variant configures one prefill replica and two decode replicas, each spanning two instances, for six instances in total. The separate replica counts show how a deployment can devote different amounts of capacity to prompt processing and token generation instead of scaling a single combined pool.

AWS recommends verifying EFA communication before benchmarking, testing representative prompt lengths, output lengths and prefix reuse, sweeping concurrency against service-level targets, and tuning topology and worker allocation from measured bottlenecks before changing engine settings.

HyperPod now handles more of the coordination and recovery work, but the benchmark conditions and deployment examples show that managed operation does not remove the serving trade-offs. The useful configuration still depends on measured prompt and generation behavior, the GPU allocation between worker pools, and the latency and cost priorities of the workload.

Credit: Source link

ShareTweetSendSharePin

Related Posts

CFTC Tells Kalshi To Stop Letting People Bet On Trump’s Speeches
AI & Technology

CFTC Tells Kalshi To Stop Letting People Bet On Trump’s Speeches

November 17, 2026
Calif Uses AI to Find Java4Shell as OpenJDK and Oracle Ship Fixes – Unite.AI
AI & Technology

Calif Uses AI to Find Java4Shell as OpenJDK and Oracle Ship Fixes – Unite.AI

November 17, 2026
Google Wallet Is Launching On iOS
AI & Technology

Google Wallet Is Launching On iOS

November 17, 2026
Cerebras Puts Disaggregated Inference Into Production With GPUs – Unite.AI
AI & Technology

Cerebras Puts Disaggregated Inference Into Production With GPUs – Unite.AI

November 17, 2026
Next Post
Israel orders ‘full lockdown’ of the West Bank

Israel orders 'full lockdown' of the West Bank

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Florida suspect caught after three-hour hideout in bushes

Florida suspect caught after three-hour hideout in bushes

November 14, 2026
Which NVIDIA GPU Is Closest To The PS5? The Answer Isn’t So Simple

Which NVIDIA GPU Is Closest To The PS5? The Answer Isn’t So Simple

November 15, 2026
FESM: Good Track Record, But Limited Upside Now

FESM: Good Track Record, But Limited Upside Now

November 17, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!