• bitcoinBitcoin(BTC)$64,003.00-1.41%
  • ethereumEthereum(ETH)$1,795.43-1.23%
  • tetherTether(USDT)$1.00-0.01%
  • binancecoinBNB(BNB)$588.91-0.51%
  • usd-coinUSDC(USDC)$1.000.00%
  • rippleXRP(XRP)$0.991.16%
  • solanaSolana(SOL)$74.310.53%
  • tronTRON(TRX)$0.3095831.23%
  • Figure HelocFigure Heloc(FIGR_HELOC)$1.010.00%
  • HyperliquidHyperliquid(HYPE)$68.71-1.27%
  • zcashZcash(ZEC)$931.39-2.48%
  • moneroMonero(XMR)$504.84-2.09%
  • dogecoinDogecoin(DOGE)$0.059302-0.23%
  • leo-tokenLEO Token(LEO)$9.330.05%
  • USDSUSDS(USDS)$1.000.00%
  • whitebitWhiteBIT Coin(WBT)$61.81-1.45%
  • chainlinkChainlink(LINK)$9.15-1.72%
  • cardanoCardano(ADA)$0.153852-0.75%
  • stellarStellar(XLM)$0.140804-0.09%
  • Ethena USDeEthena USDe(USDE)$1.00-0.03%
  • daiDai(DAI)$1.000.00%
  • nearNEAR Protocol(NEAR)$3.352.71%
  • bitcoin-cashBitcoin Cash(BCH)$208.45-1.96%
  • RainRain(RAIN)$0.004873-11.32%
  • litecoinLitecoin(LTC)$44.19-2.19%
  • CantonCanton(CC)$0.083304-0.71%
  • avalanche-2Avalanche(AVAX)$7.12-2.01%
  • uniswapUniswap(UNI)$5.23-3.36%
  • tether-goldTether Gold(XAUT)$3,757.84-2.18%
  • suiSui(SUI)$0.73-1.73%
  • the-open-networkGram (prev. Toncoin)(GRAM)$1.02-0.55%
  • USD1USD1(USD1)$1.000.03%
  • Ripple USDRipple USD(RLUSD)$1.00-0.01%
  • Global DollarGlobal Dollar(USDG)$1.000.00%
  • hedera-hashgraphHedera(HBAR)$0.057362-1.50%
  • paypal-usdPayPal USD(PYUSD)$1.000.00%
  • Ondo US Dollar YieldOndo US Dollar Yield(USDY)$1.15-0.01%
  • BittensorBittensor(TAO)$195.250.27%
  • shiba-inuShiba Inu(SHIB)$0.000004-0.63%
  • crypto-com-chainCronos(CRO)$0.043320-3.09%
  • quant-networkQuant(QNT)$154.50-1.08%
  • MemeCoreMemeCore(M)$0.88-1.81%
  • Circle USYCCircle USYC(USYC)$1.140.01%
  • Spiko Amundi Overnight Swap Fund (EUR)Spiko Amundi Overnight Swap Fund (EUR)(EURSAFO)$1.12-0.18%
  • okbOKB(OKB)$87.300.39%
  • aaveAave(AAVE)$110.73-2.58%
  • pax-goldPAX Gold(PAXG)$3,759.11-2.23%
  • Falcon USDFalcon USD(USDF)$1.000.03%
  • United StablesUnited Stables(U)$1.00-0.06%
  • BlackRock USD Institutional Digital Liquidity FundBlackRock USD Institutional Digital Liquidity Fund(BUIDL)$1.000.00%
TradePoint.io
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop
No Result
View All Result
TradePoint.io
No Result
View All Result

Microsoft Researchers Propose NUWA-XL: A Novel Diffusion Over Diffusion Architecture For Extremely Long Video Generation

July 21, 2023
in AI & Technology
Reading Time: 5 mins read
A A
Microsoft Researchers Propose NUWA-XL: A Novel Diffusion Over Diffusion Architecture For Extremely Long Video Generation
ShareShareShareShareShare

The field of generative models has recently seen a surge of interest in visual synthesis. High-quality image generation is possible in previous work. However, the duration of videos presents greater difficulties in practical applications than photos. The average running time of a feature film is over 90 minutes. The average length of a cartoon is 30 minutes. The ideal size for a video on TikTok or another similar app is between 21 and 34 seconds.

Microsoft’s research team has developed an innovative architecture for making long videos. Most existing work generates long movies segment by segment sequentially, which usually leads to the gap between training on short films and inferring large videos. The sequential generation could be more efficient. This novel method instead uses a coarse-to-fine process, where the video is generated simultaneously at the same granularity; after applying a global diffusion model to produce the range-wide keyframes, local diffusion models are used to fill in the material between adjacent frames iteratively. The training-inference gap can be narrowed through direct training on long movies, and all parts can be generated simultaneously using this straightforward yet successful approach.

The most important contributions are as follows:

🚀 Build high-quality training datasets with Kili Technology and solve NLP machine learning challenges to develop powerful ML applications
  • NUWA-XL, a “Diffusion over Diffusion” architecture, has been proposed by the research team because they see the creation of long videos as a revolutionary “coarse-to-fine” process.
  • NUWA-XL is the first model directly trained on long films (3376 frames), bridging the training-inference gap for generating such videos.
  • Parallel inference is made possible by NUWA-XL, which drastically shortens the time required to generate lengthy videos. When producing 1024 frames, NUWA-XL accelerates inference by 94.26 percent.
  • To ensure the model’s efficacy and supply a standard for extended video creation, the research team at FlintstonesHD created a new dataset called FlintstonesHD.

Methods

Temporal KLVAE (T-KLVAE)

KLVAE transforms an input image into a low-dimensional latent representation before applying the diffusion process to avoid the computational burden of training and sampling diffusion models directly on pixels. Researchers propose Temporal KLVAE(T-KLVAE) by augmenting the original spatial modules with external temporal convolution and attention layers to transfer superficial knowledge from the pre-trained image KLVAE to videos. 

Masked Diffusion in Time (MTD)

As a foundational diffusion model for the proposed Diffusion over Diffusion architecture, researchers present Mask Temporal Diffusion (MTD). While the “coarse” storyline of the movie is formed only from L prompts for use in global diffusion, the opening and last frames are also used as inputs for local distribution. The suggested MTD is compatible with global and local diffusion and can take input conditions with or without beginning and last frames. In the following, they lay out the MTD pipeline in its entirety before using an UpBlock to illustrate the fusion of various input circumstances.

There are still some restrictions, even though the proposed NUWA-XL boosts the quality of extended video creation and quickens the inference speed: First, researchers only validate the efficacy of NUWA-XL on publicly available cartoon Flintstones because open-domain long videos (such as movies and TV episodes) are not now known. With preliminary successes in creating an open-domain long video dataset, they hope to extend NUWA-XL to the open domain eventually. Second, the training-inference gap can be narrowed through direct training on long movies, but this presents a formidable obstacle for data. Finally, although NUWA-XL can speed up inference, this improvement requires a powerful graphics processing unit (GPU) to facilitate parallel inference.

Researchers suggest NUWA-XL, a “Diffusion over Diffusion” architecture, by framing the creation of long videos as an unconventional “coarse-to-fine” procedure. NUWA-XL is the first model directly trained on lengthy films (3376 frames), bridging the training-inference gap in long video production. The parallel inference is supported by NUWA-XL, which speeds up the creation of long videos by 94.26 percent while producing 1024 frames. To further verify the model’s efficacy and offer a benchmark for extended video creation, they construct FlintstonesHD, a new dataset.


Check out the Paper and Project. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 26k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.


YOU MAY ALSO LIKE

Plaid Connects Financial Accounts to Instinct’s AI Assistant – Unite.AI

NASA’s Voyager 1 Space Probe Is Now One Light-Day Away From Earth

Dhanshree Shenwai is a Computer Science Engineer and has a good experience in FinTech companies covering Financial, Cards & Payments and Banking domain with keen interest in applications of AI. She is enthusiastic about exploring new technologies and advancements in today’s evolving world making everyone’s life easy.


🔥 Gain a competitive
edge with data: Actionable market intelligence for global brands, retailers, analysts, and investors. (Sponsored)

Credit: Source link

ShareTweetSendSharePin

Related Posts

Plaid Connects Financial Accounts to Instinct’s AI Assistant – Unite.AI
AI & Technology

Plaid Connects Financial Accounts to Instinct’s AI Assistant – Unite.AI

November 18, 2026
NASA’s Voyager 1 Space Probe Is Now One Light-Day Away From Earth
AI & Technology

NASA’s Voyager 1 Space Probe Is Now One Light-Day Away From Earth

November 18, 2026
Marble Draws .5M in Seed Funding After Signing 150+ Restaurants – Unite.AI
AI & Technology

Marble Draws $5.5M in Seed Funding After Signing 150+ Restaurants – Unite.AI

November 18, 2026
Palmer Luckey Says His SLAB Seastead Is ‘Way Cooler’ Than A Floating Data Center
AI & Technology

Palmer Luckey Says His SLAB Seastead Is ‘Way Cooler’ Than A Floating Data Center

November 18, 2026
Next Post
‘Bloomberg Technology’ Full Show (01/08/20)

'Bloomberg Technology' Full Show (01/08/20)

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Search

No Result
View All Result
Danish artist holds exhibit for trash-built trolls

Danish artist holds exhibit for trash-built trolls

November 18, 2026
Trump: ‘I thought I could relax’ over rally cancellations

Trump: ‘I thought I could relax’ over rally cancellations

November 13, 2026
APA Issues Workplace AI Guidance on Trust, Oversight and Job Design – Unite.AI

APA Issues Workplace AI Guidance on Trust, Oversight and Job Design – Unite.AI

November 16, 2026

About

Learn more

Our Services

Legal

Privacy Policy

Terms of Use

Bloggers

Learn more

Article Links

Contact

Advertise

Ask us anything

©2020- TradePoint.io - All rights reserved!

Tradepoint.io, being just a publishing and technology platform, is not a registered broker-dealer or investment adviser. So we do not provide investment advice. Rather, brokerage services are provided to clients of Tradepoint.io by independent SEC-registered broker-dealers and members of FINRA/SIPC. Every form of investing carries some risk and past performance is not a guarantee of future results. “Tradepoint.io“, “Instant Investing” and “My Trading Tools” are registered trademarks of Apperbuild, LLC.

This website is operated by Apperbuild, LLC. We have no link to any brokerage firm and we do not provide investment advice. Every information and resource we provide is solely for the education of our readers. © 2020 Apperbuild, LLC. All rights reserved.

No Result
View All Result
  • Main
  • AI & Technology
  • Stock Charts
  • Market & News
  • Business
  • Finance Tips
  • Trade Tube
  • Blog
  • Shop

© 2023 - TradePoint.io - All Rights Reserved!