NVIDIA released Cosmos3-H-Dreams on October 20, 2026, a real-time, action-conditioned generative world model for robotic surgery fine-tuned on clinical cholecystectomy recordings from the CMR Surgical Versius system. The model checkpoint and companion code were published the same day.
In an NVIDIA-authored announcement on Hugging Face, the company said that, to its knowledge, Cosmos3-H-Dreams is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings to support live kinematic control. Powered by Cosmos 3 and accelerated with FlashDreams, NVIDIA’s streaming-inference library, the model runs interactively on a single NVIDIA RTX PRO 6000 GPU, according to the announcement.
The model card lists release dates of October 20, 2026 on both GitHub and Hugging Face and states the model is ready for commercial or non-commercial use under the NVIDIA Open Model License. It names surgical robotics policy evaluation in simulation, interactive teleoperated world-model exploration, and synthetic data generation for robotic cholecystectomy as intended uses, aimed at surgical robotics researchers, healthcare AI developers, academic institutions, and surgical robotics companies.
From Tabletop Suturing to Clinical Scenes
Cosmos3-H-Dreams succeeds Cosmos-H-Dreams, a model built on Cosmos-Predict2.5 that targeted dVRK tabletop suturing. As a world foundation model, the earlier system learned visual dynamics directly from synchronized video and robot kinematics rather than explicitly modeling scene geometry, tissue deformation, and physical interactions. The new release moves that approach into clinical scenes, where tissue stretches under traction, reflections shift, structures become occluded, and smoke changes visibility, and where small generation errors can accumulate over long rollouts and gradually alter instrument geometry or anatomical structure.
Architecture and Distillation Pipeline
The model builds on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model trained on the Open-H-Embodiment dataset, and specializes a bidirectional teacher on curated cholecystectomy video and kinematics. Cosmos3-H-Dreams itself has approximately 15.17 billion parameters and uses a Mixture-of-Transformers architecture that pairs a Qwen3-VL-8B-Instruct reasoning tower with a latent-diffusion generation expert, operating on Wan2.2 VAE latents with action embedding layers for kinematic conditioning.
The action interface uses 28 dimensions per time step, with 14 dimensions per arm covering anchor-relative translation, 6D rotation, jaw position, electrosurgical energy activation, a thumbstick button, and thumbstick axes. The interface was extended from 20 dimensions for the Versius embodiment, with pretrained pose and jaw weights preserved and the new control inputs zero-initialized. Training uses 832 x 480 video with actions aligned at 10 Hz, a sampling rate the announcement notes is distinct from runtime generation throughput.
The bidirectional teacher was fine-tuned first on 24 future frames for 2,000 iterations and then on a 72-frame horizon for 4,400 iterations, corresponding to 2.4 and 7.2 seconds at 10 Hz. Distillation converted the 72-frame teacher, which used 35 sampling steps, into a block-causal student through causal teacher forcing over 1,000 iterations, followed by 300 iterations of Self-Forcing with distribution-matching distillation against the fixed teacher. The released student generates video autoregressively with a key/value cache and four denoising steps per latent frame; each latent frame consumes four future action vectors (0.4 seconds at 10 Hz) and produces four new video frames, enabling streaming rollout.
A second release checkpoint uses two denoising steps. NVIDIA states the two-step model is separately trained rather than the four-step model run with fewer sampling steps, and that fewer denoising steps yield faster inference at the expense of lower visual quality. Streaming runs through FlashDreams with an adapted FadeMem module that consolidates the growing key/value cache during long rollouts, complementing Self-Forcing’s handling of generated history.
Training Data and Held-Out Evaluation
Fine-tuning used proprietary CMR Surgical Versius clinical cholecystectomy recordings drawn from 4,792 episodes totaling roughly 17 million frames at 60 Hz and 1080p resolution, subsampled to 10 Hz. Surgeons teleoperated the robot during the procedures; action labels were derived automatically from robot telemetry, and instrument visibility was labeled manually.
Sample selection retained only windows in which exactly two instruments were visible and engaged for every native 60 Hz row of each history and future window, with a left Cadiere grasper and a right monopolar hook as the canonical pairing, and excluded windows with active camera commands above a fixed threshold. The training set comprised 64 recordings and about 526 unique minutes of video, with 183,528 window starts for the 24-frame horizon and roughly 436 minutes across 106,820 window starts for the 72-frame horizon used in distillation. Recordings were split at the episode level, with about 98 percent of unique minutes used for training and about 2 percent held out.
Held-out validation covers approximately 10.6 minutes from 10 recordings, around 6,400 frames at 10 Hz. Fixed evaluation panels drawn from these recordings each contain 48 quality cases plus 12 action-control cases, supplemented by rotation probes at neutral, ±5°, and ±15° tool rotations and 30.4-second continuous rollouts for visual review. The card describes the content as intra-abdominal endoscopic video and robot telemetry from clinical procedures on human patients, showing anatomy and surgical instruments rather than faces or other identifying views, with no audio or text.
Performance, Licensing, and Code
The model card reports 30.0 frames per second at 832 x 480 with four diffusion steps and 43.2 FPS with two steps on an RTX PRO 6000 Blackwell Workstation Edition, 20.2 and 31.2 FPS on the Max-Q edition, and 16.9 FPS on DGX Spark at a reduced 512 x 288 resolution. Real-time interactive serving was validated on a single RTX PRO 6000 and on DGX Spark, and the model also runs on a Hopper H100. Supported microarchitectures are NVIDIA Blackwell and Hopper, Linux is the supported operating system, and BF16 mixed precision was used for training and native inference, with weights exported as FP32.
The companion GitHub repository is licensed under Apache-2.0 and supports replaying recorded actions to MP4 output as well as live control from a WebXR-compatible headset with two tracked controllers. The repository lists system requirements including Linux with a CUDA-capable GPU, Python 3.12, the uv package manager, and more than 32GB of VRAM at the default 480 x 832 configuration.
NVIDIA also published a step-by-step recipe for teacher fine-tuning and Self-Forcing distillation so developers can adapt the system to other datasets and embodiments with synchronized video and compatible action representations; the announcement notes controller integrations must use the same action conventions and normalization as training. The model card’s ethical considerations state that developers must hold proper rights and permissions for input image and video content, and that generated video will not blur people or personal health information included in inputs.
Stated Limitations and Research Directions
NVIDIA states that visual realism is a starting point for validation rather than proof of biomechanical accuracy, and that a plausible-looking rollout that ignores control commands cannot serve as an interactive environment. The announcement lists research directions that include modeling the full cholecystectomy procedure with clear task boundaries and success criteria, connecting generative and numerical simulation, evaluating with surgeons, supporting more instruments, and developing autonomous surgical policies coupled to the model in a closed loop. NVIDIA frames these as research directions rather than established clinical capabilities and states that Cosmos3-H-Dreams is a research simulator, not a validated system for patient care.
Credit: Source link
