Robot manipulation datasets have grown far slower than the models trained on them, mostly because collection stays closed and centralized. Expert operators gather demonstrations on lab hardware, process them offline, and ship a fixed benchmark that never grows again. A research team from Axis Robotics, UC Berkeley, Georgia Tech, NTU… is proposing a different shape for the problem. Their system, AXIS, moves demonstration collection into the browser, sends everything else to backend GPUs, and treats the dataset as something that keeps expanding rather than something that ships once.
Is it deployable? Partially. The training code is public as a patch layer over OpenPI, and the teleoperation platform is live in any browser. The dataset on Hugging Face is gated at 2.36 TB and restricted to non-commercial academic use. No policy checkpoints are released.
The browser and backend split
The core system decision is asymmetry. Contributors teleoperate a Franka Research 3 with a parallel-jaw gripper inside a MuJoCo WebAssembly frontend, using keyboard, mouse, virtual joystick or gamepad. Physics stepping and Three.js rendering run off the React UI thread, so logged state-action samples stay aligned with the simulator rather than the interface. Everything expensive happens elsewhere: rendering on 8x RTX 4090 GPUs, training and evaluation on 8x A100 GPUs.
Tasks themselves are generated rather than hand-authored. TaskGen decomposes a language instruction into task, scene and object configs, retrieves or generates meshes through an image-to-3D pipeline, rescales them to plausible physical size, then proposes a 2.5D layout. A layout supervisor validates the instantiated scene and relocates, reorients or regenerates objects when constraints fail. Every task ships with a structured success checker, which the backend re-runs rather than trusting the frontend success flag.
What the dataset contains
The released snapshot holds 207 tasks, 50,129 episodes and more than 60K task or scene variants across seven scene categories. Each trajectory carries task metadata, embodiment, simulator version, robot and object states, actions, success labels, and third-view plus wrist RGB-D observations. The paper credits more than 70,000 community members with contributions.
Cleaning is treated as a production stage. Samples with joint variation below 5e-3 are dropped as static, a Savitzky-Golay filter with window 15 and polynomial order 3 smooths continuous motion, and cubic splines resample from the 6 Hz to 8 Hz the web interface produces up to a 20 Hz target. Table 1 is honest about the tradeoff: mean acceleration drops from 1.3539 to 0.4885 and mean jerk from 11.5899 to 2.2243, while replay success falls from 100% to 86.2%.
Cleaned episodes are then replayed in IsaacSim from packed simulator state with physics stepping disabled, so the verified trajectory stays authoritative while scenes, cameras, materials and lights are randomized around it. Output is 256×256 ray-traced RGB from a fixed third-view camera and a wrist camera, with depth off by default.
Results on LIBERO-Plus
Every condition starts from the released π0.5 checkpoint, a PaliGemma Gemma-2B backbone with a Gemma-300M action expert, optionally continues pretraining on a sim corpus, then fine-tunes on LIBERO with identical hyperparameters. Pretraining is full-model with no LoRA, using a flow-matching loss over 10-step action chunks for 100,000 steps, followed by 30,000 steps of LIBERO post-training.
π0.5 plus AXIS-100% reaches 88.8 overall on LIBERO-Plus against 83.9 for vanilla π0.5 and 57.5 for a RoboCasa365 control matched on trajectory count. The abstract quotes 5.8% and 37.3%; both are relative figures normalized by the 83.9 baseline, so the point gaps of 4.9 and 31.3 are the cleaner read. Scaling holds at the aggregate level, 84.7 to 85.7 to 88.8 across the 25%, 50% and 100% snapshots.
Per axis, the biggest gains land where the augmentation pipeline actually randomizes: Sensor Noise +13.7 and Camera +11.3. Background gains 3.7, Robot pose 3.8, Layout 2.6. Light and Language regress, by 1.7 and 1.3. Camera also dips to 68.8 at AXIS-50%, below the 72.5 baseline, before recovering. Scaling is consistent in aggregate and noisy per axis.
Interactive explainer
Credit: Source link
