Every simulation/benchmark integration in the repo: what is live, what was pruned, where the minimal code lives, the command that runs each basic eval loop, and which lane carries the paper.
0025ea1benchmark_adapters/registry.py)Dispatch happens in exactly one place: src/roboprog/benchmark_adapters/registry.py, reached from python -m roboprog.run, eval_suite.py, and eval_bench.py.
| Benchmark | Status | Simulator | Adapter | Basic loop |
|---|---|---|---|---|
| LIBERO 5 suites: spatial / object / goal / 10 / 90 |
LIVE | robosuite 1.4.1 / MuJoCo 3.9 vendored at benchmarks/LIBERO |
src/roboprog/benchmark_adapters/libero/ | python eval_suite.py --suite libero_object --res 256 --seeds 0,1,2 --tag gtseg |
| LIBERO · frozen OAT harness method-of-record protocol |
LIVE | same MuJoCo, path-bound frozen runtime (CPU + OSMesa default) | scripts/method.py → configs/CURRENT_METHOD.json | python scripts/method.py eval executor --ckpt a.pt,b.pt --pool selection |
| RoboCasa kitchen pick-place, PandaOmron |
LIVE | robosuite 1.5 / MuJoCo own venv (conflicts with main venv) |
src/roboprog/benchmark_adapters/robocasa/ | python eval_bench.py --benchmark robocasa --tasks PickPlaceCounterToSink --seeds 0,1,2 |
| real FrankaPy + RealSense; mock-first |
LIVE (mock) | none (hardware); mock backend default | src/roboprog/benchmark_adapters/real/ | python -m roboprog.run real --task "pick up the red mug" --workflow agent --no-mocks --real-config real_live.json |
| Lane | Status | Simulator | Code | Basic loop |
|---|---|---|---|---|
| praxis-eval bridge OAT paper's own harness: MetaWorld MT50 + RoboMimic mt4 |
BRANCH | MuJoCo (MetaWorld) / robosuite (RoboMimic) | branch praxis-eval-bridge: scripts/praxis_bridge/ (commits b31babf, 00e984a) | MUJOCO_GL=egl PYTHONPATH=src $PE scripts/praxis_bridge/eval_metaworld.py --ckpt s0.pt,s1.pt --tasks easy --n 10 --out outputs/mw_eval |
| TacEx tactile (peg-insert / GearMesh / NutThread) home of the dp / rdp baselines |
LIVE | Isaac Lab (GPU tactile), 64–128 envs | scripts/tactile/tacex_*.py + tacex_v4/v5_study.sh | TACEX_STUDY_DIR=<run> ARCH=rdp bash scripts/tactile/tacex_v4_study.sh train_dprdp |
| ManiFeel / RoTO tactile-required benchmarks |
LIVE | ManiFeel offline (Zarr) / RoTO closed-loop (Isaac Sim) | scripts/tactile/asymmetry_*.{py,sh} · scripts/tactile/roto_*.{py,sh} | per-lane .sh drivers; own venvs (manifeel-replay-ebfa9e17, roto-venv) |
| Wan visual planner (G1) Wan2.2-TI2V-5B service |
BRANCH | video world-model service (no sim) | branch agent/wan-g1-bringup @ 4641e27 (unpushed) | pinned service + 81-frame gate; see prereg config on branch |
60f990c, 2026-08-04)Directories still exist but are empty; a regression test (tests/test_benchmark_registry.py) enforces they stay unroutable. Full source recoverable at 0025ea1.
| Benchmark | Simulator it wrapped | Why it ended |
|---|---|---|
| RoboTwin 2.0 | SAPIEN, dual-arm aloha-agilex | ours floored 0–2/30 with in-domain data (grasp trigger structural); GPC DP hit 19/20 @ 200 demos on the same data |
| MolmoSpaces | MuJoCo 3.5, Franka-DROID | CEM-always lifted MS-Pick 36.7→63.3%, then lane closed |
| Isaac DROID (sim-evals) | Isaac Lab 2.2 / Isaac Sim 5.0 | pilot 2/3 honest; not carried forward |
| SimplerEnv | SAPIEN (google_robot, widowx) | no result artifacts ever produced |
| TacSL / Taccel | IsaacGym Preview / Warp FEM | IsaacGym can't run on Blackwell; superseded by TacEx lane |
| Language-Table | Google push env | never merged (reverted 1b6b650) |
Torch-only ports living in scripts/tactile/ — dp = Diffusion Policy (ConditionalUnet1D, DDPM squaredcos 100 steps, DDIM sampling, To=2, Tp=16, EMA); rdp = Reactive Diffusion Policy (asymmetric action-chunk VAE tokenizer + latent diffusion; slow loop re-diffuses every Ta=8, fast loop re-decodes every step from tactile obs). Reference: xiaoxiaoxh/reactive_diffusion_policy @ 824c5e8.
scripts/tactile/tacex_dprdp_model.py — dp + rdp model definitionsscripts/tactile/tacex_dprdp_bc.py — training driver (--arch dp|rdp|dp_blind|dp_mono|dp_rw|…)scripts/tactile/tacex_dprdp_rollout.py — closed-loop Isaac Lab eval with tactile-ablation conditionsrc/roboprog/learned/ddpm_action.py — in-repo DP-style DDPM head on the bc3 trunk (for LIBERO/RoboCasa lanes)TRAIN_PY=/mnt/nvme1/haonan/robot_envs/roboprog-conda/bin/python
TACEX_STUDY_DIR=<run> ARCH=dp SEEDS_OVERRIDE="0 1 2" bash scripts/tactile/tacex_v4_study.sh train_dprdp
TACEX_STUDY_DIR=<run> ARCH=rdp SEEDS_OVERRIDE="0 1 2" bash scripts/tactile/tacex_v4_study.sh train_dprdp
# underlying:
"$TRAIN_PY" scripts/tactile/tacex_dprdp_bc.py --arch rdp --seed 0 \
--data prof.npz corr.npz --tactile-data corr.npz --tactile-episodes 4 \
--out ckpt/rdp_t4_s0.pt
TACEX_PY=/mnt/nvme1/haonan/robot_envs/tacex-venv/bin/python # Isaac Lab venv TACEX_STUDY_DIR=<run> CKPTS="ckpt/dp_t4_s0.pt ckpt/rdp_t4_s0.pt" OUT=eval_dprdp GPU=0 \ bash scripts/tactile/tacex_v4_study.sh eval_dprdp # underlying: scripts/tactile/isaac_watchdog.sh "$RUNS/$OUT.json" 120 -- \ env OMNI_KIT_ACCEPT_EULA=YES CUDA_VISIBLE_DEVICES=0 timeout 28800 \ "$TACEX_PY" scripts/tactile/tacex_dprdp_rollout.py \ --task TacEx-Factory-PegInsert-Direct-v0 --ckpt <ckpts...> \ --num-envs 128 --episodes 256 --sensor force_hist --force-hist-k 8 \ --seed 1000 --out "$RUNS/$OUT.json"
Current verdict (v7, PR #128): no tactile arm beats plain dp on PegInsert — dp is the unbeaten baseline there. One confirmed positive: GearMesh dp and rdp-t64 are anti-reliant 3/3 (−40 = 2.9σ, −65 = 4.9σ). Power note: per-cell session SD ≈ 6.6–6.8 episodes → ~14 seeds per arm needed to resolve 2.5pp; comparators must run in the same Isaac session.