GeoDEX — retargeting a human hand onto 23 robot joints

The learned policy clearing the table: locate a fruit, reach, close the 16-DoF hand around it, place it on the plate, return to the home posture, and go again. No resets between objects.

Report (PDF) Code Try the retargeting The runs Disturbances

23DoF

Whole body

6-DoF xArm Lite 6 + 1-DoF wrist + 16-DoF ORCA hand

2

RGB-D views

Wrist-mounted egocentric + fixed overhead

60

Action chunk

2.0 s at 30 Hz, one sub-task long

96%

SR-1, reported

Single fruit, 50 rollouts (the report's number; my re-run gets 50 %)

+66pp

SR-2 gain, reported

From State Bottlenecking alone

2.2mm

Pinch error

Re-measured on the recorded hand data

Getting a human hand onto a robot hand

A human index finger and an ORCA index finger do not have the same link lengths, the same joint axes, or the same number of joints. Mapping joint angle to joint angle is not just inaccurate, it is invalid — the resulting robot pose is not the pose the operator made.

So the retargeter does not match joint angles, and it does not really match absolute fingertip positions either. It matches geometry, in the palm frame, with three terms that pull in different directions:

J_hand(q) = γ_pos Σ_i∈K ‖x_i(q) − x̂_i‖² + γ_cos(1 − S_C(X, X̂)) + γ_rel Σ_(i,j)∈P ‖(x_i−x_j) − (x̂_i−x̂_j)‖² + γ_smooth‖q − q_t−1‖² + γ_reg‖q‖²

Cosine similarity (γ_cos = 50) over the whole flattened keypoint cloud. This is what keeps the overall posture right — a fist reads as a fist — without demanding that the robot's shorter fingers reach where a human's do.
Pairwise relative distances (γ_rel = 100) between the thumb tip and each other fingertip, with a ×5 boost whenever the operator's pair is closer than 38.7 mm. This is the term that exists for pinches: the thing that has to survive is the gap, not where the gap is.
Smoothness and L2 (0.5, 0.02) to keep the solver in a natural rest posture when the target is ambiguous.

The arm is solved separately — 7 DoF tracking the operator's wrist transform by SLSQP with λ_p = 500, λ_R = 200, λ_s = 2 — and its palm pose is then treated as a fixed base for the finger problem. Solving all 23 jointly at 30 Hz is both too slow and too full of local minima.

The retargeted hand, driven live. The operator's fingers are tracked markerlessly by the Vision Pro at 200 Hz and streamed over ZeroMQ with a two-message high-water mark and zero linger, so the planner always consumes the newest frame instead of a queue of stale ones.

Tune the retargeting yourself

Left: the operator's hand as the Apple Vision Pro recorded it, 21 landmarks in the palm frame, nine seconds of a real pinch-and-release sequence. Right: the ORCA hand, its real MuJoCo meshes posed by the exported kinematics, tracking that hand through the released cost function, solved live in your browser. Every term of the cost is a slider; fingertips share a colour on both hands so you can see which one is off and by how much.

INTERACTIVE 3-D · OPERATOR AND ROBOT, SIDE BY SIDE

OPERATOR Vision Pro landmarks, palm frame

ORCA HAND 16 joints, the real meshes

loading the hand model…

γ_rel thumb–fingertip distances100The paper's term: pairwise thumb-to-fingertip vectors, ×5 when the operator pinches. Zero it and pinches stop closing. γ_cos keypoint cosine similarity50Whole-hand posture. Zero it and the pinch sharpens while the rest of the hand drifts. γ_smooth temporal coherence0.5Penalises jumps between frames; without it the solver jitters between equivalent poses. γ_pos absolute fingertip L20The naive joint-to-joint style. Turn it up with the others down to see why it was replaced. pinch gate39 mmOperator thumb–index distance below which the ×5 boost fires.

Finger size factors, human to robot (the calibration the pipeline applies to each finger's landmarks):

thumb0.79 index0.76 middle0.79 ring0.79 pinky0.87

Playback 1× Scrub orbit both views together

Drag either view to orbit. The solver is coordinate descent on the released cost with the hand's real joint limits; the offline harness uses SLSQP on the same objective, so the numbers differ by solver, not by cost.

More runs: one to five fruits

The released checkpoint (the sequential ACT model, joint-space actions from the wrist and overhead RGB-D cameras) rolled out on the xArm Lite 6 with the ORCA hand in the MuJoCo scene, with one to five fruits dropped at random inside the training spawn zone. Fifty-two episodes, one seed per episode, nothing cherry-picked out of the tally. These are my numbers, not the report's — the report quotes 96 % on a single fruit over 50 rollouts, and re-running the checkpoint here, in the multi-fruit scene and with the fruit required to come to rest on the plate, I get half that on one fruit and much less as fruits are added. The clips are the successes and, where there were none, the runs that got furthest; clips longer than 19 seconds are sped up.

38/152

fruits placed

over 52 episodes with one to five fruits on the table

50%

one fruit cleared

6 of 12 episodes, 6.4 s on average when it worked

31/52

got at least one

12 episodes never closed on a fruit at all; 9 lifted one and never let go over the plate

14.0s

to the first fruit

3.2–94 s across the 31 episodes that placed one

Loading the recorded runs…

The protocol, the thresholds, the figures and the caveats — including why my single-fruit number and the report's differ — are written up in the rollout study (PDF).

!

The failure is the second fruit, not the first

One fruit on the table: half the episodes end with it on the plate. Two: one in ten. Three or more: none of the thirty episodes cleared the table, though they placed 26 individual fruits between them. The break is after the first success. I logged the hand and every fruit at 10 Hz: in the 25 multi-fruit episodes that placed something, the palm afterwards sits a median 14 cm from the nearest remaining fruit and comes within 4 cm of one — it does go back and engage the next fruit, and 24 of the 70 remaining fruits get visibly nudged. But only 15 of those 70 (21 %) are ever lifted clear of the table again, and only 7 (10 %) reach the plate. It approaches and paws at the next fruit without closing on it. That is the freezing the report describes and treats with the sequential sub-task encoding, and it is not fixed here — a single fruit just hides it.

Disturbances: take the fruit away mid-grasp

Right after the hand lifts a fruit, something happens to it: it is knocked out sideways, taken out of the hand and put back elsewhere on the table, pressed down while held, or removed for good. The policy never saw any of this in training and it has no memory beyond its observation window, so this is a fair test of whether the visuomotor loop is closing on what is actually in front of it. Twenty-four attempts, fifteen of which got far enough to have a grasp to disturb (one of those took away the only fruit on the table, so there was nothing left to recover with). Of the fourteen real tests, three still finished the task, and the two clean re-grasps came back within about a second. Every disturbed run is below, recovery or not.

Loading the disturbance runs…

Place the fruits yourself

The policy is a 40-million-parameter transformer over two RGB-D cameras; it does not run in a browser. What runs here is the library of recorded episodes: drop fruits on the map of the table and the page finds the recorded episode whose starting layout is closest to yours, shows that layout over your own, and plays what the policy did. Click a fruit again to remove it.

INTERACTIVE · NEAREST RECORDED EPISODE

Drop one to five fruits on the table.

What each term is worth

The report asserts that the pairwise term is what preserves precision-pinch geometry. To check it I re-ran the released cost against the real ORCA MuJoCo model on 268 frames of recorded Vision Pro hand data, once per ablation, with the operator-to-robot palm frame solved by Kabsch alignment the way a teleoperation rig anchors at start-up.

Ablation of the retargeting cost terms: direction error, pinch error and solve time — **268 recorded frames, five cost configurations, on the ORCA hand.** Absolute fingertip position is deliberately not the objective — the ORCA hand's fingers reach 76–87 % as far from the palm as this operator's do, measured — so the metrics are the scale-free ones the cost actually claims: the angle between the robot and operator fingertip vectors in the palm frame, and the error in the thumb–index gap.

Cost configuration	fingertip direction [deg]	keypoint cos sim	pinch error, all frames [mm]	pinch error while pinching [mm]	aperture error [mm]	solve [ms]
full (released)	18.94	0.871	13.54	11.30	14.6	10.3
no pairwise (γ_rel=0)	13.27	0.881	25.13	42.20	19.1	5.5
no cosine (γ_cos=0)	40.36	0.765	13.89	4.24	16.8	9.2
no smoothness	18.37	0.873	13.28	9.06	15.2	13.6
naive keypoint L2	11.77	0.878	25.58	47.40	21.5	9.1

← swipe the table →

Thumb-index distance tracked over 9 seconds of recorded teleoperation — **Nine seconds of the operator's thumb–index gap, tracked three ways.** With the pairwise term the robot follows the human curve through every closure. Without it the robot systematically undershoots exactly where it matters — below the 38.7 mm gate, where the ×5 boost would have fired.

1

The pairwise term is worth 5× on pinches

Dropping γ_rel takes the thumb–index gap error from 11.3 mm to 42.2 mm on the frames where the operator is actually pinching — nearly four times worse — and from 13.5 mm to 25.1 mm overall. Naive absolute-keypoint retargeting is worse still at 47.4 mm, despite having the best fingertip direction of any configuration, 11.8° against the full cost's 18.9°. That is the whole point: getting each fingertip pointing the right way says nothing about whether the fingers can meet.

2

The cosine and pairwise terms fight

This one I did not expect. Removing the cosine term makes pinch tracking better — 4.2 mm instead of 11.3 mm while pinching — at the cost of more than doubling the fingertip direction error, 18.9° to 40.4°, and dropping keypoint cosine similarity from 0.871 to 0.765. The two terms are optimising different things over the same 16 joints, and γ_cos = 50 against γ_rel = 100 is the compromise between them. A pinch-gated cosine weight, mirroring the boost that already exists on the pairwise term, would probably get both.

Solve times here (5–14 ms median) are slower than the 2–4 ms the report quotes, because this harness runs SLSQP with Python-side MuJoCo forward kinematics per cost evaluation rather than the tuned in-process planner loop. The ratios between configurations are the comparable quantity, not the absolute times. The solver is fully converged: raising the iteration cap from 5 to 40 changes the direction error by less than 0.01°.

!

These numbers replace an earlier set that was measured on the wrong hand

Until September 2026 this section, and the 3-D playground above it, were built from teleop/xml/hand_camera_scene.xml — a Kyber hand that happens to live in the same repository, with 20 joints and its own link lengths. Everything downstream inherited it: the meshes, the forward kinematics, the joint limits, and every number in the table. The hand on this page was not the hand in the robot.

Re-exported from the model the project actually runs, ORCA/combined_lite6_orca/task2_scene.xml — 16 joints, bodies left_palm, left_thumb_mp…dp, left_index_mp…ip — the qualitative conclusions survive intact: the pairwise term is what makes pinching work, the cosine term is what holds posture, and the two trade against each other. The magnitudes do not. Fingertip direction error is 18.9° rather than 4.7°, because the real ORCA finger has three joints and no distal one, so it simply cannot point where a human fingertip points.

The per-finger size factors were wrong in the same way, and for the same reason: they were 1.15–1.40, implying a robot hand larger than the operator's. Measured against the ORCA model they are 0.76–0.87 — it is smaller. Correcting them halved the overall pinch error, from 30.0 mm to 13.5 mm, before any of the comparisons above were made.

Loading the right model then exposed a second bug that the wrong one had been hiding. MuJoCo turns a hinge by qpos − qpos0: body_pos and body_quat describe the body at the model's reference pose, not at zero. The browser's forward kinematics rotated by qpos alone, which is correct only when every reference angle is zero — as it happened to be on the Kyber hand. On the ORCA hand the thumb abduction reference is −0.733 rad, index −0.40, pinky +0.523, so those fingers were posed wrong by up to 93 mm at the fingertip while the middle finger, whose three reference angles are all zero, was pixel-perfect. Exporting the reference angle and subtracting it brings the browser's kinematics to agreement with MuJoCo to 0.00000 mm at every keypoint. The offline ablation was never affected — it calls MuJoCo directly.

The policy, and the thing that broke it

With demonstrations in hand the policy is an Action Chunking Transformer: two ResNet-18 towers over the wrist and overhead RGB-D views, an MLP over the 23-DoF proprioceptive state, a CVAE latent, and a decoder that emits a 60-step (2.0 s) action chunk. At deployment the chunks overlap and are combined by temporal ensembling with τ = 10, so every executed action is an exponentially weighted average over every prediction still in flight.

Two implementation details mattered more than the architecture:

Depth without catastrophic forgetting. Fusing a depth channel into a pretrained ResNet by re-initialising the first conv destroys the ImageNet features. Instead the original RGB weights are copied across and the depth channel is zero-initialised, so the network starts exactly where the pretrained model was and learns the depth dependence by gradient descent.
A grasp-weighted loss. The 16-DoF hand is static during a long reach and moves for a fraction of a second during the grasp; a plain L1 is dominated by arm error and the network converges to never moving the fingers. The hand term is therefore scaled by W_dyn = 1 + α‖a^hand_τ − a^hand_τ−1‖ with α = 50, and weighted 7× against the arm.

L = λ_armL_arm + λ_handL_hand + λ_sL_smooth + βD_KL W_dyn = 1 + 50‖a^hand_τ − a^hand_τ−1‖

ACT training curves — **ACT training.** AdamW at 1×10⁻⁵ with 500 warmup steps and gradient clipping at norm 1.0 — the low base rate is there because the attention weights blow up otherwise. Up to 5,000 epochs at batch 64, keeping the lowest-validation-L1 checkpoint.

3

State Bottlenecking: the operator has to hesitate on purpose

Clear one fruit and the policy is near-perfect. Clear four and it freezes. The reason is not capacity — it is that continuous teleoperation makes the dataset genuinely ambiguous. After a placement, two apples at near-equal distance produce identical images mapping to diverging action distributions, and a CVAE trained on that averages the modes instead of committing to one. The robot hovers between the two.

The fix is a protocol, not a loss. After securing an object the operator must return to a high-clearance home posture before reaching again. That makes the visual state at the start of every reach deterministic and unoccluded, which disentangles the action distribution the policy has to fit. It costs the operator a second per sub-task and it is the difference between a policy that clears a table and one that stalls on the second object.

Training dataset	SR-1	SR-2	SR-3	SR-4
Unstructured (uncut)	90 %	22 %	0 %	0 %
State-Bottleneck	96 %	88 %	23 %	3 %
Δ	+6 pp	+66 pp	+23 pp	+3 pp

Success rate over 50 independent rollouts per condition. SR-k is clearing k objects in sequence without a reset. The bottleneck protocol rescues SR-2 almost entirely; SR-3 and SR-4 stay hard, which is where residual RL on top of the frozen policy comes in.

Reproduce it

# the retargeting ablation, from a clean checkout — MuJoCo model and # recorded Vision Pro hand data are both in the repo git clone https://github.com/thejerrycheng/GeoDEX cd GeoDEX pip install mujoco numpy scipy matplotlib python scripts/tools/retarget_ablation.py # ~2 min -> docs/figures/ python scripts/tools/export_demo_data.py # what the playground above runs on

The full stack — the Vision Pro streaming client, the three-process ZeroMQ teleoperation loop, the ACT and diffusion training, and the MuJoCo scenes — lives in a separate private repository. The public one is the part that reproduces from a clean checkout: the released cost function, the ORCA hand model, the recorded hand data, and the study above.

The browser playground loads assets/data/geodex_hand.json — the ORCA finger kinematics and 134 calibrated frames of the recorded trajectory, both produced by the exporters above.

Credits

GeoDEX was the final project for DS-GA 3001 at NYU, joint work with Yipeng Wang. The ORCA anthropomorphic hand is open-sourced by ETH Zürich; the arm is an xArm Lite 6. Everything is evaluated in MuJoCo — the sim-to-real gap is explicitly unaddressed, and is the first thing the next version has to fix.

The teleoperation stack, the policies and the report are from the project. The retargeting ablation, the frame calibration, the exporters and the browser playground on this page were built afterwards, on the released cost function and the recorded hand data.

Report Code All projects