IRIS

The thesis flew an Intel RealSense. The next IRIS carries a real camera: a full-frame Sony α7R III (35.9 × 24 mm) and a 16–150 mm zoom whose zoom and focus rings are driven by servos on a bracket under the lens, so a shot can command focal length and focus distance as continuous tracks, slew-rate limited like any other axis. The simulator models the field of view, the barrel extension, the depth of field from the thin-lens circle of confusion, and the two effects that need the arm and the lens to move together.

Dolly zoom. The arm pulls the lens back along the optical axis while the zoom servo pushes in, holding the product's size in the frame; the room behind it collapses. The reverse, pushing in while zooming out, makes it rush away. Two details decide whether that reads on screen. The first: the size that has to stay fixed is the silhouette of a body with a radius, not a point at the subject distance — a sphere of radius r at distance d subtends a half-width r/√(d² − r²), so the focal track carries the same square root. Under the point law the width instead drifts as 1/√(1 − (r/d)²) ≈ 1 + ½(r/d)², which is why every reference gives the point law and is right to: at a portrait distance, r/d ≈ 0.05, the error is under 0.15 % and nobody could see it. This is a macro shot — r/d runs from 0.43 to 0.27 — and there it costs 6.9 % of the subject's width across the move. The second: the subject has to fit in the frame. The carrot cake is 225 mm across (measured in the scene, not the 98 mm the spec's height field lists), so the old 70 mm preset framed it at 1.5 frame widths — it ran off both sides and only a 2 mm slice of it was in focus. Both shots now run the arm from 0.26 m to 0.42 m along the optical axis with the cake at 65 % of the frame width, 24 → 42 mm at f/16.

The planner's coefficient of variation on the subject's width is — for the dolly zooms against — for a plain dolly over the same travel. As executed — servo lag, joint tracking and all — the telemetry gives 0.1 % for each dolly zoom. And segmenting the cake out of the rendered frames themselves, the width the audience sees holds inside a 2.4 % band across the shot, while the plain dolly's swings 38 %.

Rack focus. Only the focus servo moves: near product, far product, back, with the aperture open to f/2.8 so the pull is visible. Object tracking. A toy car laps the desk and the wrist keeps it centred, either from the car's true position or from a pixel tracker running on the feed itself, the same loop the studio's policy tab uses to read images.

The arm in the studio is the thesis's MuJoCo model, body for body: the fixed pose of each link in its parent, the hinge axis, the joint range. A forward-kinematics and Jacobian twin in Python is checked against MuJoCo to 10⁻⁹ m; the browser runs the same equations. The tool frame is the lens's entrance pupil, 14 cm ahead of the mount on the optical axis, so a shot's camera position means the lens, not the flange.

Each frame the solver takes the target lens pose (position and a look-at orientation with a level horizon) and runs damped least squares from the previous joint vector; when a pose is at the edge of reach it retries from eight canonical postures and keeps the closest. Joints 4 and 6 are continuous-rotation motors, so paths through ±180° are taken the short way round. A 1.5 rad/s per-joint cap keeps the motion cinematic; the whip pan is allowed 3 rad/s.

The planner samples every preset at 30 Hz, solves the IK along it and asks MuJoCo for the joint torques by inverse dynamics with the 1.5 kg camera payload. It found the shots the first drafts could not do: a low 360° orbit folds the wrist past its ±100° pitch, a straight-down top view sits on the look-at singularity, and any lens position closer than about 30 cm to the base column folds the arm onto itself. The library on this page is what survived, with the numbers in the table above.

Then the framing was solved rather than guessed. A subject of width W and height H at distance d fills (f/1000)·max(H/0.024, W/0.0359)/d of a full-frame image, so once the arm decides how far away the lens can be, the focal length is the only free variable. Every preset was re-placed against the products' measured sizes — the carrot cake is 225 mm across, the cup 142 mm tall — with four constraints checked along the whole path: the arm reaches every pose, no joint exceeds 95°/s, the lens never comes closer than 100 mm to anything on the desk, and it never focuses nearer than the lens's 200 mm minimum. An earlier draft ended its hero push-in 75 mm from the coffee cup, inside both limits, and the orbits swept the lens 80 mm past it. The widest horizontal orbit that survives all four is 100°: past that the wrist runs out of twist while holding the horizon level, which is why there is no 180° orbit in the library and why a full turn is done on a motorised turntable instead.

The Record & train tab of the studio reproduces the thesis's loop at toy scale. Record saves an episode at 10 Hz: joint angles, focal length, focus distance and a 24 × 16 grey thumbnail of the feed, tagged with the shot and the product positions. Demo every preset records the whole library in one go at triple speed. Train fits a VisionJointPlanner-lite in TensorFlow.js: a five-frame window of states and the thumbnail through a tiny CNN, a goal token, a 256-wide MLP, and the next five joint and lens deltas out (action chunking); training takes seconds. Run hands the policy the arm with ACT-style temporal ensembling over the chunks, until it finishes or stalls.

It is honest about what it is: a few thousand samples and a few hundred thousand parameters, enough to reproduce a shot it has seen and to show the failure modes the thesis met at full scale, the zero-action attractor above all. Export the dataset as JSON to train something bigger offline.

Object tracking from the feed

The toy car laps the desk and the wrist keeps it centred, either from its true position or from a colour tracker reading the feed: segment saturated red, take connected components of a 132 × 88 read-back, score them by redness rather than by size, and steer the optical axis with a proportional law applied to the camera's measured pose.

Scoring by size was the first bug: the carrot cake's brown crust covers more pixels than the car, so the arm politely tracked the cake. Chromaticity separates them cleanly (car ≈ 0.85, crust ≈ 0.65). Two attempts to lead the target with an estimated angular rate made things worse, because the rate had to be estimated from the controller's own corrections and the loop amplified them.

Result. With ground truth the car stays within a fraction of a percent of frame centre. The tracker holds it to 13.4 % of the frame at 28 mm and 15.5 % at 45 mm, with residuals (what it still sees off-centre) of 9.5 % and 6.0 %, and it sees nothing in 16–18 % of frames. At 85 mm the field of view is 16°, the car leaves the frame, and a reactive tracker with no search behaviour cannot get it back: lost in 62 % of frames.

Learning a shot from demonstrations

The studio recorded 80 episodes (every non-tracking preset on all four product sets) at 10 Hz, and trained the VisionJointPlanner-lite in the tab: five past states plus a 24 × 16 thumbnail of the feed and a goal token in, the next five joint and lens deltas out, with temporal ensembling at run time.

It learns the shape of a move and then drifts. TensorFlow.js validates on the tail of the data, so the held-out set is whole shots the model never saw: the training loss reaches 0.0025 against 0.22 held out. Where it starts decides everything. Rolled out from the home pose, a state no demonstration ever visits, it ends on average 752 mm from where the demonstration ended. Started at the shot's own first pose, as a deployment would after the thesis's goal-directed start, that falls to 89 mm and the zoom lands within a few millimetres of the right focal length. That order of magnitude is the distribution shift the thesis fixed with a goal-directed start and a stagnation detector.

It is a working data-collection and training loop at toy scale, not a result: the thesis needed 17,897 samples where this has a few thousand. Record your own, export the JSON, and train something bigger offline.

# IRIS_studio/experiments
python -m pytest -q tests          # FK, Jacobian vs MuJoCo; look-at IK
python plan_shots.py               # every preset -> out/shots_plan.json + site copy
node record_videos.js              # drive iris_studio.html headless (GPU), 30 fps -> mp4 + telemetry
python make_figures.py             # docs/figures/*.pdf, site PNGs, numbers
cd ../docs && make                 # iris_studio.pdf

The spec that both sides read is assets/data/iris_studio.json: the desk, the product sets, the car's track and the 22 shots. The Python package irisstudio (kinematics, shots) and the browser modules assets/js/iris/{kin,lens,shots,scene,dof,policy,app}.js implement the same rules, so what the planner validates is what the studio plays.

Assets. Arm links: the thesis's STL meshes, decimated to 1.7 MB. Products: CC0 models from Poly Haven (apple, bananas, lemon, lime, kiwi, avocado, pomegranate, pears, two cakes, wine bottles, a vase, a succulent, two watches) plus procedural props (a takeaway coffee cup, a perfume bottle, a lipstick, a cream jar, the toy car). Camera and lens are procedural, to scale.

Not modelled. Motor dynamics and belt compliance (the arm follows the planned joints under a speed cap), self-collision, the real lens's breathing and minimum-focus curve, and the camera's exposure. The report lists the limitations.

Studio report (PDF): kinematics and the tool frame, the IK, the lens model, the planner's tables, the executed telemetry.