video-like-peter / vision Back to home
Video craft

Reading a fight: depth and skeleton

Three action clips from Peter's own projects, read frame by frame. A depth model says how far every pixel is, a pose model finds 17 joints on every fighter, and the two together show where a strike lands, who stands in front, and what a single frame can become as a camera move.

Clip 3, coat against suit. Blue = coat, orange = suit. White rings mark wrist speed peaks.
552frames read for depth and pose
17COCO joints tracked per fighter
16 of 29speed peaks within reach of another body, 1 confirmed on a SAM mask
3.24 BH/sfastest wrist across all three clips (clip 2, 3.79 s)
Synced layers

Three fights, one frame at a time

Each player switches between the source, the depth pass, a dimmed plate for the skeleton and a depth comp, all at the same frame. The skeleton is drawn live from the keypoint data, not baked into the video. Drag the timeline to scrub, click a marker or a time in the list to jump to a speed peak. A filled marker is a touch confirmed on a SAM mask, a half filled one is a peak within reach of another body by the pose box test (proximity, not proof of a hit), a hollow one touches nothing.

2.5D

One frame, pushed into space

The strike frame of each clip on a dense plane mesh. Every vertex is lifted toward the camera by its depth value, so moving the camera gives real parallax: the fighters slide over the background the way they would on a short dolly.

Strength 0 is a flat photo. Where depth jumps at a body edge, one triangle has to span the near fighter and the far wall, and it stretches into a streak. The edge cut drops those triangles (the standard fix), which leaves a dark gap where a real shot of this kind gets a clean background plate. Strength stops at 0.4 because past that the cut eats into the bodies.

What this proves

What the passes are, and how they get used

A depth map

A depth map gives every pixel a number for its distance from the camera. These passes come from Video Depth Anything Small (the vits checkpoint, input 518), recorded as depth_model in each clip's analysis.json. It predicts relative inverse depth: bigger means closer, and the scale only holds inside the shot. Metric depth models try to answer in metres, which matters when matching a CG camera or measuring a set, but on a fight they are usually less stable.

The trap is temporal flicker. Normalise each frame on its own and the background breathes. So the range is fixed per shot: the 1st and 99th percentile across the whole shot map to black and white, and that mapping holds until the next cut. A cut resets it, because a new camera means a new range.

A skeleton

YOLO11m-pose finds the 17 COCO joints on each person, from nose and ears to wrists and ankles, each with a confidence. ByteTrack keeps an id on a body within a shot by box overlap and motion, and starts over at every cut. Joining ids across a cut is a separate step: a SAM mask vote in clip 1, trunk colour clustering in clip 3 (both recorded under pose_tracking).

Raw joints jitter, so a One Euro filter smooths slow motion hard and fast motion lightly, which keeps a punch sharp instead of mushy. Occlusion is the honest limit: a hidden wrist drops below confidence 0.3 and is left out, not guessed. Side-on bodies swap left and right; a crossing test put back 33, 28 and 68 swapped pairs in the three clips.

How Peter uses them

As control maps. In ControlNet style and motion transfer workflows (Wan VACE, Animate style pose driving) the pose and depth sequences are the actual conditioning input: the skeleton locks the choreography, the depth locks the staging, the model paints the look.

As a Seedance reference. Seedance does not take control maps. It takes a reference video for motion and camera, so the source take (or a clean render of it) goes in as the reference and the prompt carries the new look. The pose pass is then the check that the beat survived.

In comp. The Composite layer is depth fog, a far plane softened as a cheap depth of field, and a warm lift on the near plane as a stand-in for a relight. The near plane plus the pose box is a fast start for a rotoscope mask.

In QA and continuity. Limb speed in body heights per second shows if a hand stops on the beat or sails through. Depth order shows real foreground and background staging. Ids, scale and depth order per shot catch a fighter who swaps sides or jumps in size across a cut.

Honest limits. Contact is a proximity test. Only clip 1 has SAM masks, so only one peak in the set is a confirmed touch; the other fifteen are within reach of a pose box, which a fist in the air between two close bodies also passes. Speeds are in screen space, so a moving camera adds to them (the orbit in clip 3, shot 2). Depth is relative, never metres. A blurred arm can drop out for a frame, arm pumping passes the speed test, and bodies at almost the same depth flip front and back. This reads staging and timing; it is not biomechanics.