Photos on the beat

I have two piles of images. One is photographs I took. The other is things I saved off Tumblr because they were worth keeping. The first pile is authorship, the second is curation, and I've come round to thinking the saved pile says as much about an eye as the shot one does.

Putting them next to each other is the actual subject. This tool is the way to do it: point it at folders of images and a track — a song, or an hour-long DJ mix — and it renders a video where the images change on chosen musical events. Snare hits, downbeats, kicks. The pool a picture came from stays a compositional variable all the way through, so the two piles can be cut against each other rather than blended into one bag.

It runs entirely on my own machine. No cloud, no accounts, no upload.

Three stages, three files

folders/  ──[ingest]──►  manifest.json
track.mp3 ──[analyze]─►  beats.json
                          │
          edit.json ──────┴──[render]──►  out.mp4

The rule that shapes everything: no stage calls the one before it. Ingest catalogues the images once. Analyze listens to the track once — and that one is expensive, tens of minutes on a long mix — and both write their results to disk.

What's left is edit.json, the choreography: which musical event triggers a cut, how strong it has to be, the minimum gap between cuts, which layouts are allowed, how long a held image lingers. That's the file you change a hundred times, and changing it costs a render and nothing else. If a render ever kicked off audio analysis, the design would be wrong.

Every experiment is seeded, so the same inputs and the same seed always produce the same video. A version you liked can be reproduced exactly instead of chased.

Listening to the track

Beat tracking uses beat_this, which handles the tempo changes a DJ mix is full of, and falls back to librosa if it isn't available. The track is then separated into stems — drums, bass, vocals, other — so the cuts can be driven by one instrument rather than the whole mix, and drum hits are sorted into kick, snare and hat by their spectral centroid.

Everything caches against a hash of the audio, so a track is analyzed exactly once, ever.

When the beat grid comes out wrong, the fix is an escape hatch rather than another hour of processing: lay a constant grid at a stated BPM, nudge the whole grid by a few seconds, or analyze only a window. There's also a mode that renders the detected grid as clicks over the track, so you can check it by ear in thirty seconds instead of by eye in a render.

The stack

Python for the two analysis stages, Remotion for the render — video composed as React components, which means the layout vocabulary is written the way a web page is. Landscape runs full-frame with a slow Ken Burns push on held images; portraits go into a triptych where one panel swaps at a time; squares pair up or sit on a matte coloured from the image's own mean.

The control surface is a local web app with a live preview embedded in it. Every knob in edit.json has a control, and the app reads and writes that same file — a hand edit and a UI edit are the same thing. Picking folders, running ingest and analyze, watching named progress rather than a spinner, browsing every ingested image in a grid to exclude the duds, and kicking off the final render all happen there. Duplicates are detected by perceptual hash and locked out, within and across pools, because the same image firing twice reads as a bug.

Stills only, 16:9 only. Both are decisions rather than gaps.