Experiments with Computational Cinematography

The Brief

  • A machine assisted editing tool would leverage the creative metadata and create multiple rough cuts which would then be exported as a Final Cut Pro project for human review. Rule based edit logic with some fuzziness baked in would provide a way to tweak rough cuts the day of the show.
  • Our “human in the loop” — an experienced senior quality control editor — could then review each rough cut, select the best and quickly adjust the edit to taste if necessary.
Behind the scenes of The Walk — Image Courtesy of AV&C

Step 1 — Acquisition

We had 3 cameras follow a model on their runway walk, with a director monitoring the shots and providing feedback to camera operators. The runway was backed by a large format video wall, providing backdrop content and dynamism. The video backdrop was designed to have 3 chapters; an intro, hero “paparazzi” moment, and an outro; each featuring various global locations, and with their own display and design logic. This provided structure and pacing and variety to each walk.

We had 1 camera on a dolly tracking our model, one on a steady cam, and one end of stage providing dynamic framing as the model walks down the runway. Image courtesy of AV&C

Step 2 — Analysis and Edit

For our analysis phase, we designed a machine learning model which understood ‘creative cinematic concepts’ such as shot framing (close up, medium, long etc). Overall, we had roughly 10 classes relating to creative cinematic terminology in our experimental model, which was trained with Tensorflow and converted to run on CoreML on a Mac Pro.

Sync — the rough cut generation software leveraged features from CoreML and produced edits for Final Cut Pro X. Note that each rough cut has different timing, but is quantized to a multiple of the BPM. If you look closely you can also see a “compressed” middle chapter with syncopated edits for the paparazzi moment as a sort of ‘climax’ in the edit. This was the result of one specific edit rule.

Review

Sync was designed to export to Final Cut Pro by leveraging FCPXML, so we were able to bake in various creative looks, slow motion and color correction, as well as place title cards and end credits. It was also of critical importance to provide head and tail for each edit so our editors had leeway to adjust the timing. This meant the editor could spend more time adjusting the structure of the edit and not noodling around with key frames or applying effects, helping us dramatically increase edit throughput.

Results

Here’s a sample of some edits we helped create:

Learnings

Some things to note about this project’s format is that we had the benefit of creative constraints that we could leverage to our advantage.

Special Circumstances is a Computational Cinematography R&D company. We build next generation tools for film makers.