How to build a Claude Code video editor that animates shorts in your brand's style

What you're building

This is the setup Cortex uses to make its own animated shorts, rebuilt here on tools anyone can install. Claude Code does the editing work, the same way an AI agent does one defined job inside your software: it reads your brand rules, plans every shot, writes the animation code and checks its own output. Remotion turns that code into a video, frame by frame, because a Remotion video is a React component that renders the same way every time.

The pipeline has six parts, and each one is a file you can open and edit:

1. A brand look file (BRAND.md): your colors, fonts, type sizes, safe zones and motion rules. This is what makes every video look like yours instead of like a template.
2. A script written for the ear: the exact words you'll say.
3. A shot plan (plan.md): what is on screen during each phrase, precise enough that two editors would build the same frame.
4. One voice take (voice.wav): your own recording or a text-to-speech read of the whole script in one go.
5. Word timestamps (words.json): when each word starts and ends, so captions and cuts land on the voice.
6. The build: a Remotion project where Claude Code turns the plan into animation, renders it and runs the checks.

Our own shorts are planned, voiced and checked exactly this way; we render them in a different engine, and Remotion is the open option you can install today.

What you need before you start

• Claude Code, Anthropic's coding agent that works in your terminal and project folder (overview).
• Node.js to run Remotion, and FFmpeg for audio work and loudness checks.
• Remotion. Check the license first: it is free for individuals, non-profits and for-profit companies with up to 3 employees; larger companies need a company license.
• A voice: a quiet room and a decent microphone, or a text-to-speech service you're allowed to use commercially.
• Your logo files (SVG if you have them) and your font files.

Write your brand look file (step 1)

This is the step most people skip, and it's why their AI videos all look the same. Create BRAND.md in an empty folder and fill in this template. Be specific: hex codes, font names, pixel sizes.

# Brand look for vertical shorts (1080x1920)

Colors
- Background: #F5F2EC (cream). Cards: #FFFFFF. Text: #111111.
- Accent: #FF571F. Use it for one thing per shot: the word or object that matters.
- Bad or waiting states are a light grey, never red.

Type (pixels at 1080x1920)
- Captions 64, medium headings 96, large 132, giant words 200-300.
- Labels and numbers in a monospace font at 28-32. Nothing smaller than 28 except fine print.
- Fonts: Inter Tight Medium (captions), Inter Display SemiBold (giant type), IBM Plex Mono (labels).

Safe zones
- Platform buttons cover the top ~250 px and bottom ~420 px. Keep key content inside x 60-1020, y 250-1500.
- Captions go in the top band (y 330-470) or the lower third (y 1180-1380), max 6 words a line, 2 lines.

Motion
- Captions type on word by word with the voice; each word appears slightly before it's spoken.
- Nothing is ever fully still: every shot has a slow push in.
- Objects land with a small overshoot and a soft shadow.
- Cuts happen right before a word starts, never in the middle of one.

End card
- The call to action, the account handle and any fine print. Hold it about 1.5 seconds after the last word.

The values above are the ones from our own Cortex look. Swap in yours, but keep the structure: colors with a job each, a short type scale, safe zones, motion rules and an end card.

Set up Remotion and its Claude skills (step 2)

In your project folder, start a Remotion project and add the official agent skills, which teach Claude Code how Remotion's API works so it writes correct animation code instead of guessing (Remotion agent skills):

npx skills add remotion-dev/skills

The skills include ones for creating compositions, animation and typography, captions, launching the Studio preview and rendering. Open Claude Code in the folder and ask it to create a 1080x1920, 60 fps composition called Short and launch Studio, so you can watch changes as it works.

Then add a CLAUDE.md to the project. Claude Code reads it at the start of every session (how project memory works), so your rules apply to every video without repeating them:

# Video editor rules
- Read BRAND.md before any visual work. Never use a color, font or size that isn't in it.
- Build exactly what plan.md says, shot by shot. If a shot can't be built as written, stop and tell me why.
- Time every caption word and every cut from words.json, never from estimates.
- Use real logo files from /logos. Never type a company name in place of its logo.
- After a render, export a still 1 frame before and 1 frame after every cut and look at them.
- Write anything you had to change from the plan into build-notes.md.

Write the script for the ear (step 3)

The script is the exact words you'll say, nothing else: no shot labels, no stage directions. What we've learned writing these:

• Keep it 70-95 words for a 30-40 second short.
• Open with the point in the first sentence, said straight to the viewer.
• Write connected sentences, the way you'd explain it across a table, using "so", "and", "but". Strings of two-word sentences sound like a movie trailer once they're voiced, and we had to rewrite our first scripts for exactly that reason.
• End with one call to action. Ours is a comment keyword, like the one that brought you here.

Have Claude write the shot plan (step 4)

The shot plan is where the quality comes from. Ask Claude Code to write plan.md from your script and BRAND.md using this format for every shot:

#### 03 · 0:04.1-0:06.8 · "the exact words spoken during this shot"
- Picture: what is on screen, with sizes and positions in pixels (e.g. a white card 760x420 centered at y 960).
- Type: the caption or giant word, its size, font and position.
- Motion: what moves, when, and on which word.
- Out: how the shot ends (cut, carry, zoom-through) and on which word.
- Sound: music hits or effects, if any.

Two rules make plans buildable: the quoted words of all the shots, in order, must equal the script word for word, and every shot must be specific enough that two editors would make the same frame. Read the plan before anything gets built. Changing a line of text here takes seconds, while changing a rendered video takes a rebuild.

Record the voice in one take and get word timestamps (step 5)

Record the whole script in one continuous take. We first generated the voice sentence by sentence and stitched it together, and it sounded choppy, so now it is always one read. Cuts then land on natural phrase starts.

Next you need the start and end time of every word. Remotion's Whisper package installs whisper.cpp and transcribes with word-level timing (install-whisper-cpp docs). Convert your audio to a 16 kHz WAV first, then ask Claude Code to run transcribe with tokenLevelTimestamps: true and save the result as words.json.

Check the transcript against your script before building. Whisper sometimes mishears brand names and URLs, or merges two short words, and if the labels drift every caption after that point is off. Have Claude match the transcript to the script word by word and fix the labels, keeping the timings.

Let Claude Code build it (step 6)

Now give Claude Code one instruction:

Build plan.md into the Short composition. Follow BRAND.md and CLAUDE.md. Use words.json for every caption
and cut, voice.wav as the audio. Build one shot at a time, render a still of each shot's key frame and
compare it with the plan before moving on. Then render the full video to out/short.mp4.

Building one shot at a time, with a still checked after each, catches most problems early: an object wider than the frame, a caption sitting on top of a card, a logo missing from your files. When something can't match the plan, Claude should write the deviation into build-notes.md rather than quietly changing the design.

Check it like an editor before posting (step 7)

Ask Claude Code to run these checks on the render and fix anything that fails:

• Cuts on time: export frames just before and just after each cut and confirm each one lands on its word.
• Nothing frozen: no stretch where the picture is completely still, except a planned hold.
• Safe zones: nothing important under the platform buttons; captions never touch other objects.
• Loudness: measure it with FFmpeg's ebur128 filter so your voice sits at a consistent level from video to video, with the voice clearly on top of any music.
• Claims: every number on screen is true or clearly labeled as an example.

Make it better every video (step 8)

The editor improves when its rules do. After each video, add one line to CLAUDE.md or BRAND.md for anything you corrected ("captions never below y 1500", "use the official logo, not a text label"). Within a handful of videos the first render is close to what you'd have made by hand, because every fix you've made is now a rule.

What usually goes wrong

• Captions drift from the voice: the timestamps were estimated or the transcript labels slipped. Use real word timings and check the transcript against the script.
• The voice sounds choppy: it was stitched from separate clips. Record or generate it in one take.
• Everything looks generic: the brand look file is vague. Hex codes, font names and pixel sizes, not "modern and clean".
• Text stands in for logos: keep a logos folder with official files and tell Claude never to type a name instead.
• Plans put things where the platform covers them: write the safe zones into the brand file and check stills.

If you'd rather have a system like this, or AI agents in the rest of your operation, built for you, start with a free AI Operations Audit.