All work
09 / 09In progress

Agentic Video Editing

AI agents that operate After Effects: they edit, render a frame, look at it, and fix their own mistakes.

Part of the same system TikTok Agent ControlTwo halves of one system. The agent that watches the account decides what to make, then hands the source and the narration over here to be edited.
Role
Solo: research, PRD, architecture, build
Year
2026
Type
AI & Agents
Status
In progress

The problem

Matching the editing style of a reference video is expert work done frame by frame, and After Effects has no API an agent can call.

What I did

Solo: the research, PRD, architecture and build, including the ExtendScript bridge that lets an agent drive After Effects.

Result

In progress, with the core loop proven: the agent edits, renders a frame, looks at it with vision and corrects itself — tested with a deliberately misplaced layer it had to catch.

Research build
In progress; runs against a local After Effects install
0
agents: an executor, and a critic that never sees its reasoning
Vision
check on a rendered frame before any edit is accepted
0
design docs: PRD, architecture and specs
0
experiment runs so far

Watch it run · sound on

the finished clip the agent made: cut, captions, treatments and cutaways
After Effects building the edit on its own: the cut, then treatments, then caption cards (sped up)

On desktop

Agentic Video Editing: A text treatment lands on the cut while the timeline fills
A text treatment lands on the cut while the timeline fills
Agentic Video Editing: Pipeline done: 12 caption cards placed, project saved to edit.aep
Pipeline done: 12 caption cards placed, project saved to edit.aep
Agentic Video Editing: The pipeline reporting each stage as it runs
The pipeline reporting each stage as it runs
Agentic Video Editing: It warns before the After Effects build starts, with no pause between stages
It warns before the After Effects build starts, with no pause between stages

Overview

The goal is to take the editing style of a reference video (pacing, typography, motion intensity, where the emphasis lands) and apply it to new footage in After Effects, without an expert doing it frame by frame.

Agents drive After Effects through a bridge I wrote, render frames, look at them with vision, and correct what's wrong. It is also the editor behind TikTok Agent Control: that agent picks the story, the source and the narration, then hands the footage here. What comes back is not a finished file but an edit sitting in an After Effects project, ready for a person to review and export.

How it fits together

Input

  • Reference video

    The style to match.

  • New footage

    The material to edit.

Agents (TypeScript + Anthropic SDK)

  • Style analystBuilt solo

    Turns measured signals into a style profile.

  • ExecutorBuilt solo

    Plans the edit and applies it in After Effects.

  • CriticBuilt solo

    Isolated. Judges rendered frames against the brief and profile.

After Effects

  • File-IPC bridgeBuilt solo

    ExtendScript: the only way agents can change the project. Every change is journaled.

How it flows

  1. 1

    Read the reference

    ffmpeg + analyst

    Shots, beats and transcript are measured, then turned into a style profile.

  2. 2

    Plan the edit

    Executor

    Choices go into the Edit Plan, each with a reason.

  3. 3

    Apply in After Effects

    Bridge

    Operations run through the bridge and come back as structured results.

  4. 4

    Render and look

    Critic

    A frame is rendered and judged with vision, without the executor's reasoning.

  5. 5

    Revise or stop

    Loop

    Fix what's wrong and try again, or stop when it no longer improves.

What I built

  1. 01Built an ExtendScript file-IPC bridge so an agent can run After Effects operations unattended and get structured results back.
  2. 02Proved the loop with an injected fault: a text layer is secretly placed off-centre, and the only way to catch it is to render a frame and look. The agent finds and fixes it.
  3. 03Architecture uses a blackboard with typed messages, with the Edit Plan as its backbone. Every change is journaled with its rationale, for audit and rollback.
  4. 04The Critic runs in an isolated context and never sees the executor's reasoning, so it judges the output rather than the argument.
  5. 05Style profiles extracted from reference videos, with deterministic signal processing (shots, beats, transcripts) kept out of the model. Agents only handle judgment.
  6. 06No agent framework: peer-to-peer coordination is the product, so it's a few hundred lines on the Anthropic SDK.
  7. 07Wrote the PRD, architecture and specs first, and ran spikes to answer the riskiest questions before building.

Challenges I solved

  1. 01After Effects has no API an agent can call.

    Solution · An ExtendScript bridge watches a folder for requests, runs the operations inside After Effects and writes back structured results.

  2. 02Tool results said everything was fine even when the frame looked wrong.

    Solution · The agent has to render a frame and look at it. A test with a deliberately misplaced layer showed it catches problems that only show up on screen.

  3. 03An agent judging its own work tends to approve it.

    Solution · The Critic runs separately and only sees the brief, the style profile and the frames, never the executor's reasoning.

  4. 04Loops that never converge waste time and money.

    Solution · Plateau detection is part of the design: the system says when it stopped improving instead of quietly shipping a twelfth attempt.

What I learned

  • Writing the PRD and architecture before code, and running spikes to rule out risky ideas early.
  • Where AI models help and where plain code wins: measuring signals versus making judgments.
  • Coordinating several agents without a framework.
Next projectBiblioteka