The problem: generation is easy, reliable rendering is hard
An AI agent can generate a storyboard in seconds. Turning that storyboard into the same correct video every time is a different engineering problem.
A production pipeline must make timing explicit, validate every asset, keep text readable, survive retries, and expose enough state for humans to review the result. If the model directly manipulates a timeline UI, those guarantees are difficult to enforce. A deterministic render contract solves this by separating creative intent from execution.
This article presents the architecture we use while building iLoveVideoEditor: CreativeBrief → ScenePlan → VideoJSON → renderer.
1. Separate planning from rendering
The model should not emit opaque renderer commands. It should first create two small planning documents:
- CreativeBrief — audience, format, duration, visual direction, call to action, and brand constraints.
- ScenePlan — ordered scenes with purpose, duration, copy, media needs, and transition intent.
Only after these are valid should the agent compile them into VideoJSON.
This boundary is useful because planning errors remain readable. A reviewer can reject a weak hook or an overloaded scene without inspecting low-level keyframes.
2. Use a deterministic composition contract
A simplified composition can look like this:
{
"width": 1080,
"height": 1920,
"fps": 30,
"duration": 12,
"scenes": [
{
"id": "hook",
"start": 0,
"duration": 3,
"background": { "color": "#0B1020" },
"layers": [
{
"type": "text",
"text": "Turn structured data into video",
"maxWidth": 880,
"maxLines": 3,
"transitionIn": { "type": "slideUp", "duration": 0.5 }
}
]
}
]
}
The important property is not the exact syntax. It is that the contract makes canvas size, frame rate, scene timing, layer bounds, and transitions explicit.
A renderer can now produce the same frames from the same input. The AI remains responsible for decisions; the renderer remains responsible for execution.
3. Validate before spending render time
Rendering is expensive compared with validation. A useful validator should reject a composition when:
- scene durations do not sum to the declared duration;
- layers extend outside their scene;
- display text has no width constraint;
- optional media is referenced without a presence gate;
- a source uses an unsafe or unsupported URL scheme;
- an easing or transition name is unknown;
- audio duration does not match the composition;
- required variables are missing.
For text over images, also enforce a strong scrim or another measurable contrast rule. "Looks readable" is not a contract.
A compact TypeScript validation result might be:
type ValidationIssue = {
path: string;
severity: 'error' | 'warning';
message: string;
};
type ValidationResult = {
ok: boolean;
issues: ValidationIssue[];
};
This format is ideal for agents: errors point to exact fields, so a model can repair the plan and retry without guessing.
4. Make retries idempotent
Agent workflows fail for ordinary reasons: timeouts, lost connections, expired asset URLs, or a worker restart. Retrying must not create several paid render jobs.
Generate an idempotency key from stable inputs:
const key = sha256(
JSON.stringify({
projectId,
composition,
assetVersions,
rendererVersion,
}),
);
The API stores that key for a limited period and returns the existing job when it receives the same request again. This makes automatic recovery safe.
5. Expose the pipeline through MCP
The Model Context Protocol is a clean boundary between an agent and the video system. Instead of teaching the model internal API details, expose narrow tools such as:
- list templates;
- compile template variables;
- validate VideoJSON;
- estimate render cost;
- submit a render;
- poll job status;
- download the result;
- start and stop a local preview.
Each tool should use strict schemas and return structured errors. The agent can then plan, validate, render, inspect, and revise without controlling a fragile browser timeline.
An open implementation is available in the iLoveVideoEditor MCP server repository.
6. Keep humans in the review loop
Deterministic does not mean creatively correct. Before final export, create review states:
- planned — brief and scenes are inspectable;
- validated — structural rules pass;
- previewed — representative frames have been captured;
- approved — a human or policy gate accepts the result;
- rendered — the final artifact is immutable and downloadable.
For high-volume workflows, preview only strategic frames: the first frame, every scene boundary, the midpoint of each scene, and the final CTA. This catches most layout and timing failures much faster than watching every draft end to end.
7. Version everything that affects pixels
Reproducibility requires more than saving JSON. Store:
- composition schema version;
- renderer version;
- template version;
- font identifiers;
- asset hashes;
- effect implementation version;
- output codec settings.
If a font file or shader changes, the same composition may produce different pixels. Versioning turns "the render changed" into a traceable engineering event.
Practical workflow
A reliable agent loop is short:
- Generate the CreativeBrief.
- Generate and validate the ScenePlan.
- Compile to VideoJSON.
- Run structural validation.
- Estimate cost and submit with an idempotency key.
- Capture preview evidence.
- Revise only the failed scenes.
- Approve and render the final artifact.
The core idea is simple: let AI propose intent, but let deterministic software enforce the contract. This separation makes video automation testable, observable, and safe enough for production.
Disclosure: I work on iLoveVideoEditor, an open video rendering toolkit and MCP integration. The examples above reflect engineering patterns used in the project.