AI-Powered Video Editing: Creator Edits Entire Video Using Only Codex

By The AI & Creative Tech Desk

In a groundbreaking experiment that blurs the lines between programming and filmmaking, a developer has successfully produced a polished video using only conversational commands given to an artificial intelligence system, completely bypassing traditional timeline-based editing software like Adobe Premiere Pro.

The project, detailed in a recent blog post and Reddit discussion by developer Adithyan (Adi), showcases a video featuring the classic "text behind the subject" effect, achieved without a green screen. The entire process—from segmentation and matting to final composition and timing animations to spoken words—was orchestrated through a dialogue with OpenAI's Codex, a sophisticated AI code-generation model.

The AI Toolchain: SAM3, MatAnyone, and Remotion

To accomplish this feat, Adi, with Codex's assistance, architected a three-stage pipeline built on open-source tools. According to the technical breakdown, the process began with SAM3 (Segment Anything Model 3), which generated a static segmentation mask identifying the subject—in this case, a person—from a single video frame.

This mask was then fed into MatAnyone, a model designed for video matting. MatAnyone used the initial mask to track the subject across the entire video sequence, producing a precise foreground alpha matte. This matte is the crucial component that allows for occlusion, enabling text to realistically appear behind the moving subject.

Finally, the composition was handled by Remotion, a framework for creating videos programmatically with React. Codex wrote the Remotion code that combined the original background video, the foreground matte, and the animated text overlays to render the final product.

A Conversation, Not a Timeline

The workflow was fundamentally different from traditional editing. Adi described working in a terminal, chatting with Codex to iterate on the project. He maintained a rough storyboard and a transcript with word-level timestamps. Instructions to the AI were as simple as "at this word, do this," and Codex would generate the corresponding code to sync animations.

"Everything in this video was done 100% through Codex. No timeline editor. Just chatting back and forth in the terminal and iterating on a Remotion project," Adi stated. His screen setup typically featured a Remotion preview on one side and the coding terminal on the other.

The Current Reality: Promise and Friction

The developer was candid about the current limitations. The project took approximately 8-9 hours, significantly longer than manual editing would have. A substantial portion of that time was spent overcoming technical hurdles, such as debugging the MatAnyone client tool, not on creative direction.

"This took longer than manual editing for me," Adi acknowledged. "Mainly because I'm still building the workflow and the primitive tools that a traditional editor gives you for free. Masking and matting is a good example. I'm basically rebuilding those pieces (with Codex) and then using them."

He also implemented a technique suggested by an OpenAI engineer: having the AI agent review its own output. By writing scripts to render specific frames, Codex could self-critique and iterate, closing the feedback loop and saving development time.

A Glimpse of an AI-Native Creative Future

Despite the current inefficiencies, the experiment highlights a compelling thesis for the future of creative software. Adi argues that while traditional software provides a rich set of built-in tools, the "harness" driving his process—the AI—is "super intelligent."

"Once Codex has the same toolkit, it'll be way more capable than any traditional editor could be," he wrote. The vision is not of AI merely assisting in a human-controlled interface, but of the human acting as a creative director and high-level feedback provider to an AI that handles the technical execution.

All code from the project has been open-sourced, though Adi cautions it is a proof-of-concept dump rather than a plug-and-play system. He plans to refine the toolchain and build more fundamental "primitives" to make AI-driven video editing more accessible and efficient.

This project stands as a significant marker in the evolution of content creation. It demonstrates that the core challenge is shifting from mastering complex software interfaces to clearly articulating creative intent to an intelligent agent capable of translating vision into code. The era of conversational video editing may be closer than it appears.

Editor's Note: A common point of confusion in writing about such projects is the past tense of "edit." As clarified by language guides from Grammarhow and Two Minute English, the correct spelling is "edited" with a single 't'. The double-'t' variant, "editted," is an incorrect spelling that does not conform to standard English conjugation rules for verbs ending like "edit."