How to sync AI music to your video, in both directions
Cutting picture to a track and writing a track to a cut are the same problem read from opposite ends. Both come down to one thing: a timing that means the same instant to every tool that reads it.
4 min read
There are only two ways a soundtrack and a cut end up agreeing. Either the music existed first and you cut to it, or the cut existed first and the music was written to it. Every editor does both, usually in the same project, and both are miserable when the timing lives in someone's head and gets retyped into four places.
The thing that has to be shared is a millisecond
The trap is expressing a beat as a fraction — "40% of the way through". A fraction means something different in every clip that reads it, and it silently changes meaning the moment anything upstream gets longer. A marker has to be a named number of milliseconds into the media: the end of the second chorus is at 0:48.320 and it stays at 0:48.320 whether the thing reading it is a video node, a timeline lane or a render.
Get that right and both directions below are the same mechanism pointed different ways. Get it wrong and you are hand-syncing forever.
Direction one: the music exists, cut to it
The classic case — a track you generated or uploaded, and picture that has to land on it.
- Get the beats out of the audio. Generate music from a composition plan and the section boundaries already exist — the node publishes the end of every section, named for it, without anyone marking anything. For audio that arrives as a file, play it back on a waveform node and mark the section bounds by ear.
- Wire that timing wherever the picture is decided: into a video node so the generated clip is told when things happen, into a composite timeline so the marks draw guides down every lane, or into a render so the animation interpolates to them.
- Cut against the guides rather than against the waveform. When you swap the track for a better take, the marks move and everything reading them re-times instead of being re-cut.
Direction two: the cut exists, write music to it
This is the direction most tools simply don't offer, and it is the more useful one once you have picture. You know the montage ends at 0:12 and the reveal is at 0:19; you want a track whose structure agrees.
Mark those moments wherever they already are — on a clip's own ruler, on the timeline's marker lane, on a waveform of the scratch narration — and wire that track into the music node's marks port. Each mark becomes a section boundary in the composition plan, so the generated track changes where your edit changes. The lyrics or instruction for each section then sit in their own row, one per section, and can be wired in from text nodes rather than typed.
Which is why the refusal is worth having at the point of wiring rather than at the point of generation. In QueyFrames a keyframe track whose marks sit closer than the minimum is rejected at the port with the reason, the offending rows are highlighted on the node that produced them, and an edit that later drags a keyframe under the limit cuts the wire instead of quietly clamping the result. You find out while you are wiring, not after you have paid for a track that does not match.
Fixing one section without re-rolling the track
The other reason to work in sections: when the second chorus is wrong, the whole track does not have to be regenerated. Inpainting replaces a span of an existing song in place, so the parts you already cut against keep their timings. Re-rolling from scratch gives you a different song that happens to have the same prompt — and every mark you cut to moves.
The same logic applies to style. Positive and negative style tags set on the whole track, merged into each section's own tags, keep a generated piece coherent across sections in a way that re-describing the mood per section does not.
Sound effects are the same problem, smaller
A door slam has one timing and it is the only thing about it that matters. Generate it, trim the silence off the front with the trim handles — which cut the audio for real rather than just moving a playback marker, so what goes downstream is the trimmed clip — and drop it on the mark it belongs to. If the mark moves because the picture changed, the effect is sitting on a timing that moved with it.
The short version
- Express beats as milliseconds into the media, never as a fraction of something's length.
- Publish them from wherever they genuinely originate, and read them everywhere else — do not retype.
- Respect the minimum section length before you generate, not after.
- Inpaint a section rather than re-rolling a track you have already cut against.
More on the audio side in AI music, voice and sound effects, and on the timing model itself in keyframes as shared timing.
Keep reading
- 5 min read
How to write AI video prompts that control pacing
Text-to-video models will give you the shot and invent the timing. Here is how to write the timing into the prompt — naming beats in order, pinning them to seconds, and letting wired keyframes place themselves in the sentence.
- 4 min read
Node-based AI video editing vs the single prompt box
One text box gives you a clip and no way to change part of it. A graph gives you the intermediate results — the still, the track, the timing — as things you can re-run one at a time. Here is what that buys, and what it costs.
- 5 min read
How to keyframe AI animation: spans, not points
Point keyframes only mean something beside their neighbors. When an AI reads your timing, a span — start, end, what changes — stands alone. How to write them.
Build the graph these posts are about.
QueyFrames is in early access. Create an account to join the list.