How to translate a video into another language and lip-sync it
Upload the video and name the language. Vizard Agent clones your voice into it, trims each translated line to the time the original one took, moves your mouth to match the new words, and erases any subtitles that were burnt into the picture in the old language.
What do you type into Vizard Agent?
Ask for the lips specifically, in those words. Otherwise Vizard Agent produces a voiceover in the new language over your unchanged mouth, which is a good deliverable but not the one you meant. Mention any burnt-in text, since it has to be erased rather than covered, and name the region: Spanish from Spain is not Spanish from Mexico.
Prompt
What does Vizard Agent actually do?
It fits each translated line to the time its original took, because a Spanish sentence forty per cent longer than the English one cannot be lip-synced into the same gap. About a third of the session went on something the brief mentioned in passing: English subtitles baked into the picture, which had to come out. In the session behind this article it:
- Transcribed the audio, surveyed every shot, extracted the audio for cloning, and generated the translated narration in the cloned voice.
- Measured each translated line's real speech length, not the file's duration, and adjusted the phrasing so each line fitted its original slot.
- Detected the burnt-in subtitles and drew a heat map of where text sat across the whole video, since subtitles wrap and shift for title cards.
- Tracked the subtitle line word by word, measured its exact height, and checked the doubtful positions, the title and the closing card.
- Built the erasure mask, erased the old subtitles, applied the colour grade, and checked whether any text survived.
- Delivered 46.29 seconds at 1280x720 and 24fps with AAC audio: a Spanish voice cloned from the speaker's own, lips moving to the Spanish words, and new subtitles.
When does this not work well?
Lip-sync is the most fragile part of this job and the most obvious when it goes wrong. It works best front-on, so profile or moving shots suffer. Languages that run long and fast speech leave little room, erasing text over faces damages the mouth, and each speaker needs their own cloned voice. Vizard Agent says up front which shots won't carry it.
Common questions
Will it still sound like me? Yes. Vizard Agent clones your voice from the video's own audio.
What if the translation is longer? It gets tightened. Vizard Agent re-phrases so the line fits its original slot.
Why not just cover the old subtitles with a bar? A black bar across a cinematic shot is worse than the subtitle was. Erasing keeps the picture.