The Lip Sync feature is a useful addition to the popular Runway AI video generation model. It's used for syncing lip movements of characters or human subjects in AI videos with speech audio, and it offers a lot of utility. But, since it's quite a new feature, many users aren't sure about how it works. This guide explores how to use Runway's Lip Sync feature in a straightforward way.
What Is Runway AI's Lip Sync?
Lip Sync is, as the name implies, a feature for syncing up speech audio with lip movements of the characters in your AI videos. In other words, you can use it to make AI videos where characters realistically move their mouths and faces to match a text script that you can either write in your prompts or upload as a separate audio file.
The uses of this are far-reaching. Creators can use Lip Sync to, for example, create realistic scenes of characters engaging in dialog with one another, or reading a speech, or making a statement. This opens up new opportunities in terms of the kind of content – such as film and TV-style content – that you could theoretically make with Runway AI.
How to Use Lip Sync in Runway AI
Now you know what the Lip Sync feature is all about, let's look at how to use it. Here's a simple step by step guide, complete with screenshots, to show you exactly how you can use Lip Sync to make AI videos with perfectly synchronized audio.
Step 1. Access the Lip Sync Tool
The first step is simple. If you want to use the Lip Sync tool, you need to click and select "Lip Sync Video" under the list of "Runway's AI Tools," which you can find on the Runway dashboard.
Step 2. Add a Video or Image
Next, you'll need an image or video to use as the baseline for your final creation. You can either drag and drop or click to upload a video or image in any of the supported formats. If you use a video, make sure that the character you want to see speaking is present in the very first frame.
Step 3. Select Your Speaker
Once your image or video has loaded into Runway, it'll try to automatically detect the face of your speaker. Most of the time, it should have no trouble spotting the key figure in the image, but it may struggle if you upload an image with multiple people in or unclear figures. If that happens, you can click on the image yourself and drag the box onto the face of your speaker.
Step 4. Choose a Voice
Click the microphone icon next to your speaker to select a voice for them. Runway gives you a whole bunch of different voice options and you can preview each one by pressing the little "Play" button. Then, when you find one you want, click to select it.
Step 5. Input Your Script or Upload Your Audio
Next, you need to actually tell the tool what you want your speaker to say. You've got two options for this. You can either click in the text box next to your speaker and type in what you want them to say. Or you can upload an audio file that already has a recording of the speech you want to hear in the final video.
Step 6. (Optional) Add Additional Speakers
This is an optional step for situations where you may want multiple people to speak in your video. You can click the "Add speaker" button at the bottom of the session window to add each one, and then click the profile icon for each speaker to match it to one of the detected faces in your video. Then, either type in your script or upload the audio you want to hear.
Step 7. Click the Generate Button
Once you've added all the desired dialog, you can just press the "Generate" button at the bottom of the page and wait a moment while Runway makes your video. You can then view it, save it, and share it around.
In addition, note that you can also add Lip Sync to AI-generated videos made with all of the different Runway models, like Gen-2, Gen-3, and Gen-3 Alpha Turbo. You just need to make your video, then click “Action” and then “Lip Sync.”
Best Practices for Using Lip Sync in Runway
The Lip Sync feature in Runway is useful and fun to use, but it's not perfect. If you want to get the best results with this feature, you'll need to follow certain recommended best practices, which are as follows:
- If you use any uploaded speech files, keep them at a maximum of 40 seconds.
- Have your characters facing forward, towards the camera.
- Use human faces only – animals may be added in the future but don't work yet.
- Have your characters' heads and shoulders in the shot, not too zoomed in or out.
- Opt for photorealistic character styles, not animated or cartoonish ones.
- Avoid any excessive movement of the body, hands, or camera.
- Keep the lighting consistent in your AI videos.
- Avoid any quick cuts or transitions that interrupt the flow of the audio/speech.
- Make sure you use the supported audio and video file types.
- Try to stick to one character in frame – it works less well with multiple faces.
Try Runway AI's Lip Sync Today with Pollo AI
If you want to test out the Runway Lip Sync feature, you can do easily and for free on Pollo AI! Pollo AI is a leading AI video generator that gives you access to a range of AI video generation models, including Runway's Gen 2, Gen 3 Alpha and Turbo, Hailuo AI, and Luma AI, plus others. It also offers a range of tools and advantages, like templates for making AI kissing videos. You can test them all out, all free, and all in one place with Pollo AI.