Skip to main content

Running SeamlessM4T locally for translating text and speech

· 3 min read
Ben Butterworth

I found a YouTube video spoken in french, and I wanted to translate it to English. This was an opportunity to try SeamlessM4T. However, when I tried to use it, it ended up taking more than an hour. After finding the path of least resistance that actually worked, I wanted to share this with others.

These steps are for the terminal. I used an ARM mac (M1 Pro). Intel macs are a bit more complicated because some dependencies need to be manually built locally (fairseq2).

SeamlessM4T is a family of models that can translate:

  • text to speech
  • speech to text
  • text to text
  • speech to speech (my use case today, translating French videos into English)

Set up SeamlessM4T locally​

  • Use Python 3.11, because 3.12 is not supported by torch or fairseq2
    • I already use pyenv, so I'll run: pyenv local 3.11
    • You can install it too, but a simpler option is to uninstall python and download and install version 3.11 from https://www.python.org/downloads/
  • Clone the repo: git clone [email protected]:facebookresearch/seamless_communication.git
  • Change directory: cd seamless_communication
  • python -m venv venv
  • source venv/bin/activate
  • pip install torch fairseq2 -r dev_requirements.txt
  • pip install -e .
    • this downloads dependencies from pyproject.toml and makes the current directory an editable package
  • brew install libsndfile
    • When running m4t_predict later, if you get an error OSError: libsndfile is not found! Use your system package manager to install it , reinstall it by running brew reinstall libsndfile

my extra step (for YouTube)​

  • Download the YouTube video, using yt-dlp
  • Convert the video into wav, using ffmpeg, I ran ffmpeg -i path/to/video.webm path/to/output-video.wav
  • Run it through seamless (in my case I'm running the speech to speech task - s2st):
m4t_predict --task s2st --model_name seamlessM4T_medium --tgt_lang eng 'path/to/input.wav' --output_path output`date +%Y-%m-%d.%H:%M:%S`.wav

Lag/crash warning​

When running the default model (SeamlessM4T-Large v2) on my MacBook Pro with 16GB memory and M1 pro chip, it started lagging for a few minutes, "your system has run out of application memory" constantly popped up, and then my computer froze. Now, I use the SeamlessM4T-Medium model, and it works.

macOS "force quit applications" window

ValueError: The input sequence length must be less than or equal to the maximum sequence length (4096), but is 11800 instead.​

If this happens, your input audio is too long. You can manually split the a few clips, depending on how many words are spoken. I'll be writing a python script to split the audio clip into chunks of 4096 max, and then run seamless on each chunk, and merge the audio back together.

A quick test is to just trim the input audio to 60 seconds chunks with ffmpeg, run ffmpeg - i input.wav -t 60 output-60s-1.wav for the first clip, then start at 60s and end at 60s ffmpeg - i input.wav -ss 60 -t 60 output-60s-2.wav, and so on.

To delete the downloaded models​

The v2 large model (seamlessM4T_v2_large.pt) is 8.4GB. You might not have enough space to keep all models locally.

On macOS, models are stored in your home folder: /$HOME/.cache/fairseq2/assets, for example, /Users/yourusername/.cache/fairseq2/assets. Open it in finder and delete files you don't want: open $HOME/.cache/fairseq2/assets and delete them.

More resources​