digital-garden-anabasis/pages/_INFORM/ffmpeg.md

ffmpeg vault

Essentials

What?

ffmpeg is a swiss army knife for everything audio/video. It can do practically every task under the sun, and in fact powers most major dedicated "video players" (VLC, MPC-HC, built-in players in Chrome and Firefox...)¹

How?

If you're on Windows, it's technically possible to install ffmpeg and use it directly ², but since the windows Command Prompt sucks ass comfort-wise and scripting-wise, it's recommended to just install Ubuntu as part of the Windows Subsystem for Linux, and then apt-get install ffmpeg.

If you're on Linux, you already know what to do 8-)

Techniques

(All commands are expected to be ran in bash or a similar Linux shell.)

Basic conversions

ffmpeg is pretty clever, it can correctly guess the codecs and reasonable default settings by the file extension, so all of the following will work as expected (and retain metadata³!):

ffmpeg -i video.avi video.mp4
ffmpeg -i video.mp4 video_sound_only.wav
ffmpeg -i video_sound_only.wav video_sound_only.mp3
ffmpeg -i song.flac song.mp3

mp3 bitrates

"Reasonable" might not be what you want though, especially in the case of mp3, where the default bitrate is V4 (!), i.e. 140-185 kbps.

If you want, for example, V[0]{.ul}, use the -q:a⁴ option, like so:

ffmpeg -i song.flac -q:a 0 song.mp3

More info at: https://trac.ffmpeg.org/wiki/Encode/MP3

video codecs

Since container/format ≠ codec, you might want to select the codec manually.

While it can reasonably assumed that mp4 ≅ h264, avi is a bit more complex. You can list all the supported codecs with ffmpeg -codecs⁵, but since there's several hundreds, you better have an idea of what you want to do in the first place.

For example, if you want an .avi with xvid codec, you just do:

ffmpeg -i original.mp4 -c:v libxvid output.avi

General codec options

This StackOverflow post explains everything: https://stackoverflow.com/a/20587693/3833159

video // image files // frames

images -> video

1. creating a list of images

ffmpeg needs a list of images in a text file in a specific format in order to convert them to a video. There's a couple ways to do this:

ls *.jpg | xargs -I xyz echo "file 'xyz'" > list.txt

for f in *.jpg; do echo "file '$f'" >> list.txt; done

It's up to preference, all end up with a list of all JPGs in current directory, in list.txt.

2. list to video

ffmpeg -f concat -r 30 -i list.txt out.mp4

-f concat tells ffmpeg to handle list.txt as a list.

-r 30 specifies resulting FPS (30 FPS)

out.mp4 is output file - autodetected as h264-encoded. (out.avi, out.gif, etc. also work - refer to ffmpeg manual)

video -> images

ffmpeg -i FILE image%05d.png

Where FILE is the video file, and image%05d.png is the format string for image filenames; this will create image00001.png, image00002.png, image00123.png, etc. (%05d means pad with 5 zeroes; %010d for padding with 10 zeroes...)

Streams

ffmpeg can also smoothly handle streams, so basic stream capture is pretty trivial, provided you grabbed the playlist/HLS url from somewhere⁶:

ffmpeg -i "https://example.com/playlist.m3u8" my_stream.mp4

Taking a screenshot of a stream

-vframes 1 is the option that tells ffmpeg to just capture one (i.e. the first) frame of the video - in the case of streams, this means the latest one anyway.

ffmpeg -i "https://example.com/playlist.m3u8" -vframes 1 capture.jpg

dropped frame re-interpolation

ffmpeg also has a rich set of filters, two of which are of interest for us now:

• mpdecimate - Drop frames that do not differ greatly from the previous frame in order to reduce frame rate.
• minterpolate - Convert the video to specified frame rate using motion interpolation.

The idea is that mpdecimate drops all near-duplicate frames, and minterpolate re-calculates them using non-duplicate frames that were left.

mpdecimate's defaults are pretty okay, but the result may not look too good if the frame drops are frequent and long. I've had pretty good results using its max parameter which limits the amount of frames dropped in a single stretch of video, e.g. -vf mpdecimate=max=15 which drops at most 15 frames (i.e. half a second assuming 30 FPS), meaning interpolation won't happen everywhere and the video will remain faithfully choppy.

minterpolate, on the other hand, defaults to semi-smart motion compensated interpolation, and that might just be what you want, but it generally gives pretty funky results. Fortunately, it also has a "blend" mode, which just averages the start and end frames and crossfades them, which gives much more agreeable outputs for simple frame drop situations. It is also generally much faster, I was getting near or above real-time speeds using "blend", whereas motion compensation dropped the processing speed to 0.01x.

TL;DR: Full command(s) including the filter pipeline:

# Fill out all frame drop gaps
ffmpeg -i choppy_video.mp4 -vf mpdecimate,minterpolate=mi_mode=blend smoother_video.mp4

# Fill out all frame drop gaps no longer than 10 frames:
ffmpeg -i choppy_video.mp4 -vf mpdecimate=max=10,minterpolate=mi_mode=blend smoother_video.mp4

# Motion interpolate the gaps and replicate a bad ketamine trip
ffmpeg -i choppy_video.mp4 -vf mpdecimate,minterpolate smoother_video.mp4

ffmpeg - skipping - remove duplicate frames after effects

what is `N/FRAME_RATE/TB`

• except the use of `FRAME_RATE` variable the `N/FRAME_RATE/TB` is equal to the example below from ffmpeg documentation ([source](https://ffmpeg.org/ffmpeg-filters.html#Examples-123))

* 

    > Set fixed rate of 25 frames per second:  
    > `setpts=N/(25*TB)`

• the math behind it perfectly explained in What is video timescale, timebase, or timestamp in ffmpeg?

  • it basically calculates timestamp for each frame and multiplies it with timebase TB to enhance precision

mp4 compatibility

h264 also has "profiles", basically sets of features

• and it turns out this can make the difference between a file working and not working on some crappy embedded media players, like TVs or pico projectors.

The -profile:v option limits the output to a specific H.264 profile. Some devices (mostly very old or obsolete) only support the more limited Constrained Baseline or Main profiles. You can set these profiles with -profile:v baseline or -profile:v main.

https://trac.ffmpeg.org/wiki/Encode/H.264#Profile

ffmpeg -i original.mp4 -profile:v baseline output.mp4

And apparently, some players are also sensitive to the pixel format⁷, i.e. can't handle anything else than YUV w/ 4:2:0 chroma subsampling, to fix this use the -pix_fmt option as follows:

ffmpeg -i original.mp4 -pix_fmt yuv420p output.mp4
# or, with the profile settings...
ffmpeg -i original.mp4 -profile:v baseline -pix_fmt yuv420p output.mp4

No silver bullet, you'll just have to try different things for different devices. A database of crappy players and appropriate ffmpeg settings would be great.

random

https://ottverse.com/ffmpeg-drawtext-filter-dynamic-overlays-timecode-scrolling-text-credits/

---

1. https://en.wikipedia.org/wiki/FFmpeg#Projects_using_FFmpeg ↩︎

2. from https://ffmpeg.zeranoe.com/builds/ ↩︎

3. like ID3 tags and their FLAC, OGG, WAV, etc. equivalents ↩︎

4. read as -quality:audio ↩︎

5. and respectively, formats with ffmpeg -formats ↩︎

6. like https://addons.mozilla.org/en-US/firefox/addon/hls-stream-detector/; or just press F12, check the Network tab, and look carefully ↩︎

7. https://trac.ffmpeg.org/wiki/Encode/H.264#Encodingfordumbplayers ↩︎