Streaming Setup: Integrating FFmpeg Overlays and Audio into a Picam feed

Lately, I’ve been setting up and refining a Raspberry Pi-based streaming setup, focusing on combining a video feed from a Raspberry Pi camera with overlay graphics and audio in real-time using ffmpeg. It’s been quite a journey, filled with trial and error as I worked through various technical challenges.

TL:DR Take me to:

The Twitch

The timelapse

I stumbled upon Restreamer (https://github.com/datarhei/restreamer) which runs in a container.

I deployed this to the Raspberry Pi and set about connecting everything up.

Initial Camera and Overlay Setup

I started by streaming a camera feed using rpicam-vid on a Raspberry Pi. The initial command streamed video at 1080p and 30 fps to a TCP connection:

rpicam-vid -t 0 --inline --listen -o tcp://0.0.0.0:8554 --level 4.2 --framerate 30 --width 1920 --height 1080 --denoise cdn_off -b 8000000

I was suitably able to add this to the restreamer software, add a secondary audio stream, connect it to a Twitch account and stream live.

Unfortunately the software has no mechanism for adding overlays to the resultant stream.

With this in mind I created another ffmpeg command that takes the TCP stream from the stream from the Pi, overlaid an image and added the contents of a text file mentioned above.

ffmpeg -loglevel debug -i tcp://192.168.1.54:8554 -i StreamOverlay.png \ -filter_complex "[0:v][1:v]overlay=0:0,drawtext=textfile='current_ track.txt':x=(w-text_w)/2:y=h-50:fontcolor=green:fontsize=24:box=1:boxcolor=black@0.5:boxborderw=10" -an -c:v libx264 -f mpegts tcp://<ip_address>:8556

It seems the Raspberry Pi 4 doesn't have sufficient resources to encode the camera feed with the overlay. I tried to reduce the incoming camera resolution to 1280 * 720, but this was still insufficient for the restreamer software to handle on the modest hardware. At this point I moved the heavy lifting over to a virtual machine on my home server and this seemed to solve the problem.

ffmpeg -loglevel debug -i tcp://0.0.0.0. -i StreamOverlay.png \-filter_complex "[0:v][1:v]overlay=0:0,drawtext=textfile='current_track.txt' :x=(w-text_w)/2:y=h-50:fontcolor=green:fontsize=24:box=1:boxcolor=black@0.5:boxborderw=10" \   -an -c:v h264 -b:v 8M -g 30 -preset veryfast -tune zerolatency -bufsize 16M -max_delay 500000 \   -x264-params keyint=30:min-keyint=15:scenecut=0 -f mpegts tcp://0.0.0.0:8554?listen

Initially, I encountered stream quality and decoding errors.

After tweaking buffer sizes, bitrate, and keyframe intervals, things began to stabilise.

Integrating Audio

Next, I focused on integrating audio into the video stream. Initially, I used a separate ffmpeg process to stream MP3 files over TCP, but I faced an issue where audio stopped after the first track ended. The ffmpeg process didn’t crash but would stall on subsequent tracks. Here’s the basic script I used:

#!/bin/bash
audio_folder="<folder where music resides>"
output_file="current_track.txt"
while true; do
  for file in "$audio_folder"/*.mp3; do
    echo "Now playing: $(basename "$file")" > "$output_file"
    cp $output_file /home/rob/$output_file
    ffmpeg -re -i "$file" -acodec copy -f mulaw tcp://0.0.0.0:8555?listen
  done
done

After switching to a local setup, with both the video and audio on the same server, I modified the overlay command to iterate through the MP3s in a folder directly.

Putting it all together

I moved the individual commands to their respective scripts and added some logic that would restart the "service" if it dropped for any reason:

It seems that the restreamer software doesn't like being on the Pi, with this in mind I bypassed that extra software entirely.

That worked, but I still had issues with audio.

#!/bin/bash

# Define the folder containing the audio files

audio_folder="/home/rob/Music"

# Define the text file where the current track info will be written

output_file="current_track.txt"

# Define the playlist file

playlist_file="playlist.txt"

while true; do

    # Generate the playlist file

    rm -f "$playlist_file"

    for file in "$audio_folder"/*.mp3; do

        echo "file '$file'" >> "$playlist_file"

    done

    # Get the first track name to display as "Now playing"

    first_track=$(basename "$(head -n 1 "$playlist_file" | sed "s/file '//g" | sed "s/'//g")")

    echo "Now playing: $first_track" > "$output_file"

    # Run ffmpeg to combine the video, overlay, and audio from the playlist

    echo "Starting ffmpeg overlay with playlist..."

    ffmpeg -loglevel level+debug -i tcp://192.168.1.54:8554 \

            -i StreamOverlay.png \

            -f concat -safe 0 -i "$playlist_file" \

            -filter_complex "[0:v][1:v]overlay=0:0,drawtext=textfile='$output_file':x=(w-text_w)/2:y=h-50:fontcolor=green:fontsize=24:box=1:boxcolor=black@0.5:boxborderw=10" \

            -c:a aac -ac 2 -b:a 128k \

            -c:v h264 -b:v 6000k -g 60 -preset veryfast -tune zerolatency \

            -bufsize 12M -max_delay 500000 -x264-params keyint=60:scenecut=0 \

            -f flv rtmp://live.twitch.tv/app/live_<stream_key>

    # Check if ffmpeg encountered an error and restart

    if [ $? -ne 0 ]; then

        echo "ffmpeg stopped. Restarting in 5 seconds..."

        sleep 5

    fi

done

This seemed to work fine for a time but then the audio would stop. I am yet to find the time to investigate.

Tidying up

I had the various scripts running in separate tmux sessions for my visibility. To make this easier, I made a script that creates the sessions and runs the respective script

#!/bin/bash

# Define script paths

camera_script="/path/to/your/camera_script.sh"

overlay_script="/path/to/your/overlay_script.sh"

# Define session names

overlay_script_session="Overlay"

camera_session="Camera"

# Start tmux session for Camera

tmux new-session -d -s "$camera_session" "bash $camera_script"

echo "Started tmux session: $camera_session"

# Start tmux session for Overlay

tmux new-session -d -s "$overlay_script_session" "bash $overlay_script"

echo "Started tmux session: $overlay_script_session"

This works great if I have to restart everything.

I'm also looking in to a way of automating the start and stop of streams based on the sunrise and sunset in my location, but for the time being I am just calculating the time in seconds between now and sunrise and adding that to the command in one line:

sleep <seconds> && sh script.sh

Timelapse Creation

During all of this, I also worked on creating a timelapse from the resultant 13-hour  off video. Using ffmpeg, I generated a 1-minute timelapse that was successfully uploaded to YouTube. The command was straightforward and effective:

ffmpeg -i input_video.mp4 -filter:v "setpts=PTS/802" -an -r 30 output_timelapse.mp4

This command sped up the video by a factor of 802 times by adjusting the presentation timestamps, producing a smooth timelapse.

Final Thoughts

This project has been a learning experience in stream handling, ffmpeg configurations, and overcoming hardware limitations. I’ve moved most of the intensive processing off the Raspberry Pi to ensure smoother streaming and a better viewer experience.

Man, formatting ffmpeg commands correctly, especially for taking multiple sources and overlaying them in the way I wanted.

While there are always more optimisations to be made, especially regarding audio stability, the progress has been rewarding. 

You can find:

The Twitch

The timelapse

A falling out, A genius idea and an idiot moment.

The modem and the router had an argument in the early hours of Thursday night.

They are no longer on speaking terms, so that's why the network has  been down.

Regardless of my attempts of mediation, the two remain inconsolable.

I thought about purchasing a replacement, but I didn't really want to spend £100-200 on a new Draytek.

That's when I came up with the genius idea of making my own router, with black jack and hookers.

I started looking up OPNsense virtual appliance installation and configuration.

Thinking about the physical aspect, cables and such, my first priority was getting the internet over to the server. On the opposite side of the house. 

The current solution was trunking from the master socket up to the office, where the Draytek router resided. From there essentially a patch lead ran from that, through the loft, down into the opposite bedroom, which is my wife's art studio.

Inside the cupboard, aside from some jackets and art supplies, reside my main server, Which provide hypervisor and file services, along with my backup server (which is not entirely live) and a unifi access point.

The first thing I thought of was a spare mikrotik router.

That's when it hit me. You idiot, just use that.

Cut to three hours later and I have it configured to give my pc an IP address and also take an IP address from the crappy nowTV modem/router.

I would have rather preserved my IP reservations, but I decided I didn't have the energy to connect back to the Draytek to note down all the reservations manually, so I plugged in the rest of the network.

I added a few NAT port forwarding rules in place, but at the time of writing, that traffic is still being blocked somewhere upstream, so will require further investigation.

At least now the network has internet access again, so my wife can stream.

I still need to get port forwarding working so that I can start using "private cloud services" and soon my Minecraft mod can get back in the server when they want to.

I'm hoping as the LAN range is the same, the existing client leases will be honoured from a new DHCP server and get given the same IP. That being the case setting the reservations will be a lot easier.

Edit: They were.

Mikrotik's web GUI takes a bit of getting used, but it is starting to make sense. I suppose that's the same with any routers GUI.

Around the outside, I replaced my POE switch, this week.

I had a really old d-link web smart switch, it really wasn't happy. After the third time of waking up to find no network access, I decided to replace it.

I got a tiny 5 port Tenda unmanaged POE switch. Its smaller, quieter and cheaper to run. Yes, I lost VLANning, but the only reason I wanted that isn't a requirement any more.

It arrived and I realised my mistake. This device was POE powered. I hastened to start the returns process, but then remembered that I still had an old POE injector. 

It was only 10/100.

Ok, back to Amazon, cancel the return and grab a gigabit injector.

What I was actually hoping to sort out this weekend, aside from mowing the lawn and doing laundry, was maybe getting cabling down to the living room, or getting a spare Cisco AP installed in the dining room.

This would have been to improve bandwidth for my raspberry pi, who was trying to talk to the unifi on the other side of the house. This resulted in a, let's say, changeable experience when streaming content.

That will have to be for future Rob.