Streaming Setup: Integrating FFmpeg Overlays and Audio into a Picam feed

Lately, I’ve been setting up and refining a Raspberry Pi-based streaming setup, focusing on combining a video feed from a Raspberry Pi camera with overlay graphics and audio in real-time using ffmpeg. It’s been quite a journey, filled with trial and error as I worked through various technical challenges.

TL:DR Take me to:

The Twitch

The timelapse

I stumbled upon Restreamer (https://github.com/datarhei/restreamer) which runs in a container.

I deployed this to the Raspberry Pi and set about connecting everything up.

Initial Camera and Overlay Setup

I started by streaming a camera feed using rpicam-vid on a Raspberry Pi. The initial command streamed video at 1080p and 30 fps to a TCP connection:

rpicam-vid -t 0 --inline --listen -o tcp://0.0.0.0:8554 --level 4.2 --framerate 30 --width 1920 --height 1080 --denoise cdn_off -b 8000000

I was suitably able to add this to the restreamer software, add a secondary audio stream, connect it to a Twitch account and stream live.

Unfortunately the software has no mechanism for adding overlays to the resultant stream.

With this in mind I created another ffmpeg command that takes the TCP stream from the stream from the Pi, overlaid an image and added the contents of a text file mentioned above.

ffmpeg -loglevel debug -i tcp://192.168.1.54:8554 -i StreamOverlay.png \ -filter_complex "[0:v][1:v]overlay=0:0,drawtext=textfile='current_ track.txt':x=(w-text_w)/2:y=h-50:fontcolor=green:fontsize=24:box=1:boxcolor=black@0.5:boxborderw=10" -an -c:v libx264 -f mpegts tcp://<ip_address>:8556

It seems the Raspberry Pi 4 doesn't have sufficient resources to encode the camera feed with the overlay. I tried to reduce the incoming camera resolution to 1280 * 720, but this was still insufficient for the restreamer software to handle on the modest hardware. At this point I moved the heavy lifting over to a virtual machine on my home server and this seemed to solve the problem.

ffmpeg -loglevel debug -i tcp://0.0.0.0. -i StreamOverlay.png \-filter_complex "[0:v][1:v]overlay=0:0,drawtext=textfile='current_track.txt' :x=(w-text_w)/2:y=h-50:fontcolor=green:fontsize=24:box=1:boxcolor=black@0.5:boxborderw=10" \   -an -c:v h264 -b:v 8M -g 30 -preset veryfast -tune zerolatency -bufsize 16M -max_delay 500000 \   -x264-params keyint=30:min-keyint=15:scenecut=0 -f mpegts tcp://0.0.0.0:8554?listen

Initially, I encountered stream quality and decoding errors.

After tweaking buffer sizes, bitrate, and keyframe intervals, things began to stabilise.

Integrating Audio

Next, I focused on integrating audio into the video stream. Initially, I used a separate ffmpeg process to stream MP3 files over TCP, but I faced an issue where audio stopped after the first track ended. The ffmpeg process didn’t crash but would stall on subsequent tracks. Here’s the basic script I used:

#!/bin/bash
audio_folder="<folder where music resides>"
output_file="current_track.txt"
while true; do
  for file in "$audio_folder"/*.mp3; do
    echo "Now playing: $(basename "$file")" > "$output_file"
    cp $output_file /home/rob/$output_file
    ffmpeg -re -i "$file" -acodec copy -f mulaw tcp://0.0.0.0:8555?listen
  done
done

After switching to a local setup, with both the video and audio on the same server, I modified the overlay command to iterate through the MP3s in a folder directly.

Putting it all together

I moved the individual commands to their respective scripts and added some logic that would restart the "service" if it dropped for any reason:

It seems that the restreamer software doesn't like being on the Pi, with this in mind I bypassed that extra software entirely.

That worked, but I still had issues with audio.

#!/bin/bash

# Define the folder containing the audio files

audio_folder="/home/rob/Music"

# Define the text file where the current track info will be written

output_file="current_track.txt"

# Define the playlist file

playlist_file="playlist.txt"

while true; do

    # Generate the playlist file

    rm -f "$playlist_file"

    for file in "$audio_folder"/*.mp3; do

        echo "file '$file'" >> "$playlist_file"

    done

    # Get the first track name to display as "Now playing"

    first_track=$(basename "$(head -n 1 "$playlist_file" | sed "s/file '//g" | sed "s/'//g")")

    echo "Now playing: $first_track" > "$output_file"

    # Run ffmpeg to combine the video, overlay, and audio from the playlist

    echo "Starting ffmpeg overlay with playlist..."

    ffmpeg -loglevel level+debug -i tcp://192.168.1.54:8554 \

            -i StreamOverlay.png \

            -f concat -safe 0 -i "$playlist_file" \

            -filter_complex "[0:v][1:v]overlay=0:0,drawtext=textfile='$output_file':x=(w-text_w)/2:y=h-50:fontcolor=green:fontsize=24:box=1:boxcolor=black@0.5:boxborderw=10" \

            -c:a aac -ac 2 -b:a 128k \

            -c:v h264 -b:v 6000k -g 60 -preset veryfast -tune zerolatency \

            -bufsize 12M -max_delay 500000 -x264-params keyint=60:scenecut=0 \

            -f flv rtmp://live.twitch.tv/app/live_<stream_key>

    # Check if ffmpeg encountered an error and restart

    if [ $? -ne 0 ]; then

        echo "ffmpeg stopped. Restarting in 5 seconds..."

        sleep 5

    fi

done

This seemed to work fine for a time but then the audio would stop. I am yet to find the time to investigate.

Tidying up

I had the various scripts running in separate tmux sessions for my visibility. To make this easier, I made a script that creates the sessions and runs the respective script

#!/bin/bash

# Define script paths

camera_script="/path/to/your/camera_script.sh"

overlay_script="/path/to/your/overlay_script.sh"

# Define session names

overlay_script_session="Overlay"

camera_session="Camera"

# Start tmux session for Camera

tmux new-session -d -s "$camera_session" "bash $camera_script"

echo "Started tmux session: $camera_session"

# Start tmux session for Overlay

tmux new-session -d -s "$overlay_script_session" "bash $overlay_script"

echo "Started tmux session: $overlay_script_session"

This works great if I have to restart everything.

I'm also looking in to a way of automating the start and stop of streams based on the sunrise and sunset in my location, but for the time being I am just calculating the time in seconds between now and sunrise and adding that to the command in one line:

sleep <seconds> && sh script.sh

Timelapse Creation

During all of this, I also worked on creating a timelapse from the resultant 13-hour  off video. Using ffmpeg, I generated a 1-minute timelapse that was successfully uploaded to YouTube. The command was straightforward and effective:

ffmpeg -i input_video.mp4 -filter:v "setpts=PTS/802" -an -r 30 output_timelapse.mp4

This command sped up the video by a factor of 802 times by adjusting the presentation timestamps, producing a smooth timelapse.

Final Thoughts

This project has been a learning experience in stream handling, ffmpeg configurations, and overcoming hardware limitations. I’ve moved most of the intensive processing off the Raspberry Pi to ensure smoother streaming and a better viewer experience.

Man, formatting ffmpeg commands correctly, especially for taking multiple sources and overlaying them in the way I wanted.

While there are always more optimisations to be made, especially regarding audio stability, the progress has been rewarding. 

You can find:

The Twitch

The timelapse

Confessional Update

Forgive me world wide web, for I have sinned, It's been 8 months since my last blog post.
As expected, I'm also terrible at this too.

So whats been happening since I last posted? Well, there was a period of unemployment.
I spent my time split between job search, being anxious about the situation and trying new things.

I finally picked up a contract position in December as a systems engineer for a public sector service.
Typical 3rd line/projects kind of thing, solution designs, meetings, cost spreadsheets and gantt charts.
Things were extremely slow going to start with. There was no training to speak of, save a wiki knowledge base and asking colleagues in the immediate vicinity. So, my understanding of the ecosystem above the fundamental was patchy at best. It quickly became apparent that things were broken. Policies, procedures, people.
It was quite a mess. The place was awash with apathy, most of the responses to "why is this like this?" would mostly be "yeah, its broken, but its not going to change, so why bother worrying"
So much time was spent going round in this particular circle.
Despite all the talk from project managers and management about reviews and change coming, it was apparent that this was a holding pattern that had been in place for a awhile.
I nearly spat my coffee all over my desk when I heard a project manager tell me "Agile is coming".

Trying to talk to anyone in the management team about the problems I had seen was a tedious and painful affair. It quickly became apparent that they really didn't care about changing anything, even though their workload was high and any seemingly broken or problematic process was met with manual hand-cranking, rather than taking a little more time to try and solve the issue. There was a lot of time spent doing the same thing over and over again, things that could have easily been automated. Any question of feasibility of process or suggestion of alternatives with benefits where met with close-mindedness and the overall attitude that there simply was no problem.

My contract was terminated at the end of march, with a "thank you for all your help, goodbye" conversation. Honestly, I knew it was coming and I was glad of it.
I couldn't have worked in that environment for much longer and I have no idea how people can spend years there.
There were some positives out of the whole experience, however. I met some interesting people and I learnt YAML.

So, here I am. Back to the job search. Back to the anxiety.
Having been doing this now for a few weeks, I find myself in the same position as last year.
Working through recruitment consultants is difficult. continually chasing for roles I've applied for, endless calls and emails from consultants for jobs that I am not experienced to do, despite being told they'd read my CV.

More and more, when talking to directors of companies at interview stage, I find that I am having to explain that I am not a Linux administrator.
I have a short sentence within my personal statement that says that I like to play around with Linux in my spare time and use it at home. I'm paraphrasing here, obviously.
I've since heavily edited this sentence.
The bulk of my CV details roles within companies that are Microsoft houses, if not Microsoft partners. I have Microsoft accreditations, albeit that they're ageing now.

Another thing I find is that it seems people cant effectively manage resource any longer, especially in the contract space.
It goes like this:
company needs more staff -> company posts a new contract -> consultant passes applicants to the client -> silence.
Would it not be better to spend a little more time on hiring, in the knowledge that they will do some of that work, theoretically leaving you with more time?
I don't really understand it, not that I've been contracting for very long, but it seems the same can be said for full-time positions too.

So, while I'm waiting, I'm trying to do constructive things.
As mentioned before, my MS accreditations are now ageing and It seems like a good idea to update them, so I'm revising to take the 740-740/1/2 exams
to update my MCSA in windows server.
My current plan is to complete these and get the accreditaion, then start looking at getting a Cisco accreditation to bulster my networking knowledge.

I briefly spent some time researching the feasibility of going freelance, setting up a small consultancy and services offering, but it quickly became apparent that there is a lot of "peopling" involved, which was exhausting and the want for a mortgage anytime in the next year or so made it kind of out of the question right now.

In addition, I have the usual plethora of half finished projects that I dip in an out of.
Most recently trying to set up a reverse Apache proxy based on domain headers as my current provider doesn't offer additional IP addresses.
This has been going on for a month or so and is proving to be somewhat awkward.

Next up is the next iteration of a hardware firewall after the last one died the night I stood it up.
This time its on a raspberry pi, with an additional network interface. I've got to the stage of setting up both interfaces and installing things like DHCP, DNS etc and its now sat next to my desk, waiting for me to get back to it.

I also recently spent a not insignificant length of time trying to stand up a windows server 2016 virtual machine on the QEMU hypervisor, this too continues to evade me.

I have a lot of respect for people who manage to blog on a daily basis. Being able to pick a topic, construct an article with hopefully some interesting content, proof read and post all in a day, alongside anything and everything else they have to do on any given day.

I'd like to say that this will be the first in a line of new blog posts, with renewed vigour! but I doubt that is the case.
I seem to spend a large portion of my time watching YouTube gameplay videos, wondering what I should do next, has it been long enough that its worth checking job boards again, etc ad nauseam.
If you do stumble upon this and are interested in hearing more of anything noted here, feel free do drop a comment below and I will do an expansion post.

Until next time.....