Raspberry PI Video Streaming to Android or Web Browser with Minimal Lag

Video streaming on the Raspberry Pi is a commonly asked question on the forums and there are many different methodologies. Although many of them work ok the biggest difference in all of them is the amount of lag in the video stream. My application is using the Raspberry Pi camera attached to a robot arm. As I moved the arm I wanted the see the video in real-time on my Android app. Any lag would be a killer blow to this working nicely.

Ultimately I was constrained in my possible solutions because I wanted to stream to an Android app and Android only supports a specific set of streaming protocols. See http://developer.android.com/guide/appendix/media-formats.html, specifically RTSP and HTTP.

With Android development a MediaPlayer view is used to display video. For streaming it needs to take its source from either a RTSP or HTTP URL. It cannot take a stream directly from a socket because the file descriptor API of MediaPlayer expects a seekable file which obviously a socket isn't.

After a lot of trial and error I managed to get a working solution using u4vl and WebView in Android. To dive straight ahead to my solution have a look at UV4L below. However, before we get there I thought it might be useful to do a quick recap of other solutions.

Things I Tried

When trying different solutions the two main things I was looking at were video lag and ease of integration with an Android app.

The Simplest - Raspvid and nc

Using raspvid and piping it into nc. See https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=87903. 

On the pi:

raspivid -t 0 -fps 24 -o - | nc -k -l 8554

and on the client:

nc 8554 | vlc --file-caching=1024 file/h264:///dev/stdin

This is certainly the simplest and easiest. However, this is only supported by a limited number of media players and not by Android. The lag with this method was 2-3 seconds.

VLC

Most of the posts suggest using VLC with something along the lines of:

raspivid -o - -t 0 -hf -w 640 -h 360 -fps 5|cvlc -vvv stream:///dev/stdin --sout '#standard{access=http,mux=ts,dst=:8090}' :demux=h264

Although this played on VLC in Windows, Linux and on VLC for Android it was not in a native format supported by the the Android MediaPlayer.

The lag was also terrible at about 5-6 seconds. This could not be reduced be even using small video resolutions.

VLC is doing a certain amount of buffering. If you need real time this is not the way to go.

GStreamer

Many posts talked about getting better performance with minimal lag using GStreamer. GStreamer is a framework for piping video and audio through various filters to do transformations and conversions. Raspvid is piped into GStreamer which in turn is piped into a RTSP server which any web browsers can connect to. RTSP can also be directly stream by MediaPlayer in Android. Thsi seemed like a promising solution. However, the biggest problem is getting a recent built version installed on your Raspberry PI. You can install GStreamer 1.0 on Wheezy using apt-get but you have to compile the RTSP server since its not part of the package install. Unfortunately version 1.2.3 of the RTSP server will not compile against the 1.2 version in the Wheezy repository. At this point I was faced with either trying GStreamer 0.10 which has lag or compiling the whole of GStreamer 1.2.3 yourself. GStreamer 1.4 is going into the next version of Debian (Jessie) but will not be installed on Wheezy. This was all starting to get a little too much like hard work.

GStreamer might still end up being a good solution but its tricky to get installed at the moment. However, here are some useful posts on the subject:

GStreamer 0.10: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=49278

GStreamer 1.0: http://rpiquadcopter.blogspot.co.uk/2014/06/raspberry-pi-camera-module-video.html

(This includes full build instructions)

Detailed description of setting up GStreamer: http://www.z25.org/static/_rd_/videostreaming_intro_plab/index.html

mjpeg-streamer

Instead of streaming video, how about capturing still images and sending them to a web server one
after each other? This is called Motion JPEG is a commonly used by web cams in surveillance systems. There is a mjpeg-streamer implementation for the Pi; see https://www.raspberrypi.org/forums/viewtopic.php?t=48597

Although this worked quite nicely and was easy to setup it still had a lag of 1-2 seconds.

UV4L

I finally stumbled upon a clear winner; UV4L. UV4L was easy to setup and I could get a solution with very minimal lag. It comes with a server that can stream MJPEG. Since MJPEG is supported by web browsers there is an easy way to integrate a web page into an Android app using WebView. Its  as simple as adding the WebView to your layout then setting to URL to the server running on the Pi. See below.

For setup, follow the instructions on: http://www.linux-projects.org/uv4l/installation/

Install all modules including the optional ones.

For streaming follow the instructions in here http://www.linux-projects.org/uv4l/tutorials/streaming-server/

and you're done!

I had to set the frame rate and the video resolution to be quite low to reduce the lag.

To configure UV4L edit the configuration file in /etc/uv4l/uv4l-raspicam.conf. The key part of the file is:

encoding = mjpeg
width = 320
height = 240
framerate = 10

I also specified some server options to allow more connections:

server-option = --max-streams=5
server-option = --max-threads=10
server-option = --thread-idle-time=5

As soon as I get Raspberry PI 2 I'm going to give the WebRTC option a go and check out its performance.

Android App

The most simplest method is to use a WebView. If you can view it in a web browser you can view it in WebView. First add the WebView to your layout. In in your activity call loadUrl() on the WebView to start streaming.

Note: the URL is loaded in a call to post(). This anonymous method is call after the layout has been drawn. This is important because I'm passing the width and heigtht of the WebView into the URL and the dimensions are only known after the layout has been drawn. The size of the WebView is different for different layouts (landscape/portrait) for phones and tablets.

There is an MJPEG view class available from https://bitbucket.org/neuralassembly/simplemjpegview but I could not get his app to work with my UV4L stream so was not tempted to integrate the classes into my app.

The WebView worked fine and in particular was able to deal with the different windows sizes used on the app's multiple layouts.