Sunday, April 12, 2015

Raspberry PI Video Streaming to Android or Web Browser with Minimal Lag

Video streaming on the Raspberry Pi is a commonly asked question on the forums and there are many different methodologies. Although many of them work ok the biggest difference in all of them is the amount of lag in the video stream. My application is using the Raspberry Pi camera attached to a robot arm. As I moved the arm I wanted the see the video in real-time on my Android app. Any lag would be a killer blow to this working nicely.

Ultimately I was constrained in my possible solutions because I wanted to stream to an Android app and Android only supports a specific set of streaming protocols. See http://developer.android.com/guide/appendix/media-formats.html, specifically RTSP and HTTP.

With Android development a MediaPlayer view is used to display video. For streaming it needs to take its source from either a RTSP or HTTP URL. It cannot take a stream directly from a socket because the file descriptor API of MediaPlayer expects a seekable file which obviously a socket isn't.

After a lot of trial and error I managed to get a working solution using u4vl and WebView in Android. To dive straight ahead to my solution have a look at UV4L below. However, before we get there I thought it might be useful to do a quick recap of other solutions.

Things I Tried

When trying different solutions the two main things I was looking at were video lag and ease of integration with an Android app.

The Simplest - Raspvid and nc

Using raspvid and piping it into nc. See https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=87903
On the pi:
raspivid -t 0 -fps 24 -o - | nc -k -l 8554
and on the client:
nc 8554 | vlc --file-caching=1024 file/h264:///dev/stdin
This is certainly the simplest and easiest. However, this is only supported by a limited number of media players and not by Android. The lag with this method was 2-3 seconds.

VLC

Most of the posts suggest using VLC with something along the lines of:

raspivid -o - -t 0 -hf -w 640 -h 360 -fps 5|cvlc -vvv stream:///dev/stdin --sout '#standard{access=http,mux=ts,dst=:8090}' :demux=h264

Although this played on VLC in Windows, Linux and on VLC for Android it was not in a native format supported by the the Android MediaPlayer.
The lag was also terrible at about 5-6 seconds. This could not be reduced be even using small video resolutions.
VLC is doing a certain amount of buffering. If you need real time this is not the way to go.

GStreamer

Many posts talked about getting better performance with minimal lag using GStreamer. GStreamer is a framework for piping video and audio through various filters to do transformations and conversions. Raspvid is piped into GStreamer which in turn is piped into a RTSP server which any web browsers can connect to. RTSP can also be directly stream by MediaPlayer in Android. Thsi seemed like a promising solution. However, the biggest problem is getting a recent built version installed on your Raspberry PI. You can install GStreamer 1.0 on Wheezy using apt-get but you have to compile the RTSP server since its not part of the package install. Unfortunately version 1.2.3 of the RTSP server will not compile against the 1.2 version in the Wheezy repository. At this point I was faced with either trying GStreamer 0.10 which has lag or compiling the whole of GStreamer 1.2.3 yourself. GStreamer 1.4 is going into the next version of Debian (Jessie) but will not be installed on Wheezy. This was all starting to get a little too much like hard work.

GStreamer might still end up being a good solution but its tricky to get installed at the moment. However, here are some useful posts on the subject:

(This includes full build instructions)
Detailed description of setting up GStreamer: http://www.z25.org/static/_rd_/videostreaming_intro_plab/index.html

mjpeg-streamer


Instead of streaming video, how about capturing still images and sending them to a web server one
after each other? This is called Motion JPEG is a commonly used by web cams in surveillance systems. There is a mjpeg-streamer implementation for the Pi; see https://www.raspberrypi.org/forums/viewtopic.php?t=48597

Although this worked quite nicely and was easy to setup it still had a lag of 1-2 seconds.

UV4L


I finally stumbled upon a clear winner; UV4L. UV4L was easy to setup and I could get a solution with very minimal lag. It comes with a server that can stream MJPEG. Since MJPEG is supported by web browsers there is an easy way to integrate a web page into an Android app using WebView. Its  as simple as adding the WebView to your layout then setting to URL to the server running on the Pi. See below.

For setup, follow the instructions on: http://www.linux-projects.org/uv4l/installation/

Install all modules including the optional ones.

For streaming follow the instructions in here http://www.linux-projects.org/uv4l/tutorials/streaming-server/
and you're done!

I had to set the frame rate and the video resolution to be quite low to reduce the lag.

To configure UV4L edit the configuration file in /etc/uv4l/uv4l-raspicam.conf. The key part of the file is:

encoding = mjpeg
width = 320
height = 240
framerate = 10
I also specified some server options to allow more connections:

server-option = --max-streams=5
server-option = --max-threads=10
server-option = --thread-idle-time=5

As soon as I get Raspberry PI 2 I'm going to give the WebRTC option a go and check out its performance.

Android App

The most simplest method is to use a WebView. If you can view it in a web browser you can view it in WebView. First add the WebView to your layout. In in your activity call loadUrl() on the WebView to start streaming.



Note: the URL is loaded in a call to post(). This anonymous method is call after the layout has been drawn. This is important because I'm passing the width and heigtht of the WebView into the URL and the dimensions are only known after the layout has been drawn. The size of the WebView is different for different layouts (landscape/portrait) for phones and tablets.

There is an MJPEG view class available from https://bitbucket.org/neuralassembly/simplemjpegview but I could not get his app to work with my UV4L stream so was not tempted to integrate the classes into my app.

The WebView worked fine and in particular was able to deal with the different windows sizes used on the app's multiple layouts.

9 comments:

  1. I could lend you my Pi2 for awhile, I don't have any project in mind for it at the moment.

    ReplyDelete
  2. Excellent work.
    Seen Your Video https://www.youtube.com/watch?v=b49IQAWUIJM.

    You could publish your file uv4l-raspicam.conf settings 1920x1024 at 30 FPS?

    ReplyDelete
    Replies
    1. See comments on latest blog post.

      /etc/uv4l/uv4l-raspicam.conf just has three lines changed:

      width = 1920
      height = 1024
      framerate = 30

      Restart your sever using:

      sudo /etc/init.d/uv4l_raspicam restart

      Delete
  3. Your website is really cool and this is a great inspiring article.
    streaming de video on demand

    ReplyDelete
  4. Will the UV4L work for a webcam? Or does it only work for Raspberry Pi cameras?

    ReplyDelete
  5. Hi Paul. I can view the uv4l stream from RPI camera via chrome. But not in android, the webview keeps showing white blank screen. Could you please help? Thank you

    ReplyDelete
  6. Things to try:
    URL should be singular to http://192.168.1.80:8080/stream
    It had to be the stream url. The root site may not work.
    Try viewing this URL on pc browser then try a browser on your phone. If you have problems see this stage then you have a problem with uv4l or a connectivity issue.
    If you have problems with Webview try pointing at a simple web site. If this doesn't work you need to get on the android forums on stack overflow.

    ReplyDelete
  7. Hello,

    could anyone advice me how to stream FullHD stream over UDP from Raspicam V2?
    I am using following pipeline, but the video is always cropped and not FULLHD:
    rpicamsrc preview=false !video/x-raw, width=1920,height=1080,framerate=24/1 ! clockoverlay ! omxh264enc target-bitrate=1000000 control-rate=variable ! video/x-h264,width=1280, height=720, framerate=30/1, profile=(string)high ! h264parse ! rtph264pay name=pay0 pt=96

    I tried to change the caps to obtain video/x-h264 directly from rpicamsrc without explicitly using omx codec, but I face the same problem.
    There is not problem obtaining FULLHD video via raspivid application, but I need to transfer the stream over UDP and piping RASPIVID to gstreamer is of no use for I am using gst-rtsp-server and I don't think the pipeline could be assembled in such a way it allows external program to feed stream to gstreamer

    Thank you
    Best regards,
    Ivo

    ReplyDelete