A dual-camera server for Zoneminder

A while ago I managed to acquire a used Asus WL500G Premium wireless router for $50. Its a rather useful device as it is supported by OpenWRT and has two USB 2.0 ports. That’s quite unusual for a wireless router and it was a lucky find. My idea was to use it to run two USB webcams as IP cameras connected over the household LAN to a machine I have running the very good Zoneminder security software. The idea was to get better resolution than the conventional analog CCD camera + capture card system that I was using, which tops out at about 640 x 480.

I’m going to describe the process I went through to get this working in the hope that it might be useful to anyone else trying to assemble an OpenWRT-based multi-camera server. Its a pity that relatively few devices are available that will run OpenWRT and have multiple USB 2.0 ports, but they do exist and I hope to show they are worth pursuing.

So, I bought a Logitech S5500 webcam, which was capable of 960 x 720, connected it to the router, installed OpenWRT and mjpg-streamer, and… got very mediocre results. The video stream kept breaking up, triggering Zoneminders alarms, and the image couldn’t be expanded much beyond 512 x 384 before the load on the Zoneminder machines CPU went over 1.0, indicating Imminent Problems. I assumed that the issue was with the Zoneminder machine being underpowered; it was a 256Mbyte 1.2GHz Pentium III.

So when I eventually replaced it with a nice new Pentium dual-core E6500 with 1Gbyte, I was surprised to find that the problems remained. Some checking of the Zoneminder logs soon turned up some evidence; repeated complaints of “Invalid JPEG file structure: two SOI markers”. So. Clearly either the camera, or mjpg-streamer was generating bad JPEGs…

There is a moral here; always Google your error messages. I omitted to do this, or wasn’t thorough enough.

So I proceeded to build the latest stable version of OpenWRT Backfire for the WL50oG. OpenWRTs build process has been steadily improving; here is all that was needed. I did this build on a Atom-based netbook running Ubuntu Lucid:

sudo apt-get update
sudo apt-get install subversion build-essential libncurses5-dev zlib1g-dev gawk flex git-core quilt
mkdir openwrt
cd openwrt
svn checkout svn://svn.openwrt.org/openwrt/branches/backfire

When svn completes its downloads you can edit buildsystem/feeds.conf.default and save some time and bandwidth by taking out feeds you won’t use.  For this build I commented out the feeds for LuCI and the Xwrt WebUI.

cd backfire
./scripts/feeds update -a
./scripts/feeds install -a
make defconfig
make menuconfig

This will present a menu interface, in which I selected the following items: (pressing space twice to set to ‘*’, not ‘M’). I could have done all the customisation of OpenWRT after building with the default settings, but its easy to do here. I didn’t want to use the wireless connectivity, and I wanted mjpg-streamer and the ability to write to external USB drives (in case I wanted to configure the router as a time-lapse camera system), so:

Target System: (Broadcom BCM947xx/953xx)
Target Profile: (ASUS WL-500g Premium v1 (Atheros WiFi))
Base System:
     select block-mount
     deselect dnsmasq, firewall
Network:
     deselect iptables,ppp,wpad-mini
Kernel modules:
     select USB support: kmod-usb-core, kmod-usb-storage, kmod-usb2, kmod-usb-video, kmod-uhci
     select Video Support; kmod-video-core, kmod-video-uvc
     select Filesystems->kmod-fs-vfat, kmod-nls-xxxx
Multimedia:
     select mjpg-streamer

make V=99 2>&1 | tee build.log | grep -i error

After this build (which may take some time, as it needs to download quite a lot of source), the firmware image can be found in backfire/bin/brcm47xx/openwrt-brcm47xx-squashfs.trx. The fancy make invocation ensures that if things go wrong the console should show errors and the build.log file has the complete log of the build.

Installing the image, given that OpenWRT was already running on the router, was easy. First, I set up an ftp server on the build machine with anonymous access enabled:

sudo useradd -d /home/ftp/ftp -s /bin/false ftp
sudo mkdir -p /home/ftp/upload
sudo apt-get install vsftpd
sudo vi /etc/vsftpd.conf

I edited the configuration to set anonymous_enable=YES, then restarted the server to get it to reread the configuration file.

sudo service vsftpd restart

and copied the image into the server directory so the router could find it:

cp bin/brcm47xx/openwrt-brcm47xx-squashfs.trx /home/ftp/ftp

From here its a matter of logging in to OpenWRT on the router, and telling it to load the image over ftp and update to it:

cd /tmp
wget ftp://192.168.2.3/openwrt-brcm47xx-squashfs.trx
sysupgrade /tmp/openwrt-brcm47xx-squashfs.trx

192.168.2.3 happened to be the address the development machine had been given by my DHCP server; your mileage may vary.

OpenWRT Backfire doesn’t actually have the latest version of mjpg-streamer, but it swiftly became apparent that it hadn’t solved the problem. So my next step was to add some code to mjpg-streamer to filter out bad frames, on the assumption that these were coming from the webcam. The JPEG format is well documented (the Wikipedia article is a good source), and it it isn’t difficult to parse the basic structure without diving into the compressed data. Two SOI markers should be easy to detect.

To cut a long story short, my filter code, once running on the router, didn’t find any such thing.

At this point I did what I should have done much earlier, which was to Google “zoneminder” and “Invalid JPEG file structure: two SOI markers” a bit more thoroughly. And there in the Zoneminder FAQ is the answer:

What causes “Invalid JPEG file structure: two SOI markers” from zmc (1.24.x)

Some settings that used to be global only are now per camera. On the Monitor Source tab, if you are using Remote Protocol “HTTP” and Remote Method “Simple”, try changing Remote Method to “Regexp”.

This was indeed the fix. It was nothing to do with the camera or mjpg-streamer, but an internal problem in Zoneminder. I could now use the camera, but after only a day of testing a new problem became apparent. Some webcams, and the S5500 is one of them, have trouble with darkness. Here is a sample frame from the S5500 at night.

Night, badly rendered.

The little 8×8 blocks in this image are a giveaway that this really is a problem with the JPEG encoding. This time it was easy to prove that it was coming from the webcam, whose internal processor evidently can’t encode a uniformly black frame. Unfortunately, it doesn’t produce a consistent image like the one shown above, but a kaliedoscopic variety of them. This triggers Zoneminders motion detection, and you wind up with huge alarm events that last for hours and eat up your storage.

Fortunately, this problem is common enough that mjpg-streamer has a solution. The bad frames are all relatively small, there being no detail to encode, so a simple size test can detect them. The -minimum-size parameter allows us to throw away frames that are smaller than a certain size, and a little testing quickly revealed that at 960 x 720, any frame under 21000 bytes could be assumed to be bad. So:

/usr/bin/mjpg_streamer -i "input_uvc.so -d /dev/video0 -r 960x720 -m 21000" -o "output_http.so -p 80"

should be the solution? Well, nearly. The trouble is that what mjpg-streamer does with the bad frames is skip over them and not return a result until it gets a good frame. Which is fine for occasional bad frames, but when the camera produces a continuous stream of them – at night – mjpg-streamer may not return a result for hours. Zoneminder doesn’t like this either, and is prone to disable the camera altogether. This problem is apparently addressed in Zoneminder 1.25, but I’m running 1.24.2, so another solution was needed. The easiest thing to do was to edit input_uvc.c in the mjpg-streamer source so that instead of skipping the frame, mjpg-streamer returned the last good frame that it had. This was a matter of changing the lines

        if(pcontext->videoIn->buf.bytesused < minimum_size) {
            DBG("dropping too small frame, assuming it as broken\n");
            continue;
        }

to:

        if(pcontext->videoIn->buf.bytesused < minimum_size) {
            DBG("dropping too small frame, assuming it as broken\n");
            /* Provide the previous frame, as this condition might last some time... */
            pthread_cond_broadcast(&pglobal->in[pcontext->id].db_update);
            continue;
        }

This change isn’t really suitable for all uses as it means that the camera server may return an image of what was visible several hours ago when the scene was adequately lit, rather than current darkness. A more elegant solution would be to return a synthetic JPEG that was just a uniform field of black, but this was a quick workaround for my purposes.

Changing the code in the OpenWRT source and getting it recompile without ignoring or overwriting the changes was a bit tricky. The combination of OpenWRTs interesting build system as it interacts with mjpg-streamers makefile means that just changing input_uvc.c deep down in the build_dir directory and doing a make won’t work. After some trial and error I wrote a script to do the rebuild:

#!/bin/sh
MJPGDIR="build_dir/target-mipsel_uClibc-0.9.30.1/mjpg-streamer-r136"
touch $MJPGDIR/mjpg_streamer.c
rm $MJPGDIR/ipkg-brcm47xx/mjpg-streamer/usr/lib/*.so
rm $MJPGDIR/*.so
make V=99 >build.log

Having fixed this, I bought a second webcam (a Logitech C905) as it was now clear the whole idea would work as intended. The differences between the two webcams when looking at the same scene were surprising. Here is the S5500:

The C905 wins on detail, but also has quite different colour rendering.

The final step was to get both webcams to start up when OpenWRT booted.

OpenWRTs initialisation is a little complicated, and is controlled by the files in /etc/rc.d, /etc/init.d, and /etc/config. The mjpg-streamer package includes appropriate entries in each of these directories to automatically start up a single instance, but there is nothing to prevent us from running two versions provided that we tell them to serve to different TCP ports. The two webcams appear as /dev/video0 and /dev/video1. So we edit /etc/config/mjpg-streamer and add a couple more config options for device2 and port2:

config mjpg-streamer core
        option device           "/dev/video0"
        option device2          "/dev/video1"
        option resolution       "960x720"
        option minimumsize      "21000"
        option fps              "10"
        option port             "80"
        option port2            "8080"                                
        option enabled          "true"

Then we add an entry in init.d; the easiest way is to duplicate the existing script:

cd /etc/init.d
cp mjpg-streamer mjpg-streamer2

and edit the second script to use the device2 and port2 options, plus its own PID file name.

#!/bin/sh /etc/rc.common
# Copyright (C) 2009 OpenWrt.org
START=50

SSD=start-stop-daemon
NAME=mjpg_streamer
PIDF=/var/run/$NAME.2.pid
PROG=/usr/bin/$NAME

start() {
        config_load mjpg-streamer
        config_get device core device2
        config_get resolution core resolution
        config_get fps core fps
        config_get port core port2
        config_get_bool enabled core enabled
        config_get minimumsize core minimumsize
        [ $enabled -gt 0 -a -c $device ] && sleep 3 && $SSD -S -m -p $PIDF -q -x $PROG\
         -- --input "input_uvc.so --device $device --fps $fps --resolution $resolution"\
         --output "output_http.so --port $port" &
}

stop() {
        $SSD -K -p $PIDF
}

Finally we need to add a link in the rc.d directory to the new init.d script:

cd /etc
ln rc.d/S50mjpg-streamer2 ../init.d/mjpg-streamer2

This will start up the WL500 with both cameras if both are present, and serve them on port 80 and port 8080. The images and movie streams can therefore be found at:

http://<IP address>/?action=snapshot

http://<IP address>:8080/?action=snapshot

and

http://<IP address>/?action=stream

http://<IP address>:8080/?action=stream

respectively. Entering the snapshot URLs above into Zoneminder has produced a working system. The only remaining issue was setting up  Zoneminder to minimise the CPU usage, which is an art in itself and better covered in the Zoneminder forums…

Advertisement