A dual-camera server for Zoneminder

A while ago I managed to acquire a used Asus WL500G Premium wireless router for $50. Its a rather useful device as it is supported by OpenWRT and has two USB 2.0 ports. That’s quite unusual for a wireless router and it was a lucky find. My idea was to use it to run two USB webcams as IP cameras connected over the household LAN to a machine I have running the very good Zoneminder security software. The idea was to get better resolution than the conventional analog CCD camera + capture card system that I was using, which tops out at about 640 x 480.

I’m going to describe the process I went through to get this working in the hope that it might be useful to anyone else trying to assemble an OpenWRT-based multi-camera server. Its a pity that relatively few devices are available that will run OpenWRT and have multiple USB 2.0 ports, but they do exist and I hope to show they are worth pursuing.

So, I bought a Logitech S5500 webcam, which was capable of 960 x 720, connected it to the router, installed OpenWRT and mjpg-streamer, and… got very mediocre results. The video stream kept breaking up, triggering Zoneminders alarms, and the image couldn’t be expanded much beyond 512 x 384 before the load on the Zoneminder machines CPU went over 1.0, indicating Imminent Problems. I assumed that the issue was with the Zoneminder machine being underpowered; it was a 256Mbyte 1.2GHz Pentium III.

So when I eventually replaced it with a nice new Pentium dual-core E6500 with 1Gbyte, I was surprised to find that the problems remained. Some checking of the Zoneminder logs soon turned up some evidence; repeated complaints of “Invalid JPEG file structure: two SOI markers”. So. Clearly either the camera, or mjpg-streamer was generating bad JPEGs…

There is a moral here; always Google your error messages. I omitted to do this, or wasn’t thorough enough.

So I proceeded to build the latest stable version of OpenWRT Backfire for the WL50oG. OpenWRTs build process has been steadily improving; here is all that was needed. I did this build on a Atom-based netbook running Ubuntu Lucid:

sudo apt-get update
sudo apt-get install subversion build-essential libncurses5-dev zlib1g-dev gawk flex git-core quilt
mkdir openwrt
cd openwrt
svn checkout svn://svn.openwrt.org/openwrt/branches/backfire

When svn completes its downloads you can edit buildsystem/feeds.conf.default and save some time and bandwidth by taking out feeds you won’t use.  For this build I commented out the feeds for LuCI and the Xwrt WebUI.

cd backfire
./scripts/feeds update -a
./scripts/feeds install -a
make defconfig
make menuconfig

This will present a menu interface, in which I selected the following items: (pressing space twice to set to ‘*’, not ‘M’). I could have done all the customisation of OpenWRT after building with the default settings, but its easy to do here. I didn’t want to use the wireless connectivity, and I wanted mjpg-streamer and the ability to write to external USB drives (in case I wanted to configure the router as a time-lapse camera system), so:

Target System: (Broadcom BCM947xx/953xx)
Target Profile: (ASUS WL-500g Premium v1 (Atheros WiFi))
Base System:
     select block-mount
     deselect dnsmasq, firewall
Network:
     deselect iptables,ppp,wpad-mini
Kernel modules:
     select USB support: kmod-usb-core, kmod-usb-storage, kmod-usb2, kmod-usb-video, kmod-uhci
     select Video Support; kmod-video-core, kmod-video-uvc
     select Filesystems->kmod-fs-vfat, kmod-nls-xxxx
Multimedia:
     select mjpg-streamer

make V=99 2>&1 | tee build.log | grep -i error

After this build (which may take some time, as it needs to download quite a lot of source), the firmware image can be found in backfire/bin/brcm47xx/openwrt-brcm47xx-squashfs.trx. The fancy make invocation ensures that if things go wrong the console should show errors and the build.log file has the complete log of the build.

Installing the image, given that OpenWRT was already running on the router, was easy. First, I set up an ftp server on the build machine with anonymous access enabled:

sudo useradd -d /home/ftp/ftp -s /bin/false ftp
sudo mkdir -p /home/ftp/upload
sudo apt-get install vsftpd
sudo vi /etc/vsftpd.conf

I edited the configuration to set anonymous_enable=YES, then restarted the server to get it to reread the configuration file.

sudo service vsftpd restart

and copied the image into the server directory so the router could find it:

cp bin/brcm47xx/openwrt-brcm47xx-squashfs.trx /home/ftp/ftp

From here its a matter of logging in to OpenWRT on the router, and telling it to load the image over ftp and update to it:

cd /tmp
wget ftp://192.168.2.3/openwrt-brcm47xx-squashfs.trx
sysupgrade /tmp/openwrt-brcm47xx-squashfs.trx

192.168.2.3 happened to be the address the development machine had been given by my DHCP server; your mileage may vary.

OpenWRT Backfire doesn’t actually have the latest version of mjpg-streamer, but it swiftly became apparent that it hadn’t solved the problem. So my next step was to add some code to mjpg-streamer to filter out bad frames, on the assumption that these were coming from the webcam. The JPEG format is well documented (the Wikipedia article is a good source), and it it isn’t difficult to parse the basic structure without diving into the compressed data. Two SOI markers should be easy to detect.

To cut a long story short, my filter code, once running on the router, didn’t find any such thing.

At this point I did what I should have done much earlier, which was to Google “zoneminder” and “Invalid JPEG file structure: two SOI markers” a bit more thoroughly. And there in the Zoneminder FAQ is the answer:

What causes “Invalid JPEG file structure: two SOI markers” from zmc (1.24.x)

Some settings that used to be global only are now per camera. On the Monitor Source tab, if you are using Remote Protocol “HTTP” and Remote Method “Simple”, try changing Remote Method to “Regexp”.

This was indeed the fix. It was nothing to do with the camera or mjpg-streamer, but an internal problem in Zoneminder. I could now use the camera, but after only a day of testing a new problem became apparent. Some webcams, and the S5500 is one of them, have trouble with darkness. Here is a sample frame from the S5500 at night.

Night, badly rendered.

The little 8×8 blocks in this image are a giveaway that this really is a problem with the JPEG encoding. This time it was easy to prove that it was coming from the webcam, whose internal processor evidently can’t encode a uniformly black frame. Unfortunately, it doesn’t produce a consistent image like the one shown above, but a kaliedoscopic variety of them. This triggers Zoneminders motion detection, and you wind up with huge alarm events that last for hours and eat up your storage.

Fortunately, this problem is common enough that mjpg-streamer has a solution. The bad frames are all relatively small, there being no detail to encode, so a simple size test can detect them. The -minimum-size parameter allows us to throw away frames that are smaller than a certain size, and a little testing quickly revealed that at 960 x 720, any frame under 21000 bytes could be assumed to be bad. So:

/usr/bin/mjpg_streamer -i "input_uvc.so -d /dev/video0 -r 960x720 -m 21000" -o "output_http.so -p 80"

should be the solution? Well, nearly. The trouble is that what mjpg-streamer does with the bad frames is skip over them and not return a result until it gets a good frame. Which is fine for occasional bad frames, but when the camera produces a continuous stream of them – at night – mjpg-streamer may not return a result for hours. Zoneminder doesn’t like this either, and is prone to disable the camera altogether. This problem is apparently addressed in Zoneminder 1.25, but I’m running 1.24.2, so another solution was needed. The easiest thing to do was to edit input_uvc.c in the mjpg-streamer source so that instead of skipping the frame, mjpg-streamer returned the last good frame that it had. This was a matter of changing the lines

        if(pcontext->videoIn->buf.bytesused < minimum_size) {
            DBG("dropping too small frame, assuming it as broken\n");
            continue;
        }

to:

        if(pcontext->videoIn->buf.bytesused < minimum_size) {
            DBG("dropping too small frame, assuming it as broken\n");
            /* Provide the previous frame, as this condition might last some time... */
            pthread_cond_broadcast(&pglobal->in[pcontext->id].db_update);
            continue;
        }

This change isn’t really suitable for all uses as it means that the camera server may return an image of what was visible several hours ago when the scene was adequately lit, rather than current darkness. A more elegant solution would be to return a synthetic JPEG that was just a uniform field of black, but this was a quick workaround for my purposes.

Changing the code in the OpenWRT source and getting it recompile without ignoring or overwriting the changes was a bit tricky. The combination of OpenWRTs interesting build system as it interacts with mjpg-streamers makefile means that just changing input_uvc.c deep down in the build_dir directory and doing a make won’t work. After some trial and error I wrote a script to do the rebuild:

#!/bin/sh
MJPGDIR="build_dir/target-mipsel_uClibc-0.9.30.1/mjpg-streamer-r136"
touch $MJPGDIR/mjpg_streamer.c
rm $MJPGDIR/ipkg-brcm47xx/mjpg-streamer/usr/lib/*.so
rm $MJPGDIR/*.so
make V=99 >build.log

Having fixed this, I bought a second webcam (a Logitech C905) as it was now clear the whole idea would work as intended. The differences between the two webcams when looking at the same scene were surprising. Here is the S5500:

The C905 wins on detail, but also has quite different colour rendering.

The final step was to get both webcams to start up when OpenWRT booted.

OpenWRTs initialisation is a little complicated, and is controlled by the files in /etc/rc.d, /etc/init.d, and /etc/config. The mjpg-streamer package includes appropriate entries in each of these directories to automatically start up a single instance, but there is nothing to prevent us from running two versions provided that we tell them to serve to different TCP ports. The two webcams appear as /dev/video0 and /dev/video1. So we edit /etc/config/mjpg-streamer and add a couple more config options for device2 and port2:

config mjpg-streamer core
        option device           "/dev/video0"
        option device2          "/dev/video1"
        option resolution       "960x720"
        option minimumsize      "21000"
        option fps              "10"
        option port             "80"
        option port2            "8080"                                
        option enabled          "true"

Then we add an entry in init.d; the easiest way is to duplicate the existing script:

cd /etc/init.d
cp mjpg-streamer mjpg-streamer2

and edit the second script to use the device2 and port2 options, plus its own PID file name.

#!/bin/sh /etc/rc.common
# Copyright (C) 2009 OpenWrt.org
START=50

SSD=start-stop-daemon
NAME=mjpg_streamer
PIDF=/var/run/$NAME.2.pid
PROG=/usr/bin/$NAME

start() {
        config_load mjpg-streamer
        config_get device core device2
        config_get resolution core resolution
        config_get fps core fps
        config_get port core port2
        config_get_bool enabled core enabled
        config_get minimumsize core minimumsize
        [ $enabled -gt 0 -a -c $device ] && sleep 3 && $SSD -S -m -p $PIDF -q -x $PROG\
         -- --input "input_uvc.so --device $device --fps $fps --resolution $resolution"\
         --output "output_http.so --port $port" &
}

stop() {
        $SSD -K -p $PIDF
}

Finally we need to add a link in the rc.d directory to the new init.d script:

cd /etc
ln rc.d/S50mjpg-streamer2 ../init.d/mjpg-streamer2

This will start up the WL500 with both cameras if both are present, and serve them on port 80 and port 8080. The images and movie streams can therefore be found at:

http://<IP address>/?action=snapshot

http://<IP address>:8080/?action=snapshot

and

http://<IP address>/?action=stream

http://<IP address>:8080/?action=stream

respectively. Entering the snapshot URLs above into Zoneminder has produced a working system. The only remaining issue was setting up  Zoneminder to minimise the CPU usage, which is an art in itself and better covered in the Zoneminder forums…

Advertisement

A Battery-powered Time-Lapse Camera

Introduction

The time-lapse camera system described in my earlier post is fine where mains power is available, but for remote locations and longer durations it rather falls down. At 5-6W power consumption it will flatten even a large wet-cell lead-acid battery in 3-4 days. Taking pictures less frequently does little to reduce the power consumption as the camera and NSLU2 run all the time.

Unless we turn the system off — or rather, get it to turn itself off. The NSLU2 has an onboard power supply switch controlled by a separate piece of circuitry that turns power on to the main processor when the power button is pressed. The power button cannot, however, turn the switch off. Instead the switch is designed so that the processor itself has to commit electronic suicide and turn its own power off by sending a signal to the switch. In the ‘off’ state — which as you can see, isn’t really off — only the power switch circuitry uses power, and it uses very little, only 1.3mA. The main processor and anything connected to the USB ports are simply disconnected from the supply voltage. So turning the power off is easy – OpenWRT provides the poweroff command, which does just what it says. The question then becomes, how do we turn the power back on at regular intervals?

A hardware modification

The NSLU2 has an X1205 real-time clock chip that keeps the time and date. It runs continuously, even when the power supply is physically unplugged – that is its 3V battery in the middle of the board. This chip has an alarm facility; it has a pin that can be programmed to send an interrupt signal to the main processor at a specific time and date, up to a year ahead. On the NSLU2 board, this interrupt signal pin is not connected to anything. If we connect it to ‘on’ input of the power supply switch, the X1205 can turn the NSLU2 on when the alarm time is reached. This turns out to be a fairly simple hardware mod. A wire connecting pin 3 of the X1205 to the U15 side of R94 will do the job. The wire has to run from one side of the board to the other, and requires a fine-tipped soldering iron as R94 and the X1205 are both surface mount devices. The photographs below show where it goes.

Software changes

Having added this wire, all we need is some software to set the alarm time and all sorts of interesting possibilities appear. The software turned out to be a little more complicated than I had expected, as it turned out that the low-level Linux driver for the X1205 wasn’t quite working properly. It didn’t correctly set the alarm values or provide a way to enable the interrupt pin. Nor, having fixed the driver, is there a standard Linux tool for setting up the alarm in the way we need. Fortunately there is a program out there called rtcwake that does nearly the job we want, and I was able to modify it to make a similar program I have called rtcalarm.

With rtcalarm, the fixed driver, and the wire in place, we can now have the NSLU2 turn on, take a single photo, and turn itself off again at any interval. Well, almost any interval. It takes a little over a minute to start up, so the smallest sensible interval would be longer than that, say 90 seconds. The longer the interval, the lower the power consumption.

There was one small remaining problem with this setup. When the time came to make a movie out of the contents of the flash drive, the file renaming strategy described in my earlier post on time-lapse cameras no longer worked. Now the serial number part of the filename is always 000000000, because MJPG-streamer is being restarted every time. The solution to this was to make a small change to the output_file plugin of MJPG-streamer to omit the serial number and the underscore characters separating the time fields. The file names now have the form yyyyymmddmmhhss.jpg. This makes converting them to movies with QuickTime Pro straightforward – no file name changes are required.

The end result: a power-cycling time-lapse camera

An OpenWRT firmware build with the rtcalarm tool and the driver patch, openwrt-nslu2-uvc-webcam3.bin is available here. It provides the same three modes as the build described in A MegaPixel Time-Lapse Camera System – webcam, dual webcam, and time-lapse. In time-lapse mode, however, it now requires the hardware mod, and takes pictures every six minutes, turning itself off for five minutes out of six.

The net power consumption of the new time-lapse mode is a little tricky to work out, as the camera only starts using power quite late in the boot-photograph-shutdown cycle, so the full 1100mA is only used for about ten seconds at the end. I used a data acquisition system to monitor the current draw at 5V over a single photo cycle, and it looks like this:

This shows that it takes 47.6 A/secs to take a photo, if I can be excused for employing such units. The profile above is for a unit not connected to Ethernet. With Ethernet, power use is slightly higher but the cycle runs some ten seconds shorter and uses 44.6 A/secs. At 5V thats 238W/secs per photo with no network. With these figures in hand, we can estimate how many photos we can get from a battery. For example, a 12V 7AH gel cell running the NSLU2 through my switching voltage regulator contains 12×7 = 84W/hours. If the regulator is 88% efficient we get 73.9W/hours delivered to the NSLU2, which is 266112W/secs and therefore about 1118 photos. This number is largely independent of the interval between photos, because the power consumption of the NSLU2 while off is so low.

Internal details

This build is based on a later version of the OpenWRT trunk, revision 11566. It has a later version (r63) of the MJPG-streamer software, which has some bug fixes and should behave better when streaming to a web browser. I had to apply a patch I found here to the /sbin/usb-storage script to get the flash drive to mount correctly at power-on. As usual, most of the controlling code is in /etc/init.d/done. The only exception is the script at /root/whoa which sets the RTC alarm and turns the power off. If you want to change the number of seconds between pictures, edit the rtcalarm command in this script. The /root/whoa script is called by the MJPG-streamer output_file plugin via the -c option

The source for the rtcalarm tool can be found here, along with the patched rtc-x1205.c driver and the other patches.

A Megapixel Time-lapse Camera System

Introduction
The NSLU2 running OpenWRT and the QuickCam Pro 9000 combine to make a very versatile device. In this post I’m going to describe how to use them as a time-lapse camera which takes photos every ten seconds at 1600×1200 resolution and stores them on a flash drive. The photos occupy anywhere from 80K to 400K each depending on the complexity of the scene. This means that around 10,000 to 15,000 will fit on a 4GByte flash drive, corresponding to 20-40 hours of coverage. All that is required is an NSLU2, a UVC-compatible webcam, a flash drive, and an OpenWRT firmware build, which is available here. A brief sample movie (30 seconds, 8.5Mbytes, DivX) is available here.

Further notes on the NSLU2 and QuickCam Pro 9000

I have been playing around with the webcam firmware that was the subject of my last post and have a number of random notes that I should put up here on using it.

A High-Resolution IP Webcam

Web cams are fairly ubiquitous things these days and by no means expensive. They can be good or bad depending on how much money you want to spend, but there is one almost-universal rule, which is that they connect to a host PC over USB. IP-based cameras that connect to a LAN via an RJ45 connector or wirelessly over 802.11 are quite a bit more useful, because they can be put almost anywhere, but they tend to cost a surprising amount and not provide much resolution. A low-end one like the LevelOne FCS1030 is NZ$260, and they go a lot more expensive than that. For example, the wireless D-Link DCS5300 sells for NZ$930 and it only does 320×240 pixels.