R-Pi Security Webcam Dropbox Remix

Having recently had a bad experience involving some local hoodlums, some tins of spray paint, mistaken identity and a pristine white garage door, I decided I needed some slightly better security at my home.

R-Pi to the rescue!


WARNING! If you are allergic to a command line interface to a computer you might choose not to go any further! I've tried to show *exactly* what I did in the hope it helps someone, so for the less fainthearted... read on...

Also note that it's *really* useful to have SSH set up and running on the Pi first, so you can connect to it from another computer and transfer files and stuff!

How to set up a Raspberry Pi to use an EDIMAX USB wireless adapter and connect to a WPA2-PSK secured wireless access point

But I had a few problems to overcome first. I had a Model B, with Raspian installed via NOOBS - but no ethernet where I wanted to put the camera. So I first had to work out how to get the R-Pi connected to a WPA2-PSK secured Wifi Router.

Luckily others have also already sorted this issue already, the trick is to get the networking sorted enough via use of wpa_supplicant to a) have the pre-shared key which is generated by your wifi router's password, and b) to then run a script that checks for connection and continually reconnects if unavailable, thus avoiding the dreaded wifi dropout.

I shoved my Edimax wifi USB adapter into the Pi and powered it up.

 ) details the setup - having the pre-up and post-down lines in /etc/network/interfaces is key. As is doing the wpa_passphrase <YOUR-ESSID> <YOUR-PASSPHRASE> step.

So let's do it:

sudo nano /etc/network/interfaces

Which will then let you modify your config file until it looks something like the below.

auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0
iface wlan0 inet dhcp
wireless-essid <YOUR-WIFI-ESSID>
#wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
pre-up wpa_supplicant -B w -D wext -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf
post-down killall -q wpa_supplicant
iface default inet dhcp
pi@raspberrypi ~ $ auto lo
iface lo inet loopback
iface eth0 inet dhcp


(but obviously with your wireless network name instead of <YOUR-WIFI-ESSID>) . Ctrl-X to save.

Then generate your network configuration:

wpa_passphase <YOUR-WIFI-ESSID> <YOUR-WIFI-PASSPHRASE> >>/etc/wpa_supplicant/wpa_supplicant.conf

This appends the wireless key onto the end of your wpa_supplicant file. You then need to edit the file to amend the proper security settings:

sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

And edit the file until it looks something like:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
        ssid="<YOUR-WIFI-ESSID>"
        proto=WPA RSN
        scan_ssid=1
        key_mgmt=WPA-PSK
        pairwise=CCMP TKIP
        group=CCMP TKIP
        psk=<A-WHOLE-LOAD-OF-HEXADECIMAL-GUFF-WHICH-IS-THE-NETWORK-KEY>
}

Ctrl-X to save. You need to make sure you retain the ssid= and psk= lines. Ensure you have quotes around your ESSID.

You can then test the wireless with sudo ifup wlan0 and sudo ifdown wlan0

Then I configured MrEngman's script ( http://www.raspberrypi.org/forums/viewtopic.php?t=16054 ) to continually reconnect, which i saved as network-monitor.sh in /home/pi, then amended /etc/rc.local to call it.

nano /home/pi/network-monitor.sh

Make sure the file looks like:

#!/bin/bash

while true ; do
   if ifconfig wlan0 | grep -q "inet addr:" ; then
      sleep 60
   else
      echo "Network connection down! Attempting reconnection."
      ifup --force wlan0
      sleep 10
   fi
done


CTRL-X to save. Now set it to be executable via chmod and set it to auto-start by modifying /etc/rc.local:

sudo chmod +x /home/pi/network-monitor.sh

sudo nano /etc/rc.local

Add the lines at the bottom before exit 0. This runs the script, which is now executable, and redirects it's output to nowhere and makes sure it runs in the background. (If you encouter problems with it, once it's running, hit <CTRL-C> at a command prompt and then you should be able to kill the process.)

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

# Print the IP address
_IP=$(hostname -I) || true
if [ "$_IP" ]; then
  printf "My IP address is %s\n" "$_IP"
fi

#sleep for 5s
sleep 5

#auto-restart wifi in the background, we dont care about output
bash /home/pi/network-monitor.sh >/dev/null &

exit 0


Phew! Hopefully you can now see that your R-Pi connects to a network by using ifconfig to see that it has an IP address. And you can turn your wifi router off and on again, and it'll reconnect. Yay!

Installing Motion with motion-mmal.

It's all detailed here ->


I built a lego housing for my Pi to make sure it pointed in the right direction, and followed the instructions at the link above, which I had no issue with whatsoever, except that it takes a *long* time to update all the packages.

So it was just a case of connecting the Pi-NOIR camera and setting it up as per the post above.

Here's my Pi looking cool in my window. The camera sits in a little cradle of lego at the top of the arm, and looks down at my driveway. It's got a good view from here:



I had intended to use an IR illuminator for a bit of Pi-Night Vision, but the LEDs glow red and it'd look a bit out of place in my street. I'm going to get a second unit and have the night vision version pointing out the back window for late night wildlife photography instead!

This is the IR illuminator. It lights things up pretty well, for £25!



And here's an example picture of what motion picks up when a pigeon flies across my driveway!


There's a few reflections because it's a sunny day, but if it's fast enough to catch a flying bird... 

The Science-Bit: Connecting the output to Dropbox so I can view it from anywhere

All this is great. Except that if some scumbag actually does breaks in , then they can just steal the Pi and all the evidence.

Dropbox API to the rescue. What we need to do is copy all the stills to a staging location periodically (every couple of minutes max), and then we'll kick off a process using the Dropbox Java API to upload em to Dropbox.

Step 1: Some scripting to hang it all together

First things first, then, lets modify a few lines in /etc/motion.conf to make sure we:

a) put all the picture files into a correct location: /home/pi/camera
b) output 2 fps as stills
c) have set a correct resolution
d) amend the filename a bit so that it outputs alphanumerically sequential filenames so you can just swipe through them in the right order in Dropbox when we get that far.

Set up the directories we'll use:

mkdir /home/pi/camera
mkdir /home/pi/output
mkdir /home/pi/java

Then we need to tweak our motion.conf. Set up the following lines by editing motion.conf with the usual sudo nano /etc/motion.conf

# Start in daemon (background) mode and release terminal (default: off)
daemon on

...


# Use a file to save logs messages, if not defined stderr and syslog is used. (default: not defined)

logfile /home/pi/motion.log



...

# Image width (pixels). Valid range: Camera dependent, default: 352, 1024x576 is also valid for R-Pi Camera Module.
width 1280

# Image height (pixels). Valid range: Camera dependent, default: 288, 1024x576 is also valid for R-Pi Camera Module.
height 720

# Maximum number of frames to be captured per second.
# Valid range: 2-100. Default: 100 (almost no limit).
framerate 2

# Minimum time in seconds between capturing picture frames from the camera.
# Default: 0 = disabled - the capture rate is given by the camera framerate.
# This option is used when you want to capture images at a rate lower than 2 per second.
minimum_frame_time 0

...


# Switch this setting to "on" to use the still image mode of the Pi's camera
# instead of video. This gives a wider field of view, but requires
# a much slower frame-rate to achieve exposure stability
# (e.g. 0.25 fps or slower). You can use the minimum_frame_time
# parameter above to achieve this

mmalcam_use_still yes


...


# Specifies the number of pre-captured (buffered) pictures from before motion
# was detected that will be output at motion detection.
# Recommended range: 0 to 5 (default: 0)
# Do not use large values! Large values will cause Motion to skip video frames and
# cause unsmooth movies. To smooth movies use larger values of post_capture instead.
pre_capture 2

# Number of frames to capture after motion is no longer detected (default: 0)
post_capture 2

# Event Gap is the seconds of no motion detection that triggers the end of an event.
# An event is defined as a series of motion images taken within a short timeframe.
# Recommended value is 60 seconds (Default). The value -1 is allowed and disables
# events causing all Motion to be written to one single movie file and no pre_capture.
# If set to 0, motion is running in gapless mode. Movies don't have gaps anymore. An
# event ends right after no more motion is detected and post_capture is over.
event_gap 60

# Maximum length in seconds of a movie
# When value is exceeded a new movie file is created. (Default: 0 = infinite)
max_movie_time 0

...


# Target base directory for pictures and films
# Recommended to use absolute path. (Default: current working directory)
target_dir /home/pi/camera

# File path for snapshots (jpeg or ppm) relative to target_dir
# Default: %v-%Y%m%d%H%M%S-snapshot
# Default value is equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d/%H/%M/%S-snapshot
# File extension .jpg or .ppm is automatically added so do not include this.
# Note: A symbolic link called lastsnap.jpg created in the target_dir will always
# point to the latest snapshot, unless snapshot_filename is exactly 'lastsnap'
snapshot_filename %Y%m%d%H%M%S-%v-snapshot

# File path for motion triggered images (jpeg or ppm) relative to target_dir
# Default: %v-%Y%m%d%H%M%S-%q
# Default value is equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d/%H/%M/%S-%q
# File extension .jpg or .ppm is automatically added so do not include this
# Set to 'preview' together with best-preview feature enables special naming
# convention for preview shots. See motion guide for details
picture_filename %Y%m%d%H%M%S-%v-%q

# File path for motion triggered ffmpeg films (movies) relative to target_dir
# Default: %v-%Y%m%d%H%M%S
# Default value is equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d/%H%M%S
# File extension .mpg or .avi is automatically added so do not include this
# This option was previously called ffmpeg_filename
movie_filename %Y%m%d%H%M%S-%v

# File path for timelapse movies relative to target_dir
# Default: %Y%m%d-timelapse
# Default value is near equivalent to legacy oldlayout option
# For Motion 3.0 compatible mode choose: %Y/%m/%d-timelapse
# File extension .mpg is automatically added so do not include this
timelapse_filename %Y%m%d-timelapse


And that's about it. Note how we tweaked the number of frames to store, framerate, camera resolution, and the filenames, and file path to store them, in. If you want a full copy of my motion.conf, it's attached.

Now I need a script to copy things around. This is just a variation on MrEngman's auto-wifi-reconnect script. I also run it at the end of /etc/rc.local

nano /home/pi/java/dropbox-auto-upload.sh

and make it look like:

#!/bin/bash

while true ; do
# cd to the camera directory
cd /home/pi/camera

# remove all avis we dont want to keep them
rm -f *.avi

# now move all the jpg files to output
mv *.jpg ../output/.

# call the dropbox upload, moving directory first
#cd /home/pi/java
#java -jar RpiCameraToDropbox.jar config.properties

# finally sleep for 2 minutes before we go again
sleep 120

# and repeat..
done

Then make it executable. Note that we're calling a java executable we haven't set up yet...

sudo chmod +x /home/pi/dropbox-auto-upload.sh

Now we can add the following lines to auto-run it in rc.local, just before the exit 0 line.

#and then run the auto-upload in the background, logging output
bash /home/pi/java/dropbox-auto-upload.sh >/home/pi/java/upload_log.txt &

i.e. we'll run this script, logging any output to a new file just in case, and do it in the background (the & bit).

Step 2: Set up Dropbox and get the API Jars installed on the Pi.

In order to upload to dropbox, you need to set up an app to have it's own area in your dropbox, and then set up the application keys.

Register for Dropbox and log in. https://www.dropbox.com/

Once logged in, go to the developer app console. https://www.dropbox.com/developers/apps


Hit the Create App button and set up an app.
It should be a DropBox API app, Development type, set to allow Only You to access it. Dropbox should generate a key and an app secret for you. You'll need these bits of cryptographic goo later, so copy and paste them somewhere.

Great. Now we need to get hold of the Dropbox JAR files so we can write some Java. Either navigate to the Core API section, and find the Java download, of go direct to download the Dropbox Java API:


Unzip this file, and find the libs folder, extract the two JAR files you find within it.


You then need to copy these files into your java installation on the Pi.

Simply copy them using an SSH file transfer client like WinSCP to the Pi (you'll need to know the Pi's network address to do it - ifconfig!) and deposit the files into /home/pi/java directory.

On the Pi itself, now copy them across to where they are needed:

sudo cp /home/pi/java/*.jar /usr/lib/jvm/jdk-7-oracle-armhf/jre/lib/ext/.

This copies the java dependencies into the right place on the Pi.

Step 3: Write some Java! Or install my little JAR file.

Here's the source. You can compile this and turn it into a JAR file in the usual Java way. If you're not comfortable doing that, just take the JAR file provided.

RpiCameraToDropbox.java

package uk.co.rpicameratodropbox;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.List;
import java.util.ListIterator;
import java.util.Locale;
import java.util.Properties;

import com.dropbox.core.*;
import com.dropbox.core.DbxEntry.WithChildren;

public class RpiCameraToDropbox {

/**
* @param args
*/
public static void main(String[] args) {
if (args.length<1) {
System.out.println("RpiCameraToDropbox");
System.out.println("Usage: java -jar RpiCameraToDropbox.jar <configuration-file>");
System.out.println("<configuration-file> is the full path of a configuration file written in the form of a java .properties file");
System.out.println("watch-directory=<string, required: the full path of teh directory to watch. Note that multiple directores can be configured by running multiple instances of this program.>");
System.out.println("dropbox-api-key=<string, required : Copy and paste this from the Dropbox Api Console>");
System.out.println("dropbox-secret=<string, required : Copy and paste this from the Dropbox Api Console>");
System.out.println("dropbox-access-token=<string, optional initially, then required : Copy and paste this from the authorisation you are requested to do on first use.>");
System.out.println("days-to-keep=<numeric, optional : number of days worth of files to keep. Each day is given it's own directory within the Dropbox root folder. Folders older than this threshold (in days) are deleted automatically.>");
return;
}

Properties props = new Properties();
try {
props.load(new FileInputStream(new File(args[0])));
} catch (FileNotFoundException e) {
System.out.println("RpiCameraToDropbox initialisation failure: configuration file ["+args[1]+"] not found. Exiting.");
return;
} catch (IOException e) {
System.out.println("RpiCameraToDropbox initialisation failure: configuration file ["+args[1]+"] could not be loaded (IOException). Exiting.");
return;
}

if (props.containsKey("watch-directory")==false) {
System.out.println("RpiCameraToDropbox initialisation failure: You must specify a directory to watch.");
return;
}

if (props.containsKey("dropbox-api-key")==false) {
System.out.println("RpiCameraToDropbox initialisation failure: You must specify the dropbox api key.");
return;
}

if (props.containsKey("dropbox-secret")==false) {
System.out.println("RpiCameraToDropbox initialisation failure: You must specify the dropbox secret for this app.");
return;
}

String apikey = props.getProperty("dropbox-api-key");
String secret = props.getProperty("dropbox-secret");
String accessToken="";
DbxAppInfo appInfo = new DbxAppInfo(apikey, secret);
DbxRequestConfig config = new DbxRequestConfig("RpiCameraToDropbox/1.0", Locale.getDefault().toString());
if (props.containsKey("dropbox-authorisation-token")==false) {
DbxWebAuthNoRedirect webAuth = new DbxWebAuthNoRedirect(config, appInfo);

if (props.containsKey("dropbox-access-token")==false || "".equals(props.getProperty("dropbox-access-token"))) {

String authorizeUrl = webAuth.start();
System.out.println("RpiCameraToDropbox Setup - you need to provide the dropbox access token");
System.out.println("1. Go to: " + authorizeUrl);
System.out.println("2. Click \"Allow\" (you might have to log in first)");
System.out.println("3. Ensure this token is provided in the \"dropbox-access-token\" field of the configuration properties file you are using.");        
return;
}


DbxAuthFinish authFinish;
try {
authFinish = webAuth.finish(props.getProperty("dropbox-access-token"));
} catch (DbxException e) {
System.out.println("RpiCameraToDropbox could not use the access token ["+props.getProperty("dropbox-access-token")+"] to access Dropbox. Are you sure this is correct? Please remove it from the configuration file and try setting it up again.");
return;
}

accessToken = authFinish.accessToken;
props.put("dropbox-authorisation-token", accessToken);
File propsFile = new File(args[0]);
propsFile.delete();
try {
props.store(new PrintWriter(new FileWriter(propsFile)),"RpiCameraToDropbox authorisation complete.");
} catch (IOException e) {
System.out.println("RpiCameraToDropbox encountered a problem re-writing the configuration file with Dropbox's returned access token. You may need to configure everything all over again.");
return;
}
} else {
accessToken = props.getProperty("dropbox-authorisation-token");
}
        DbxClient client = new DbxClient(config, accessToken);
        
        String localFolder = props.getProperty("watch-directory");
       
        String ndaystr = props.getProperty("days-to-keep");
        int ndays = new Integer(ndaystr).intValue();
        
        Calendar cal = Calendar.getInstance();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
String today = sdf.format(cal.getTime());
Date now = cal.getTime();
cal.add(Calendar.DATE, -ndays);
        Date threshold = cal.getTime();
try {
System.out.println("Dropbox Login Success: " + client.getAccountInfo().displayName);
// have we got a folder for today? If not, create one.
DbxEntry folder = client.getMetadata("/"+today);
if (folder==null) {
client.createFolder("/"+today);
folder = client.getMetadata("/"+today);
}
if (folder.isFolder()) {
// now we can upload any files in our root path, deleting them as we go.
File dir = new File(localFolder);
 File[] localFiles = dir.listFiles();
 if (localFiles != null) {
   for (File localFile : localFiles) {
    String remoteFilename = localFile.getName();
    FileInputStream localInputStream = new FileInputStream(localFile);
    DbxEntry.File uploadedFile = client.uploadFile("/"+today+"/"+remoteFilename, DbxWriteMode.add(), localFile.length(), localInputStream);
    System.out.println("Uploaded: " + uploadedFile.toString());
    localInputStream=null;
    System.gc();
    localFile.delete();
   }
 }  
}
// clean up: get a list of all folders in the root and delete any older than NDAYS (from config file)
WithChildren entry = client.getMetadataWithChildren("/");
ListIterator<DbxEntry> rootIterator = entry.children.listIterator();
while (rootIterator.hasNext()) {
DbxEntry rootEntry = rootIterator.next();
if (rootEntry.isFolder()) {
String rootFolderName = rootEntry.name;
try {
Date folderDate = sdf.parse(rootFolderName);
if (threshold.compareTo(folderDate)>0) {
System.out.println("Deleting '"+rootFolderName+"'");
client.delete("/"+rootFolderName);
}
} catch (ParseException e) {
System.out.println("Warning: could not parse folder name '"+rootFolderName+"' as date. Ignoring.");
}
}
}
System.out.println("Done. Exiting.");
        } catch (DbxException e) {
        System.out.println("RpiCameraToDropbox: There was some problem in the Dropbox API during processing. Abort!");
        e.printStackTrace();
} catch (FileNotFoundException e) {
        System.out.println("RpiCameraToDropbox: There was some problem getting an inputstream handle to a local file. Abort!");
e.printStackTrace();
} catch (IOException e) {
System.out.println("RpiCameraToDropbox: There was some problem uploading a file. Aborting.");
e.printStackTrace();
}
        
}

}



and here's the pre-compiled JAR file.

 https://dl.dropboxusercontent.com/u/61615472/RpiCameraToDropbox.jar
Put the JAR file into the /home/pi/java directory on the Pi.

 You then need to set up an initial config.properties file in the /home/pi/java directory, so we can enter all the security gubbins Dropbox needs later and configure our upload directories etc.

 nano /home/pi/java/config.properties

 And make sure the file looks like:

dropbox-secret=<DROPBOX_SECRET_TOKEN_FROM_APP_CONSOLE>
watch-directory=/home/pi/output
days-to-keep=7
dropbox-api-key=<DROPBOX_API_KEY_FROM_APP_CONSOLE>

Substituting the API Key and Secret from the Dropbox Console earlier.

Now we're ready to try and connect and populate the rest of the security so we can connect automatically in future.

Step 4: Authorise the app and run it.

cd /home/pi/java
java -jar RpiCameraToDropbox.jar config.properties

 You'll get a message like:

 RpiCameraToDropbox Setup - you need to provide the dropbox access token
1. Go to: https://www.dropbox.com/1/oauth2/authorize?locale=en_GB&client_id=qsh42834xuve7c7&response_type=code
2. Click "Allow" (you might have to log in first)

3. Ensure this token is provided in the "dropbox-access-token" field of the configuration properties file you are using.

Copy and paste the url into a browser.



Click Allow. You'll then be prompted with an auth code:
Enter this code into RPiCam2 to finish the process.
THhZT07J_7YAZAAAAAfFwvflXhTjGwff9BEcHV3j6MQ

Now edit the config.properties file again and add the following key:

 dropbox-access-token=<YOUR-RETURNED-TOKEN>
Save the file and run the process again:

 java -jar RpiCameraToDropbox.jar config.properties
You should get a confirmation. If you don't, try again, removing the line from the config.properties file.

 Dropbox Login Success: <YOUR DROPBOX ID>
Done. Exiting.

Dropbox will also send you a nice mail notifying you that they've linked an app.

Step 5: Stand back and hope!

Time to run around outside to trigger the camera and check it all appears on Dropbox after a while. 

 Now I can access Dropbox on my phone from anywhere and see 
within 2 minutes of anyone stepping on my driveway. Mission accomplished.
You can also amend the config.properties file to delete files after a set number of days by the "days-to-keep" parameter in config.properties. I tend to run it for about 7 days at a time, but clearly you may need to change this depending on how much Dropbox space you have, and how active your driveway is.

Finally, a bit more optimisation and tidying up - image masking and backups


 I found that I was picking up every slight little movement in the street which led to my camera using up 1 GB of space in a day on Dropbox one particularly busy Saturday.

 There's a way to mask out certain areas of the image using a mask file. Basically what you do is generate a black and white image like the one below, and save it as a .PGM file (most image editors will let you do this).

So you can see where we'll be detecting motion if you compare it against the image the camera takes:


Black means motion is not detected in this area. White means "detect motion in this area". This mask needs to have exactly the same resolution (i.e. 1280x720) as your camera is recording. When you save it, you need to save it as BINARY or RAW format PGM file. I used GNU Image Manipulation Program to do this (GIMP).

 Save the file on the pi in /home/pi somewhere.

 Then you can amend the motion.conf file to point to the mask file. Note that this line exists in the file already, you just need to uncomment it and give it the right mask file name.

 # PGM file to use as a sensitivity mask.
# Full path name to. (Default: not defined)

mask_file /home/pi/driveway_mask_1280.pgm

 I did this and got my data rate right back down to 50-100MB a day.

 Finally, once you've done all of this, you wont want to lose your hard work, so BACK UP YOUR SDCARD IMAGE. That way you can reinstate the whole R-Pi if its file system corrupts, and they do, sometimes frequently if you have a poor SD Card.

 Happy camming!








Subscribe to: Posts (Atom)