Create your own face tracking, pan and tilt camera on the Raspberry Pi!
This tutorial will demonstrate use of the OpenCV (computer vision) library to identify and track faces on the raspberry pi using two servos and a USB webcam. For the interested, I previously covered a more thorough overview of the installation of OpenCV from source here, however, I have found that the apt package is sufficient for all but the most bleeding edge of projects.
This project is based on the OpenCV face tracking example that comes along with the source-based distribution. In short, it performs face detection using haar-like features to analyze the video frame, and locates any faces present within it. In the live video window, identified faces are surrounded with a red rectangle. For additional information about how face detection works as well as the specifics of face detection with OpenCV, I recommend this article by Robin Hewitt.
Using the coordinates of the rectangle vertices, my script calculates the (X,Y) position of the center of the face. If the face is sufficiently on the left side of the screen, the pan servo will progressively rotate leftward, on the right side, rightward. This is likewise performed for the tilt servo as well, if the face is in the upper portion of the screen, it will pan upward, in the lower portion, pan downward. If the face is detected reasonably within the center of the image, no action is performed by the servos. This prevents unnecessary jitter once the camera has locked itself on the face.
- 512 MB raspberry pi
- 2x Hobby servos (Turnigy 9g fom Hobby King)
- Pan & tilt bracket (from Foxtech FPV)
- USB webcam (Microsoft LifeCam Show from Amazon)
- Power supply
- Hook-up wire
- Raspberry Pi enclosure (from Built to Spec)
Connect the red, power lines of the servos to +5v, the black ground lines to GND, and the yellow signal lines to the desired output pins, GPIO pins 22 and 23 in the example. Here is a diagram of the completed circuit (created with Fritzing):
And here is how it looks all put together:
Get the source.
The first step of this procedure is to install the required libraries and packages using the Raspberry Pi package manager. Open up terminal shell and run:
sudo apt-get update && sudo apt-get install git python-opencv python-all-dev libopencv-dev
This command will pull down all of the required packages (about 215 MB worth) including the git version control system, as well as the OpenCV development headers and Python bindings. The next step is to configure the Raspberry Pi for use with multiple pulse width modulation (PWM) outputs. Normally, the Raspberry Pi has only one channel of PWM output. However, thanks to the efforts of Richard Hirst, eight channels of PWM can be used through the use of this servoblaster kernel driver.The driver creates a device file, /dev/servoblaster to which commands can be sent to control the servos. The commands take the form “=” with servo number representing the desired servo (0-7 in this case) and servo position representing the pulse width in units of 10 µs. For example, to send servo 3 a pulse width of 120 µs: echo 3=120 > /dev/servoblaster To configure the servoblaster on the Raspberry Pi, first pull down the sources from Richard’s Github repo:
git clone https://github.com/richardghirst/PiBits.git
Then change into the servo blaster directory, and install the module:
This command also sets up the necessary udev rules for accessing the /dev/servoblaster device. Note: using the ’install_autostart ’ command will set up a Raspberry Pi to load the servoblaster kernel module on every boot. If you don’t want this behavior, execute ‘make install’ instead. In either case, the module will not yet be loaded so go ahead and load it into the kernel using modprobe:
sudo modprobe servoblaster
Now that all the prerequisites have been installed and the servo blaster device configured, it’s time to get the actual face tracking and servo moving code. Clone my repo from Github here:
git clone https://github.com/mitchtech/py_servo_facetracker
Now change directory into the created folder, and run the script like this:
python ./facetracker_servo_gpio.py 0
If you have a different servo bracket configuration, the pan and tilt axis may need to be inverted. To do so, invert the sign on the values of panStepSize and tiltStepSize. Similarly, increasing or decreasing these values will change the sensitivity of the movement, larger numbers corresponding to more degrees moved per face detection frame.
Google Talk/Chat/Messenger is normally used by humans to interact with other humans. However, its underlying technology can also be used as a mechanism to implement software robots. Internet bots, also known as web robots, WWW robots or simply ‘bots’ can also utilize the technology to perform automated functions over the web. There are many such bots in existence, offering a diverse spectrum of services from jokes (firstname.lastname@example.org) to URL Shortening using bit.ly (email@example.com), even mathematical calculation (firstname.lastname@example.org). Using such bots is quick and easy to configure, all that must be done is to add the bot as a contact to your messaging account. Then, whenever you desire data from the service, simply text the command to the bot and it will respond with the respective message.
The Raspi Bot is essentially the same as any other automated Internet robot. To configure it, it must first have its own e-mail address associated with a Google talk account. This e-mail address must also be added as a contact with the account that wishes to communicate to the bot. Then, whenever the script is running on the remote machine, it will log into Google chat and appear as a friend in your contact list.
The software itself is essentially just a Python daemon script that is a wrapper around the XMPP protocol. When executed, the script will sign in to Google talk using its own username and password. The Python script is derived from the open source project pygtalkrobot: An open source python gtalk(google talk) bot framework using XMPPPY and PyDNS libraries, that also references the source code of python-jabberbot.
The Raspi Bot requires several additional Python modules for use. The easiest way to install these is with the python pip package manager. If you don’t have it installed, you can install them using apt-get:
sudo apt-get install python-pip git-core python2.7-dev
Then update the easy_install index:
sudo easy_install -U distribute
And install the GPIO, xmpppy, and pydns modules:
sudo pip install RPi.GPIO xmpppy pydns
Then clone my repo for the Raspi Gtalk robot:
git clone https://github.com/mitchtech/raspi_gtalk_robot.git
Now change into the newly created directory:
Finally, you will need to configure the Raspi Bot’s Gtalk username and password. This is done by editing the fields BOT_GTALK_USER, BOT_GTALK_PASS, and BOT_ADMIN, on lines 31-33 in the raspiBot.py file. It is recommended, though not required, to give the Raspi Bot its own Gmail account. Since access to the Raspberry Pi GPIO pins is restricted, the script needs to be run with sudo:
sudo python ./raspiBot.py
This basic sample script supports the following commands:
- [pinon|pon|on|high] [pin] : turns on the specified GPIO pin
- [pinoff|poff|off|low] [pin] : turns off the specified GPIO pin
- [write|w] [pin] [state] : writes specified state to the specified GPIO pin
- [read|r] [pin]: reads the value of the specified GPIO pin
- [available|online|busy|dnd|away|idle|out|xa] [arg1] : set gtalk state and status message to specified argument
- [shell|bash] [arg1] : executes the specified shell command argument after ‘shell’ or ‘bash’
For example, sending the message “pinon 10” will turn on GPIO pin #10, “read 8” will read the current state of GPIO pin 8, or “bash ps” to execute the shell command ‘ps’.
The video demonstration uses a slide switch connected to GPIO pin 8 and an LED connected to GPIO pin 10. Here is a diagram of how this is wired (created with Fritzing):
Use case #1: Home automation
One of the most obvious usages of this technology is for home automation purposes. The Raspi Bot can be accessed anywhere with Google talk, which to my understanding, is nearly every system in existence. Send the Raspi Bot messages to turn on and off lights and other electrical appliances.
This is also useful to provide immediate notification in the event of intrusion detection. The Raspi Bot can be supplemented with additional security sensors, including infrared motion, and ultrasonic distance sensors. If any pre-programmed sensor violates any predefined condition, you can be immediately notified via message from the Raspi Bot.
Use case #2: Remote shell
The Raspi Bot can be used essentially as a remote shell. In this configuration, every message sent to the Raspi Bot will be interpreted as a shell command with the output piped back to the user in the form of a response message. Obviously, this could raise some security concerns. To protect against misuse, the Raspi Bot will only respond to Google chat messages from the Google user designated as the administrator of the bot. By default, messages from any other user will simply be ignored.
The following is a small subset of the relatively benign commands possible to be run remotely via the Raspi Bot:
- vmstat – system activity, hardware and information
- uptime – how long the system has been running
- w – logged in users and their process activity
- ps – reports a snapshot of the current processes
- free – physical and swap memory usage
- iostat – average CPU load, disk activity
Arguably, disclosure of any amount of information about system can be considered a security issue such as that reported by some of the above tools. For users more concerned about convenience over security exposure, much more elaborate commands can be run, such as executing additional scripts or accessing private data.
Use case #3: Remote Reboot
Another problem that can be solved by the Raspi Bot is frozen remote machines. We’ve all been there before, attempting to access a remote machine only to find it to be completely non-responsive to any form of remote login. These cases, we (or maybe a system administrator somewhere) would usually have to make a trip to the physical location of the server and push ‘the big red button’ to reboot the affected machine. The situation can be eliminated completely by deputizing a Raspi Bot as a remote reboot agent. This can be done by adding bot controlled relay(s) to the power supplies of the machines. In the event any of the machines controlled by the Raspi Bot becomes non-responsive, simply send the appropriate Google talk message to flip the respective relay, and reboot the affected machine. Here’s a diagram of how this would look:
This tutorial demonstrates how to easily use a Raspberry Pi to drive 12V RGB LED strips using Pulse Width Modulation (PWM). Out of the box, the Raspberry Pi has only one GPIO pin that is capable of pulse width modulation (PWM). However, thanks to the efforts of Richard Hirst and his servoblaster kernel module, standard GPIO pins can be used to perform PWM.
Note: The flashing of the LED strip due to PWM is only noticeable in the uploaded video; in reality, the colors progress smoothly without any visible flashing.
- Raspberry Pi
- 3 x TIP120 power transistors
- RGB LED strip
- Hook-up wire
Connect a 12V power supply to the 12V pad on the RGB LED strip, connect the base of each TIP120 power transistor to its respective GPIO pin (pins 18, 23, and 24 in this example), connect the collector of each to its matching pad on the LED strip, and finally, connect the emitters to common ground for both the Raspberry Pi and the 12V power supply. Here is a diagram of the circuit (created with Fritzing):
The next step is to configure the servoblaster kernel driver. The driver creates a device file, /dev/servoblaster to which commands can be sent to control the servos. The commands take the form “=” with servo number representing the desired servo (0-7 in this case) and servo position representing the pulse width in units of 10 µs. For example, to send servo 3 a pulse width of 120 µs:
echo 3=120 > /dev/servoblaster
To setup the servo blaster on the Raspberry Pi, we will need to have git installed to pull down the sources. If you don’t have it installed already, open a terminal and run:
sudo apt-get install git-core
Then pull down the sources from Richard’s Github repo:
git clone https://github.com/richardghirst/PiBits.git
Now change into the servo blaster directory:
And compile and install the module:
This command also sets up the necessary udev rules for accessing the /dev/servoblaster device. Note: using the ’install_autostart ’ command will set up a raspberry pi to load the servo blaster kernel module on every boot. If you don’t want this behavior, execute ‘make install’ instead. In either case, the module will not yet be loaded so go ahead and install it using modprobe:
sudo modprobe servoblaster
Sample code usage.
Now that all the prerequisites have been installed and the servo blaster device configured, on to the actual sample code. The python sample script uses PWM to fade from blue to violet, to red, to yellow, to green, to teal, back to blue.
Easily connect your Raspberry Pi to web services and social networks! This tutorial demonstrates how to painlessly send and receive Gmail on the Raspberry Pi from Python, which in turn, allows you to easily connect it to web services and social networks like Facebook, Twitter, and more! This would normally be well beyond the abilities of most users due to the inherent complexities of programming through social media APis, client/server authentication, etc. However, with the easy-to-use web service Swiss Army knife ifttt (if this then that) anyone with even the most basic programming skills can dramatically expand the capabilities of their Raspberry Pi.
How does this work?
If you are not yet aware, ifttt is a great tool to simplify interaction with many social networks and other web services. It operates with the premise that when some specific action occurs it should perform some other predefined action. The interface is intuitive and very easy to quicky understand. The bridge then is connecting the Raspberry Pi to any of these supported services, which in turn, enables the use of all other services that can respond to that respective trigger. In an earlier tutorial, I previously demonstrated how to connect your Raspberry Pi to your Dropbox account using SSHFS. This is another perfectly viable option to establish two way communication with ifttt. However, instead of using email as the primary communication medium, it relies on Dropbox and the filesystem itself. Depending on your application, this may or may not be a better option.
The tutorial uses the most recent Raspbian wheezy image, (2012-08-16-wheezy-raspbian)but should largely generalize for not only other Raspberry Pi distributions, but most other linux distributions as well (especially those derived from Debian, such as Ubuntu and Mint). The majority of the complexity and functionality comes from the feedparser Python module, which should be available essentially anywhere with Python support, even including Windows and OSX distributions too. In any case, begin by setting up the necessary accounts.
As you would undoubtedly expect, the ability to send and receive Gmail is predicated upon having a Gmail account. You are welcome to use your personal Gmail account if you wish, however, I recommend creating a separate account for the Raspberry Pi as it provides the most flexibility. If you would like to expand beyond the capabilities of Gmail on the Raspberry Pi, you will also need an account on ifttt. As with Gmail, you can either share your accounts with the Raspberry Pi, or create a separate account specifically for this purpose. Your decision here might be dependent on which services will be necessary for your own specific application.
Independent of whether you are using one or multiple Gmail or ifttt accounts, you will first need to install some required packages. Open up a terminal on the Raspberry Pi and install the Python development headers and the pip package manager:
sudo apt-get install python-pip python2.7-dev
Next, for pip to work correctly you will need to update to a newer version of distribute using the easy_install utility:
sudo easy_install -U distribute
Once that completes, install the feedparser module with the pip package manager:
sudo pip install feedparser
Now that all of the prerequisites are installed, we can move on to the actual code!
Checking Gmail by the Raspberry Pi
This is an example script that will check the Gmail of the specified user, and display the subject line of all unread e-mails.
Sending Gmail from the Raspberry Pi
This is an example script that can be used to send plain text e-mails with Gmail to the specified mail to e-mail address.
Sending Attachments through Gmail from the Raspberry Pi
This is a more complex example script that can be used to attach files to emails sent from Gmail. It sends an array of attached files to an array of recipients.
Scheduling tasks with cron
Often, we want to run tasks at periodic intervals, like to poll Gmail for example. Cron, the UNIX time-based job scheduler, is an easy way to run regular tasks on the Raspberry Pi (or other *nix). Cron is a daemon (like a Web server) that is used to execute commands or scripts automatically at a specified time and date interval. To use it, open the global crontab (cron table) for editing:
sudo crontab -e
In the text editor, you’ll see a commented out section of text describing how to configure tasks with cron. To add your own task, simply add a line to the end of the file. For example, to run the check_gmail.py script above every minute:
* * * * * python /home/pi/check_gmail.py
Here is a breakdown of what that line actually means, and how you can use to create your own periodic tasks at the desired frequency:
* * * * * command to be executed - - - - - | | | | | | | | | | | | | | +----- day of week (0 - 6) (0 is Sunday, or use names) | | | +---------- month (1 - 12) | | +--------------- day of month (1 - 31) | +-------------------- hour (0 - 23) +------------------------- min (0 - 59)
Expanding functionality with ifttt
Now that the Raspberry Pi is capable of both sending and receiving e-mails and running scripts at regular intervals we can move on to fun things! The following are some quick examples of the types of things this interface enables. This list is FAR from inclusive, there are literally thousands of possible combinations!
Facebook and Twitter
Update your Facebook status, or post tweets based on raspberry pi events.
Alternatively, create a Facebook or Twitter account specifically for the Raspberry Pi, and have it update its own status messages and post its own tweets.
Automatically send alerts for events that match certain criteria within your Google calendar to the Raspberry Pi. Use this to make things like physical notifiers for holidays or other calendar events.
SMS and Phone
Receive SMS notifications or phone calls based on events from the Raspberry Pi. Notify based on certain sensor values, or create your own DIY security system, for example.
Weather and Stocks
It’s also easy to control the Raspberry Pi based on the weather forecast. For example, send notifications of impending severe weather or extreme temperatures.
Or create physical notifiers that perform actions whenever a specified stock price exceeds or falls below a specified threshold.
There are, of course, a few limitations from using Gmail and cron as the primary basis for communication to the cloud. Cron jobs can only be scheduled at minute granularity, that is, they can be scheduled to occur at maximum frequency of one minute intervals. This is acceptable for most tasks, though it does however leave a bit to be desired for the more ‘real time’ applications like turning on and off lights, or remotely operated robotics.
This tutorial will demonstrate how to mount Dropbox (or any filesystem) over the network on the Raspberry Pi using SSHFS (Secure SHell FileSystem). For this procedure to work for your Dropbox share, you will need another machine somewhere that is running Dropbox, and is accessible to the Raspberry Pi via SSH.
Note: The following is not actually specific to the Raspberry Pi, nor to Dropbox. The tutorial generalizes for other systems and architectures that are not officially supported by Dropbox, as well as for mounting of other non Dropbox shares over the network.
How it works
SSH is a secure protocol for communicating between machines. SSHFS is a tool that uses SSH to enable mounting of a remote filesystem on a local machine; the network is (mostly) transparent to the user.
On the local computer where the SSHFS is mounted, the implementation makes use of the FUSE (Filesystem in Userspace) kernel module. The practical effect of this is that the end user can seamlessly interact with remote files being securely served over SSH just as if they were local files on his/her computer.
Installation (remote host)
The first step is to configure the remote host that the Raspberry Pi will connect to via SSH. It will need to be running Dropbox, if you need to install it, follow the instructions for your respective OS here. If you are not yet a Dropbox user, and this has finally persuaded you to join, signup for Dropbox here.
Next, the remote machine will need to be running OpenSSH server. For Windows and Mac instructions on how to set up OpenSSH server, I recommend this tutorial on Lifehacker. For Linux users, OpenSSH server is available in most every package manager. To install on Ubuntu, for example:
sudo apt-get install openssh-server
Installation (Raspberry Pi)
Now that the remote host is configured, you can setup the mount on the Pi. This first requires installation of the sshfs package. Open a terminal on the Pi and install it like this:
sudo apt-get install sshfs
Then add the user pi to the FUSE users group:
sudo gpasswd -a pi fuse
Once added to the fuse group, log out and log back in again for the change to take effect. Next, create a directory to mount Dropbox (or other remote share)
Now use sshfs to mount the remote share on the newly created mountpoint. Be sure to change the user@remote-host and path to Dropbox to match your own settings:
sshfs -o idmap=user user@remote-host:/home/user/Dropbox ~/Dropbox
For example, connecting to another machine on your local network will look something like this:
sshfs -o idmap=user email@example.com:/home/michael/Dropbox ~/Dropbox
The idmap=user option ensures that files owned by the remote user are owned by the local user. If you don’t use idmap=user, files in the mounted directory might appear to be owned by someone else, because your computer and the remote computer have different ideas about the numeric user ID associated with each user name. idmap=user will not translate UIDs for other users.
That’s all there is to it! To unmount,
fusermount -u ~/Dropbox
Automount Dropbox on boot
To configure the Dropbox SSHFS to automatically mount at startup, we first need to enable SSH keyless remote login. The first part of this task is to generate an RSA crypto key so we can securely login to the remote machine running Dropbox without entering a password. In a terminal on the Pi, run:
ssh-keygen -t rsa
Hit enter three times when prompted, accepting the default settings for the RSA ssh keys. Now copy the public part of the key to the remote host using the ssh-copy-id command:
ssh-copy-id -i ~/.ssh/id_rsa.pub user@remote-host
You will be prompted for the password on the remote one last time. Once entered, terminal output will confim the key was added sucessfully.
Now that you can login remotely without password, the final task is to configure the share to automatically mount on startup. There are a few ways this could be accomplished, I decided to use cron for the task. Open the global crontab for editing:
sudo crontab -e
And add a line to the end like this:
@reboot sshfs user@remote-host:/home/user/Dropbox /home/pi/Dropbox
Then press CTRL and X to exit the editor, then Y to confirm the changes (if using nano, the default text editior).
That’s it! Reboot the Pi, and your Dropbox share will mount automatically on startup.
Another method to accomplish this task would be to add a line to /etc/fstab to automatically mount the Dropbox SSHFS share.