Intruder/Guest Alert System

Have you ever dreamt of building your own "Intruder alert system" which has the following capabilities?

1. Recognizing people who enter your room in a real-time video recording.

2. Notifies you and your friends/neighbors on social media if some one entered your room without your permission.

[UPDATE]

3. Along with status update, new feature uploads a picture of intruder to your facebook timeline in real-time.

You can find the updated code in my GitHub Repository.

[UPDATE]

Then this tutorial is for you.
You can watch the attached video of completed system which automatically posts a status to your facebook timeline in case some one enters into your room/ house.

For building this system I am using my Laptop running on Ubuntu OS. I will be using Sublime text 3 for writing the code. I am gonna build this system in Python language using OpenCV to tackle with Computer vision problem.
Confused, don't know all this jargon? You are going to learn about all of these things by the end of this tutorial.

Setting up the system

1. Position your camera (USB/ built-in) pointing towards the main door and start the program. The program starts recording the video and analyses it to detect any humans in real-time.

2. When some one enters your room it detects that person and checks with our database to recognize him/her.

3. It notifies you/your friends by posting a status on social media.

This tutorial is heavily code based and you should have some prerequisites if you want to successfully build your own Intruder alert system.

Skills

1. You need to have working knowledge of Python language. Even if you don't know python, you can understand the code easily if you have knowledge of C/C++/Java.

2. As we are dealing with recognizing intruders, knowledge of Image Processing/Computer Vision is a plus point.

Hardware

1. A laptop/desktop with built-in camera or a USB-camera.

2. Your system running on Ubuntu is preferred. You can manage with windows but, installing all the programs required, on windows machine is a bit difficult.

Software

1. As we are coding in python, we need to install python on your machine.

2. We need to install OpenCV bindings for Python, so that we can tackle with computer vision stuff.

3. And lastly, some Python modules like 'requests', which we will be using to post a status update on social media to notify you and your friends.

Now, we will move on to the next step on how to install all the required programs.

If your machine is running on Ubuntu, follow the instructions given in the Ubuntu section to install the required the programs. Similarly, follow the Windows section if you are on a Windows machine.

Ubuntu

Out of the box, Ubuntu comes with python and many extension modules pre-installed. So, there's no need to install python if you are on Ubuntu OS. I am using python version 2.7 for this tutorial.

I used sublime-text-3 as text editor for this project. You can use your favorite text editor.

Enter the following command into Ubuntu terminal to get all the required packages (dependencies)-

sudo apt-get -y install libopencv-dev build-essential cmake git libgtk2.0-dev pkg-config python-dev python-numpy libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff4-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev libtbb-dev libqt4-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils unzip<br>

Enter the following commands in the terminal to download latest version of OpenCV and to install it on your machine-

mkdir opencv<br>cd opencv
wget <a href="https://github.com/Itseez/opencv/archive/3.0.0-alpha.zip" rel="nofollow">  https://github.com/Itseez/opencv/archive/3.0.0-al...> -O opencv-3.0.0-alpha.zip
unzip opencv-3.0.0-alpha.zip

Now, install downloaded opencv files

cd opencv-3.0.0-alpha<br>mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -DBUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D WITH_QT=ON -D WITH_OPENGL=ON ..
make -j $(nproc)
sudo make install

Final steps

sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'<br>sudo ldconfig

By now, you have almost got all the modules and softwares. So, get started with building our alert system.

Windows

Firstly, you need to download Pyhton from here.
Download OpenCV for windows from here. You need to download numpy from here.
You can get any text-editor you like. I am using sublime-text-3 for this purpose.

Follow the below steps to install the downloaded files:

1. First install Python 2.7 by running the downloaded file. Leave all settings as default. In that case, Python will be installed in default folder C:\Python27\

2. Now install Numpy. Again leave everything as default. Numpy will find Python directory and will be installed to most appropriate folder.

3. Now double-click OpenCV.exe. It will ask for extraction folder. Give it as just C:\. It will extract all files to C:\opencv\ . Wait until everything is extracted.

4. Now copy everything in the folder C:\opencv\build\python\x86\2.7\ ( most probably, there will be only one file cv2.pyd ) and paste it in the folder C:\Python27\Lib\site-packages\

5. Now open your "Python IDLE" ( from Start > All Programmes > Python 2.7 > Python IDLE ) and just type following 'import cv2'

If everything OK, it will import cv2 module, otherwise an error message will be shown.

If the last step works without any errors, you have successfully installed opencv on your machine. Let's move on to next step.

You can find the whole code used in this tutorial at my GitHub repository here.

In the coming steps, I am going to explain in detail about every component of the code. If you got all the prerequisites and wanted to implement the system without knowing the concepts, you can proceed by skipping the next couple of steps. But, it is not recommended as you may not know how to tackle with errors (if you encounter, any) while implementing by yourself.

You can always post in comment section below, if you din't understand a concept or if you have encountered a problem and are unable to solve it.

The Concept

These are the main concepts, how we are proceeding to write the code for intruder alert system:

1. Firstly, we need to get some images of friends to fed into out system so that it can learn and recognize people in video recordings. We need to do some pre-processing work on images and save them in separate folders which I am gonna explain in up coming steps.

2. Now, as we have, all the required softwares and pictures of our subjects/people, we will start getting into our code:

• Firstly, we will access all the images saved in a folder as mentioned (will mention in next step) from our code.
• Now, as we can access images, we will send commands to OpenCV functions to train from out database. This will return a trained model which we can use to recognize people in video. Basically, in this step we are saying to our system to analyse all the images and learn to recognize the people in them so we can use this functionality in future to recognize intruders/ guests.
• We will write a small code snippet so that our code can automatically post a status to your facebook timeline in case of an intruder attack.
• As our program learned to recognize people in images, we will start recording the video and leave it by itself by pointing the camera at your room door or your main door.
• We have no work to do now, except to catch the intruder when our program notifies us.

3. Underlying process running in our code while detecting intruder attack and notifies you:

• As we left out program running, it will records video of your main entrance. It analyses every frame of the video, if it could find any human faces.
• If any human faces are found, it will send commands to out database of saves images if this matches any of our saved guests. This would return a name of guest or 'intruder' if it couldn't recognize this new face.
• After making sure that some one entered into your home, it will send requests to social media (facebook in this case) servers to post a new status.

In this step, we are going to collect images of some friends and save them is a specific format so that out code can get to them to learn to recognize faces. You can see screenshots annotated, how to organize collected pictures into folders:

1. input_images- Make a folder named "input_images" and store all of your friends' images in that folder. There's no need to do any pre-processing here. Just collect your friends' pictures and save all the images here.

2. output_images- Make a folder named "output_images" and leave it empty. After running the pre-processing code, it will crop faces from images in "input_images" folder and automatically saves them here. You can see the annotated image: 'output_images' with saved images after running the code provided in this step.

3. sorted_images- Now, we need to manually pick your friends' pictures and save them in separate folders with names same as your friends. You can see, I have collected pictures of 6 of my friends and stored them in separate folders.
Why we need to manually pick and organize out friends' images? I will answer this question in next step.

After running this code, your folder "output_images" should be full of cropped images. So, now move on to sub-step 3 mentioned above where we have to manually put all of friends' images into separate folders.

With this sub-step, we have successfully completed pre-processing & organizing our data (images).

Code

Save this as .py and run it from command line. I assume you have working knowledge of coding in python, so I am not explaining how to run it here.

Note: You need have images saved in 'input_images' folder before running this code and the pre-trained xml file attached to this step saved with the name- "haarcascade_frontalface_default.xml" in the same folder as this code is in.

I have briefly commented what the code is doing at each part.

You can view the same code here.

You can alternatively download the same code below as I have attached it to this step.

'''<br>Surya Teja Cheedella
shine123surya[at]gmail[dot]com
BITS Pilani, Hyderabad Campus
Takes all the images in "input_path" and analyses them for any faces.
If face(s) is(are) found, it crops and saves them at "output_path".
'''
import cv2
import cv2.cv as cv
from os import listdir
import time
def cropImage(img, box):
    [p, q, r, s]= box
    # crop and save the image provided with the co-ordinates of bounding box
    write_img_color= img[q:q+ s, p:p+ r]
    saveCropped(write_img_color, name)
# save the cropped image at specified location
def saveCropped(img, name):
    cv2.imwrite(output_path+ name+ ".jpg", img)
if __name__== "__main__":
    # paths to input and output images
        input_path= "input_images/"
        output_path= "output_images/"
    # load pre-trained frontalface cascade classifier
    frontal_face= cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
    input_names= listdir("/home/suryateja/Documents/GitHub/FaceRecognizer/"+ input_path)
    print("Starting to detect faces in images and save the cropped images to output file...")
    sttime= time.clock()
    i= 1
    for name in input_names:
        print(input_path+name)
        color_img= cv2.imread(input_path+ name)
        # converting color image to grayscale image
        gray_img= cv2.cvtColor(color_img, cv2.COLOR_BGR2GRAY)
        # find the bounding boxes around detected faces in images
        bBoxes= frontal_face.detectMultiScale(gray_img, scaleFactor=1.3, minNeighbors=5, minSize=(30, 30), flags = cv.CV_HAAR_SCALE_IMAGE)
        #print(bBoxes)
        for box in bBoxes:
            #print(box)
            # crop and save the image at specified location
            cropImage(color_img, box)
            i+= 1
    print("Successfully completed the task in %.2f Secs." % (time.clock()- sttime))

In the following five steps (Step 5 to Step 9), I am going to explain the main code used in our "Intruder alert system". You can find the complete code explained in the steps (5 to 9) here. And I have attached the file to this step.

By this step, you should have three folders namely "input_images", "output_images", "sorted_images". From this step we will be using only the last folder "sorted_images". So, you may delete other folders, if you wish.

In this step I am gonna explain how to access images saved and neatly organized in out last folder. This code will takes all the sub-folders (folders named after our friends) and gets all the images in each folder. It automatically gives a unique label (a number starting form 0) to each of our friends and stores in memory. So, this is the reason we should organize our friends' pictures.

This part of code returns an array with the elements.

1. images- This code re-sizes & converts every image into an nparray (numpy array). You don't need to know what numpy array is. But, if you are interested, you can know more about it here.

2. labels- This is an array of the Unique labels we have discussed. This is given to every picture in out sub-folders.

3. people- This is an array of actual name of our friend. (This returns the names of sub-folders) This is the reason we need to name our sub-folders after our friends.

We will send the returned array from this code snippet to our next step.

Code

Note:
This code is a small part of the completed code and doesn't work individually. I have used it here to explain the code.

def get_images(path, size):<br>    '''
    path: path to a folder which contains subfolders of for each subject/person
        which in turn cotains pictures of subjects/persons.
    size: a tuple to resize images.
        Ex- (256, 256)
    '''
    sub= 0
    images, labels= [], []
    people= []
    for subdir in listdir(path):
        for image in listdir(path+ "/"+ subdir):
            #print(subdir, images)
            img= cv2.imread(path+"/"+subdir+"/"+image, cv2.IMREAD_GRAYSCALE)
            img= cv2.resize(img, size)
            images.append(np.asarray(img, dtype= np.uint8))
            labels.append(sub)
            #cv2.imshow("win", img)
            #cv2.waitKey(10)
        people.append(subdir)
        sub+= 1
    return [images, labels, people]

From the last step, we will receive an array of three elements namely, images, labels and people.

Now, in this step we will use images and labels as parameters to train a faceRecognizer model. Basically, this model is an OpenCV function and we are asking this function to remember & learn to make a co-relation between images and labels. So, after training, we can ask the function to predict the label by giving an image as input and it will return a predicted label.

This part of code takes one input:

1. path- The path to images folder.

And returns two outputs:

1. eigen_model- The trained model after analyzing all of the images is returned.

2. people- An array which contains names of our friends. (names of our sub-folders)

This code snippet is the crucial part in our whole code. This part of code may take a while to complete if you have a lot of images to train. I have a total of 200 images and it took nearly 7 seconds on my Dell laptop with Intel 3rd generation i5 processor with 1.8GHz speed running Ubuntu.

So, we have completed training a model with the help of OpenCV functions and we can proceed on to the next steps.

Code

Note:
This code is a small part of the completed code and doesn't work individually. I have used it here to explain the code.

def train_model(path):<br>    '''
    Takes path to images and train a face recognition model 
    Returns trained model and people
    '''
    [images, labels, people]= get_images(sys.argv[1], (256, 256))
    #print([images, labels])
    labels= np.asarray(labels, dtype= np.int32)
    # initializing eigen_model and training
    print("Initializing eigen FaceRecognizer and training...")
    sttime= time.clock()
    eigen_model= cv2.createEigenFaceRecognizer()
    eigen_model.train(images, labels)
    print("\tSuccessfully completed training in "+ str(time.clock()- sttime)+ " Secs!")
    return [eigen_model, people]

Now, it is the time to start recording video.

This code snippet also uses some basic functions of OpenCV library. Here we will initialize the built-in camera and start recording the video. For every frame recorded by the camera, this code changes its color to grayscale and sends request to two functions which I will mention in next two steps.

1. detect_faces

2. post_on_facebook

The first function will return a name if it recognizes any faces in the gray-scale frames sent and an array of values of bounding boxes. We will take that array of bounding boxes and draw a rectangle around the recognized face and print subject's/ friend's name.

After confirming that there is an intruder attack, this code snippet sends a request to the second function 'post_on_facebook' to post an update on social media.

So, we will see how those two functions work in next steps.

Code

Note: This code is a small part of the completed code and doesn't work individually. I have used it here to explain the code.

if __name__== "__main__":<br>    if len(sys.argv)!= 2:
        print("Wrong number of arguments! See the usage.\n")
        print("Usage: face_detrec_video.py <fullpath to root images folder>")
        sys.exit()
    arg_one= sys.argv[1]
    eigen_model, people= train_model(arg_one)
    #starts recording video from camera and detects & predict subjects
    cap= cv2.VideoCapture(0)
    counter= 0
    last_20= [1 for i in range(20)]
    final_5= []
    box_text= "Subject: "
    while(True):
        ret, frame= cap.read()
        gray_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        gray_frame = cv2.equalizeHist(gray_frame)
        bBoxes= detect_faces(gray_frame)
        for bBox in bBoxes:
            (p,q,r,s)= bBox
            cv2.rectangle(frame, (p,q), (p+r,q+s), (225,0,25), 2)
            crop_gray_frame= gray_frame[q:q+s, p:p+r]
            crop_gray_frame= cv2.resize(crop_gray_frame, (256, 256))
            [predicted_label, predicted_conf]= eigen_model.predict(np.asarray(crop_gray_frame))
            last_20.append(predicted_label)
            last_20= last_20[1:]
            '''
            counter modulo x: changes value of final label for every x frames
            Use max_label or predicted_label as you wish to see in the output video. 
                But, posting on facebook always use max_label as a parameter.
            '''
            if counter%10== 0:					
                max_label= majority(last_20)
                #box_text= format("Subject: "+ people[max_label])
                box_text= format("Subject: "+ people[predicted_label])
                if counter> 20:
                    print("Will post on facebook timeline if this counter reaches to 5: "+ str(len(final_5)+ 1))
                    final_5.append(max_label)		#it always takes max_label into consideration
                    if len(final_5)== 5:
                        final_label= majority(final_5)
                        print("Intruder is "+ people[final_label])
                        print("Posting on your facebook timeline...")
                        post_on_facebook(people[final_label])
                        final_5= []
            cv2.putText(frame, box_text, (p-20, q-5), cv2.FONT_HERSHEY_PLAIN, 1.3, (25,0,225), 2)
        cv2.imshow("Video Window", frame)
        counter+= 1
        if (cv2.waitKey(5) & 0xFF== 27):
            break
    cv2.destroyAllWindows()

In this step, we will have a look at the code snippet which takes a single frame of the video recorded from the last step and analyzes it for any faces with the help of OpenCV function cascadeClassifier. If any faces are found, it sends the cropped part of the image to another function asking to predict the face in the image.

The name of subject/ intruder is sent to the code snippet in previous step which draws a bounding box around the subject while printing name of subject.

Right now the whole detection and recognition system has an accuracy of 70 to 80 percent only. So, rather than changing the name of subject based solely on present label, we will take the last twenty labels into consideration. So we will store the last twenty labels and asks for most repeated label from that last_20 array. We will take the name of intruder this way and pass on to previous step which draws bounding box.

Code

Note: This code is a small part of the completed code and doesn't work individually. I have used it here to explain the code.

def detect_faces(image):
    '''
    Takes an image as input and returns an array of bounding box(es).
    '''
    frontal_face= cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
    bBoxes= frontal_face.detectMultiScale(image, scaleFactor=1.3, minNeighbors=4, minSize=(30, 30), flags = cv.CV_HAAR_SCALE_IMAGE)
    return bBoxes</p><p>def majority(mylist):
    '''
    Takes a list and returns an element which has highest frequency in the given list.
    '''
    myset= set(mylist)
    ans= mylist[0]
    ans_f= mylist.count(ans)
    for i in myset:
        if mylist.count(i)> ans_f:
            ans= i
            ans_f= mylist.count(i)
    return ans

This is the last step, explaining the code for our Intruder Alert System. This code snippet takes the name of the intruder as an input and notifies your neighbours on social media.

As we are posting on facebook, we need to get an access_token from facebook developers page. You need to press on get_access_token button in that site and need to tick mark publish_actions in Extended_permissions tab. Then copy the access_token and paste that in this code snippet in 6th line here.

token= "paste your access_token in the code below"

You can change the message to post on facebook by changing the values of strings my_messsage1 and my_message2 in the code below. Finally, you will get a conformation messages printed on terminal if the message is successfully posted on your facebook timeline.

Code

Note: This code is a small part of the completed code and doesn't work individually. I have used it here to explain the code.

<p>def post_on_facebook(intruder):<br>    '''
    Takes name of intruder and posts on your facebok timeline.
    You need to get access_token from facebook GraphAPI and paste it below.
    '''
    token= "get GraphAPI token with Extended_permission- publish_actions"
    url= "https://graph.facebook.com/me/feed"
    my_message1= "Surya is not in his room at present and '"+ intruder+ "' entered into his room without permission."
    my_message2= "PS: This is automatically posted by 'intruder alert system' built by Surya!"
    params= {"access_token": token, "message": my_message1+"\n\n"+my_message2}
    posted= requests.post(url, params)
    if str(posted)== "<response [200]="">":
        print("\tSuccessfully posted on your timeline.")
    else:
        print("\tPlease check your token and its permissions.")
        print("\tYou cannot post same message more than once in a single POST request.")</response></p>

If you have followed all the steps as mentioned and came until here, you have successfully built your own "Intruder alert system". Congratulations!

Please let me know if you face any difficulties while doing this instructable in the comment section below.