Augmented Book Prototype

Intro:

This tutorial is for an augmented reality book using a Kinect 360 and a projector. The Kinect is tracking the position of the book via its camera, so you can move it around and the projection will follow the motion of the book. You can for example project 3D elements that will appear complete with shadows and will be able to rotate and change shape, colour, texture.

We always thought augmented reality to be a great technology however it is always required to experience it while looking through a device. We wanted to try to use it in combination with projection-mapping to create a seamless and magical experience.

As we decided to go for a large format (A3) we wanted to use a Japanese Stab Binding technique. There are several versions of this technique, and we are explaining the most simple one which is perfect for beginners. You also do not need any adhesives or complicated tools. The downside of this binding technique is that it does not open completely flat which is why we added another step of prefolding the pages using a bone folder. If you decide to go with a smaller format that allows you to fold your pages in the middle you could also use single, multi-section or coptic binding which naturally open flat.

We hope you enjoy the tutorial, we would love to see the type of books and projections you come up with and welcome any feedback. If anything is unclear, please comment.

• Printed pages for the inside of your book (number of pages according to your needs)
• 2 Cover papers
• Thread (for example waxed cotton or linen thread)

• Pencil
• Ruler
• Awl
• Bookbinding needle
• Scissors
• Clips
• Bone folder (optional)

With this binding technique the pages do not normally open flat. For the AR book, we needed the book to open relatively flat, which is why we pre-folded the front cover and all pages 2cm from the edge using a bone folder.

You can either use a geometry set square or make marks 2cm away from the edge using a small steel ruler. Then use the ruler as a guide for the bone folder and to deepen the fold by bending the page over the ruler. Continue to use the bone folder to complete the fold as pictured. This way the page will lie as flat as possible when turned over.

Proceed to do this with your front cover and all inside pages, however not your back cover, as it will not need to fold over.

Prepare a template for making the holes using a paper the same size of your book. Mark it 1.8cm away from the spine edge. This distance is flexible, but you should make sure that the binding will not interfere with the fold.

Fold the paper in half three times. Continue by unfolding it and marking 4 holes starting at the top fold and then jumping one fold for each whole, so the distance between the holes towards the top and bottom edge is smaller than the distance between the holes.

Hold your pages together using clips to ensure the holes are all in the same position. To make sure the clips are not leaving any marks on your cover, place a bit of scrap paper underneath them.

Place the template on your pages to use as a guide. Make all four holes using your awl.

Use a single thread such as a waxed cotton or linen thread. Do not make a knot.

To ensure you have enough thread use roughly five times the height of your book. (image 1)

Take a section of your pages, to hide the thread inside, and start by using the second hole from the bottom. Leave approximately 10cm on sticking out on the end. This is where you will tie your knot at the end. (images 2 and 3)

Wrap around the book and go back through the same whole. Pull the thread gently so it sits tight around the edge of the book. (images 4 and 5)

Continue on to the next hole above the current one, wrap the thread around the edge and back through. Move onto the last whole, wrap the thread around the end and back through the hole. Then wrap it around the top edge and again back through the hole. (images 6-10)

Now continue to weave back through the holes, wrapping around the edges like you did at the top and finally back to the hole you started with. Once you reach this point return the needle back between the pages where the beginning of your thread is. (images 11-15)

Gently tuck both threads to ensure they are tight. Then tie the two ends together in a couple of knots. Cut the ends and you are done! (images 16-18)

Now the book is finished, we will explain the augmented reality part of the project.

• Kinect 360
• Projector (we used an Epson EBW12) + cables
• Projector mount
• Recent computer with a screen (we used a pc with windows 10)
• Cable ties

Our project relies on openFrameworks and different add-ons from its community. You will need to be familiar with openFrameworks and coding as the setup procedure is complex! Additionally you will need to know some basics to create your augmented reality content.

You will first need to download and instal openframeworks http://openframeworks.cc Then download the modified add-ons and the example from the following address: http://www.convivial.studio/arbook.zip Add the modified 'ofxFern' (this is based on https://github.com/ofTheo/ofxFern ( based on http://cvlab.epfl.ch/software/ferns/index.php )) folder to your openframeworks addons folder, then add the folder 'reProj' to your folder 'apps/myApps'.

You will also need to download the following addons

• https://github.com/kylemcdonald/ofxCv (openCv for openFrameworks)
• https://github.com/thomwolf/ofxProjectorKinectCalibration
• https://github.com/rezaali/ofxUI

Once you have added all the required addons to your openframeworks folder you can open the project file in "reProj" folder. The project is built for Microsoft visual studio 2015, if you are not on windows you can try using openFrameworks project generator, you will need to include all the following addons to your project

• ofxOpenCv
• ofxCv
• ofxFern (modified)
• ofxUI
• ofxKinect
• ofxAssimpModelLoader
• ofxXmlSettings

In order to identify and track your pages you will need to transform them into markers for ferns.

In your data folder you can find maker templates.

You will need to create a .bmp image (640px x 480px ) by taking a picture of the marker you wish to use. If you name your image model.bmp you need to create a text file called model.bmp.roi. The next step is to edit this text file with the 4 corner pixels of your region of interest (roi) coordinates such as:

0 0

320 0

320 640

0 640

When you launch your app, fern will generate two files:

• model.bmp.detector_data
• model.bmp.marker_data

Generating the files will take a few minutes and you may have to try different pictures and test different marker images/objects.

You will need to define how many markers you are using in ofApp.h at the line

#define PAGES 3

We recommend not using more than 5 different markers with ferns AR library.

Note: As detection markers not all images will work, for instance photography with a lot of contrast, asymmetric patterns and shapes works great.

It is essential to have the projector mounted relatively high. You could for example use a ceiling mount or construct a DIY solution to fix the projector to.

The kinect attached to the projector. Note: The kinect should not be too far from the tracked object, as it will not track the object if it is too far away. This parameter might make the calibration (step 3) challenging and you might have to try different setups of your space.

First you will need to input your screen resolution in the code by editing ofApp.h

#define FIRST_SCREEN_W 1920

#define FIRST_SCREEN_H 1080

#define PROJ_W 1280

#define PROJ_H 800

Then you will need to compile the app and follow this procedure:

https://vimeo.com/75415111

Re-adjust your projector and kinect position until you get a good calibration result.

Once everything is setup create your scenes in openFrameworks and link them to your markers. Have fun!

the project is at an early experimental state, and the example application is not completely refined yet. Please do not expect everything to perfectly work out of the box.

A few improvements or alternatives could be tried out:

• using a kinect v2 with a better camera
• track hand movements with the kinect in order to get touch events on the book
• We used ferns as an AR library however if you want more than 5 pages to be tracked you might need to use instead an AR solution with markers ( for instance https://github.com/arturoc/ofxAruco )