Object Interaction With Touchscreens

We were interested in finding ways to get our flat screens to interact with physical 3d objects and surfaces.

Touchscreens are omnipresent, a multitude of devices and screen formats exist with thousands of apps however we only found a few options that enable object interaction.

We can imagine a lot of creative opportunities to bridge these two worlds - objects and materials you can touch with digital applications able to display rich interactive content.

Our tutorial suggests ways to create the object tracking with capacitive touchscreen using an openFrameworks application. The application can run on Windows or iOS. You will need coding experience with openFrameworks to run the object detection application and to create your own interactive content.
Please note this tutorial is at an experimental level and there is room for improvement. We hope you will have suggestions and questions!

Technology Background

To understand where we come from we need to go back to infrared acrylic projection touchscreen using camera, there are systems able to detect touch as well fiducial markers and objects.

( How to make one :

https://www.instructables.com/id/Multi-touch-Table

https://www.instructables.com/id/How-to-build-a-Multi-Touch-surface/

http://ccv.nuigroup.com/#home

In 2012 we created one using ccv )

As this system is using a camera for touch tracking it was fairly simple and natural to track objects or fiducial markers.

Now the technology is used less and capacitive touchscreens (same as used in our smartphone touchscreens) are widespread. Capacitive touchscreens are very accurate, cheap to get and simple to use. However unlike infrared acrylic projection there are no common ways to create interaction with an object. The only instance of such technology we have seen is from the company Volumique, we believe they are using a similar technology as what we are showing in this tutorial.

For our demo, we used the following materials (affiliate links)

• A capacitive touchscreen http://amzn.to/2ctA8ZL Iiyama ProLite T2336MSC-B2 or http://amzn.to/2ctA8ZL iPad

Note you must choose a touchscreen or device with capacitive touch and more than 3 points of touch (you need 3 points of touch per object interacting on the screen, usually recent capacitive touch screen support 10 point touch, so potentially 3 objects detected at the same time + 1 finger )

• Pads to create contact points with the touchscreen - http://amzn.to/2ctzgEp Floor Protector Felt Pads

Note we first tried using plastic pads http://amzn.to/2ctzgEp , however it was not sliding nicely on the screen so unless you want the object to stick on your screen use softer materials like felt pads

• Electric Paint http://amzn.to/2cmS4Jm or http://amzn.to/2cmS8Ja Bare Conductive electric paint
• Brush for the paint http://amzn.to/2cmR1cx
• Objects with one flat surface

We are creating three touch points forming a triangle on the touchscreen.

The software is detecting all touch points and looking for isosceles triangle patterns.

We chose to detect isosceles triangles with different apex angles so we can detect different objects. Also, knowing the apex of the triangle we can find out the orientation/rotation angle of the object.

The center of the object is estimated at the center of the triangle.

We suggest using 1 to 13 different objects. If you want to use as many as 13 different objects or more you will need to create very accurate markers. You might need to experiment with different triangles that feature different angles before getting satisfying results.

You can create different size of triangles, what is important for the small software we provide is to keep the same apex angle.

The object needs to have at least one flat surface, that can rest on the screen.

In our demo, we created geometric shapes using mold making techniques. We worked with alginate to create the molds as it sets really quickly and works with most materials such as ceramic powder, which we mixed with pigments to make our objects. We look forward to seeing what kind of objects you will use!

• Following as accurately as possible the printed triangle template, stick the 3 raised pads onto the flat surface of your object

• Using the conductive ink, paint the pads surface

• Connect the 3 pads and the object with conductive ink

If you are not using a conductive object you will need to create a connection point you can reach with your hand at the top or sides of the object. This point will also need to be connected to the pads.

When the conductive ink is dry, you are ready for the next part.

• You need to have openFrameworks setup and running on your system.
• Download the touch application from:

https://github.com/paul-ferragut/oscTouchObject (windows and openFrameworks, relies on external add-ons https://github.com/nneonneo/ofxWin8Touch )

https://github.com/paul-ferragut/iOSoscTouchObject (iOS and openFrameworks, unfortunately, you will need to be a licensed Apple developer in order to compile the app on iOS)

• Before you run the app you will need to modify the file bin/data/settings.xml. Write the apex angle of the isosceles triangle marker you are using in the settings.xml file and save the file (eg: on the example image we use 4 objects with apex angles of 38 , 91 , 150 and 165 )
• Optionally you can use OSC protocol to communicate with another application, you will need to change the localhost address and port in the settings.xml file. Additionally, there is flag called #USE_OSC in the file touchObject.h, you will need to keep the flag uncommented. If you don't need OSC you can comment #USE_OSC
• When the app launch you will see a dark screen with a GUI. Depending on your screen resolution and the width of your triangle markers you will need to adjust the slider "distanceTriangle". This variable changes the minimum distance for a triangle to be detected. You can simulate a marker using 3 fingers on the screen and form a triangle (see gif).

• After you were able to test your objects and the markers detection you can start to create your own interactive content!

In our demo we had some fun with the following openFrameworks addons:

https://github.com/vanderlin/ofxBox2d

https://github.com/bakercp/ofxLight2D

https://github.com/tado/ofxLiquidFun

https://github.com/jacres/of-DeferredRendering