Intangible Surface

Intangible Surface is a gesture and IoT based touchless interface to interact with the digital world.

A button is the most common interface to interact with the digital world. It could be as simple as a light switch or at a pedestrian crossing or it could be as complex as in the elevator or a kiosk. These buttons can act as a catalyst in the spread of viruses. The problem I am trying to solve is to reduce the touch-based interfaces a.k.a the catalysts by replacing it with a gesture and IoT based interface that is intangible, simple, and intuitive to operate.

My solution is divided into two parts

Method 1: Replace the tangible buttons with an intangible gesture-based interface. A person will then interact with the digital world (light, fan, elevator, etc.) using gesture(s) instead of touch.

Method 2: No matter how much you try you will always touch your smartphone. So why not enable your smartphone to complement the gesture-based interface. It will make some complex interfaces like an elevator easy to use (to see the elevator control using a smartphone please go to the elevator application section i.e. end of the page).

A person will swipe right to turn ON a light/fan and swipe left to turn it OFF. Rotate hand clockwise or anticlockwise to increase or decrease a fan's speed or change a light's intensity. The same device can also be used for some complex interfaces like an elevator. Swipe up or down to call the elevator based on the direction you are going. Swipe or rotate your hand to select the floor. And the magical part is that all of these actions are possible using a single interface. No need for a separate knob to control the fan speed. No need for multiple buttons to select the floor numbers. Selecting 2 digit floor on an elevator using gestures could be a multi-step process, therefore, you also have the option of controlling an elevator using your smartphone. Not only the elevator but you can control almost anything using your smartphone.

The main elements of my design are

• Gesture sensor
• Arduino Wi-Fi 1010
• LCD
• Smartphone

The electronic modules I used for prototyping are grove based. Connecting them is a child's play. The sensor and LCD both use I2C protocol and the Arduino I am using has only one I2C port so I have used an I2C hub to connect all three modules. Please refer to the below schematics for connecting the modules.

I have used a product box of a charger to assemble my project. Ideally, I would love to design and print a custom enclosure but I don't have access to a 3D printer. I have slightly modified the box and it does a fantastic job of housing the circuit. It also hides the cluttered wiring and makes it look clean.

Code for this project is available here.

A person can control any device using intangible surface via two methods

• Gesture
• IoT (I don't want to call it IoT, please suggest me a better name)

5.1 Gesture

The gesture sensor I am using can detect 9 different gestures

1. Left
2. Right
3. Up
4. Down
5. Clockwise
6. Anti-clockwise
7. Forward
8. Backward
9. Wave

The sensor detects the gesture and sends it to the Arduino. Arduino then translates the gesture into a specific meaning based on the device we want to control. Also, the number of gestures used depends on the device we want to control. The interface at pedestrian crossing needs only 1 gesture and light needs 2 whereas a fan needs 4 gestures.

5.2 IoT (Someone Else's Computer)

In this method, your smartphone acts as a remote control. The app on your phone uses the MQTT protocol over WiFi to communicate with the intangible surface. When you click a button on your mobile app, it publishes a message to a topic. The Intangible Surface reads that message by subscribing to that topic. After processing the message it sends a control signal to the device it is controlling. Intangible Surface then publishes the feedback message to the topic which is then reflected by the mobile app.

To know more about MQTT protocol please click here. I used the Mosquitto broker installed on a RaspberryPi (acting as a server) to test this design. I have also tested it with CloudMQTT. You can use any MQTT broker of your choice.

A clear benefit of this method is that multiple people can control the same device at the same time. So you and your friend can select different floor numbers in an elevator at the same time using your smartphones.

The smartphone app I am using to talk to the Intangible Surface is called MQTT Dashboard. I have designed various control interfaces in it according to the different applications.

I have developed some application interface demonstrating the capability of the Intangible Surface.

You might have seen and used a switch at a pedestrian crossing which looks like the one in the above image.

It's a very simple interface. You press the button and wait for a while until the signal turns green. Once the signal is green, it tells you to walk. This functionality can be replicated using intangible surface. In the new interface just swipe LEFT or RIGHT to activate the signal. Voila!

You must have used an on/off switch to control a light or any other electrical appliance. It can also be replaced with our touch-less interface. Simply swipe RIGHT to turn ON the light and swipe LEFT to turn it OFF.

You might have seen a fan switch and a knob to control its speed(Sorry! I can't find a royalty-free image). Lets replicate that interface using Intangible Surface.

Swipe RIGHT to turn ON and swipe LEFT to turn OFF. Rotate your hand CLOCKWISE to increase the speed and ANTICLOCKWISE to decrease the speed.

A common elevator has two interfaces, one to summon it (outside), and another to select a floor (inside). The elevator summon interface look like the one in above image.

To call the elevator using Intangible Surface, simply swipe UP or DOWN based on the direction you want to go.

The floor selection interface inside the elevator is generally quite complex. You will find lots of buttons like the one shown in the image above.

These buttons are divided into 3 main category

1. Floor selection buttons.
2. Door control buttons.
3. Alarm/Emergency button.

We can replace these 3 functionalities using our Intangible Surface.

1. Using the touch-less interface we can select the floor by rotating our hand CLOCKWISE or ANTICLOCKWISE. If a building has too many floors then you can swipe LEFT or RIGHT to decrease or increase the floor number by a factor of 5, respectively.

2. For door control, simply swipe UP to open the door and swipe DOWN to close.

3. To activate the alarm, wave your hand.

I have implemented the Elevator control using IoT also. Using this method, one can interact with the elevator using a smartphone.

I want to prototype one more feature and that is to "auto-detect Intangible Surface devices around you in the mobile App".

• If Intangible Surface is deployed indoors then use BLE's advertise feature to inform users around it about the device's identity.
• If Intangible Surface is deployed outdoors then use GPS in your smartphone to complement the above method.

Unfortunately in Arduino MKR 1010, you can use either WiFi or BLE. I am using WiFi to communicate with the MQTT server therefore I will need a separate BLE radio which I don't have at the moment.

Intangible Surface is an extremely versatile and cost-effective design that can be used to replace a number of touch-based interfaces, simple as well as complex. This device makes you touch less stuff and do more things. Replace the touch-based interface using this and Protect yourself and the people around you from COVID19.

Which interface you would like to replace?

Open Source is LOVE. Happy Making and Take Care:-)