Making of Kindbot: Customized Ambient Computing With Flask-Ask

Kindbot is powered with Flask-ask to control an arduino and raspberry pi. Flask-ask, a flavor of Flask, is a relatively new tool that allows you to set up your own local server to control anything with Amazon Alexa. Many projects showcase how to use Lambda as the brains for your Alexa skill. This entails setting up lambda functions and any other services needed to support the application (S3, dynamodb, etc.) This flask-ask tutorial probably offers the fastest way to test your home automation application using local compute resources.

Get into your pi and open up a new terminal. It's easy to install flask-ask by simply running

sudo pip install flask-ask 

If you run into any issues, clone the flask-ask repo, cd into the flask-ask directory and run

git clone  https://github.com/johnwheeler/flask-ask.git
cd flask-ask
python setup.py install

An Intro to Flask and Flask-Ask

Flask is a micro framework designed to make application building with python fast and simple. Flask-ask swiftly adds Alexa to any project. Here's a quick rundown on how to structure your app using flask and flask-ask: Initialization: You need to import the modules and basic functions, initialize the application and the alexa functionalities.

#!/usr/bin/env python
from flask import Flask                  #Importing the main class
from flask_ask import Ask, statement     #Importing Alexa class
app = Flask(__name__)                    #Initializing the app
ask  = Ask(app, '/')                     #Initializing Alexa functionality 
                                         #around app

"app" and "ask" are instances of the two main classes (Flask and Ask) used to build the application. These instances will provide us with decorators, @app and @ask, respectively, to declare any function to be a web application endpoint or an Alexa skill intent.

Let's start with a simple application for our new project - CentiPi - another garden variety Alexa enabled raspberry pi camera.

For our first trick, we want to make CentiPi say hello when we say hello.

@app.route('/')
def home():
    return 'Hi! I am CentiPi'
@ask.intent('helloCentiPi')
def hello():
    speech_text = 'Hello there!'
    return statement(speech_text)

The first function uses the @app decorator. Notice that it is calling the .route( ) method. This method tells the application to serve a function at that address. Here, @app.route('/') tells the application to run the function attached to it when the user navigates to http://sample-application.com/. If this decorator was instead @app.route('/centipi'), the application would run its function when you navigate to the 'centipi' endpoint, i.e. http://sample-application.com/centipi. Here, the home( ) function will print "Hi! I am CentiPi" on the browser. The hello( ) function uses the @ask decorator, pointing to a skill intent-'helloCentiPi'. This decorator tells the application to run the function associated to it when Alexa recognizes the intent invoked. In this case, when Alexa recognizes that the 'helloCentiPi' intent was invoked, Alexa will say "Hello there!" in return. To make Alexa speak, use the statement( ) function with a string of text.

If you have one of the Echos with a screen, you can take advantage of more functionality by displaying text and images. Let's have our echo also write out some statements on the screen using the method .simple_card( ) . To our hello function we add:

@ask.intent('helloCentiPi')
def hello():
    speech_text = 'Hello there!'
    return statement(speech_text).simple_card(title='CentiPi says', content= 
                                               speech_text)

This method adds a gray card in the background and prints whatever string you declare in the content argument.

Suppose we want CentiPi to take a picture of our garden. When invoked, our CentiPi will take a photo with its tiny camera and display it on the screen. For this application we'll need to install python-picamera on our pi. Run

sudo apt-get install python-picamera 

Let's use it to render a picture in the echo display.

import picamera
@ask.intent('photoCentiPi')
def photo():
    camera = picamera.PiCamera()    #initialize the camera
    camera.capture('my_pic.jpg')    #take a photo!
    pic = '/home/pi/my_pic.jpg' 
    speech_text = 'Here is your picture!'
    return statement(speech_text).display_render(template='BodyTemplate7',
                                                 title='CentiPi',
                                                 backButton='HIDDEN',
                                                 background_image_url=pic)

When we say, "Alexa, ask CentiPi to take a photo", the application will run the photo( ) function which:

• enables your picam
• takes an image and saves it to your current directory (here assuming your application lives in /home/pi/).
• tells Alexa to display it.

Amazon Alexa has many different Body templates for displaying images on the Echo Spot and Echo Show. Here
is the documentation showcasing all the templates you can use. Once you've found one you like, replace 'BodyTemplate7' with the template name.

We will now identify key words when invoking our skill. We want Alexa to be able to identify when we tell her a date and time. We need another library to schedule a script to run using crontab. Run:

sudo pip install python-crontab

We'll use the python-crontab module to schedule a camera script to run at a desired date and time.

from datetime import datetime
from crontab import CronTab
@ask.intent('scheduleCentiPi')
def schedule(date, time):
    cron = CronTab(user='pi')
    job = cron.new('python /home/pi/take_picture.py')
    dd = date + ' ' + time
    job_date = datetime.strptime(dd, '%Y-%m-%d %H:%M')
    job.setall(job_date)
    cron.write()
    speech_text = "Ok. I've got it scheduled for {} at {}".format(date,time)
    return statement(speech_text)

Here, Alexa is hearing out for a date and a time when you invoke the
'scheduleCentiPi' skill. Alexa parses out these values stored in the date and time variables, constructs a datetime object and schedules a crontab job to run at that date and time. Once it sets the crontab for the take_picture.py script, she'll tell you the time she schedule it to run. You can collect other slots and process them however you wish. Here we used this to take a date and time to schedule an event. You can get the finished application app.py below.

When you run app.py, you can access your site now by navigating to http://127.0.0.1:5000 on your browser. You should see "Hello there!" printed on the screen. However, your Amazon skill won't be able to access it yet. You need an https connection off your local network and served publicly on the open internet. Ngrok is a super quick and dirty way to get your application hosted without having to set up an apache server or configure ssl certificates. Go to the ngrok download page and download the arm linux zip package. Once you unzip the package, it is installed and ready to use. Run

./ngrok http 5000

This command will randomly generate a https link for which your local app will be routed to. If you navigate to this link, you'll be able to see your app! Try it on another device not on your local network and you'll be able to access it there too.

Setting up the Alexa skill, navigate to the Amazon developer dashboard and log in. Click on Alexa and get started with Alexa Skill kit. Then create a new skill by clicking Add a new skill. Start completing the tabs on the left following their instructions.
Follow the instructions given by the gui.

• Under the Interaction Model, fill out the Intent Schema box with the following:

{
 "intents": [
   {
     "intent": "helloCentiPi"
   },
   {
     "intent": "photoCentiPi"
   },
   {
     "intent": "scheduleCentiPi",
     "slots": [
                {"name": "time",
                 "type": "AMAZON.TIME"},
                {"name": "date",
                 "type": "AMAZON.DATE"}
              ]
   },
   {
     "intent": "AMAZON.PreviousIntent"
   },
   {
     "intent": "AMAZON.NextIntent"
   },
   {
     "intent": "AMAZON.HelpIntent"
   },
   {
     "intent": "AMAZON.ScrollUpIntent"
   },
   {
     "intent": "AMAZON.ScrollLeftIntent"
   },
   {
     "intent": "AMAZON.ScrollDownIntent"
   },
   {
     "intent": "AMAZON.ScrollRightIntent"
   },
   {
     "intent": "AMAZON.PageUpIntent"
   },
   {
     "intent": "AMAZON.PageDownIntent"
   },
   {
     "intent": "AMAZON.MoreIntent"
   },
   {
     "intent": "AMAZON.NavigateSettingsIntent"
   },
   {
     "intent": "AMAZON.StopIntent"
   }
 ]
}

• In the Sample Utterances box, write:

helloCentiPi say hello
helloCentiPi say hi
helloCentiPi hello
photoCentiPi take a picture
photoCentiPi take a photo
photoCentiPi take a pic
scheduleCentiPi schedule a picture for {date} at {time}
scheduleCentiPi take a pic {date} at {time}
scheduleCentiPi snap a photo on {date} at {time}

• In the Configuration tab, make sure to choose the service end point to HTTPS. Copy your https link (from ngrok earlier) and paste it in the Default box underneath. Account linking can be left to No.

• In the SSL Certificate pick the middle option, "My development endpoint is a sub-domain of a domain that has a wildcard certificate from a certificate authority".

• The Test tab will allow you to test the new skill by typing in one of your sample commands.

Finish filling out the last two tabs until all the check marks are green. Then launch your skill with the Beta Testing feature. This allows you to host your skill on any echo device before publishing it. Follow the instructions on the email link to install the skill on your echo device.

Congratulations! You have a full amazon skill hosted on your raspberry pi. You can program it to do anything on the raspberry pi and other peripherals you might want to add (Arduino etc).