A Hearing MeArm, Google Coral TPU Accelerator Driven

In the following I would like to describe a voice-controlled version of the MeArm, a small xyz robot arm with a gripper. I used the MeArm Pi from MIME industries, but the system should be applicable to any version of he MeArm, or similar servo-driven devices.

Using the Google Coral TPU Accelerator allows to run rapid offline TensorFlow voice recognition scripts on the Raspberry Pi, and hereby control physical devices by spoken orders, with a latency below one second.

The device described herein is a combination, and extension, of concepts described in two previous instructables. It is a extension of an earlier implementation of the Google Coral voice control, a Jumping Jack, described here and a vast improvement of a Google AIY voice controlled MeArm described here.

The voice-controlled MeArm using the Google Voice AIY system required online access, was not easy to implement, required to press a button to activate listening for voice orders and had a long latency time. The Google Coral TPU Accelerator used now allows to run TensorFlowLite models offline with a high velocity on a Raspberry Pi or other Linux devices. Among the examples on the Google Coral Github page there is an example called "hearing snake" for a voice recognition system that can understand 140 key phrases (Sept 2019) , which are then mapped to virtual keystrokes. Coupling these "keystrokes" with the execution of some functions programmed in Python makes it possible to build a a voice command controlled device. I recently had described a first implementation, a voice-controlled electromechanical jumping jack.The implementation herein is a bit more complex and allows to control all four servos of the MeArm to either move the MeArm continuously or have it to move to a number of predefined positions, or to perform some more complex tasks.

Using the script provided here as example, it should be relatively simple to construct other voice-controlled devices, e.g. robotic cars or assistive tech units.

• MeArm. Used here: MeArm Pi from MIME Industries
• Raspberry Pi 4
• Google Coral TPU Accelerator
• Adafruit 16 channel servo bonnet
• some jumper cables
• optional: capacitor for servo bonnet, about 400 µF for 4 servos (recommended by Adafruit)
• 5-6 V power source for servo bonnet. I here used an old 6V charger, a 4x AA battery pack works as well
• Microphone. I used an old Microsoft HD3000 webcam as microphone.

Download the preconfigured Raspian image for the Google Coral TPU Accelerator from the Google Coral Github page and install it on a µSD card. The image contains also a number of example scripts. Set up the Pi as indicated.

Install the example Keyword spotter from the Google Coral GitHub site, if not included in the image, and all required programs. Attach the microphone to the Pi. I would recommend to play with the "Hearing Snake" example to make sure everything is working.

Download and install the Adafruit 16 channel bonnet software, as described here.
Install the bonnet and play with the Adafruit examples to ensure everything is working properly.

Download the files attached to this instructable and copy them to the "Project Keyword Spotter" folder. The "commands_v1_MeArm.txt" file must be copied to the "config" subfolder.

Connect your MeArm's servos to the servo bonnet as indicated. I used port 15 for up/down, port 11 for forward/backwards, port 7 for turn and port 3 for the gripper servos.

Within the script you may have to adjust the min/center/max values for each servo to your configuration, These settings help to avoid damage to the servos. You may also have to modify the included "positions", "transport1" and "transport2" lists.

Run the script. So far I had been running it from the IDE.

In case you would like modify the key phrases that evoke a certain function according to your need.
A complete list of available KeyPhrases are found in the "labels_gc2 raw.txt" file in the config subfolder.

The system has a latency time of about 1 second, but depending much on which actions are performed. In some cases the key phase has to be repeated, accuracy of recognition is not always 100%.

If everything is set up and checked, you may run the device.

A current limitation is that a given order is executed repetitively as long as it is not stopped (using "stop game") or another order is given. Complex multistep tasks, e.g. "transport1" (evoked by the phrase "launch game") are always executed to the final step.

So by "turn right" the device will move in small steps to the right until stopped, or the preset maximum value is reached. "launch game", "next game" or "start_video" will start a series of moves that are defined by lists containing the setting for each servo at a given step. "random game" will the device to jump from one to another step, picked randomly from a list of settings.

As you may see in the accompanying video, I had build a diabolo shaped object from LEGO that can be picked up by the MeArm and be transported from one location to another by a predefined set of movements. You may define your own functions by modification of the 'transport1' or 'transport2' lists.

The script listed here is a modification of the "Hearing Snake" example from "Project Keyword Spotter". The example has been stripped down to a minimum, then the part for driving the servos was added, based on the software and examples provided for the Adafruit servo bonnet.

The script has not been optimized by now. Use on your own risk, feel free to modify and optimize.

In addition to the python script there is the commands-file and the used labels-file. Place it in the config-subfolder.

As mentioned before, several adjustments of parameters might be required to adapt the script for your special MeArm or some other device.

# Copyright 2019 Google LLC#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     href="https://www.apache.org/licenses/LICENSE-2.0" href="https://www.apache.org/licenses/LICENSE-2.0"  https://www.apache.org/licenses/LICENSE-2.0

#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the original "hearing_snake" code was modified for an implemention for the MeArm by Dr H.

'''
Instructions

My implementation uses a Raspbery Pi 4 with a Google Coral accelerator and
an Adafruit 16 channel servo bonnet attached.
The servos of a MeArm (MIME industries) were attached to ports 3, 7, 11 and 15 of the bonnet.
For details please have a look on the "Hearing MeArm" Instructable.
Commands:
"position x", x= 0 to 9, moves the device to a given predefined position.
"move/go up", "move/go down", "go/turn forwards", "go/turn backwards",
"turn/go left" and "turn/go right" evoke a slow, stepwise movement in the given direction,
"stop game" stops the movements.
"open tab" and  "close tab" opens or closes the gripper.
"start video" evokes the device to go follow a preset order of positions,
defined by the list 'positions'.
"random game" results in a random pattern of movements, "stop game" ends it.
"launch game" starts another series of moves predefined by the list 'transport1',
"next game" the reverse operation predefined by 'transport2'

Use on your own risk.
'''

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import argparse
import os
from random import randint
from threading import Thread
import time
from edgetpu.basic.basic_engine import BasicEngine
import model
import pygame
from pygame.locals import *
import queue

from random import randrange
from adafruit_servokit import ServoKit

import board
import busio
import adafruit_pca9685
import time

i2c = busio.I2C(board.SCL, board.SDA)
hat = adafruit_pca9685.PCA9685(i2c)
hat.frequency = 60

kit = ServoKit(channels=16) # set number of channels
#kit.servo[0].actuation_range = 160
#kit.servo[0].set_pulse_width_range(1000, 2000)

# min, center and max settings 
up_l = 145  # servo up/down: up
md_l = 95
dn_l = 45

up_r = 135 # servo forward/backward
md_r = 90
dn_r = 50

ri_t = 30 # turning arm right or left: right position
md_t = 90 # turning arm right or left: center position
le_t = 150

op_g = 65 # gripper open
md_g = 90 # gripper centered
cl_g = 130  # gripper closed

vert = 15  # number of servo port, servo up/down
forw = 11  # number of servo port, forward/backward moving servo
turn = 7   # servo port for turning servo
grip = 3 # servo port for grip servo

#list of arm settings for nine positions
position = [(md_l,md_r,md_t,op_g),
            (up_l,md_r,ri_t,op_g),(up_l,md_r,md_t,cl_g),(up_l,md_r,le_t,cl_g),
            (md_l,md_r,md_t,op_g),(md_l,md_r,md_t,md_g),(md_l,md_r,md_t,cl_g),
            (dn_l,dn_r,ri_t,op_g),(dn_l,dn_r,md_t,md_g),(dn_l,dn_r,le_t,md_g)]
# defines 10 base positions, indicated by integers 0-9

# transport procedures [vert/forward/turn/grip]
transport1 = [(140,70,65,op_g),(110,50,65,op_g),(65,50,65,op_g),(65,70,65,cl_g),(120,70,65,cl_g), #get object
              (100,70,135,cl_g),(100,80,135,cl_g),
              (100,80,135,md_g),(100,80,135,op_g),(140,70,135,op_g),(140,70,90,op_g),(140,70,65,op_g)]</p><p>transport2 = [(140,70,65,op_g),(140,70,135,op_g),(95,70,135,op_g),(95,80,135,op_g),
              (95,80,135,cl_g),(110,70,135,cl_g),(110,70,65,cl_g),(70,70,65,cl_g),
              (70,70,65,op_g),(80,50,65,op_g)]</p><p>dance1 =(0,8,7,4,1,2,3,6,9,8,5,2,1,4,7,8,9,6,3,2,0) # a "dance"
  
#moving MeArm to Zero position

status =[md_l,md_r, md_t,md_g]
kit.servo[vert].angle = status[0]
kit.servo[forw].angle = status[1]
kit.servo[turn].angle = status[2]
kit.servo[grip].angle = status[3]
print (status)

class Controler(object):     #Callback function
    def __init__(self, q):
        self._q = q

    def callback(self, command):
        self._q.put(command)

class App:
  def __init__(self):
    self._running = True
  def on_init(self):
    pygame.init()
    self.game_started = True
    self._running = True
    return True

  def on_event(self, event):
    if event.type == pygame.QUIT:
      self._running = False

  def MeArmPos(self, keys):    # drives MeArm to preset positions, keywords: "position x"
        key = int(keys)
        p = position[key]
        a = p[0]
        b = p[1]
        c = p[2]
        d = p[3]
        print ("Positions: ", key, "  vert/forw/turn/grip: ",a,"/",b,"/",c,"/",d,"degrees")
        status = [a,b,c,d]  # documents current status
        print (status)
#       sys.stdout.write("Position: ", key, "  left/right: ",a,"/",b,"degree")
        kit.servo[vert].angle = a
        kit.servo[forw].angle = b
        kit.servo[turn].angle = c
        kit.servo[grip].angle = d
        time.sleep(0.5)
        
  def DancingMeArm(self):    # controls MeArm dance, keyword: "start_video"
        dnce = dance1
        sp=(len(dnce))
        for r in range (sp):     #dancing order of positions, sp steps
         dc = dnce[r]
         p = position[dc]
         a = p[0]
         b = p[1]
         c = p[2]
         d = p[3]
         kit.servo[vert].angle = a
         kit.servo[forw].angle = b
         kit.servo[turn].angle = c
         kit.servo[grip].angle = d
         time.sleep(1)  # sets velocity of movements
        time.sleep(0.5) # break at the end of procedure 
         
  def TransMeArm1(self):    # controls MeArm transport 1, keyword: "launch game"
        tr1 = transport1
        sp=(len(tr1))            #calculate number of steps
        for r in range (sp):     #go to any step
           p = tr1[r]
           a = p[0]
           b = p[1]
           c = p[2]
           d = p[3]
           kit.servo[vert].angle = a
           kit.servo[forw].angle = b
           kit.servo[turn].angle = c
           kit.servo[grip].angle = d
           print (p)
           time.sleep(1)  # sets velocity of movements
        time.sleep(0.5)
        
  def TransMeArm2(self):    # controls MeArm  dance, keyword: "next game"
        tr2 = transport2
        sp=(len(tr2))
        for r in range (sp):     #dancing order of positions, sp steps
           p = tr2[r]
           a = p[0]
           b = p[1]
           c = p[2]
           d = p[3]
           kit.servo[vert].angle = a
           kit.servo[forw].angle = b
           kit.servo[turn].angle = c
           kit.servo[grip].angle = d
           print (p)
           time.sleep(1)  # sets velocity of movements
        time.sleep(0.5)
        
  def RandomMoves(self):    # jumps randomly between predefined positions, keyword: "random game"
         dr= randrange (9) #randomly selects a position 
         p = position[dr]   # reads position parameters
         a = p[0]
         b = p[1]
         c = p[2]
         d = p[3]
         kit.servo[vert].angle = a
         kit.servo[forw].angle = b
         kit.servo[turn].angle = c
         kit.servo[grip].angle = d
         time.sleep(1)  # sets velocity of movements

  def MoveUp(self):        # lifting gripper in small steps
        u0 = status[0]     # read current status
        u1 = u0 + 5        # plus x degrees
        if (u1 > up_l):    # tests if not exceeding min/max parameters
            u1 = up_l      # otherwise set to min/max value
        kit.servo[vert].angle = u1 # move servo
        status[0] = u1     # adjust status value
        print ("up ", status)
        time.sleep (1)     # sets velocity
        
  def MoveDown(self):
        d0 = status[0]
        d1 = d0 - 5 #minus x degrees
        if (d1 < dn_l):
            d1 = dn_l
        kit.servo[vert].angle = d1 # move servo
        status[0] = d1
        print ("down ", status)
        time.sleep (1)

  def MoveForw(self):
        f0 = status[1]
        f1 = f0 + 5 #plus x degrees
        if (f1 > up_r):
            f1 = up_r
        kit.servo[forw].angle = f1 # move servo
        status[1] = f1
        print ("forward ", status)
        time.sleep (1)
        
  def MoveBack(self):
        b0 = status[1]
        b1 = b0 - 5 #minus x degrees
        if (b1 < dn_r):
            b1 = dn_r
        kit.servo[forw].angle = b1 # move servo
        status[1] = b1
        print ("back ", status)
        time.sleep (1)

  def MoveLeft(self):
        l0 = status[2]
        l1 = l0 + 2 #plus x degrees
        if (l1 > le_t):
            l1 = le_t
        kit.servo[turn].angle = l1 # move servo
        status[2] = l1
        print ("left ", status)
        time.sleep (0.2)
        
  def MoveRight(self):
        r0 = status[2]
        r1 = r0 - 2 #minus x degrees
        if (r1 < ri_t):
            r1 = ri_t
        kit.servo[turn].angle = r1 # move servo
        status[2] = r1
        print ("right ", status)
        time.sleep (0.2)

  def OpenGrip(self):
      kit.servo[grip].angle = op_g # set grip to "open" position: "open_tab"
      time.sleep(0.5)
      status[3] = op_g

  def CloseGrip(self):
      kit.servo[grip].angle = cl_g # set grip to "closed" position: "close_tab"
      time.sleep(0.5)
      status[3] = cl_g
      
  def StopMove(self):  # does nothing, but stops movements
      print ("stop ", status)
      time.sleep(0.25)
  
  def spotter(self, args):
    engine = BasicEngine(args.model_file)

    mic = args.mic if args.mic is None else int(args.mic)
    model.classify_audio(mic, engine,
                         labels_file="config/labels_gc2.raw.txt",
                         commands_file="config/commands_v1_MeArm.txt",
                         dectection_callback=self._controler.callback,
                         sample_rate_hz=int(args.sample_rate_hz),
                         num_frames_hop=int(args.num_frames_hop))

  def on_execute(self, args):
    if not self.on_init():
      self._running = False

    q = model.get_queue()
    self._controler = Controler(q)

    if not args.debug_keyboard:
      t = Thread(target=self.spotter, args=(args,))
      t.daemon = True
      t.start()

    item = -1
    while self._running:
      pygame.event.pump()
      if args.debug_keyboard:
        keys = pygame.key.get_pressed()
      else:
        try:
          new_item = q.get(True, 0.1)
        except queue.Empty:
          new_item = None

        if new_item is not None:
          item = new_item

      if (args.debug_keyboard and keys[pygame.K_ESCAPE]) or item == "stop":
        self._running = False

#      if (args.debug_keyboard and keys[pygame.K_SPACE]) or item == "go":
#        self.MeArmPos(7)
#
      if (args.debug_keyboard and keys[pygame.K_RIGHT]) or item == "right":  # turn right
          self.MoveRight()

      if (args.debug_keyboard and keys[pygame.K_LEFT]) or item == "left":    # turn left
          self.MoveLeft()

      if (args.debug_keyboard and keys[pygame.K_UP]) or item == "up":
          self.MoveUp()

      if (args.debug_keyboard and keys[pygame.K_DOWN]) or item == "down":
          self.MoveDown()
          
      if (args.debug_keyboard and keys[pygame.K_B]) or item == "b": # backwards
          self.MoveBack()

      if (args.debug_keyboard and keys[pygame.K_F]) or item == "f":   # forwards
          self.MoveForw()

      if (args.debug_keyboard and keys[pygame.K_O]) or item == "o":  # open grip: 
          self.OpenGrip()

      if (args.debug_keyboard and keys[pygame.K_C]) or item == "c":  # close grip: 
          self.CloseGrip()

      if (args.debug_keyboard and keys[pygame.K_S]) or item == "s":  # stop movement: "start_game"
          self.StopMove()

      if (args.debug_keyboard and keys[pygame.K_0]) or item == "0":
          self.MeArmPos(0)

      if (args.debug_keyboard and keys[pygame.K_1]) or item == "1":
          self.MeArmPos(1)

      if (args.debug_keyboard and keys[pygame.K_2]) or item == "2":
          self.MeArmPos(2)

      if (args.debug_keyboard and keys[pygame.K_3]) or item == "3":
          self.MeArmPos(3)

      if (args.debug_keyboard and keys[pygame.K_4]) or item == "4":
          self.MeArmPos(4)

      if (args.debug_keyboard and keys[pygame.K_5]) or item == "5":
          self.MeArmPos(5)

      if (args.debug_keyboard and keys[pygame.K_6]) or item == "6":
          self.MeArmPos(6)

      if (args.debug_keyboard and keys[pygame.K_7]) or item == "7":
          self.MeArmPos(7)

      if (args.debug_keyboard and keys[pygame.K_8]) or item == "8":
          self.MeArmPos(8)

      if (args.debug_keyboard and keys[pygame.K_9]) or item == "9":
          self.MeArmPos(9)
          
      if (args.debug_keyboard and keys[pygame.K_a]) or item == "d":
          self.DancingMeArm() #dancing MeArm, on "next_game"

      if (args.debug_keyboard and keys[pygame.K_r]) or item == "r":
          self.RandomMoves() #random dance "random game"
     
      if (args.debug_keyboard and keys[pygame.K_j]) or item == "j":
          self.TransMeArm1() # transport object: "lunch_game"
      
      if (args.debug_keyboard and keys[pygame.K_k]) or item == "k":
          self.TransMeArm2() # transport object reverse direction: "next_game" 
 
      '''
      if (args.debug_keyboard and keys[pygame.K_l]) or item == "l":
          self.JumpingJack2(1) #LED blink "target"
      '''
            
      time.sleep(0.05)
    self.on_cleanup()

if __name__ == '__main__':
  parser = argparse.ArgumentParser()
  parser.add_argument(
      '--debug_keyboard',
      help='Use the keyboard to control the MeArm.',
      action='store_true',
      default=False)
  model.add_model_flags(parser)
  args = parser.parse_args()
  the_app = App()
  the_app.on_execute(args)