Real Life J.A.R.V.I.S Well.. Almost

Summary

I've recently started to learn how to code/program and by coincidence my CAD class at school decided to make one of the projects be to participate in an instructables contest. Then as a fan of Tony Stark and Computer Engineering the idea of creating a J.A.R.V.I.S like assistant was extremely intriguing to me. So then this Roxy came into life. Using my knowledge of Python and my 3D printing skills I created a robot with an implemented voice assistant to aid you through your journey of life. My program has currently over 50+ available commands and more will be coming! After all a robot is a good choice for a life long companion.

Notes

When the project was assigned to us I only had less than 5 days left to enter this contest. Please do not take this as an excuse because it is not. But I just want you to keep in mind that this project has a large amount of potential to become better and more technologically advanced. I will be actively working on and updating this project. If you are interested in seeing where Roxy will go check out my Github Repo Well that's it for this notes section, have fun reading this instructable! :)

- Raspberry Pi or Arduino Robot (Servos, Wheels, Motors, Power Source, etc.)

- Audio output/input device of your choice

- Python 3.6 or above

- An IDE of your choice

For this project I decided to use an old robot I had that is powered by a Raspberry Pi and lithium batteries. I chose a Raspberry Pi robot because in the future I am going to make the program for the voice assistant runnable purely from the raspberry pi. This robot has an ultrasonic sensor and a camera module with implemented face recognition using opencv. This robot also came with a line tracking module but I decided to not include it. I bought this robot from Amazon. "Freenove 4WD Raspberry Pi Robot"

Right now my robot just looks like what a phone would look like if it were naked and all the components were revealed. So I decided to 3D print a casing/shell for my robot to protect the Raspberry Pi from damage and add a place for holding things. I 3D designed this casing using Solidworks.

To install Python go to Download Link Please install Python 3.8.8 to ensure stability of the code but I believe any Python version 3.6 and above should work. Also, when installing python you will need to install pip alongside it.

If you are interested in learning how the code works read this, if not skip this step and step 5. The full code will be linked at the very end of the instructable.

Alright now that I've gathered all my fellow Philomaths let's start with the explanation of the code. Also before any confusion arises I wrote this code and the name of the voice assistant is "Roxy"

import winshell
import subprocess
import pyttsx3
import json
import random
import operator
import datetime
import ctypes
import time
import wikipedia
import newsapi
import wolframalpha
import smtplib
import webbrowser
import os
import requests
import selenium
import pyjokes
import feedparser
import shutil
from bs4 import BeautifulSoup
from twilio.rest import Client
from urllib.request import urlopen
from clint.textui import progress
import win32com.client as wincl
import speech_recognition as sr

This first section of the code is where all the packages that are needed are being imported. But you don't have to worry about having to install each package one by one. I've included a setup.py file so you can install it all within a click.

setting up voice engine
engine = pyttsx3.init('sapi5')
voices = engine.getProperty('voices')
print(voices)
engine.setProperty('voice', voices[1].id)
# set which browser you wish to use
chrome_path = "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe"
webbrowser.register('chrome', None, webbrowser.BackgroundBrowser(chrome_path))
webbrowser = webbrowser.get('chrome')

In this section of the code the we are defining which voice engine we are using as well as which voice we would like to include. Below that is where we register which browser we would like Roxy to use.

def speak(audio):
    engine.say(audio)
    engine.runAndWait()


def wishMe():
    hour = int(datetime.datetime.now().hour)
    if hour >= 0 and hour < 12:
        speak("Good Morning Master")

    elif hour >= 12 and hour < 18:
        speak("Good Afternoon Master")

    else:
        speak("Good Evening Master")

    assistantname = ("Roxy 1 point o")
    speak("I am your Assistant")
    speak(assistantname)


def usrname():
    speak("What is your name master")
    global uname
    uname = takeCommand()
    speak("Welcome" + uname)
    columns = shutil.get_terminal_size().columns

    print("#####################".center(columns))
    print("Welcome Master", uname.center(columns))
    print("#####################".center(columns))

    speak("How can i Help you, Master" + uname)


def takeCommand():

    r = sr.Recognizer()

    with sr.Microphone() as source:

        print("Listening...")
        r.pause_threshold = 1
        audio = r.listen(source)

    try:
        print("Recognizing...")
        query = r.recognize_google(audio, language='en-ca')
        print(f"User said: {query}\n")

    except Exception as e:
        print(e)
        print("Unable to Recognize your voice.")
        return "None"

    return query

def takeSilentCommand():

    r = sr.Recognizer()

    with sr.Microphone() as source:

        r.pause_threshold = 1
        audio = r.listen(source)
    try:
        print("Recognizing...")
        query = r.recognize_google(audio, language='en-ca')
        print(f"User said: {query}\n")
    except:
        return "None"
    return query

def sleeping():
    r = sr.Recognizer()

    with sr.Microphone() as source:
        r.pause_threshold = 1
        audio = r.listen(source)

    z = 1

    while z == 1:
        print("Sleeping...")
        x = takeSilentCommand().lower()
        if "roxy" in x or "hey roxy" in x or "wake up roxy" in x or "yo roxy" in x or "hi roxy" in x:
            print("Waking up...")
            z = 0

This whole section is explaining each function.

speak(audio) - This function is just to setup the voice assistant to speak.

wishMe() - This function is the the first words Roxy will say to you. Roxy will greet you depending on what time of the day and introduce her/himself.

usrname() - In this function Roxy will ask for your name and set the name you provide into a variable so they can use your name in the future.

takeCommand() - This function is used to listen for commands coming in from your input device.

takeSilentCommand() - This function is pretty much the same as takeCommand but I removed the "listening" string.

sleeping() - This function starts right as you start the python program. This makes it so you have to wake up Roxy with a certain phrase before using the program.

Welcome to the main part of the program. This is where all the voice commands are defined and what Roxy will do depending on the voice command. There is far too many things to explain so I suggest you go out and learn everything for yourself :)

Before running the code you will need to open it in an IDE and generate api keys for certain functions to work. You can do this very easily using the find action (in most cases it's Ctrl+F) and find where the word api is found and enter your personal api codes.

You will also need to change the path of where some files are located.

if __name__ == '__main__':
    def clear(): return os.system('cls')

    clear()
    sleeping()
    wishMe()
    usrname()

    while True:

        # qeury is the proccess of taking the voice input of a user and turning it into a command
        # .lower() makes the input all lowercase for easier recognition

        assistantname = ('Roxy')
        query = takeCommand().lower()

        if 'wikipedia' in query:
            speak('Searching Wikipedia...')
            query = query.replace("wikipedia", "")
            results = wikipedia.summary(query, sentences=3)
            speak("According to Wikipedia")
            print(results)
            speak(results)

        elif 'open youtube' in query:
            speak("Here you go to Youtube\n")
            webbrowser.open("youtube.com")

        elif 'open google' in query:
            speak("Here you go to Google\n")
            webbrowser.open("google.com")

        elif 'open stack overflow' in query:
            speak("Haha looks like this loser needs to use stack overflow")
            webbrowser.open("stackoverflow.com")

        elif 'open reddit' in query:
            speak("Here you go to Reddit\n")
            webbrowser.open("reddit.com")
        elif 'open crunchyroll' in query:
            speak("Anime Time!")
            webbrowser.open("crunchyroll.com")

        elif 'play music' in query or "play song" in query or "play my music" in query or "play my songs" in query:
            speak("Alright bet, cutie")
            # music_dir = "*drive*\*user*\*music folder"
            music_dir = "D:\louis\ilovemusic"
            songs = os.listdir(music_dir)
            print(songs)
            random = os.startfile(os.path.join(music_dir, songs[1]))

        elif 'the time' in query or 'what is the time' in query or 'tell me the time' in query or "what time is it" in query or "what is the current time" in query:
            strTime =  time.strftime("%H:%M:%S")
            speak(f"Master, the time is {strTime}")
            print(strTime)

        if 'how are you' in query:
            speak("I am fine, Thank you")
            speak("How are you" + uname)

            if 'fine' in query or "good" in query or "great" in query or "amazing" in query:
                speak("You better be")

        elif "change name" in query:
            speak("What's wrong with my name! Fine, what would you like to change it to ")
            assistantname = takeCommand()
            speak("Thank you for naming me")

        elif "what's your name" in query or "what is your name" in query:
            speak("My friends call me")
            speak(assistantname)
            print("My friends call me", assistantname)

        elif "i'm sad" in query or "i hate everyone" in query or "fuck i'm sad" in query or "i'm so tired" in query or "i hate my life" in query:
            webbrowser.open('https://www.youtube.com/watch?v=dyUDQGUQPck')

        elif "i'm happy" in query:
            webbrowser.open('https://youtu.be/_dpYeMZUKRM')

        elif 'exit' in query:
            speak("Thanks for using me")
            exit()

        elif "who made you" in query or "who created you" in query:
            speak("I was created by the homie Louis.")

        elif 'joke' in query or "tell me a joke" in query:
            speak(pyjokes.get_joke())

        elif "calculate" in query:
            # make an account in wolfram alpha in order to create an api key
            # then place api key in app_id
            app_id = "<api key>"
            client = wolframalpha.Client(app_id)
            indx = query.lower().split().index('calculate')
            query = query.split()[indx + 1:]
            res = client.query(' '.join(query))
            answer = next(res.results).text
            print("The answer is " + answer)
            speak("The answer is " + answer)

        elif "who am i" in query:
            speak("You are" + uname + ", my master!")

        elif "how were you created" in query:
            speak("Thanks to Louis. further It's a secret")

        elif 'what is love' in query:
            speak("It is 7th sense that destroy all other senses")

        elif 'reason for you' in query or "who are you" in query:
            speak("My name is Roxy, I was created as a Minor project by the homie Louis ")

        elif 'change background' in query:
            ctypes.windll.user32.SystemParametersInfoW(20,
                                                       0,
                                                       "Location of wallpaper",
                                                       0)
            speak("Background changed succesfully")

        elif 'open steam' in query:
            appli = r"D:\Program Files (x86)\Steam\Steam.exe"
            os.startfile(appli)
            speak("Opening steam")

        elif 'news' in query:

            try:
                jsonObj = urlopen('https://newsapi.org/v2/top-headlines?sources=bbc-news&apiKey=<enter your api key>')
                data = json.load(jsonObj)
                i = 1

                speak('here are some news')
                print('''=============== NEWS ============''' + '\n')

                for item in data['articles']:

                    print(str(i) + '. ' + item['title'] + '\n')
                    print(item['description'] + '\n')
                    speak(str(i) + '. ' + item['title'] + '\n')
                    i += 1
                    if i == 4:
                        break

            except Exception as e:

                print(str(e))

        elif 'lock window' in query:
            speak("locking the device")
            ctypes.windll.user32.LockWorkStation()

        elif 'shutdown system' in query:
            speak("Hold On a Sec ! Your system is on its way to shut down")
            subprocess.call('shutdown / p /f')

        elif 'empty my recycle bin' in query or 'empty recycle bin' in query:
            winshell.recycle_bin().empty(confirm=False, show_progress=False, sound=True)
            speak("Recycle Bin Emptied")

        elif "don't listen" in query or "stop listening" in query:
            speak("for how many seconds do you want to stop Roxy from listening commands")
            a = int(takeCommand())
            time.sleep(a)
            print(a)

        elif "where is" in query:
            query = query.replace("where is", "")
            location = query
            speak("User asked to Locate")
            speak(location)
            webbrowser.open(
                "https://www.google.ca/maps/place/" + location + "")

        elif "restart" in query:
            subprocess.call(["shutdown", "/r"])

        elif "hibernate" in query:
            speak("Hibernating")
            subprocess.call("shutdown / h")

        elif "log off" in query or "sign out" in query:
            speak("Make sure all the application are closed before sign-out")
            time.sleep(5)
            subprocess.call(["shutdown", "/l"])

        elif "write a note" in query or "write note" in query:
            speak("What should i write, master")
            note = takeCommand()
            file = open('notes.txt', 'a')
            speak("Master, Should i include date and time")
            snfm = takeCommand()
            if 'yes' in snfm or 'sure' in snfm:
                strTime = datetime.datetime.now().strftime("%m-%d-%y %H:%M:")
                file.write("\n" + strTime)
                file.write(" - ")
                file.write(note)
                speak("Successfully written note")
            else:
                file.write("\n" + note)
                speak("Successfully written note")
        elif "clear notes" in query:
            file = open('notes.txt', 'w')
            file.write("NOTEPAD")
            speak("Sucessfully cleared notes")

        elif "show note" in query or "show notes" in query:
            speak("Showing Notes")
            file = open("notes.txt", "r")
            print(file.read())
            speak(file.read(6))
        elif "save link" in query or "save a link" in query:
            speak("What is the link master")
            link = input("Paste link: ")
            file = open('links.txt', 'a')
            speak("What is this link master")
            linkinfo = takeCommand()
            file.write("\n" + link + " - " + linkinfo)
            speak("Successfully saved link")

        elif "show links" in query or "show my links" in query:
            speak("Showing links")
            file = open("links.txt", "r")
            print(file.read())
            speak(file.read(6))
        elif "clear links" in query:
            file = open('links.txt', 'w')
            file.write("LINKS")
            speak("Sucessfully cleared links")

        elif "i have an idea" in query or "write down my idea" in query or "write down this idea" in query or "save this idea" in query:
            speak("What is your brilliant idea master")
            idea = takeCommand()
            file = open('ideas.txt', 'a')
            file.write("\n" + idea)
            speak("Successfully saved idea")

        elif "show me my ideas" in query or "show ideas" in query or "show my ideas" in query or "show me my amazing ideas" in query or "show me my brilliant ideas" in query:
            speak("Showing ideas")
            file = open("ideas.txt", "r")
            print(file.read())
            speak(file.read(6))

        elif "clear ideas" in query or "clear my ideas" in query:
            file = open('ideas.txt', 'w')
            file.write("IDEAS")
            speak("test")

        elif "thank-you" in query or "thank you" in query or "thanks" in query:
            speak("I am honored to serve you" + uname)

        elif "Roxy" in query:
            speak("Roxy 1 point o in your service Mister")
            speak(assistantname)

        elif "weather" in query:
            #\\dev note// find a way to make the output of temperature an int rather than float. why? cuz float looks ugly af
            # openweathermap.org to get api
            api_key = "<insert api key>"
            base_url = "http://api.openweathermap.org/data/2.5/weather?"
            speak("What is the name of the city ")
            city = takeCommand()
            complete_url = base_url + "q=" + city + "&appid=" + api_key
            response = requests.get(complete_url)
            data = response.json()

            if response.status_code == 200:
                main = data["main"]
                temperature = main["temp"] - 273.15
                pressure = main["pressure"]
                humidity = main["humidity"]
                report = data ['weather']
                print(f"Temperature: {temperature}")
                print(f"Humidity: {humidity}")
                print(f"Pressure: {pressure}")
                print(f"Weather Report: {report[0]['description']}")
                speak(f"Temperature: {temperature}")
                speak(f"Humidity: {humidity}")
                speak(f"Pressure: {pressure}")
                speak(f"Weather Report: {report[0]['description']}")

            else:
                speak(" City Not Found ")

        elif "send message " in query:
            # twillo account needed in order to use this
            account_sid = 'Account Sid key'
            auth_token = 'Auth token'
            client = Client(account_sid, auth_token)

            message = client.messages \
                .create(
                    body=takeCommand(),
                    from_='Sender No',
                    to='Receiver No'
                )

            print(message.sid)

        elif "Good Morning" in query:
            speak("A warm" + query)
            speak("How are you Master")
            speak(assistantname)

        elif "Good Night" in query:
            speak("Goodnight master" + uname)
            speak("Sweet Dreams")
            exit()

        elif "will you be my gf" in query or "will you be my bf" in query or "will you be my girlfriend" in query or "will you be my boyfriend" in query:
            speak("ask me again in a couple days master")

        elif "i love you" in query or "i am in love with you" in query or "i'm in love with you" in query or "im in love with you" in query:
            speak("Hahaha, who would ever love you back")
        elif "i hate you" in query or "fuck you" in query:
            speak("that is not very nice!")

        elif "what is" in query or "who is" in query:

            # use same wolfram api key here
            client = wolframalpha.Client("<api key>")
            res = client.query(query)

            try:
                print(next(res.results).text)
                speak(next(res.results).text)
            except StopIteration:
                print("No results")

        elif "go to sleep" in query:
            speak("okay master")
            sleeping()
            r = 1
            if r == 1:
                wishMe()


        # elif "" in query:
            # Command go here
            # For adding more commands

Before running the code you will need to open it in an IDE and generate api keys for certain functions to work. You can do this very easily using the find action (in most cases it's Ctrl+F) and find where the word api is found and enter your personal api codes.
You will also need to change the path of where some files are located.

Okay now that I have explained all the code, I will now explain how to use Roxy. After running setup.py to install all needed packages and generating and adding your personal api keys you should be all setup to use Roxy. You can either run Roxy by double clicking roxy.py or using your desired terminal. The program should begin to load and you will be able to tell when the program has loaded once you see the image attached to this step in your terminal window.

Once you see that image Roxy is now on stand by/sleeping and waiting for you to call for her. The following phrases can be used to wake her up:

"hey roxy", "roxy", "wake up roxy", "yo roxy", "hi roxy"

Of course you are free to add or change it to your liking but once you wake her up or call for her the program will start. Bam! You have your very own voice assistant.

Hello, congratulations on making it to the end of this instructable. Thank you for taking the time out of your day to read this and I hope it turned out for you well. If you liked this please leave a comment and favorite this article <3

If you have any questions or issues regarding this project please visit my github for further help.

A vote for the automation contest is very much appreciated!

CODE FOR PROJECT

Until the next project, El Psy Congroo

- Louis