Speech-to-text Keyboard Via WiFi Using an ESP8266

In this Instructable, I will show how to use a Teensy LC along with an ESP8266 board to emulate a keyboard controlled from a mobile app on your smartphone. The mobile app converts speech to text (using the corresponding google API) and sends that to the emulator to be "typed" on your PC. Let's begin!

• Teensy LC & programming cable - This acts as the keyboard emulator
• ESP8266-01 (The ESP8266 comes in many packages, you can use any, however I have used the ESP-01) - This will act as an access point and receive the text to be typed from the mobile app
• USB to UART converter capable of running at 3.3v (Also called a USB to Serial converter) and programming cable - This will be used to program the ESP826
• Smartphone running Android
• Connecting wires and breadboard

The ESP8266 is going to act as an access point for your phone to connect to. It will also host a server running on port 8080 (although you may change this to anything), which the app connects to as a client and sends your speech data to. In this step, we will go through the code along with the circuit in detail.

We will be using the Arduino IDE to write the program and upload it to the board. If you are unsure of how to do this, click here to be taken to an instructable on how to set up the Arduino IDE to be able to compile and upload programs to the ESP8266.

First, start by including the ESP8266 Wifi library header. Then create strings to store your access point's SSID (the name used to identify it) and password, along with a server object running on port 8080. You may change this value to anything in the range 0 to 65535, just make sure to remember it for later on. Another integer called numStations has also been created, although this is only for debugging potential problems.

#include <ESP8266WiFi.h>

const char *ssid = "ESP-8266", *pass = "abcd1234";
int numStations = 0;
WiFiServer server(8080);

Next, go to the setup and start the Serial monitor (making sure to set the baud rate to 115200 and not anything else). Keep note of the baud rate, no other value will work with the ESP8266. Also start the access point and the server. I have also collected the IP of the access point, however this is for debugging purposes only.

void setup()
{
	delay(5000);
	Serial.begin(115200);

	WiFi.softAP(ssid, pass);
	IPAddress myIP = WiFi.softAPIP();

	server.begin();<br>}

Finally, enter the following code in the loop to keep checking for new clients who connect to the server, and keep checking for any input from them. On the availability of input, the individual characters are read and merely forwarded through the Serial port of the ESP8266. The program also makes sure that the only one client is served at a time, in the order in which they connected. There is also some extra code for counting and updating the number of stations connected to the AP, and this too, is for debugging purposes only.

void loop()
{
	if(numStations != WiFi.softAPgetStationNum()) numStations = WiFi.softAPgetStationNum();

	WiFiClient newClient = server.available();
	if(newClient)<br>	{
		while(newClient.connected())
		{
			while(newClient.available()) Serial.write(newClient.read());
		}
	}
}

Here is how the finished code should look.

#include<ESP8266WiFi.h>

const char *ssid = "ESP-8266", *pass = "abcd1234";
int numStations = 0;
WiFiServer server(8080);

void setup()
{
	delay(5000);
	Serial.begin(115200);

	WiFi.softAP(ssid, pass);
	IPAddress myIP = WiFi.softAPIP();

	server.begin();
}

void loop()
{
	if(numStations != WiFi.softAPgetStationNum()) numStations = WiFi.softAPgetStationNum();

	WiFiClient newClient = server.available();
	if(newClient)
	{
		while(newClient.connected())
		{
			while(newClient.available()) Serial.write(newClient.read());
		}
	}
}

Next, we will have to make the circuit to program the ESP8266. For this you will need a few connecting wires and a USB to UART converter, to convert from USB to Serial which can be used to program the board. Wire the boards as shown above, i.e.

Tx of ESP -> Rx of converter

Rx of ESP -> Tx of converter

Vin of ESP -> 3.3v

GND of ESP -> GND of converter

Chip enable of ESP -> 3.3 v of converter (To allow the board to power up)

Reset of ESP -> 3.3v of converter (connecting to ground resets the board)

GPIO0 of ESP -> GND

A few things to make sure before powering this,

1. Make sure that the GPIO 0 of the ESP8266 is connected to ground, as this puts it in programming mode, ready to accept new code.

2. Make sure that the power pins are connected the right way, otherwise you risk permanently damaging your board.

3. Make sure that the USB-UART is capable of 3.3v output ON ALL PINS. The ESP8266 CAN NOT be powered by 5v, and none of its pins can tolerate >3.3v.

Now, go to the Arduino IDE and select the right board (Generic ESP8266) along with the right Flash size (Mine is 1 MB) and the right Serial Port and hit upload.

In case your Serial port is not showing up, you can re-check all connections and make sure your PC has the appropriate device drivers set up. It is a common issue for Serial ports to not show up and you can click here to find out more on troubleshooting the problem.

After doing this, you can now check if the network created by the board shows up and your smartphone is capable of connecting to it, in which case you can move onto the next step.

The teensy LC emulates a keyboard and forwards the Serial messages it receives from the ESP8266 as Key strokes. We will be programming this with the Arduino IDE as well. If you are unsure of how to do this, click here to be taken to the official PJRC tutorial on the same. Once you are done setting up the environment, you can move ahead.

In the setup, start Serial communication, but on the second serial port of the teensy, making sure to select the baud rate as 115200, which is the same as the ESP8266.

void setup()
{
	Serial2.begin(115200);
}

In the loop, simply enter the following code which checks for the presence of any input on the Serial port, and forwards it as a key stroke in case it is available.

void loop()
{
	if(Serial2.available()) Keyboard.write(Serial2.read());
}

The code is quite straightforward, and should look like this.

void setup()
{
	Serial2.begin(115200);
}

void loop()
{
	if(Serial2.available()) Keyboard.write(Serial2.read());
}

To program the teensy, simply connect it to your PC through a USB cable and hit upload. You might have to select the right serial port, although this should mostly happen automatically. In case the upload freezes midway, try to press and release and the white button the teensy, which starts the bootloader to accept programs. You will also have to select the "USB type" as keyboard (It is Serial by default) for this to work.

This finishes the coding of the device, and we can start making the app and the final circuit.

In this step, we will connect the two boards together. Shown above is the final circuit.

First, we need to connect the ground pins on the ESP8266 and Teensy to a common ground, as shown. Then connect the Tx Pin on the ESP8266, to the Rx2 pin (pin 9) on the teensy. It is better to leave the other half of the Serial line disconnected (ESP8266 Rx -> Teensy Tx2 (pin 10)), even though I have shown it connected in the picture. This is because the Teensy LC operates at 5v, which can cause permanent damage to the ESP8266, in case something is accidentally sent. You can even connect them to ground, if you are uncomfortable with leaving them floating.

The ESP8266 may be powered by any source (including the USB-UART chip used previously), however it must be capable of providing upto 300 milli amps, which the board is known to draw on some occasions.

This finishes the final circuit. We can now move onto making the app.

In this step, we will be creating the mobile app for the project. Since it is a very simple app, we will use the cloud based MIT app-inventor software for this, as it provides a drag-and-drop block interface instead of code, which is beginner friendly and faster to get started with.

Start by going to the MIT App inventor website by clicking here (and creating an account if you haven't already). Then, create a new project and give it a name.

You can now drag and drop the following components onto the window-

• A Textbox
• A Connect button
• A Record button
• A Send button
• A Speech to Text converter (This is a non-visible component)

And adjust their sizes to match the following layout (This step will not affect the working of the app, it is cosmetic only). I have also changed the text on the buttons to "Connect", "Record" and "Send".

You will also need a way to create TCP sockets, for which I have used ClientSocketAI2 extension by Jean-Rodolphe Letertre. Here is the link to the download (click here). To import it into the environment, simply click the "import extension" button on the bottom left of the webpage and browse to the downloaded file.

Drag and drop the socket component onto your application as well.

This completes the GUI part of the app, and given above is a picture of how it should look. We can now move onto the blocks. Simply switch from "designer" to "blocks" on the top right of the webpage.

Next, you'll want to drag and drop the following blocks (as shown above).

• Button1.click, Button2.click, Button3.click (The blocks under each are executed whenever the corresponding button is pressed and released)
• SpeechRecognizer1.getText (This tells the speechRecognizer component to start the converter)
• SpeechRecognizer1.afterGettingText (The blocks under this are executed whenever the speech recognizer has completed getting one set of commands)
• TextBox1.setText (This changes the text in the text box to whatever is supplied to it at the end, which in this case are the results of the speech recognizer)
• SpeechRecognizer1.result (This returns the result of the speech to text conversion process)
• set ClientSocketAI2Ext1.serverAddress and ClientSocketAI2Ext1.serverPort (These are used to set the address of the server which we wish to connect to, in this case of the ESP8266)
• ClientSocketAI2Ext1.connect and ClientSocketAI2Ext1.sendData (These connect the client to, and send data to the server. The data sent in this case, is the text in the text box)

This does 3 things-

1. Connect to the remote server on the IP 192.168.4.1 (This is the default IP of the ESP8266 Access point and the server) at the port 8080 (Change this to whatever value you put in the Arduino Program).
2. Record speech data, convert it to text and set the text in the textbox to this upon pressing the record button.
3. Send the text in the textbox (typed or recorded) to the server.

This completes the mobile App. To be able to use this, you can either download the APK to your PC and transfer it to your phone or generate a QR code and scan it with your Phone which directly takes you to the download link. The APK then needs to be installed. Your phone might warn you about the App coming from an unknown source, but this is perfectly normal and it is safe to install it. You may also face some issues with permissions the first time but allowing them will allow normal functioning.

To know more about MIT App inventor, click here to be taken to a nice instructable and here to be taken to some additional resources. This completes the creation of the mobile app and the whole project. We can now move onto testing !

Start by connecting the circuit to the PC. After powering up you should be able to see the Wifi network of the ESP8266. Connect to it along with the password. If you have used exactly the code given at the beginning, it should be named "ESP-8266".

Now, start the app and click connect. If you get a popup with some exception, recheck your connection and the power source to the ESP8266. Otherwise, you are good to go.

You can now click the record button, which should cause the google speech recognition popup to appear and convert your voice to text, which appears in the text box. You can alternatively type something into the textbox directly. After pressing the send button, this text should appear on your computer as key strokes. You can open a text editor (I have used notepad) and confirm this.

This completes the project !