Play AVI With ESP32

This instructables show how to play AVI video file with ESP32 and Arduino IDE.

Any ESP32 dev device with display and I2S audio output should be ok. Dev devices that with SD/MicroSD card slot are preferred.

This instructables mainly test and develop with Lilygo T-Deck:

https://www.lilygo.cc/products/t-deck

And also further tested with some decent dev devices:

1. https://www.aliexpress.com/item/1005006729377800.html
2. https://www.aliexpress.com/item/1005007566332450.html
3. https://www.aliexpress.com/item/1005007338742875.html

Note:

Since the arduino-esp32 upgrade from v2.x to v3.x has a big change of the interfaces supported. I highly recommend using SPI/QSPI display only during this period.

Audio Video Interleave (AVI) is a proprietary multimedia container format and Windows standard introduced by Microsoft in November 1992 as part of its Video for Windows software. AVI files can contain both audio and video data in a file container that allows synchronous audio-with-video playback.

AVI is a common video file format in last century. It is designed for playing video with the computer in that time. The disk reading speed and CPU processing power are very limited, the software engineer at that time use various techniques to optimize the reading, decoding and playing process.

Thanks to this thoughtful design, now we can use a ESP32 dev device to play AVI files.

Ref.:

https://en.wikipedia.org/wiki/Audio_Video_Interleave

Above flow chart illustrate a simple AVI Player design. First read file list from a file system folder, loop each item in file list. If it is an AVI file, open, read, decode and play it.

1. The file system can be SD card or built-in flash LittleFS or FATFS, most AVI video size requires SD card
2. Decode MP3 audio also a CPU intensive task, so it is better offload to another EPS32 core

AVI file is a container, it can contains video and audio data encoded in various codec.

Decoding video and audio is always a CPU intensive task, ESP32 is not powerful enough to decode most recent codec e.g. H.264, H.265.

This project concentrate on the codecs developed for last century computer only, i.e. Cinepak and Motion JPEG(MJPEG) video codec; PCM and MP3 audio codec.

Ref.:

https://en.wikipedia.org/wiki/Codec

https://en.wikipedia.org/wiki/Cinepak

https://en.wikipedia.org/wiki/Motion_JPEG

https://en.wikipedia.org/wiki/Pulse-code_modulation

https://en.wikipedia.org/wiki/MP3

AVI file data reading

"avilib" library is part of transcode, a linux video stream processing tool. And Github user lanyou1900 put it on Github. This project use avilib for reading avi file data:

https://github.com/lanyou1900/avilib.git

Cinepak Codec Decoder

ScummVM ported Cinepak decoder for decoding video in retro game. Now I further ported the decoder to Arduino, so we can decode Cinepak codec in Aduino.

MJPEG Codec Decoder

MJPEG is simply a sequence of JPEG image, so we can use normal JPEG decoder like JPEGDEC or ESP32_JPEG for decoding:

https://github.com/bitbank2/JPEGDEC.git

https://github.com/esp-arduino-libs/ESP32_JPEG.git

PCM Codec Decoder

Pulse-code modulation (PCM) is a method used to digitally represent analog signals.

PCM also the format feeding audio data to I2S audio module, so actually no further decoder is required. But it still requires some bit size conversion, audio gain manipulation and audio channel data duplication.

https://en.wikipedia.org/wiki/Pulse-code_modulation

https://en.wikipedia.org/wiki/I²S

MP3 Codec Decoder

Github user pschatzmann ported Helix MP3 decoder from RealNetworks as a simple Arduino Library. This decoder also can decode the MP3 audio stream in AVI file:

https://github.com/pschatzmann/arduino-libhelix.git

Cinepak developed for early computer, so it requires relative low CPU power for decoding. But it tradeoff the encoding time, it is much higher than all the other codec. Moreover, the color space is also limited, so it only suitable for simple color anime video.

Motion JPEG(MJPEG) is simply a sequence of JPEG images, so the quality can be as good as JPEG image and capable decoded by ESP32. Since each frame is independent, no further compress between frames, so the size is relatively larger compare with other advanced video codec.

ESP32-S3 is able to decode:

1. 320x240@30fps MJPEG video
2. 480x272@15fps MJPEG video
3. 480x272@30fps Cinepak video

You can test and select the suitable codec for you. T-Deck MP3+Cinepak:

T-Deck MP3+MJPEG:

JC4827W543 MP3+Cinepak:

Note: Since read data from SD and feed audio may not well distributed in time, some frame skip is expected. The skipped frame should not notable in most case.

T-Deck PCM+Cinepak:

PCM audio format eliminated the decode time and should improve the frame rate. However, since we had offloaded the MP3 decode to another core so it did not have positive effect in ESP32-S3. PCM should help improve the frame rate for the single core device, e.g. ESP32-C3.

Arduino IDE

Download and install Arduino IDE latest 1.x version if not yet:

https://www.arduino.cc/en/software

Arduino-ESP32

Follow installation step to add Arduino-ESP32 support if not yet:

https://docs.espressif.com/projects/arduino-esp32/en/latest/installing.html

Arduino_GFX Library

Open Arduino IDE Library Manager by selecting "Tools" menu -> "Manager Libraries...". Search "GFX for various displays" and press "install" button.

You may refer my previous instructables for more information about Arduino_GFX.

avilib

Download and import the avilib library to Arduino IDE:

https://github.com/lanyou1900/avilib.git

JPEGDEC

Open Arduino IDE Library Manager by selecting "Tools" menu -> "Manager Libraries...". Search "JPEGDEC" and press "install" button.

ESP32_JPEG

Download and import the ESP32_JPEG library to Arduino IDE:

https://github.com/esp-arduino-libs/ESP32_JPEG.git

arduino-libhelix

Download and import the arduino-libhelix library to Arduino IDE:

https://github.com/pschatzmann/arduino-libhelix.git

Note:

You may refer to Arduino Documentation for the details on how to import library to Arduino IDE.

We can use ffmpeg convert most video file to AVI format.

Convert to 320x240 dimension 30 FPS video with PCM audio and Cinepak video codec:

Convert to 320x240 dimension 15 FPS video with MP3 audio and Motion JPEG video codec:

Then copy all converted AVI files to the SD card defined avi_folder.

1. Please download the source code at Github: https://github.com/moononournation/aviPlayer.git
2. Open AviPlayer/AviPlayer.ino in Arduino IDE
3. The default pins settings is for T-Deck, you may comment out the code `#include "T_Deck.h"` and use your own
4. Then compile and upload the program

It's time to show off this fancy device on your desk!

Let's say during Olympics 2024, you may want to loop the related Youtube Shorts on your desk. You can prepare the video with:

Or if you are a cat lover:

Then convert the video to AVI format:

And then copy the AVI files to the Micro SD card and play it.

Note:

Arduino file system do not recognize the file with emoji characters, so please rename the file before copy to the Micro SD card.