ESP32 VMU Handheld Console - Yes, It Plays Doom!

This is my first Instructables project so be gentle! So heres a quick introduction into the ESP32 VMU. Its a combination of things I've been working on recently such as RetroGo, the amazing software which runs on the ESP32, as well as the Null Nano designed by Ampersand (All info here) and also a device that was super popular a few years ago, the Circuit Gem by Kite, which fits a Raspberry Pi Zero into a Dreamcast VMU.

I had recently built up a Null Nano, which was an awesome project. I then went down a little rabbit hole of trying to port the latest RetroGo software to the Null Nano so had a good understand of how it worked. If I'm being honest, if you want something to play for longer, just go ahead and build the Null Nano!

One day, I looked up at my Circuit Gem and wondered if I could fit a RetroGo ESP32 into a VMU shell.

I'm not great with electronics, I wouldnt be able to look at a bunch to component datasheets and know what resistors or capacitors are required to properly make the circuit but I can copy/paste & drag/drop :) so I basically took Ampersand's Null Nano schematics and re-arranged it to fit in a VMU, thankfully it all worked out. Obviously massive credits go to him to creating an amazing device and also opening up the schematics for idiots like me.

For the VMU, I would recommend searching eBay, there are usually plenty on there. Ideally get one with a cap, its not essential but does look better. Colour is up to you, the standard white ones tend to be cheaper but the translucent ones look nicer in my opinion. The VMU itself doesnt actually have to work, you only need the shell/buttons and membranes.

There are also repo cases now, these do look nice but not sure on availability outside of the US as looks like preorders are in progress. They should be compatible and could potentially be a cheaper option: Link

Here's a list of the components used. I've tried to list parts from Aliexpress Choice service but some are not.

ESP32 16MB N16R8: https://www.aliexpress.com/item/1005007549782099.html

1.69" IPS Display: https://www.aliexpress.com/item/1005006916285489.html

Micro USB Port (Micro-23): https://www.aliexpress.com/item/1005007360060905.html

Micro Slide 3Pin Switch (2mm): https://www.aliexpress.com/item/1005005659497951.html

MicroSD Socket: https://www.aliexpress.com/item/1005006210048232.html

S8050 J3Y: https://www.aliexpress.com/item/1005006307277263.html

AP2112: https://www.aliexpress.com/item/32998839398.html

PAM8302A: https://www.aliexpress.com/item/1005006033828553.html

MCP73831/OT: https://www.aliexpress.com/item/1005006809469170.html

0805 600 Ohm Ferrite Bead: https://www.aliexpress.com/item/1005003124199167.html

0805 LED Kit: https://www.aliexpress.com/item/1005006417226949.html

USB UART Programmer: https://www.aliexpress.com/item/1005006445462581.html

Speakers:

15X3.5 Speaker: https://www.aliexpress.com/item/1005007674241739.html

Or 8R 0.5W 10mm Speaker: https://www.aliexpress.com/item/1005006682276648.html

More info on which to choose in below.

Batteries:

403035 400mAh battery

or 403048 600mAh battery

Aliexpress is not the best for batteries in my opinion, they dont tend to be covered by Choice. I usually use eBay. Just search for '403035 lipo battery' or similar

More info below but ties into the speaker choice.

0805 Resistors:

https://www.aliexpress.com/item/1005005600798857.html (Individual)

https://www.aliexpress.com/item/1005003737088895.html (Kit)

0805 Capacitor :

https://www.aliexpress.com/item/32964553793.html (Individual)

https://www.aliexpress.com/item/1005002782324319.html (Kit)

Probably best to buy a kit for the resistors and individual for the capacitors.

You can order the PCB from your chosen PCB manufacturer. JCLPCB and PCBWay are popular ones in China, or maybe Oshpark if you're US based.

Download the gerber file, upload to your chosen provider. The most important part is that you select a 1.0mm PCB if its not 1.0mm it wont fit in the case properly. All other settings apart from a funky colour, I leave standard personally.

You can download the gerber file from here: https://github.com/DynaMight1124/retro-go/releases/tag/VMU

You will need to be pretty good at soldering and have a good soldering iron with a suitable tip for some fairly small soldering plus the usual leaded solder, flux, solder wick.

I personally start off by soldering the ESP32, I would tack down one corner to make sure its very well lined up, this does need to be pretty accurate as its very easy to bridge pads. Once you're lined up, solder one side but I would suggest using a multimeter to check for bridges before soldering the other side. Once both sides are soldered without any bridges, if you have a good PCB holder, I'd probably recommend doing the MicroUSB socket next, but if you dont, its easier to have a flat base to solder the rest of the components on that side so go onto the Micro SD socket.

Now continue to solder the components however you see fit, just try to be smart so you have space, I tend to work from the inside outwards. Once the back side is done, start on the front side, it doesnt matter massively on the order, except for the screen, which should be last.

When you get to R3, this resistor controls the charge rate. The 'kit' resistor is 5.6k which gives 179mA charge rate however this can be adjusted to charge quicker. The max the chip supports is 500mA (2k resistor) and ideally you shouldnt go above 1C (this depends on the battery size but basically 1C on a 300mAh battery would be 300mA, & 400mAh battery would be 400mA etc). Other common resistor values could be 4.7k = 217mA or 2k = 500mA (max). 5.6k is very safe but can be quite slow. You can use this to calculate your desired charge rate: 1000/X = Y e.g. 1000/5.6 = 179mA.

R19 can control the volume as the original Null Nano had a variable resistor for its sound, a lovely slider which worked really well however that wouldnt had fitted the VMU without lots of additional shell hacking but luckily the software has built in volume control. I was a little concerned when designing the PCB that 10k might be way too loud but 10k seems a good volume to me with plenty of range to go lower from within the software so havent felt the need to adjust.

So the screen. It is quite important this is lined up as accurately as possible else the screen position will be off and it may bug you. It can be de-soldered if you're careful but its not super easy with a flex cable so probably best to get it right first time!

Heres the resistors and capacitors for each location:

5x 10uF: C1,C2,C3,C4,C11

3x 1uF: C5,C9,C10

3x 0.1uF: C6,C7,C8

2x 220pF: C12,C13

2x 600 Ohm Ferrite Bead: FB1,FB2

1x 0805 Green LED: LEDG

1x 0805 Red LED: LEDR

2x 470R: R1,R2

1x 5.6k: R3 or different if you want to change the charge rate, see above.

3x 100K: R4,R5,R9

9x 10K: R6,R7,R8,R10,R11,R12,R14,R18,R19

1x 10R: R13

1x 1K: R15

2x 100R: R16,R17

The remaining components are:

T1 - S8050

U2 - AP2112

U3 - PAM8302A

U6 - MCP73831/OT

This can be a tricky step, there is a flow to getting it done and its not always obvious why its not working, trust me! Theres no onboard flashing chip, hence the need of the external UART programmer. In theory it should only need to be done once!

You can download the firmware image file here: https://github.com/DynaMight1124/retro-go/releases/tag/VMU

You will need the .img file to flash to the ESP32, but you will also want VMU SD Files.zip as this contains the folder structure of the SD card. Theres no roms or BIOS files included so you'll have to find those yourself.

I have been using this app to flash: https://github.com/Jason2866/ESP_Flasher/releases/tag/v3.0.4 I find it simple to use. The official app 'Flash Download Tool' is quite clunky and requires a lot of settings to be set properly to allow proper flashing whereas ESP Flasher just sorts all this out for you, plus it supports other OS's.

Software wise, simply plug in your external programmer so it picks up the COM port, choose the .img file (you may need to select 'All files') then click Flash ESP when ready. If you get an error about COM ports being used/taken, I tend to unplug/replug the programmer or worst case close/re-open ESP Flasher.

Hardware wise, you need to make sure its done in a proper order. First get the VMU into Boot mode, power on the VMU while holding down the Mode button to set it into Boot mode for flashing, you only need to hold the button for a second while turning on, the display should be black (but backlight lit), next make sure all your wires are hooked up correctly, GND to GND, but RX>TX and TX>RX (note the what I've just said, the wires need to be crossed, trust me I've previously spent a lot of time before I realised this!!) between the VMU and programmer. Its also important that the programmer is set to 3.3v via the header.

So once you have the programmer plugged in, the wires connected, click on 'Flash ESP' after a few seconds it should connect and flash.

If you get an error, its probably not in Boot mode, turn the VMU off, re-hold the Mode button and switch on, try again. Obviously if you keep having issues, check the wiring. It is sometimes easier to program with the front shell in place as easier to hold everything plus easier to press the Mode button.

You can solder directly to the Prog vias on the PCB however its not needed, I have used Dupont connectors and just angle them into the un-soldered vias, it's worked fine for me and only needs to make a connection for like 30 seconds to flash, obviously you're welcome to solder, on my original test PCB, I soldered the pin header then bent it a little, it did allow it to fit into the case afterwards so you're welcome to do that.

P.S. I believe you need to connect the battery for the flashing to work rather than powering via the MicroUSB port, but not 100% sure tbh.

Now that the image is flashed, its time to work on the shell. This could be the trickiest part depending on your experience as it requires quite a bit of work. If you're not super confident on this part, I would recommend the white shell as you can hack the inside a lot and it wont show any of that in the outside, however the translucent shells will show through so you have to be way more careful and only cut the minimum, even a scratch from a hobby knife can be seen from the outside.

Theres two parts to this section. First is the front shell, this requires a lot less cutting so best to start here. If you make a total mess of this, then it's a good learning experience for the hell that awaits!

Front Shell

You can trim the front shell using just some flush cutters and hobby knife, like the ones you get with 3D Printers: https://www.aliexpress.com/item/1005005544592970.html or you can just go straight in with a rotary tool/Dremel if you prefer. It doesnt matter how you get there, you just need to make sure it doesnt interfere with the screen. I found it easier to cut vertical in small sections then rock back and forth until it broke away, then I used a hobby knife clean up the edges. Check the pics for the amount of material I cut away, which was mostly to allow the screen to fit.

Rear Shell

This part requires quite a bit of cutting as pretty much the whole back section needs to be cut away to allow space for the battery and speaker, even without a battery a lot of the VMU's original coin cell battery compartment needs to be removed due to the ESP32's thickness so once you start you may aswell keep going.

This part does require the careful use of a rotary tool/Dremel (theres a pic of the two attachments I used, I didnt use a cutting disc, just two different sanding attachments for different angles), I did try using only flush cutters when creating this project but in my opinion it wouldnt had left enough space unless you used a very small battery. I spent quite a bit of time cutting away small bits to check for fitment and space to try to cut away the minimum but ended up with a very similar setup to the Circuit Gem i.e pretty much everything.

If you have a translucent shell you need to be super careful not to scuff up the inside as it will show through the shell, you can see this in the above images around the battery door. If you have a white shell, then just go to town.

The battery door, needs to be trimmed flat and essentially superglued into place, I place on a flat surface, glue a few corners and make sure its even, then once thats set, I tend to run a bead around it to ensure its secure. Obviously try not to use too much so its not thick. You can retain the original screw on the battery cover by cutting it flush and super gluing it on.

I have tried to show this in the pictures and a few badly edited to show where I have cut, one ended up looking at person so got legs ha!

Also the top cap does need the middle section removed, just use some flush cutters and its all good.

Just adding a new section to show the shells after all the cutting as its easier to show here rather than mixed in the above section. Also included a white shell which is quite a bit more hacked up but totally fine as not translucent.

This has its own section because you do have some choice here. There isnt a lot of space so there needs to be some compromise. The battery needs to be a maximum of 4mm thick and 30mm wide but depending on your choice of speaker can be quite long.

Initially I had intended on a 403035 400mAh battery, but when hunting around I found quite a few 4030XX batteries that will fit, pretty much up to around 700mAh can be found, but once you go above 403035 you lose the space for the 15mm speaker.

I found a 10mm speaker, which was 3mm thick, that happens to be the perfect size to fit under the screen, however they have a slight hiss so you may not like this. Also the 10mm speaker will fit just above the ESP32 between the PCB and battery but obviously still have that hiss :) I picked up a 403048 600mAh for testing and it does fit great with the 10mm speaker.

The 15mm speaker is lovely and crisp but limits the battery choice. There is still a small hiss but thats probably related to my poor PCB design. Its certainly a lot more noticeable on the 10mm, even when crammed between the PCB and battery.

Theres a 3mm spacer needed between screen and PCB. If you use the small 10mm speaker then use that, you'll notice theres a speaker solder point top and bottom of the PCB for this purpose.

If you dont, I have created a small simple 3D printable spacer. Just place it using the locating hole and it should keep the screen in the right place. You can use something else if you wish, like foam, needs to be 3mm from my testing.

Place the whole unit into the VMU shell, screw the PCB into place using the original VMU's smaller screws, then place the back cover on using the original VMU's 4 bigger screws and its done.

The screen might need a little moving, just you your finger to push it around. Its a fine line between too tight in the shell and causing issues with the display and too loose where the screen moves around. I tried to get it close! I didnt design it for use with double sided tape as I felt it wouldnt be easy to line up before placing in the shell, but you are welcome to use tape but you'll probably want to either sand down or adjust the spacer so its not too thick.

To be honest, the screen flapping in the breeze while its out of the shell is my worst part about this project, I was thinking off a cradle for the screen to slot into for perfect alignment but the space around the screen is quite limited however it is fairly solid when its in the shell even if its only help in my a little pressure.

So its all together, ensure you have copied the file structure from VMU SD Files.zip onto your FAT32 formatted MicroSD card. You dont need a massive card, it doesnt support a large amount of systems and even if you had the entire library of every console supported it wouldnt be any bigger than about 4GB. I've actually been using a 512MB card with my curated games list, which still manages to be over 1,300 games as some of the smaller systems have the whole library, all of those games including all the romart etc still fit on a 512MB card.

The SD structure has all the folders setup but no roms, so you will need to obtain these yourself, however I have included the shareware version of Doom to get you going. Once you have backed up all your legally acquired games, copy them to the Roms directory. They can be in a zip file if needed. There is a BIOS directory in the retro-go folder, I believe this is only needed for MSX so you will need to locate these yourself also.

Theres lots of information about supported systems and features on the RetroGo Github page: https://github.com/ducalex/retro-go

When you boot up you should be greeted with the menu screen. Left and right scroll through the menu, A to enter a console, B to back out. When in a game, Mode acts as Start and Sleep acts as Menu, there are also 2 virtual buttons Select and Option. You can use these by pressing Mode and A or Mode and B together. You can also get to Options by pressing Sleep and going down to Options.

The rest of the system is fairly easy to use, have a play around. If you mess something up, it'll likely be easy to fix by resetting the settings or overwriting the files again from the VMU SD Files.zip.

As for charging the device. Plugging it in will charge it obviously, I wouldnt recommend playing and charging at the same time. When its charged the green LED will come on. The LED's are only for charging, no LED's are visible when the device is on (and not charging) this is normal.

The ESP32 isnt super powerful and although there are SNES and MegaDrive emulators, the device will struggle with these higher end consoles. You can try overclocking the ESP in the options. I have also disabled sound in the MegaDrive emulator as this did help get it somewhat playable from my fairly limited testing.

Thats it, its all done. Now play on it for a bit, realise its a nice novelty, put in on a shelf or in a drawer and never play it again :)

The controls on the VMU were never gonna be amazing due to the size of the device, obviously Sega never built it to be a proper gaming device so the DPad is very small, the other buttons are just rubber so dont give great feedback however there are some things that can cause issues and things we can do to get the most out of it. The VMU is over 25 years old now and I'd imagine most were made almost 25 years ago also, so depending on how much use your one had over the years or how it was stored it could need a little freshen up.

For me the main thing I came across is the conductive pads, these need to be fairly clean else you will run into a lot of problems, unlike large devices with large conductive pads these are quite small so if they are dirty they wont make good contact with the pads on the PCB and can cause incorrect or missing inputs.

I find the best way to clean these is using paper as its just a little abrasive. As per the pic above, I usually rub one of the pads across the paper, not lots of force as you dont want to remove all the carbon, just a bit of the top layer. When I've done this I also clean with some IPA & swab to remove any residue. I found this massively helped to get the best out of the device.

I'll be honest, it probably still wont be perfect, I found the original VMU also missed some inputs and some devices may have had a lot more use than others leading to softer button presses and worn out conductive pads so I'm not taking all the blame but some of it could be the design as well :) Its probably something I notice more in the menu system as you tend to use light presses to navigate through the games list but when you play a game you tend to use firm longer presses and I dont get many issues.