Accurate Wiimote Light Gun for Raspberry PI

Normally, the Wii Remote used as a light gun isn't accurate enough for retro games like NES Duck Hunt, because the Wii Remote does not in fact select the point on the TV it's pointed at. It can't! The Wii Remote has an infrared camera in its front that sees the line of infrared LEDs in the sensor bar, but it can't know how far (or in what direction) the TV is from the bar or how big the TV is. Emulators and games work around this by showing cross-hairs or other targeting indicator, but that's not an accurate target-shooting experience.

To make the Wii Remote work as an accurate light gun that you can sight along to select a target on a TV requires four infrared LEDs arranged in a known quadrilateral pattern (not a straight line) in the same plane as the TV. The Wii Remote then sees the four LEDs and the camera image can be used to calculate a homography that allows us to figure out where the camera is pointing at.

The hardware for this project is simple. There are four infrared LEDs in simple 3D-printed housings that can be glued to the top and bottom of the TV housing and plug into a USB charger. Plus, in case you don't have Wii gun housing, I have a simple 3D printed handle and sights that you can attach to the Wii Remote (though to save plastic, I made mine a hybrid between wood and 3D printed plastic).

The python-based software was harder to make than the hardware and is currently Linux-only. It calibrates the LEDs and the Wii Remote and then uses homography calculations to emulate an absolute mouse that works quite well in Retroarch's fceumm NES emulator (and probably some other emulators) on my Raspberry PI 3B+.

• Wii Remote
• Four 940nm 5mm infrared LEDs
• Old USB cable with a working type A plug
• Raspberry PI 3 or other Linux computer with Bluetooth support
• 3D printer and filament (optional)

Get an old USB cable with a working type A male socket (usually my phone charging cables break on the micro USB end, so I have leftover cables with a working type A male socket).. Actually, it's even OK if the data cables are broken as long as the power lines work. Cut off the other end. In theory the red cable should be +5V and the black should be ground, but check it with a multimeter (plug it into a charger, and then check the voltage between the red and black wires).

Since infrared LEDs have around a 1.2-1.3V voltage drop, I just soldered four of them in series loop to the USB cable. Make sure the wires you solder are long enough that you can put LEDs at the bottom of the TV and two at the top, with a decent amount of horizontal space between LEDs (about 10 inches or so).

More precisely to make the LED loop:

• solder the minus side (cathode, shorter leg, with flat edge) of the first LED to the +5V USB wire
• join the plus side of the first LED (anode, longer leg, with round edge) to the minus side of the second LED
• repeat to join the second LED to the third and the third to the fourth
• then connect the plus side of the fourth LED with a wire to the ground USB wire.

To make things neater, you can use heat shrink tubing when you do the connections. Otherwise, use electrical tape to avoid shorts.

Make sure you have no short circuits. Then plug it into a USB charger and check that it's emitting infrared light by looking at the LEDs with a phone camera. (Many phone cameras are infrared sensitive.)

Now, attach two of the LEDs to the underside of the TV and two to the upper side. The horizontal spacing should be about ten inches. If it's too much, you may have problems with the Wii Remote camera's field of view capturing them all. But if they're too close, then my geometric intuition says you'll have lower precision.

For testing, I taped the LEDs with electrical tape, and then for a permanent connection, I designed and printed four neat little LED clips (files are here) which I hot glued to the TV. You should make the LEDs be as close to the plane of the TV display as you can, without the bezel obscuring them from the location where you will be shooting.

Currently the software is Linux-only. The following setup is designed for the Raspberry PI 3 with Raspbian Stretch. Other Linux systems will require some changes. On earlier models you'll need a Bluetooth dongle and you'll need to run this from a commandline as well:

sudo get-apt install bluetooth

Step A: udev

Next, create a file in /etc/udev/rules.d/wiimote.rules that contains the single line:

KERNEL=="uinput", MODE="0666"

You can do that, for instance, with a text editor or by typing the following on the commandline:

sudo sh -c 'echo KERNEL==\"uinput\", MODE=\"0666\" > /etc/udev/rules.d/wiimote.rules'

And then restart udev:

sudo /etc/init.d/udev restart

Step B: cwiid

Next, you'll need my modified cwiid package. Here it gets a bit hairy as ideally you'd need to build it on your Raspberry PI, but I have to confess that I've lost track of what packages you need to install to make it work. There are three options for doing this.

Option B1: Build yourself

cd ~
git clone https://github.com/arpruss/cwiid-1
autoconf
./configure
make -C libcwiid
sudo make -C libcwiid install
make -C python
sudo make -C python install

Unfortunately, there is a pretty good chance you're missing a bunch of stuff needed for building this, and ./configure will complain. You can look at all the things it complains about and run sudo apt install on all of them.

Option B2: Use my binaries

cd ~
wget https://github.com/arpruss/cwiid-1/releases/download/0.0.1/cwiid-rpi.tar.gz
tar zxvf cwiid-rpi.tar.gz
cd cwiid
sudo make install

Step C: python libraries

Finally, get support stuff for my lightgun python script:

sudo pip3 install uinput numpy pygame opencv-python
sudo apt-get install libatlas-base-dev
sudo apt-get install libjasper-dev
sudo apt-get install libqtgui4
sudo apt-get install python3-pyqt5

Step D: lightgun.py

Finally, get my lightgun python script:

cd ~
git clone https://github.com/arpruss/cwiid-lightgun.git

If all has gone well, you now have ~/lightgun.py which you can use to calibrate the lightgun.

There are two aspects to calibration. The first is to calibrate the center of the camera on each Wiimote. This requires using the camera to take two images of the LEDs around your TV screen, one with the remote right-side up and the other with it upside-down.

To avoid pressing the buttons when you lay the Wii Remote on its front, and in order to make the Wii Remote have consistent elevation, you can 3D print the calibration tool I included here. You basically need things that are are 10.5mm thick that you can put under the Wii Remote when it lies on its front. I actually used some scrap plywood to save on plastic.

Turn on your LEDs and make sure your Raspberry PI or other computer is displaying on the TV. Connect a keyboard (this won't work over ssh) or use VNC. Then run:

python3 ~/lightgun/lightgun.py -M

If all goes well, you will get a full-screen display asking you to press 1+2 on the Wii Remote. Do that. Lights will flash on the Wii Remote, and then lights 1 and 4 will stay on. You will also see a little green rectangle at the top of the screen, with the view from the Wii Remote camera. Point the Wii Remote at the LEDs and if all goes well, you will see the four LEDs, numbered 1 through 4.

Now you need to find a solid surface with a sharp edge, like a coffee table, that you can point at the TV screen and that can allow the Wii Remote to see all the LEDs with the Wii Remote aligned against the edge. Begin by aligning the Wii Remote right side up, with the Remote's side aligned against the surface edge, making sure all four LEDs are seen. Then press SPACE on your keyboard (or attach a Nunchuck and press C if that's more convenient). You will then be prompted to rotate the Wii Remote. Now, make sure it is elevated 10.5 mm up from your surface, using the calibration tool or something else, and in as close to the same location as before (e.g., aligned against the same edge of your surface). Press SPACE again.

If all goes well, you will now go to the LED calibration step. Yup, this is complicated! But you're going to have a very precise lightgun. That's just the price.

Note: If like me you have a Wii under the TV, the Wii needs to be turned off for two reasons: first, if the Wii is on, it will connect to the Wiimote and, second, the sensor bar's infrared LEDs will interfere with this project. For similar reasons, while you use the Wii it's a good idea to unplug the LEDs around the TV.

Now you need to tell the software where the LEDs are located around the TV's edge. You will see a calibration screen showing four arrows, one of them selected (bright) and three of them grayed out, around the edge of the TV. You use +/- to switch to change which arrow you are adjusting.

For each of the four arrows around the edge, do this:

1. press left/right on the Wiimote to move the arrows until they point as precisely as you can towards the corresponding LED;
2. press up/down on the Wiimote to change the arrow's length until the length of the arrow matches the distance between the LED and the edge of the TV display; in other words, the length of the arrow needs to be equal to the distance from the tip of the arrow to the LED.

Once your four arrows are correct (and maybe even earlier) you will see a red crosshair when you point the Wiimote at the screen. You can check that this is where it should be. (Remember that you need to be far enough away that the Wiimote can see all the LEDs. It's also important that there be no other sources of infrared in the field of view. I once had trouble because of sunlight reflecting off a screw head on the TV stand.)

Finally, there is a fifth arrow, that only shows up when you press + from the fourth LED arrow or - from the first (and it by default has zero length, so it's just a pixel). This arrow adjusts how far above the camera of the Wii Remote the shot will be registered. The issue is this: you will be sighting along the top surface of the Wii Remote. But the camera is actually located some distance below that surface, in the middle of the black rectangle in the front of the Wii Remote. If we registered the shots where the camera is pointing, they would be registered about 8 mm below the top surface of the Wii Remote. You can check this by noting that as you sight along the top surface, the center of the cross-hairs is hidden by the camera.

You can live with this, or you can grow this fifth arrow to software align the shots with the top of the Wii Remote, or you can adjust the 3D printable files for the iron sights to compensate for this (but the compensation will only work for one particular distance to the TV). I went for the software alignment myself.

Press HOME on the Wii Remote to exit calibration and save all data to the ~/.wiilightgun directory.

You probably want to try your light gun now. Just run in a terminal emulator (or a script):

python3 ~/lightgun/lightgun.py -t

You will need to press the 1+2 buttons at the same time, and after that if all goes well, as long as lightgun.py is running, the lightgun will emulate a two-button absolute mouse. The trigger button is mouse button 1, and the A button is mouse button 2. Press ctrl-c to exit.

You now need only configure your emulators and/or games to work with an absolute mouse. Unfortunately, that's not always going to be that easy.

One fun thing you might try is my mod of iminurnamez's duck-duck-shoot:

cd ~
git clone https://github.com/arpruss/duck-duck-shoot
cd duck-duck-shoot
python play_game.py

For NES games, I use the libretro fceumm core in Retroarch. Go to the Options menu, and configure the Zapper to be a touchscreen. (Configuring it as a mouse doesn't actually work, as fceumm expects a relative-movement rather than absolute-position mouse.)

If you start your games with a script, you can edit the part that starts the game or emulator to say:

python3 ~/lightgun/lightgun.py -t -B 30 "command to start game"

Then during the first 30 seconds of the game's execution (hence the -B 30 option), you can connect your lightgun by holding down 1+2.

By the way, the lightgun.py script can also be used for general Wii Remote gaming with Retroarch. Just add the -o option and the lightgun functions will be turned off, and instead the Wii Remote will work horizontally, with the three buttons being 1, 2 and B respectively. There are other Retroarch-related functions in lightgun.py's mappings which you will discover by reading the code. For instance, the minus key acts as a shift, and together with the dpad controls saving and loading (up/down = change save number; left=restore; right=save).

You can use the Wii Remote by itself as a gun, sighting along the top. You can also buy one of the commercial gun casings for it. But because the original Wii Remote wasn't capable of use as a sightable gun, the casings tend not to come with iron sights, and iron sights greatly improve accuracy.

I designed a simple three-part 3D-printable system: a slide-on handle that sits just behind the trigger (so it looks a bit like a Star Trek Original Series phaser), and slide-on sights. Printable files are here. If you want to save on plastic at the expense of scrap wood, you can also do what I did and instead of printing the whole handle, print just the part that holds the Wiimote, and cut out a wooden piece and screw it on.

To sight, focus your eyes on the sights. Align the front sight's bump between the rear sight's bumps so that the air space on either is equal and all three bumps stick out equally high. Then align the middle of the target with the top of the bumps.

If you want even more precision, you can run:

python3 ~/lightgun/lightgun.py -d

(for demo) and look carefully whether the sights align with the cross-hairs. If they don't, exit and manually edit ~/.wiilightgun/wiimotecalibration, and tweak the x and y coordinates of the camera center slightly to adjust sighting. For instance, my gun was shooting slightly to the right, so I ended up changing the x coordinate from 529 to 525. Everybody's numbers will probably be different.

The mouse emulation code works roughly as follows.

• Process button presses.
• Get data from camera and adjust for camera centering calibration.
• If fewer than three LEDs are visible in the camera:
  • Keep last mouse position.
• If three or four LEDs are visible:
  • Use the Wiimote accelerometer data to get the Wiimote orientation and identify which LED camera image corresponds to which physical LED.
  • If four LEDs are visible:
    • Calculate homography between LED camera images and LED locations (in screen coordinates).
    • Use homography to calculate what screen location corresponds to the center of the camera field of view.
    • Do Y-adjustment to adjust for the center of the virtual gun barrel below the sight-line. This is a somewhat kludgy algorithm but it works.
    • Set the mouse position to the adjusted screen location.
  • If three LEDs are visible:
    • Use OpenCV to solve the P3P problem between the LED camera images and the physical LED locations. This generates up to four solutions.
    • If successful:
      • If we have a previous successful location calculation, choose the solution that makes the missing LED be closest to the last observed or computed position of that LED.
      • If we do not have a previous successful location calculation, choose the solution that best predicts the accelerometer heading.
      • Use the best solution to calculate where the fourth LED should go.
      • Do the rest as in the four LED case.
    • If not successful:
      • Keep last mouse position.