Star Recognition Using Computer Vision (OpenCV)

This instructable will describe to you how to create a computer vision program to automatically identify star patterns in an image. The method uses the OpenCV (Open-Source Computer Vision) library to create a set of trained HAAR cascades that can be used to recognise specific star patterns. Although this guide is in the context of star pattern recognition, the OpenCV process I describe can be applied to other applications too - so it will hopefully be useful!

The project is summarised in this video:

Why did I write this instructable?

1. The star pattern identification method I am developing I believe has potential to be applied to a wide range of amateur astronomy projects - whether that's telescope orientation, automatic image classification, or even ultimately a star sensor on an open source or amateur CubeSat.
2. There are lots of good OpenCV instructables here, but even so I found it a very difficult process to learn initially, so I hope that this guide will be a good reference for other people looking to train HAAR classifiers for OpenCV (not necessarily to do with astronomy maybe!).
3. I'm not a trained programmer myself, so this project really pushed my understanding. Hopefully by writing this Instructable other, more experienced, makers will be inspired to work on this concept and contribute to the GitHub and this instructable via comments on this page.
4. Amateur astronomy and orientation methods are a big interest of mine, see my previous instructable featuring an Arduino Star-Finder for Telescopes.

The cover photo of this Instructable is of a concept 3U CubeSat design that I participated in the design of. I used it to illustrate this instructable as the original application of the computer vision star recognition system was to be for an orientation sensor for amateur-made CubeSats, using a Raspberry Pi V2 Camera. There are many other potential applications of computer vision star recognition I believe, but I think this is the coolest!

A Small Glossary:

Learning about computer vision is made slower by the stupid amount of specialist terms used, so I'll define some for us here:

Cascade - A classifier trained to identify a specific target object.

Fiducial Marker - A marker adding a point of visual reference to an image.

HAAR - Haar-like features are a type of image feature used for classifier training.

OpenCV - Open Source Computer Vision, a library of computer vision tools.

Stellarium - Open Source astronomy software.

OpenCV is a Linux based library, so although it is supposedly possible to operate it well on Windows, you will have a much easier time running it in a Linux environment (take this from me and a lot of days trying to get it fully working on Windows!). As an experiment, I downloaded and ran OpenCV on my Raspberry Pi 3B+, which was successful, although classifier training is a very RAM intensive process, so if you wish to do it at any speed, the recommended route is to hire a Linux Virtual Server (which can be surprisingly inexpensive actually) for a few days/weeks/months and use that as a dedicated environment in which to run the classifier training. You will be able to control the server from a Windows PC using an SSH client such as Putty. Once the cascades are trained using the VPS, they can be downloaded to your Windows PC, and Python can be used to run the image recognition program in a Windows environment.

Linux Virtual Server:

• A Linux Virtual Server (VPS) is needed to perform the HAAR cascade training processes. Initially I hired a server with 8GB of RAM and Ubuntu 16.04.6 (LTS) x64, and later a second one to double the rate at which I could train cascades, although you'll only need a minimum of one.

Software:

• Stellarium - this is virtual planetarium/astronomy software, freely available. It will be used to gather simulated star images for use in testing.
• Putty - This is an SSH client that is used to control the VPS via command line.
• WinSCP - this is used to perform file transfer from the Windows PC.

There is a small setup process to get the VPS up and running. The first time it may take a little while for you, but it's not too tricky if you follow the steps closely. This tutorial was a great reference for me, I'd recommend you read this too whilst working through this instructable. It covers the specifics of the linux commands line by line, which is necessary to follow to the letter.

Roughly, the process involves:

1. Creation of Linux server with correct Ubuntu version.
2. Upgrading and Updating of the server.
3. Creation of a workspace directory, into which OpenCV is installed.
4. Installation of some essentials, namely a compiler, various libraries, and Python bindings.

After this stage, you're ready to begin preparing for the training process.

The whole process of computer vision using HAAR cascades is quite confusing at first, so this Step describes the logic in a little more detail:

Basic Process

1. A negative image dataset exists, consisting of several thousand images that don't contain the object of interest. This will need to be uploaded to the VPS.
2. A single positive image is created that contains the object of interest. This will also need to be uploaded to the VPS.
3. The single positive image is distorted, warped, rotated, etc., by a set of chosen parameters and overlaid upon a selection of the negative images. This is an artificial way of creating a large positive dataset out of a single image. (For other real-world applications, such as identifying a cat, you could simply use several thousand images of cats, but this method is not always appropriate if you don't have such a large set of positive images. The artificial approach used here will be less effective, but it is the only option for a use-case such as this).
4. A training process is run, that works in stages. Each stage will train a cascade to identify different HAAR-type features within the imagesets. Each stage takes exponentially longer to complete, and the effectiveness of the classifier increases each time (it is also possible to over-train just so you know!).
5. A single trained cascade will be able to look for a single target object. If you wish to identify multiple unique objects you will need a trained cascade for each. In this case, I trained about 50 different cascades for unique star patters, to create a set that could cover the northern celestial hemisphere.
6. Lastly, a detection program is used that runs each cascade of a set against an input image. The cascade will look for its given target object within the input image.
7. If successful, the target object will be identified within the input image.

Negatives

A really key aspect of cascade training is to have as large a dataset of negative images as possible. We're talking thousands, ideally tens of thousands of images. It doesn't really matter what they contain, the aim is just to provide variety of visual information. The Classifier Training folder contains a variety of different negative image datasets that I compiled. Initially these were comprised solely of simulated star field images gleaned from Stellarium, but I later augmented the dataset with as many randomised images as I could find (yes, including my holiday photos ...). The largest dataset there includes almost 9000 images, which was the largest I've created so far. By using this it'll save you compiling your own.

Positives

The positive image (that is the target star pattern that the cascade will be trained to recognise) begins as a screenshot of a star pattern in Stellarium. A python program then identifies the brightest stars in the image, and overlays markers (explained later in this instructable) onto these star positions. This image is then shrunk to 50x50 pixels. This is small, but the training time required for the cascades will increase exponentially as this size increases, and so this is a good compromise between quality and time.

The Stellarium Scripts folder of the GitHub repository contains three programs I wrote to control the use of Stellarium. To use them, place them in the scripts folder of your Stellarium install folder. To run them, you can open the scripts window from within the Stellarium menu, or just by double clicking on the program in the scripts folder, which will launch Stellarium and immediately run the selected program.

thesis_4 and thesis_5 capture about 2000 images each of the northern and southern, respectively, celestial hemispheres. These were used to form databases of negative images, to train the positive image against. The distinction between north and south was a simple way to ensure that the target (positive) star pattern would not be present in the negative dataset by training northern hemisphere star patterns against the southern celestial hemisphere image dataset and vice versa. (If a positive image is also present within the negative image dataset it will affect the quality of the classifier).

thesis_setup is also useful - this sets up Stellarium to be appropriate for capturing images - the images used to simulate a view from Space. It does actions such as hiding menus, gridlines, labels etc automatically to save you needing to each time you want to grab an image.

The first cascades I trained were unable to correctly identify any star patterns. They were very unreliable and were very prone to false positives. My assumption was that effectively the star field images from Stellarium (basically just white dots on a black background) just did not contain sufficient visual information to contain enough HAAR-type features for successful classifier training. I think it was late at night, but I decided to try the idea of writing a program to automatically place a small thumbnail image over the location of each bright star in a star field image.

Elton

This was a silly test, but by adding a small picture of Elton John's face to each bright star location, training the classifier against this positive image, and then running the cascades against the original image, it was much much more effective at correctly finding the right pattern. I knew I was onto something!

Although the 'Eltons' proved the theory, I needed a marker that had full rotational symmetry, so that the star pattern would appear the same no matter at what orientation it was presented. I tested a range of marker types, and found that the type at bottom right was most effective, with the contrasting black and white rings. The python program presented in the positive folder of the GitHub repo shows how the brightest stars in a given image are identified, and these markers automatically overlaid in those positions. We have now created a representation of the key star patterns that can be trained against.

When you have trained a set of cascades, you need to know how to use them to identify an object in an image!

Look at the Star Identification folder of the GitHub, where you will find the cascade_test19.py program. This catchily-named program takes a set of cascades from a given folder, and runs them all against an input image, and reports on the detections made. The 'detectMultiScale' function is the core of this, and it takes a variety of arguments that define the detection process. Altering these is critical to the performance of the cascade classifier, and more discussion of this can be found in the following step, where we look at how to eliminate false positives.

This could be applied in a satellite orientation system by correlating the pixel value in the centre of the bounding box to the Ra/Dec celestial coordinate of the identified star patter, and then correlating this to the angular displacement from the centre of the image (camera axis). From this, using an understanding of the lens distortion (approximated to a gnomonic projection), the angle of the satellite can be found from just two positive identifications.

These two images show the results of testing the cascade set against an identical image, but with different parameters. Clearly, the first image contains the true identification, but also an enormous number of false positives, whereas the second image contains only the correct identification.

The cascade_test19.py program within the Star Identification folder of the GitHub repo uses two methods to sort the results. Firstly, the detectMultiScale functon sets a Miminum and Maximum size of result that can be found, which is sensible, as the approximate size of the target star pattern within the window (for the given lens and magnification - my simulated Stellarium images use the properties of the Raspberry Pi V2 Camera) is known. Secondly, the code will select the result with the largest bounding box (within the previous limits). In testing, this was found to be the true positive. Thirdly, the program sets a minimum 'levelWeights' (effectively 'confidence value') that is required to treat this ID as a true positive. By this method, the cascades were effective at finding the correct result.

As well as the star field images, I also tested this against pictures of my desk for instance, training cascades to identify my notebook, mug etc., to practice eliminating false positives. The above methods worked well in all circumstances which was encouraging.

Areas for Improvement

This has been a complex project for me, and really pushed my understanding of the topic. It has involved a total of several months nearly full-time work to get the project to this point when I can share it with you, but there is plenty of more work to be done to improve the performance of the method. As it stands, it can function well within certain constraints. I have worked to identify which areas are needing additional work, and will hopefully be able to spend time addressing these in the months to come. They are:

Angle - This is a complex area, the idea that the results of the classifiers must be rotationally invariant, i.e., it should reliability identify the target star pattern regardless of the angle at which it is presented the image containing the target star patter. A cascade trained using an input image at a single orientation will not be able to identify that image at randomised orientations, so variance of the positive image angle must be introduced into the training process to train cascades that can accept a range on input angles. The parameter 'maxzangle' in the cascade training commands takes an argument in radians, that controls a limit of the angle that the input positive image will be overlaid on the negative images provided, so the resulting positive image set will contain a range of orientations of the positive image. However, as this maxzangle increases, the acceptance ratio (broadly speaking, quality) of the cascade will reduce sharply. I believe the solution is to train cascades using a significantly greater database of negative images than what I was using to ensure that a good quality cascade classifier can be created even incorporating a large orientation spread.

Another potential solution would be to train a number of cascades for a specific target, each cascade governing a certain portion of a full 360 degree rotation. That way the quality of each cascade can be maintained at a high level, but on the other hand this will result in far more cascades, and hence the identification process will be slower.

The parameter 'levelWeight', which is a value provided by the 'detectMultiScale' function, is analogous to a confidence value in the detection that has been made. Studying this, the above graph was created, which shows how the confidence of positive identification decreases sharply as the orientation of the image increases in either direction, confirming the thoughts that this is a weak-point.

Pixel Placement - A much simpler, but also problematic point is that of pixel placement, illustrated by the following two images, showing an enlarged view of a star image, so that the individual pixels of two stars can be clearly seen. The erosion process used in the program to scrub all except the brightest stars from the image will keep the first star, and discard the second, despite them being of equal brightnesses. The reason for this is that the first star is centred on a pixel, whereas the second is not as such. The erosion function strips concentric rings of pixels from around the central pixel of a group, and so the first star will have the central pixel survive the erosion function, but the second star will be completely removed from the image. Therefore fiducial markers will only be placed on the first star, and not the second. This will cause inconsistencies relating to which bright stars in a given star field will receive markers (and hence be compared to the trained classifiers) - hence it is possible that a correct positive observation will not be possible.

Thank you for reading my instructable, I hope that you found this project intriguing. It has been a very interesting process working on it, it's been over a year since I began working on the concept, and I'm encouraged by the results to this point. From the literature I read, this is a pretty original concept, and with more development can certainly be applied in a range of applications for amateur astronomy or more.

This project was a steep learning curve for me, and so I hope that some readers with more programming experience may be inspired to contribute to the continuation of the project through the GitHub page, and we can continue to develop this open-source tool. I look forward to reading any comments you may have, but please don't ask too many difficult questions!