Beginners Guide to an Optical Flow Sensor With OV7670 Camera

This Project focuses on building a cheap Optical Flow sensor using the OV7670 VGA camera Module in combination with a Teensy 4.0 micro-controller. The system basically calculates the pixel displacement between subsequent frames

Ultimately this system either outputs Pixel displacement, or Approximated real world displacement. We'll go into it deeper a little later.

This sensor package is highly ideal to enable localization, homing-actions in cheap robotic-platforms, especially in GPS-denied scenarios.

My previous instructable covers the basics of interfacing the OV7670 camera module with a Teensy 4.0, going through that should bring the readers up to speed on the basics of the camera module, still we'll look into some of it here ;)

-> The Teensy 4.0, although may work with Arduino variants at lower speeds

-> OV7670 VGA (non-FIFO camera module)

-> VL53L0x TOF Li-DAR module

-> A custom PCB or Circuit-Board is recommended over breadboard connections for this setup. The camera's external clock signal generated from the Teensy is 13MHz, High frequency signal transmission through Breadboards generally leads to signal bleeding. Something like the one in the image shown above should do. If still using breadboard, you could experiment with different frequencies (Probably within 6-17MHz) and find out which one suits you the best !

-> Li-ion cell x 2 and a 5V buck converter (for a wireless package)

-> NRF24L01 x 2 (for a wireless package)

-> any compatible Arduino, wired with the receiving NRF24L01 module, hooked to a computer to receive the data wireless ( again, for a wireless package :D )

-> Python image visualizer (GitHub link included)

-> Processing coordinates visualizer (GitHub link included)

In the image shown, I created circuit to also contain connectors for an MPU6050. It could be used for higher accuracy, but is beyond the scope of this instructable, hence will not be discussed here. The Camera's SCCB protocol is similar to I2C protocol, and requires pull-up resistors. But when connecting the VL53L0x sensor (which has Pull-ups onboard the breakout board) to the same I2C line as the camera, we need not use external pull-ups.

OV7670 to Teensy 4.0:

connect the power pins (3.3V and GND)

VSYNC_PIN 4

HREF_PIN 5

PCLK_PIN 6

XCLK_PIN 3 (try to keep the tracks as short as possible)

D7_PIN 20

D6_PIN 23

D5_PIN 22

D4_PIN 16

D3_PIN 17

D2_PIN 15

D1_PIN 14

D0_PIN 0

RESET 21

PWDN GND

SCL 19

SDA 18

VL53L0x to Teensy 4.0

connect power pins (Vin-5V, GND)

SCL 19

SDA 18

NRF24L01 to Teensy 4.0 (only for wireless)

connect power pins (3.3V, GND)

CE 9

CSN 10

SCK 13

MISO 12

MOSI 11

Connect the Li-Ion cells to the buck converter and power the Teensy 4.0 (only for wireless). If connecting USB to the Teensy on battery, then break the bridge beneath the Teensy board, given to cut power jumper from USB. This ensures that power doesn't flow back to the Computer USB port from the Buck converter. This happened to me and my laptop blacked-out for a few hours.

For power-control, I used a simple male jumper as a power switch, to break or make the GND connection from the batteries in the circuit board.

The OV7670 VGA camera is a cheap camera sensor, that is pretty much straight forward to deal with. According to the spec-sheet, it outputs a max resolution of 640x480 (VGA standard), and supposedly maxes out at 30 FPS, though my tests with the module yielded more than 48 FPS continuous (comment out the VL530x's distance function and estimateMotionSSD function to view camera-only FPS).

The module need only be supplied with a 3.3V, GND, external clock signal (5MHz - 48MHz), and a few dozen parameters (through it's SCCB interface) for it to start sending out pixel data, which only requires reading after.

The module has two other signal pins for telling us when to read, HREF and PCLK. HREF goes high when a Horizontal Line of Pixel data is going to be given out (eg, pixels 0-159 or 160-319 in a 160x120 image), and PCLK goes high when a valid Byte of pixel data is available in the data pins.

The module has 8 data pins, D0-D7, representing 1 Byte of data. In this setup we'll be using RBG565 format of the module, as it offered a good balance between speed and details in my tests. In this format, each pixel offers 2 Bytes of data. So for each pixel, we need to read 2 bytes of data from the 1-Byte data-pins, over 2 consequent PCLK cycles.

The Basic SCCB parameters are already set in the software, further details can be approached in the module's documentation.

Optical flow basically refers to the various techniques used to compute motion from images. These techniques vary among many flavors. While some of them are very computationally intense, like the Lucas-Kanade method, or the Farneback method, some are very much achievable in the Domain of micro-controllers, like Block-Matching with Sum of Squared Differences (SSD) or Sum of Absolute Difference (SAD) or simple Edge Detection and tracking.

This sensor uses Block-Matching with SSD, i.e., Sum of Squared Differences. It considers a central block from the latest image, with candidate blocks of known offsets from the previous image. By changing the offsets systematically in every cycle, this loop searches over a certain window of the previous image, eventually returning the displacement of the block that matches the current block the closest (where sum of squared differences between respective pixels is the smallest). The nuances are explained pretty well in the code with comments.

So, now we have the pixel displacement values dx and dy, Next we can move onto extracting real world coordinates from the obtained values. this is where we need the sensor's distance from the subject in frame. The camera captures images at a constant resolution, for simplicity let's assume 50x50 pixels (like here). Here the FOV horizontally and Vertically is the same, and is derived experimentally, which you may need to do, considering inaccuracies between different modules.

From the above setup, you can find the FOV of the camera, Given, height 'h' is in centimeters, the output will be in cm. Now essentially we need only compute a factor which when multiplied with the pixel displacement value will output real world displacement in cm. The factor is,

F = (2 * tan(θ/2) * h) / pixel width

Displacement = (dx * F) cm

Here, I found θ/2 to be 0.10567 rad. This measurement is crucial in yielding accurate values and its advised that proper care be taken when measuring. The height 'h' is taken from the distance measured by the VL53L0x sensor. the library Adafruit_VL53L0X.h is used. The Library's continuous measurement function computes and updates the measurement asynchronously enabling resources to be vastly shifted to more real-time computation. In the main program, the functions dealing with it's setup and loop logic is small and simple enough to understand.

This section discusses, the performance this sensor package and it's known/observed issues with their possible sources.

The Teensy's relatively modest computational capabilities were initially a concern, but it performed significantly better than expected, achieving a precision of at-most ±1 cm during testing and general using. While the system generally behaves predictably and consistently, it does show systematic errors under certain conditions.

One notable observation is that, although the sensor remains precise (i.e., low random noise), its accuracy is influenced by the velocity of the sensor. In particular, higher velocities introduce drift, likely due to limitations in how quickly the sensor can calculate motion, and maybe lower velocities too, likely due to lack of legible details and pixels merging, originating from the camera's output resolution.

This drift can be mitigated in two main ways:

Modeling the error based on velocity and applying compensations to the output.

Operating the sensor within an optimal velocity range—where its performance is stable and consistent.

This limitation stems from the nature of the sensor's block-matching algorithm (using SSD), which is affected by:

the camera’s frame rate,

the time required to compute SSD,

the minimum and maximum velocities the system can resolve,

and the search radius, which defines the maximum displacement detectable between frames.

A larger search radius improves motion detection range but increases computation time, which further limits the maximum velocity that can be tracked accurately. Initially, scaling-down the image to a more manageable resolution instead of cropping for the SSD function was considered, but was then rejected as it reduce the sensitivity of the Output. The current code-flow and logic was one among 5 slightly different code-layouts, chosen as it was the most performant and stable. This one fires at 41.9 FPS with the VL53L0x and the Algorithms running, and around 48.5 FPS with the camera alone running, methods included in the code. I tried making the camera faster by increasing the X_CLK but the algorithms and VL53L0x measuring seems to be a bottle-neck.

Practically speaking the sensor is precise enough to yield very much usable results with very minimal error rates. As seen in the video, the sensor is able to air-write, mapping it's position accurately enough. Although there is room for improvement (help is always welcome both here and on GitHub), this sensor package and the methods used, would serve best as a cheap and accurate positioning system for small robots, UAV's and many other robotic platforms, and most importantly, I'm looking forward to the creations the readers of this instructable may showcase using this sensor package! Happy making ;)

OptiFlo GitHub Repository