No-Stitch Panorama Picture

Hugely wide panorama picture kept fascinating me since I was a kid. Photo stitching software seem to have done a lot of progress these days. However, I was intimidated by the process of shooting the pictures (slightly overlapping, horizontally aligned) and doing the manual stitching adjustments, etc. and never made the step⁽¹⁾.

An alternate method, based on capturing a rotating vertical scanline, and then letting some software do all the job, has been in my head since about two years.

Having received a whole set of Fischertechnik ⁽²⁾ components helped me decide to experiment it, just for the sake of verifying the idea.

Then, although the whole is still a work in progress, the results are so good that I wanted to share them with you. It has room for many improvements to increase resolution and usability, but it basically works pretty well.

Read on, make one, and show your results!

_{(1) In short, I have much more interest in spending time engineering something (that can later be effortlessly reused), rather than tediously executing manual work.
(2) Fischertechnik is to mechanic what breadboard is to electronic --Well it's more than just that, but you get the idea.}

The principle
A movie consists of a succession of pictures shot rapidly (30 per second). We will only consider the vertical central line (1 pixel thick), that we'll call the scanline. When slowly rotating the camera, each frame will be like the last one, but very slightly shifted. So the scanline will sweep the whole scene.

Being in the center of the frame, the scanline will have no lateral barrel distortion. The scanline will also be very well horizontally aligned (relative to the rotation axis).

In order to collect more vertical pixels (i.e. to have a taller scanline, hence a higher vertical resolution), the camera is rotated by 90°. This rotation will have to be taken into account during the post-processing.

In summary, there are two phases:

1. Capture: The first part consists of shooting the movie in slow and continuous rotation. It only needs a camera able to shoot video clips, and a motorized rotating mount.
2. Post-processing: The second part needs a computer program. It consists of extracting the central column of pixels (corresponding to the scanline) from each movie frame, and stitching them together to form the panorama image.

Enough theory, now let's test this principle

So if you want to experiment too, you must be (for now) familiar with computer command-line tools:

• the installation of command-line tools,
• the operation of command-line tools,
• the edition of script files, to tweak some parameters.

I will only provide instructions for experienced such users, and not how to get acquainted to the command line.
Tested on Linux (Ububtu) and Mac OSX so far.

But, upon popular request, I may write a user-friendly program that anybody can user.

1. What rotation speed and precision is required, and can it be reached with toy motors?
2. Is the quality and resolution of a video clip sufficient?
3. What kind of camera is required?
4. What computer power is necessary?
5. Can it be achieved with open-source software?
6. What is the relationship between the final image resolution and the lens aperture, the movie frame-per-second, and the rotation speed?
7. What is the effect of exposure adjusts during the recording?
8. How will moving objects appear?
9. What kind of perspective distortions will appear?

Answers in the last step. So please read on...

There is nothing complex with this mount:

• Red plate: the fixed part, four feet (for use on a table), a fixture for a tripod, a battery holder.
• A spirit level
• The motor with gears, rotating the turret
• The rotating turret, consisting of a big gear and a chair for the camera

The only tricky part is to obtain a very slow rotation (approx one round in two minutes or more).

The depicted setup gives one round in one minute:

Later, I added more gears. Now one round takes 3 minutes, but it is less reliable. The definitive version will use a servo modified for continuous rotation.

As said, this is still very experimental, and meant for power users who know how to operate the command-line and edit text files.

For the other more casual users, wait for a user-friendly version of the program, that I may write soon.

If you want to try the experimental way, read on. You need four things:

1. FFMPEG

• On Mac OSX: first install XCode (from your "Mac OSX Install DVD"), then svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg, then read the INSTALL file (./configure && make && sudo make install).
• On Debian/Ubuntu Linux: sudo apt-get install ffmpeg do the same as for Mac OSX, because you need a more recent version than what apt-get provides.
• On Windows: follow the "FFmpeg Windows Builds" on http://www.ffmpeg.org/download.html . Be sure to add the path of the binaries directory to your PATH environment variable.
• Then, on any platform, when done, check from a shell (aka terminal on OSX, aka CMD or DOS window on Windows):
  type: ffmpeg -version
  and it should say something like:
  FFmpeg version SVN-r26402, Copyright (c) 2000-2011 the FFmpeg developers
  built on Apr 22 2011 09:07:41 with gcc 4.2.1 (Apple Inc. build 5664)
  etc.

2. ImageMagick

• On Debian/Ubuntu Linux: sudo apt-get install imagemagick.
• On Mac OSX, and Windows: follow the corresponding page under "Binary Releases".
• Then, on Windows, be sure to add the path of the binaries directory to your PATH environment variable.
• Then, on Mac OSX, edit /home/Laxap/.profile and add:
  export MAGICK_HOME=/Users/Laxap/Downloads/ImageMagick-6.6.7
  export PATH="$MAGICK_HOME/bin:$PATH"
  export DYLD_LIBRARY_PATH="$MAGICK_HOME/lib/"
  Replacing Laxap by your username. Also replace .../Downloads/... by the location where you decompressed the ImageMagick archive.
• On any platform, when done, check from a shell:
  type: convert -version
  and it should say something like:
  Version: ImageMagick 6.5.7-8 2010-12-02 Q16 http://www.imagemagick.org
  Copyright: Copyright (C) 1999-2009 ImageMagick Studio LLC
  etc.

3. Python

• On Linux (Debian and Ubuntu) and Mac OSX, Python comes already installed because many system utilities use it, so there is nothing to do.
• On Windows, install it, then be sure to add the path of the binaries directory to your PATH environment variable.
• On any platform, when done, check from a shell:
  type: python --version
  and it should say something like:
  Python 2.6.1

4. My script:

• Download the script panoramize.py attached to this step, and save/move it to a wanted location.
• On Linux and Mac OSX, make it executable: chmod +x panoramize.py

Find a good place.

Install tripod, attach mount on it.

Tune tripod to be lever.

Set camera on movie mode. Place camera on the rotating mount. Check every mechanical parts. Test rotation.

Turn motor on. Press camera trigger.

Wait for one complete rotation, plus some margin. Stay behind camera.

Stop camera. Stop motor.

Inspect movie for proper sharpness, exposure, and scene scrolling.

(Like in the past, it is thrilling to come back home, work a bit, and see the results. Or you could take a good netbook on the go.)

Transfer your movies to the computer.

Open a shell and type:
./panoramize.py IMG_12345.AVI

You can specify more than one file as command argument.

You may have to add "python" in front:
python panoramize.py IMG_12345.AVI

To adjust parameters, edit the panoramize.py file, and change:

• ROTATE
  0: landscape, 90: portrait. Acc. to camera orientation.
• RATIO
  e.g. "80%", acc. to lens aperture and rotation speed
• MIRROR
  True or False. Acc. to camera and rotation direction

ROTATE and MIRROR only depend on the construction of your mount.

RATIO depends on the rotation speed, but also on the view angle (i.e. the focal distance) of the zoom value that you are using. If you use the same default zoom factor, you won't need to change the value once tuned.

Landscape panorama just look like... panoramas!

• First image: A collage of three panorama images. See a bigger version. In reality these images are approx 4,000 pixel wide!

Indoor panoramas give a very unusual view of familiar places.

• Second image: The object being close, the perspective distortion effect is very strong.
• Third image: Cropping and keeping only the middle half will eliminate the strongest distortion areas and give a more natural aspect.

The only problems I could see are the vertical dark/light bands due to readjustment of the exposure.

When trying to have people on the picture: it's hard to be really still. Moving will yield strange or funny effects. Try it during parties!

Answers to the questions posed in Step 2

1. What rotation speed and precision is required, and can it be reached with toy motors?

• Speed strongly depends on the video resolution, FPS, and focal distance. Approx one round in 2-3 minutes for 640 pixels, 30 fps, f 2.8.
• Toy motors are OK but require considerable gear ratio to achieve this low rotation speed. This will result in noisy mechanics, but with the advantage of a considerable torque with inexpensive motors. The gears should work reliably, else distortions will be generated. Servo modified for continuous rotation may give the best results in all respects.
• With battery or accumulator power, the rotation speed might not be extremely precise, but sufficiently stable.

2. Is the quality and resolution of a video clip sufficient?

• Yes it is largely sufficient as long as you target web resolutions. For poster prints, the resolution will not be sufficient.

3. What kind of camera is required?

• Cheap cameras are sufficient. I used a Canon Digital IXUS 95 IS. In fact it is quite ideal due to its compactness.

4. What computer power is necessary?

• With a 2.4 GHz Intel dual core, a 4423 x 512 panorama takes 3 minutes to compute. This can be shortened by a better Python script.

5. Can it be achieved with open-source software?

• Yes, in fact very easily, at very high quality. ffmpeg is blazing fast at converting MJPEG into indifidual JPEGs, and imagemagick is ideal for cropping/stitching. Python is great at orchestrating the whole (but for this many other languages would fit). They have ports for many major platforms.

6. What is the relationship between the final image resolution and the lens aperture, the movie frame-per-second, and the rotation speed?

• What matters is that the final pixels represent a view angle that is the same in both vertical and horizontal direction.
• In vertical direction, i.e. along the scanline, the view angle is determined by the F-number of the zoom value, and the movie pixel resolution.
• In horizontal direction, the view angle is determined by the movie FPS, and the mount rotation speed.

7. What is the effect of exposure adjusts during the recording?

• This really depends on your camera, and how abruptly/smoothly the adjust is performed. But it generates vertical bands of different exposures. This is a bit problematic. Can some cameras be told not to readjust the exposure during a movie? Maybe some cheap/old ones?

8. How will moving objects appear?

• As long as they do not cross the scanline, moving objects will not be seen (unless they are near and affect the overall scene luminosity).
• Fast moving object will appear as a single column of pixels.
• An object moving around the camera in sync with its rotation will appear as an horizontal band.
• Objects moving "at reasonable speed" in the same direction as the camera rotation will appear thicker, and in the opposite direction, thinner.

9. What kind of perspective distortions will appear?

• Vertical objects will have no distortion at all, because they correspond to the scanline, which is at the center of each frame, having no lateral distortion.
• Horizontal objects will have fish-eye-like distortion. Horizontal parallel straight lines will appear as hyperbolic lines, meeting at a vanishing point on the horizon line.
• In life, although we constantly see them, we no longer notice the distortions because (1) when checking the straightness of big objects we tend to move our eyes to focus along straight lines and (2) our brains compensates and removes the distortions.

Conclusion

This little experiment has proved to be a more than viable solution. It is easy to obtain good results. It can be combined to pole camera mount, for aerial panorama, or anything else you can imagine. The strong perspective distortion can either be used as a wanted effect, or removed by cropping to keep the central 50% of the image height.

There are numerous possibilities for building a good mount, including laser cutting of plywood or acrylic.

Instead of a camera, use the lens of an old reflex camera and the sensor of a scanner. You should get insane resolutions! Rotation must be very very slow.

Thanks for reading!

Next steps

• Slower rotation, usage of a servo modified for continuous rotation.
• Usage of a camera with higher resolution.
• More compact and robust mount.
• Microcomputer (e.g. Arduino) to ease operations: return to given position, motor and movie start/stop sequence; remote control for a one-button operation.
• Guessing the ideal rotation speed, given the resolution, FPS and focal distance.
• A user-friendly program with GUI, to compute the final image and adjust parameters (on top of the python script) -- see 2nd picture of this step.

Acknowledgments

I must warmly thank Udo, my wife's Uncle, who donated us more than 20 Kg of Fishertechnik. As a teacher, he was using it at school; now he's retired and we're the happy recipients of this awesome technology!