Using Photogrammetry to Create Models

Who am I?

I am Samuel Conklin and I am a sophomore at E.L. Meyers High School. I have experimented with photogrammetry for the last couple of months and I hope to help you if you choose to recreate this product. I am interested in seeing how other people can use this new and ever-evolving technology.

What is Virtual Reality and SteamVR?

Virtual Reality is a new form of artistic expression and experience. It's still a new frontier of technology as countless companies and people work to improve the medium. You're doing yourself a great disservice if you've never tried VR, as it totally encapsulates its user's sense of sight and sound. One of the major platforms for Virtual Reality is Valve's SteamVR. SteamVR allows users to freely share environments and set pieces with each other at no cost. It is mostly open-sourced and allows a myriad of headsets. These include, but are not limited to the HTC Vive series, Oculus Rifts, Oculus Quest, Valve Index, Windows Mixed Reality, etc. SteamVR is the platform we're using to create our environment.

What is Photogrammetry and how can it be used?

Photogrammetry is the process of taking photos or videos and turning the images into a 3D model. The most famous example of the use of this technology is Google Earth. Google uses satellite images and street views to reconstruct landscapes and buildings. Looking at Google Earth, it is clear that they had few images to work from, so some buildings will look misshapen. When used correctly, photogrammetry is an extremely convincing way to create 3D models. This project uses this technology to preserve historical landmarks. Erosion and being exposed to the elements means that historical landmarks will eventually fall into disrepair and people will forget about them. Using photogrammetry, we're able to preserve everything digitally, so future generations will be able to experience and appreciate these structures. This also means that more people worldwide could experience other cultures without the cost or risk of traveling.

What supplies do I need?

For this project, I used Reality Capture, SteamVR, and Krita. There are Autodesk, Agisoft, and Zephyr programs that can be used instead of Reality Capture if you don't have the required Nvidia GPU to run it. Photoshop or other photo editing programs can be used instead of Krita, but since it's free and public to use, I chose to use it for this project. You will also need a Desktop VR headset that can plug directly into your computer. Finally, a high-quality camera is needed. A mobile camera is decent, but I used a Nikon D7000 since Reality Capture can automatically detect the aperture of the camera, which leads to higher quality models.

This is the most important part of this project as you need to get it correct in one shot. If you don't you will need to go back to the location and retake everything.

First, you will need to find a suitable location and structure. The structure should be large and easily identifiable. What this is means the photogrammetry software will be able to detect the differences between the structure and the background.

Typically, taking pictures on an overcast day is the best way to go. This is because a sunny day has the possibility to overexpose some photos and underexpose others. It could also create unwanted shadows. A model with little to no shadows will make lighting easier in the level development section. Additionally, the software needs the color balance to remain the same with every photo, as it uses colors to calculate the size of the object. Taking photos on an overcast day, however, will improve the quality of the structure. The clouds in the sky will act as a natural diffuser for the camera. It will distribute the light more evenly and means the model will become precise during the rendering process.

The final piece of advice is to take as many photos as possible. Most programs will pick and choose which photos it is going to use for the final product. By taking more photos, everything will be more exact and gives the program some breathing room. I had about 150-300 photos for each model.

Now, we're going to import the photos into the software. Note that Reality Capture requires an Nvidia graphics card. You cannot get past the Align Images section without it.

If you have a folder with all of the images you want to use, choose the Folder option in the top left corner. This will take you to File Manager and you can choose the folder. Inputs will allow you to choose individual photos with File Manager.

Next, we're going to start the process of building the model. At the top of the screen, under Workflow, there is a Process section. We want to press the larger Start button. This will start to align the images. The fourth photo briefly shows what is going on during this process. It will recognize the left eye of the deer and compare the distance with other photos. It will continue to compare different photos until it has gone through all the possible combinations. This step is complete if you can see small dots that resemble the structure.

The next step is to calculate the model. To do this, we click on the Calculate Model in the Process section. This will directly translate whatever it finds in the photos into 3D form. The results are decently close to what we want the model to look like, but there are three issues with this version of the model. First, there are holes in the model that are not in the real-life structure. Second, the surface is way too bumpy. Since the program is approximating its calculations, it will wrongly make some smooth surfaces bumpy. Third, there are too many triangles. Every model in Reality Capture is created with small triangles. Each triangle represents a small surface of the model. We want as least triangles as possible to optimize the model. If there are too many triangles, it can cause some performance issues.

These issues can be solved with the Simplify tool. All models will have 3,000,000 triangles after the Calculate Model stage, and we can choose to make that number higher or lower. For all models like the ones found in this project, I lowered the triangle count to 1,000,000. The result can be found in the second image, where all the bumpy surfaces are smoother and there are fewer holes. Additionally, with fewer triangles to render, performance will be better when we get into VR.

Now the model will need color and texture to make it even more real to life. These two parts are similar in what they achieve but are different in the way they work. Using Color, which is found in the Process section, the program will color in the model. The tool will take the pictures it used to create the model and wrap your images around the different surfaces of your model. Texture, on the other hand, will color in the simplified parts of the model. Looking at the first image, it is clearly visible how the program wrapped the images around the model, but in the second image after the Texture process, the sharp sides of the images are now smoothed out.

In Reality Capture, we are going to transfer the models to SteamVR. At the top of the screen, under Export, select Mesh, which is located next to the Process section. You should see this pop-up which allows you to change the settings and parameters of the model. It's best to keep all the settings set to default, except to change the texture format JPEG. The program will now export a file in the OBJ format, which is the mesh file, and a JPEG, which is the texture file for the model. For reference purposes, the JPEG should look like the second image in this step.

Now, we need to import our models into SteamVR. First, open SteamVR, then navigate to Create/Modify an Environment, which is under the Workshop tab. The SteamVR Workshop Tools will now open. Choose Create Empty Addon and click on Launch Tools.

The Asset Browser will now open with different materials and models already included. In the top left, there is an icon with a Hammer symbol. This is the built-in SteamVR Hammer editor, which gives you the ability to create maps. Maps are the environment the VR user will inhabit during gameplay.

With the Hammer editor open in SteamVR, we can now create a cube. We want to create a cube in order to begin building the environment. The program allows you to change the size of the cube before it is secured in place. Right now, only the outside surfaces of the cube are visible. By pressing CTRL+F the surfaces will all be inverted and will turn the cube into a room. Additionally, you will need to add an object called info_player_start, which will tell Hammer where to spawn the user upon loading the program.

To make the room more realistic, we also need to add textures. We can add some custom images through the Material Editor. This is the circular icon to the right of the Hammer button in the Asset Browser. Selecting the Material Editor will take you to a new window which will allow you to import an image to be made into a texture. You can use your own images or get images from stock texture websites such as textures.com. Click the File Button under the Color Selection to select the image you want to use. After selecting your image, make sure you save it by pressing the Floppy Disk icon in the top left of the Material Editor window. Now, close the Material Editor and navigate back to the Asset Browser. Here, search for the texture file you want to use and then click and drag it into the Hammer Editor. In this window, you can apply your texture to the different faces of the cube.

We also need to add lighting to the room. To do this, we put a light_omni or a light_spot on our map, which is found in the Entity Section to the left of the map view. A light_omni will shine a light in all directions while a light_spot shines light in one specific direction. For this project, I used at least two light_spots for each object and one light_omni to help light up the room.

Now, it's time to start moving our objects into our map. In the Asset Browser, to the right of the Material Editor icon is the Model Editor which resembles a humanoid icon. Upon opening the Model Editor, you will get the option to browse or to make a new VDML, which is the Hammer Editor's version of an OBJ file. Select the option to make a new VDML and it will require you to make a folder. This folder will store the VDML file. Make sure it's in the Add On folder and press continue.

This will take you to a screen with your model, which has a white texture. To add the texture for this model, select Materials from the toolbar on the left. This will take you to the Material Editor and you can select the JPEG from the Reality Capture export. Make sure to save the texture and the model by clicking the Floppy Disk icon. Return to Hammer Editor.

In Hammer, you can drag and drop the model from the Asset Browser to the map. The model will most likely be tiny and you will need to resize it. By pressing T on your keyboard, you can change its position on the map. Pressing E will change the proportions of the model while R will rotate it.

The Object Labels seen next to each landmark provide context and history to enhance the user's experience. For my Object Labels, I simply took photos of the informational plaques at each location to easily record important information I wanted to use later on. I then used Krita, a free photo editing software program, in order to create the Object Labels including additional images and text.

For the Miner Park and Anchor side of the room, there were no issues adding labels as textures. Simply add them to your Asset Browser through Material Editor and add the image to a block which can be created in the Hammer Editor. Problems arose when I created boards for the other side of the room. All the text and images were backward and there was no clear way to reverse the textures within the editor. To fix this, I went back into Krita and reversed the boards for the deer and the bear and thankfully this fixed the issue.

In addition, I added miscellaneous images of Wilkes-Barre, as I wanted to create a room that had the feel and ambiance of a boutique museum.

In conclusion, both Virtual Reality and Photogrammetry have the power to vastly improve countless industries, including but not limited to sports, real estate and forensics. Today, I used photogrammetry to preserve some small yet important parts of my hometown's local history. VR and Photogrammetry are both still experimental technologies that can sometimes challenge you. Despite some shortcomings, the amount of innovative uses for both processes is exciting and limitless. I thank you for reading about this project and I hope this inspired you to try something new with Virtual Reality.

The download link for the project can be found here.