Reefshape: A System for the Efficient Collection and Automated Processing of Time-Series Underwater Photogrammetry Data for Benthic Habitat Monitoring

ReefShape is a protocol for the collection and automated processing of geo-referenced time series underwater photogrammetry data sets benthic habitat monitoring. Photogrammetry, or large area imaging, is a tool that has become very popular among coral reef researchers for creating 3D digital models and high resolution 2D orthorectified photo mosaics of sections of the benthos that can later be analyzed on a computer to measure things like community composition or structural complexity. Collecting this kind of data and time series for coral reef monitoring is incredibly useful in the context of the rapid global decline of coral ecosystems. Underwater photogrammetry as a concept is fairly simple. It involves taking hundreds to thousands of highly overlapping images of an area that are input into structure from motion photogrammetry software to recreate the orientations of the photos in the reef scape in 3D. A mesh model digital elevation model and 2D top-down orthomosaic can then be generated. While many protocols for collecting photogrammetry data for reef monitoring exist, they often focus on image collection and leave processing options up to individual researchers, which can be very time consuming and has steep learning curve. Automation of this process is a challenge because the models must be properly scaled, located, and precisely aligned to other time points to be particularly useful for ecological data analysis. We have developed ReefShape as a way to address these issues. Our method provides a solution for establishing permanent monitoring plots with real world geo referencing data, efficiently collecting time series imagery and processing data using a set of custom Python scripts that fully automate and dramatically simplify the processing pipeline. With an Agisoft Metashape Pro, the industry standard photogrammetry software. Our scripts yield scaled and geo-referenced time series data that are exported in the proper formats for common analysis workflows and GIS software and tag lab, a purpose-built application for rapid annotation of coral reef orthomosaics. We rely on the use of four permanent auto detectable ground control markers that are fixed to the substrate when a plot is first established. The depths and GPS locations of these markers are collected at the first time point with a Bluetooth GPS and smartphone with a custom survey that is freely available and emails pre formatted location data to the user. For data collection, temporary scale bars are placed within the plot and imaging is done by a diver or snorkeler with a single camera and wide angle lens. Subsequent time points require only relocating and cleaning the permanent corner markers, placing scale bars, and collecting images. On the computer, our custom ReefShape scripts facilitate the automatic processing and time series alignment of the imagery data, saving the researcher significant time and effort that can be invested instead into ecological data analysis. The process begins with equipment preparation, assembling a GPS kickboard kit, creating scale bars, and making a text file with their precise lengths, procuring corner markers, assembling a camera system, and configuring the correct settings, and setting up the Metashape software and installing the ReefShape scripts onto the processing computer. In the field, a suitable plot must first be chosen. As with any underwater research protocol, safety should be prioritized throughout this process. This protocol can be executed by a single researcher or buddy pair and can be adapted for any plot size between about 25 square meters and upwards of a thousand square meters. But we recommend a base plot size of 10 by 10 meters. Once a plot is chosen, the researcher installs the four corner markers while on scuba using a sledgehammer and masonry nails, taking care not to fracture the substrate or damage any living corals. Ideal locations are areas of relatively flat non-living substrate that are easily visible from directly above and not likely to be damaged or quickly bio eroded such as very porous coral skeletons. Install marker one in the northeast corner, marker two, in the southeast, three, in the southwest, and four in the northwest, for consistency and ease of relocation, using a compass and tape measure if needed. Using a depth gauge, record the depths of each marker on a dive slate. If permitting or necessity prevents the use of permanent markers, temporary ones can be placed and retrieved after photo and GPS data collection. After the corner markers are laid out, place three to five scale bars throughout the plot and use a dive weight or small rock to secure each one and prevent movement. Like any photogrammetry protocol, image acquisition is the key step. This protocol can function with most any underwater camera with a wide angle lens capable of taking photos at a rate of one photo per second, but we specifically recommend a camera system with a mirrorless camera and wide angle rectilinear lens with a field of view of around 90 to a hundred degrees or a full frame equivalent focal length of 18 to 22 millimeters, paired with an underwater housing and a dome port that is well matched to the lens. The key goal is maintaining sharp images. And therefore, we recommend using manual mode with an aperture of F8.0, a shutter speed of 1/500th of a second, and automatic ISO, to achieve correct exposure for every frame. In darker and deeper conditions, an aperture of F5.6 and a shutter speed of 1/320th of a second can be used to increase the amount of light and reduce image noise levels. White balance should be set to custom. An interval timer of one second should be used and an auto-focus mode that focuses at the beginning of image collection and remains constant throughout the photography process should be used. The camera should be set to record JPEG and RAW images simultaneously. Navigate to the plots median depth and set a custom white balance on the camera using a gray card or one of the scale bar ends. Beginning at one corner, position the camera pointed down 1.5 to two meters above the substrate. Auto-focus the camera on the reef and begin collecting photos at one frame per second, swimming toward an adjacent corner. In the graphic, the blue rectangles represent the photo positions for this plot and the red highlighted rectangle demonstrates the swimming pattern. Collect photos covering the whole plot in at least a half meter buffer around the perimeter in a series of anti-parallel passes, spaced about one meter apart, consistently, one and a half to two meters above the reef. When you complete this first set of passes, turn 90 degrees and collect a similar set of passes, completing a grid pattern. Photos should be downward facing except in high relief areas where the camera should be tilted slightly obliquely to point perpendicular to the substrate surface. After photography, clean up the plot, leaving only the installed corner markers. Back at the surface, retrieve the GPS kit and swim over the plot using the ReefShape survey in the ArcGIS Survey123 to collect GPS locations above each of the four corner markers. Back on the boat, input the depth information corresponding to each marker and submit. For subsequent time points, first relocate the plot and find the corner markers using the original GPS data or a printout of the original time point photo mosaic, as a reference if needed. Using a plastic card, scrape off any biotic growth from the marker surfaces. Replace any lost or damaged markers with the proper target number and make note of it. Place scale bars, set the white balance, and collect images as before. For processing the first time point of a plot, choose the full ReefShape workflow from the ReefShape custom menu bar. Name the project as the plot name. Name the chunk with the image collection date. Import your photos and save the project. Next, in the general panel, set the coordinate system to WGS 84 plus EGM 96. Select project settings are presented in this dialogue box for the user to modify if needed. The defaults should be appropriate for most situations. Select the folder for data exports and tick the boxes for desired outputs, either for standard GIS software or tag lab. In the geo referencing panel, select yes for using markers and locate your scale bar file and the geo referencing file that was emailed to you. Finally, click okay to begin the photogrammetry process. The custom pipeline will auto save after each step, allowing the user to rerun it from any point in the process without losing progress. Upon completion, the script will export all requested data products, a processing report, and a Shape file of the automatically generated region of interest bounded by the corner markers. Individual functions useful to this process are included as standalone tools. in the ReefShape menu, in particular, a function is included to automatically calculate the 3D surface area to planar area ratio, a common metric to study structural complexity on reefs. Each time point of a plot will be stored as a new chunk in the same Metashape project. For subsequent time points, open the plots Metashape project and run the aligned time points script from the ReefShape menu. Click create chunk and name it with the date of image collection. Import the photos and then select the original time point as the reference chunk and the new one as the active chunk. If any corner markers were replaced under water, note this in the dropdown box. Click okay. This will detect markers and import the geo referencing for the corner markers from the original time point. Inspect the markers and geo referencing in the reference panel. If any markers were not detected, place them manually on at least three images and then rerun the aligned timepoint script to reimport the reference data properly. Next, run the full ReefShape workflow script. Only the settings in the general panel need to be adjusted, especially which data products to export and where to save them. The geo referencing and project setup panels can be left blank. Clicking okay will complete the photogrammetry process and export time series aligned data products. For each time point, it's important to inspect the alignment and data products for accuracy. If any issues arise, manual intervention may be needed. Specific data pieces such as the orthomosaic DEM 3D model or tie points can be deleted in the full ReefShape workflow script rerun to regenerate them properly. Once the user is satisfied, they can bring the exported data products into GIS software or tag lab for various analysis pipelines. These visuals demonstrate representative results. Time .1 one included 1,299 photos and time .2 included 1,974. All photos were properly aligned for each time point, and the resulting data have holes or obvious image quality issues. The data appears in the correct real world location and the geo referencing error for the initial time point is about 30 centimeters. The depth error is 2.3 centimeters. The scaling error is 0.14 millimeters and the reprojection error is 1.12 pixels. The error between time points is about one millimeter. The plot covers 208 square meters at 0.5 millimeter resolution. The entire process for each time point was completed in about eight hours on a low end computer, and in less than two hours, on a high end gaming computer. This protocol was designed to address some of the main challenges in underwater photogrammetry. In particular, through the use of detectable permanent corner markers and scale bars. Coupled with real world geo referencing information, the entire photogrammetry process within Metashape becomes automated, saving significant time and effort. Our ReefShape scripts include improvements to the standard photo alignment procedure that increase the number of aligned photos in difficult conditions and help facilitate precise time series alignment between data sets. They're also designed to increase efficiency in the photogrammetry processing pipeline to avoid the need to use very expensive computers for processing. Our overarching goal is to remove barriers to make underwater photogrammetry more accessible and functional for researchers looking to extract important ecological data to inform coral reef conservation efforts.