Name: High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques (Video) | JoVE Journal
Uploaded: 2013-12-03T13:10:00.000+00:00
Duration: 11 min 34 s
Description: Learn digital fringe projection techniques for high-resolution 3D video imaging using a high-speed binary defocusing system. This method captures over 300,000 data points per frame for dynamic surface measurements.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

playerTrialDesktop

This research was an accumulated effort that began more than 10 years ago when Dr. Zhang was a graduate student at Stony Brook University. The current and previous students in our team at Iowa State University have contributed tremendously toward advancing this technology to where it is today. This work was partially sponsored by National Science Foundation under project number CMMI 1150711, and the William and Virginia Binger Foundation.

1. System Configuration
A schematic of the system is shown in Figure 1.
<ol>
	<li>Generate the fringe patterns for projection. These can be prepared well in advance by using an image programming environment such as MATLAB, OpenCV, or QT. Construct the patterns according to the three-step phase-shifting algorithm in18. Make three images, shifted in phase offset from each other by 2&#960;/3. For binary defocusing, use a dithering technique to generate sinusoidal patterns using only black and white pixels as described in19.</li>
	<li>Select the digital light processing projector. A high-speed binary defocusing system requires a faster, specialized projector such as the DLP LightCommander with ALP High Speed module. Be sure to use the binary or monochromatic setting to project the fringe images. Since the image values are purely on or off, nonlinearity adjustments are not necessary. Utilize the projector&#8217;s software program to upload the patterns for phase shifting.</li>
	<li>Select the camera. Choose a black-and-white CCD or CMOS camera with the correct capture rate for the system. Avoid color cameras for 3D capture since color is not needed and color cameras require nonlinearity and gamma adjustments. Keep in mind that the camera will need to capture the entire set of fringe images for each 3D video frame. High-quality sinusoidal systems require precise synchronization between the projector and the camera; for binary defocusing systems this requirement is more relaxed20.</li>
	<li>Determine the desired maximum (x, y) range and the distance from the projector to the object (d0). Choose a range that makes sense for the application, but be sure that the area is slightly larger than the subject to reduce any optical boundary effects. Since the light output of a projector is a frustum, this (x, y) range will drive d0. Move the projector relative to a large flat projection surface until the desired (x, y) range is found, then measure d0 with a tape measure.</li>
	<li>Select the camera lens with the proper focal length. Using the camera&#8217;s sensor size, find the focal length such that the field of view at the distance d0 is the same as the desired imaging range (x, y).</li>
	<li>Determine the separation distance between the projector and the camera. A trade-off occurs here between noise and shadowing. At a large angle between these components, triangulation between feature points is more obvious, but more features get lost in shadow from the camera&#8217;s perspective. At a small angle, triangulation becomes difficult, increasing noise in the results. Typically, 10-15&#176; is a good compromise.</li>
</ol>
2. System Calibration
This reference plane calibration is the simplest and most readily accessible method of calibration for the system. Therefore, it is the best for getting started. More accurate calibration methods are available in the literature for specific sinusoidal16 and binary defocusing17 applications. For maximum accuracy, calibration should be performed just before data capture. After calibration, the camera and projector should not be displaced relative to each other.
<ol>
	<li>Focus/defocus the projector. Carefully defocus the projection lens until the patterns at the imaging plane resemble high-quality sinusoids. This may require an iterative process of examining the data quality (Section 4) and adjusting the lens.</li>
	<li>Capture fringe images of a reference plane. Place a flat white board at the focal plane of the projector and in the field of view of the camera. A 3/16 in&#160;(5 mm) thick white foam core board works well, provided that the surface facing the system is not shiny or significantly blemished or torn. Record and save fringe images of this board for the data processing step.</li>
	<li>Capture fringe images of a reference object of known dimensions. For this step, a rigid foam cube is one simple example. Cover the cube with squares of 1/16 in&#160;(1.5 mm) white adhesive foam to make it diffuse. Place it at the focal plane of the camera in the camera&#8217;s field of view and record fringe images for the processing step.</li>
</ol>
3. Data Acquisition
<ol>
	<li>Place the object or invite the subject to sit at the focal plane of the camera. For a human subject, warn him or her that the projector light may be bright. A black cloth backdrop can be used behind the subject to hide extraneous surroundings.</li>
	<li>Adjust the camera aperture to optimize the light level. Sample fringe images should be as bright as possible, but not saturated. Dark images will have too much noise, while saturated areas (significant regions of maximum brightness) in images will result in the loss of details in the saturated region.</li>
	<li>Capture the desired number of frames. Be sure to bring a hard drive large enough to hold all of the captured images for both the subject and the calibration datasets. With the .OBJ file format, a 3D video recorded at 30 Hz for 1 min at a resolution of 640 x 480 could be over 50 GB.</li>
</ol>
4. Data Analysis and Visualization
With software optimized for speed such as our in-house GUI, this step can take place during data capture. Real-time processing allows the user to immediately detect if the resulting data is desirable for the application and adjust if necessary. However, post processing can be more flexible and higher in accuracy. Post processing is also much simpler to implement and the best place to begin.
<ol>
	<li>Compute the wrapped phase for both the calibration and subject data. In the three-step phase-shifting algorithm in18, phase describes the position of a point within the cosine function. Since we have three equations and three unknowns, we can solve the equations used to generate the fringe images in step 1.1 for the phase at each point. Because of the arctangent function, the computed phase is in the range (-&#960;, &#960;]; hence it is called &#8220;wrapped phase.&#8221; To improve the processing speed, we developed a fast phase wrapping algorithm that is discussed in21.</li>
	<li>Unwrap the phase maps. Adopt a phase-unwrapping algorithm that detects the 2&#960; jumps in the phase and removes them by adding or subtracting multiples of 2&#960;. We have used the fast algorithm in22 in previous systems to unwrap the phase robustly yet quickly. In the video, we demonstrate the multifrequency technique described in15, which uses additional sets of three phase-shifted patterns at differing frequencies. The wrapped phase maps from each set of three can be combined to robustly yield a single unwrapped phase map. This technique increases the depth range for accurate capture with binary defocusing.</li>
	<li>Optional: Compute the 2D texture. Averaging sets of three neighboring fringe images will wash out the fringe stripes and generate the 2D texture map. This can be mapped onto the 3D data during visualization if desired.</li>
	<li>Convert the unwrapped phase maps into depth. As described in23, depth can be computed for the calibration cube as the difference between the calibration cube phase map and the reference plane phase map. Compare this computed depth to the known depth to compute the correct depth scaling factor c0. Then, compute the depth for the subject by subtracting the reference plane phase from the subject&#8217;s phase and multiplying the results by c0.</li>
	<li>Compute the x- and y-coordinates. Apply the scaling factor c0 to the calibration cube depth map. Determine the conversion factor &#961; from the cube dimensions in pixels to the known cube dimensions in the xy plane. Multiply the pixel count in the subject data by &#961; to compute the x and y coordinates.</li>
	<li>Visualize the data. Individual frames can be saved in our in-house format and viewed with simple MATLAB code or saved in .OBJ format and viewed with a commercial 3D modeling program such as Blender. Due to the large amount of data in each frame, these applications may be sluggish on some computers. For more responsiveness or for live video display, write software using a computer graphics library such as OpenGL or Direct3D. This software can take advantage of the graphics processing unit (GPU) to rapidly generate the x, y, and z coordinates from the unwrapped phase, form a triangle mesh, compute lighting normals, and display the results. Using the GPU, we have achieved up to 30 Hz live 3D data visualization with approximately 300,000 points per frame.</li>
</ol>

<ol><li>Salvi, J., Fernandez, S., Pribanic, T., Llado, X. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+state+of+the+art+in+structured+light+patterns+for+surface+profilometry%5BTitle%5D)+AND+%22Patt.+Recogn%22%5BJournal%5D)">A state of the art in structured light patterns for surface profilometry.</a> Patt. Recogn. 43, 2666-2680 (2010).</li>
<li>Gorthi, S. S., Rastogi, P. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Fringe+projection+techniques%3A+Whither+we+are%5BTitle%5D)+AND+%22Opt.+Laser+Eng%22%5BJournal%5D)">Fringe projection techniques: Whither we are.</a> Opt. Laser Eng. 48 (2), 133-140 (2010).</li>
<li>Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Recent+progresses+on+real-time+3-D+shape+measurement+using+digital+fringe+projection+techniques%5BTitle%5D)+AND+%22Opt.+Laser+Eng%22%5BJournal%5D)">Recent progresses on real-time 3-D shape measurement using digital fringe projection techniques.</a> Opt. Laser Eng. 48 (2), 149-158 (2010).</li>
<li>Su, X., Zhang, Q. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Dynamic+3-D+shape+measurement+method%3A+A+review%5BTitle%5D)+AND+%22Opt.+Laser+Eng%22%5BJournal%5D)">Dynamic 3-D shape measurement method: A review.</a> Opt. Laser Eng. 48 (2), 191-204 (2010).</li>
<li>Geng, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Structured-light+3D+surface+imaging%3A+a+tutorial%5BTitle%5D)+AND+%22Adv.+Opt.+Photonics%22%5BJournal%5D)">Structured-light 3D surface imaging: a tutorial.</a> Adv. Opt. Photonics. 3 (2), 128-160 (2011).</li>
<li>Zhang, S., Yau, S. -T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((High-resolution%2C+real-time+3-D+absolute+coordinate+measurement+based+on+a+phase-shifting+method%5BTitle%5D)+AND+%22Opt.+Express%22%5BJournal%5D)">High-resolution, real-time 3-D absolute coordinate measurement based on a phase-shifting method.</a> Opt. Express. 14 (7), 2644-2649 (2006).</li>
<li>Zhang, S., Royer, D., Yau, S. -T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((GPU-assisted+high-resolution%2C+real-time+3-D+shape+measurement%5BTitle%5D)+AND+%22Opt.+Express%22%5BJournal%5D)">GPU-assisted high-resolution, real-time 3-D shape measurement.</a> Opt. Express. 14 (20), 9120-9129 (2006).</li>
<li>Lei, S., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Flexible+3-D+shape+measurement+using+projector+defocusing%5BTitle%5D)+AND+%22Opt.+Lett%22%5BJournal%5D)">Flexible 3-D shape measurement using projector defocusing.</a> Opt. Lett. 34 (20), 3080-3082 (2009).</li>
<li>Mehta, R. P., Zhang, S., Hadlock, T. A. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Novel+3-D+video+for+quantification+of+facial+movement%5BTitle%5D)+AND+%22Otolaryngol.+Head+Neck+Surg%22%5BJournal%5D)">Novel 3-D video for quantification of facial movement.</a> Otolaryngol. Head Neck Surg. 138 (4), 468-472 (2008).</li>
<li>Wang, Y., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((High+resolution+acquisition%2C+learning+and+transfer+of+dynamic+3D+facial+expressions%5BTitle%5D)+AND+%22Comput.+Graph.+Forum%22%5BJournal%5D)">High resolution acquisition, learning and transfer of dynamic 3D facial expressions.</a> Comput. Graph. Forum. 23 (3), (2004).</li>
<li>Laughner, J. I., Zhang, S., Li, H., Efimov, I. R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Mapping+cardiac+surface+mechanics+with+structured+light+imaging%5BTitle%5D)+AND+%22Am.+J.+Physiol.+Heart+Circ.+Physiol%22%5BJournal%5D)">Mapping cardiac surface mechanics with structured light imaging.</a> Am. J. Physiol. Heart Circ. Physiol. , in press (2012).</li>
<li>Ekstrand, L., Wang, Y., Karpinsky, N., Zhang, S. Superfast 3D profilometry with digital fringe projection and phase-shifting techniques. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aEkstrand%2C+L+%22Handbook+of+3-D+machine+vision%3A+Optical+metrology+and+imaging%22">Handbook of 3-D machine vision: Optical metrology and imaging</a>. , Taylor & Francis. Boca Raton, FL. (2012).</li>
<li>Zhang, S., Huang, P. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((High-resolution+real-time+three-dimensional+shape+measurement%5BTitle%5D)+AND+%22Opt.+Eng%22%5BJournal%5D)">High-resolution real-time three-dimensional shape measurement.</a> Opt. Eng. 45 (12), 123-601 (2006).</li>
<li>Li, Y., Zhao, C., Qian, Y., Wang, H., Jin, H. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((High-speed+and+dense+three-dimensional+surface+acquisition+using+defocused+binary+patterns+for+spatially+isolated+objects%5BTitle%5D)+AND+%22Opt.+Express%22%5BJournal%5D)">High-speed and dense three-dimensional surface acquisition using defocused binary patterns for spatially isolated objects.</a> Opt. Express. 18, 21-635 (2010).</li>
<li>Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Superfast+multifrequency+phase-shifting+technique+with+optimal+pulse+width+modulation%5BTitle%5D)+AND+%22Opt.+Express%22%5BJournal%5D)">Superfast multifrequency phase-shifting technique with optimal pulse width modulation.</a> Opt. Express. 19 (6), 5143-5148 (2011).</li>
<li>Zhang, S., Huang, P. S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Novel+method+for+structured+light+system+calibration%5BTitle%5D)+AND+%22Opt.+Eng%22%5BJournal%5D)">Novel method for structured light system calibration.</a> Opt. Eng. 45 (8), 083601(2006).</li>
<li>Merner, L., Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Accurate+calibration+for+3D+shape+measurement+system+using+a+binary+defocusing+technique%5BTitle%5D)+AND+%22Opt.+Laser+Eng%22%5BJournal%5D)">Accurate calibration for 3D shape measurement system using a binary defocusing technique.</a> Opt. Laser Eng. 51 (5), 514-519 (2013).</li>
<li><a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=%22Optical+Shop+Testing%22">Optical Shop Testing</a>. Malacara, D. , 3rd, John Wiley and Sons. New York. (2007).</li>
<li>Wang, Y., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Three-dimensional+shape+measurement+with+binary+dithered+patterns%5BTitle%5D)+AND+%22Appl.+Opt%22%5BJournal%5D)">Three-dimensional shape measurement with binary dithered patterns.</a> Appl. Opt. 51 (27), 6631-6636 (2012).</li>
<li>Ekstrand, L., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Autoexposure+for+three-dimensional+shape+measurement+with+a+digital-light-processing+projector%5BTitle%5D)+AND+%22Opt.+Eng%22%5BJournal%5D)">Autoexposure for three-dimensional shape measurement with a digital-light-processing projector.</a> Opt. Eng. 50 (12), 123-603 (2011).</li>
<li>Huang, P. S., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Fast+three-step+phase-shifting+algorithm%5BTitle%5D)+AND+%22Appl.+Opt%22%5BJournal%5D)">Fast three-step phase-shifting algorithm.</a> Appl. Opt. 45 (21), 5086-5091 (2006).</li>
<li>Zhang, S., Li, X., Yau, S. -T. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Multilevel+quality-guided+phase+unwrapping+algorithm+for+real-time+3-D+shape+reconstruction%5BTitle%5D)+AND+%22Appl.+Opt%22%5BJournal%5D)">Multilevel quality-guided phase unwrapping algorithm for real-time 3-D shape reconstruction.</a> Appl. Opt. 46 (1), 50-57 (2007).</li>
<li>Xu, Y., Ekstrand, L., Dai, J., Zhang, S. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Phase+error+compensation+for+3-D+shape+measurement+with+projector+defocusing%5BTitle%5D)+AND+%22Appl.+Opt%22%5BJournal%5D)">Phase error compensation for 3-D shape measurement with projector defocusing.</a> Appl. Opt. 50 (18), 2572-2581 (2011).</li>
</ol>

The authors have no competing financial interests.

This high-resolution, real-time to superfast 3D video imaging technology is a platform technology that could potentially benefit numerous and diverse scientific fields ranging from biological science to engineering practice. Biomedical applications include precision measurements of facial movements and organ surfaces.&#160;Other applications include 3D automated quality control with detection of warped surface features; 3D enhanced videoconferencing; detailed digitization of facial features for movies and videogames; dense and rapid deformation measurements for the design and analysis of structures; and fluid surface characterization.&#160;Many biological and engineering applications (e.g. beating rabbit hearts, fluid shockwaves) require the superfast imaging rates of a binary defocusing system to correctly resolve features without aliasing artifacts.&#160;
Nevertheless, many challenges remain to the widespread adoption of this technology.&#160;Conventional DFP technology requires the projector to display 8-bit grayscale sinusoidal fringe patterns.&#160;The speed of this technique is limited by the projector&#8217;s refresh rate (typically 120 Hz).&#160;This speed is sufficient for slow motion capture such as that in facial expressions.&#160;However, numerous applications exist that require faster capture rates.
Binary defocusing technology has relaxed this speed limitation, and we have successfully created a superfast 3D video imaging system.&#160;However, this system has two drawbacks.&#160;First, it requires an expensive projector such as the DLP Discovery platform and a costly high-speed video camera such as the Vision Research Phantom v9.1.&#160;Second, since it generates sinusoidal patterns via the defocusing of squared binary patterns, the binary defocusing technique has difficulty generating sinusoidal fringes of the same quality as the traditional DFP technique and a reduced depth measurement range (for further explanation, see23).&#160;Recent investigation indicates that dithered binary sinusoidal patterns can significantly alleviate the limitations on depth measurement range19.&#160;Future research will focus on overcoming the remaining issues while preserving the merits of binary defocusing.&#160;
Another challenge is compressing and storing the large amount of data generated by high-speed, high-resolution 3D video imaging systems.&#160;Uncompressed 3D videos are drastically larger than uncompressed 2D videos.&#160;For instance, for a 3D video recorded at 30 Hz for 1 min at a resolution of 640 x 480, the .OBJ file size could be over 50 GB, making it extremely difficult to store.&#160;Since little progress has been made in the 3D video compression field, we will continue to focus on this in the future.

Digital fringe projection (DFP) techniques are based upon correlation and triangulation between two views of the same scene at different angles, the same principle employed by the human eyes and brain to achieve stereo vision. However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras1. The projector rapidly projects a known sinusoidal pattern onto the object that the object&#8217;s surface distorts in the camera&#8217;s view. Three such distorted patterns (fringe images) at differing phase shifts from each other can be analyzed to retrieve the depth via triangulation. The use of a known pattern eliminates the difficult computational problem of identifying correspondence points, allowing the capture of depth measurements at the camera resolution. For example, with a 576 x 576 camera, the technique can capture 331,776 points. This allows DFP systems to measure very fine details such as the movement of facial muscles in human emotions.
3D optical imaging techniques for static or quasi-static events have been extensively studied over the past few decades and have seen great success in video game design, animation, movies, music videos, virtual reality, telesurgery, and many engineering disciplines5. Though numerous 3D profilometry techniques exist, they can be classified into two categories: surface contact methods and surface noncontact methods. Both the coordinate measurement machine (CMM) and the atomic force microscope (AFM) require contact with the measuring surface to obtain 3D profiles at high accuracy. This requirement places severe restrictions on the speed of contact methods. They cannot reach kHz measurement speed with thousands of points per scan.
Surface noncontact techniques typically utilize optical triangulation methods (e.g.&#160;stereo vision, spacetime stereo, structured light). By actively projecting known patterns onto the objects, structured light techniques can be used to measure surfaces without strong local texture variations1. Fringe analysis is a special group of structured light techniques that uses sinusoidal structured patterns (also known as fringe patterns). Because these patterns have intensities that vary continuously from point to point in a known manner, they boost the structured light techniques from projector-pixel resolution to camera-pixel resolution12. In the recent past, fringe analysis techniques were instrumental in achieving high-resolution 3D imaging.
The digital fringe projection (DFP) technique uses digital video projectors to generate sinusoidal fringe patterns. This technique has the merits of lower cost, higher speed, and simplicity of development, and it has been a very active research area within the past decade. Recent developments in DFP and similar digital structured light techniques are summarized in1-5. To achieve high-speed applications, a digital-light-processing (DLP) projector is preferable due to its fundamental operation mechanism. The speed and flexibility of this technique has allowed us to acquire 3D video at 40 Hz&#160;13 and then later at 60 Hz&#160;6,7.
Nevertheless, a fundamental speed limit exists for the traditional DFP technique. A DLP projector can only swap 8-bit color images at its maximum refresh rate (typically 120 Hz). Since the traditional fringe patterns are 8-bit grayscale images, we can encode three of them into one color image as the red, green, and blue color channels. The projector will swap each channel (and therefore each fringe pattern) at three times the refresh rate (typically 360 Hz). However, since each 3D video frame requires three fringe patterns, the maximum rate of 3D video capture is still only the refresh rate (120 Hz)3,14. To break past this hardware limitation, we have invented a modified DFP technique that uses binary defocusing8. Instead of 8-bit grayscale fringe patterns, this technique uses computer-generated 1-bit binary structured patterns. These patterns are defocused using the projector lens to become pseudo-sinusoidal patterns for DFP. Because DLP projectors can display binary images orders-of-magnitude faster than 8-bit grayscale images, the binary defocusing technology permits tens of kilohertz&#160;3D video imaging speed with the same resolution as the conventional DFP techniques15.
The overall goal of the following protocol is to demonstrate the basic implementation and operation of a binary defocusing three-step phase-shifting DFP system. First, the protocol will cover the selection and integration of the necessary components. Then, it will discuss the simplest, most readily accessible method of calibration for the system; more complex calibration methods are available in the literature for specific applications16,17. The protocol will then focus on the procedure for 3D video capture with the system and the process for converting the fringe images into visualized 3D measurements. Finally, we will present some representative results from our real-time and high-speed systems.

<table><tbody><tr><td colspan="4">&lt;p class=&quot;jove_content&quot;&gt;No materials or equipment used.&lt;/p&gt;</td></tr></tbody></table>

high-resolution, high-speed, three-dimensional video imaging with digital fringe projection techniques

High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces.&#160;Like the human eyes and brain, DFP uses triangulation between matching points in two views of the same scene at different angles to compute depth.&#160;However, unlike a stereo-based method, DFP uses a digital video projector to replace one of the cameras1. The projector rapidly projects a known sinusoidal pattern onto the subject, and the surface of the subject distorts these patterns in the camera&#8217;s field of view. Three distorted patterns (fringe images) from the camera can be used to compute the depth using triangulation.
Unlike other 3D measurement methods, DFP techniques lead to systems that tend to be faster, lower in equipment cost, more flexible, and easier to develop.&#160;DFP systems can also achieve the same measurement resolution as the camera.&#160;For this reason, DFP and other digital structured light techniques have recently been the focus of intense research (as summarized in1-5). Taking advantage of DFP, the graphics processing unit, and optimized algorithms, we have developed a system capable of 30&#160;Hz 3D video data acquisition, reconstruction, and display for over 300,000 measurement points per frame6,7.&#160;Binary defocusing DFP methods can achieve even greater speeds8.
Diverse applications can benefit from DFP techniques. Our collaborators have used our systems for facial function analysis9, facial animation10, cardiac mechanics studies11, and fluid surface measurements, but many other potential applications exist.&#160;This video will teach the fundamentals of DFP techniques and illustrate the design and operation of a binary defocusing DFP system.

Figure 1 shows the schematic of the system. The high-speed binary defocusing system in this video consists of a Logic PD DLP LightCommander projector and a Phantom v9.1 CMOS camera.
Figure 2 presents a single frame from our 3D real-time system of a human face. This system uses a 640 x 480 camera. Thanks to the aforementioned known sinusoidal pattern, we can capture 640 x 480 = 307,200 measurements, enough resolution to record very fine details.
Figure 3 shows an example of measuring human facial expressions in 3D at 60 Hz. Here, four frames selected from a video sequence clearly demonstrate the capability of the real-time system to capture dynamic changes in finely detailed geometry.
Figure 4 demonstrates our live visualization software used in conjunction with our real-time binary defocusing 3D video system. The 3D captured video of the subject is displayed in real time on the computer monitor to his right. This software was written in C++ using the OpenGL library, GLSL, and QT. The computer used is a Lenovo laptop.
Figure 5 shows 3D frames from live rabbit heart measurement with our newly developed superfast binary defocusing system. This system can record 3D frames at 667 Hz with an image resolution of 576 x 576. A superfast rate is required to measure the heart surface without motion-induced artifacts. The heart measurement research is in collaboration with Prof. Igor Efimov at Washington University-St. Louis (see11 for further details); note that the rabbit was humanely killed and that the images were taken while the heart was still beating.
<img alt="Structured light imaging diagram with projection equations; projector-camera setup analyzes patterns." fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig1highres.jpg" src="/files/ftp_upload/50421/50421fig1.jpg" data-altupdatedat="1787630408000" data-alt="Figure 1"> 
Figure 1. Layout of the 3D video imaging system. In this system, a high-speed DLP projector projects three binary dithered phase-shifted images in rapid succession onto the subject. A high-speed CMOS camera is used to capture the three fringe images one by one for computation of the depth.
<img alt="3D facial reconstruction process; mesh visualization; digital modeling technique; anthropometric study." fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig2highres.jpg" src="/files/ftp_upload/50421/50421fig2.jpg" data-altupdatedat="1787630408000" data-alt="Figure 2"> 
Figure 2. 3D measurements of a human face at a resolution of 640 x 480, revealing fine details. Left to right shows the simultaneously captured texture perfectly aligned with the geometry, a shaded view of the geometry, the wireframe view depicting the density of the points, a close-up view of the nose area, and a close-up view of the eye region.
<img alt="3D facial expression morphing, digital transformation, visual analysis, sequence diagram." fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig3highres.jpg" src="/files/ftp_upload/50421/50421fig3.jpg" data-altupdatedat="1787630408000" data-alt="Figure 3"> 
Figure 3. Four selected frames from 3D video of the formation of a facial expression. The video was captured at 60 Hz with a resolution of 640 x 480. These frames highlight the geometric changes in the woman’s face as she moves from a neutral expression to a smile.
<img alt="Facial recognition setup with optical scanning and 3D imaging experiment displaying real-time results." fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig4highres.jpg" src="/files/ftp_upload/50421/50421fig4.jpg" data-altupdatedat="1787630408000" data-alt="Figure 4"> 
Figure 4. Live 3D video capturing, processing, and rendering. The 3D measurements are displayed in real time on the computer screen to the subject’s right.
<img alt="3D heart models; diagram showing different rotational views for medical imaging analysis." fo:content-width="5in" fo:src="/files/ftp_upload/50421/50421fig5highres.jpg" src="/files/ftp_upload/50421/50421fig5.jpg" data-altupdatedat="1787630408000" data-alt="Figure 5"> 
Figure 5. Capturing a live rabbit heart with our superfast 3D video imaging system. The heart is beating at approximately 200 beats/min. The 3D capture rate was 166 Hz with an image resolution of 576 x 576. See11 for further details.

This video describes the fundamentals of digital fringe projection techniques, which provide dense 3D measurements of dynamically changing surfaces.&#160;It also demonstrates the design and operation of a high-speed binary defocusing system based on these techniques.

Digital fringe projection (DFP) techniques provide dense 3D measurements of dynamically changing surfaces.&#160;Like the human eyes and brain, DFP uses ...

The overall goal of this procedure is to capture high resolution 3D video at or above realtime speed. This is accomplished by first projecting sinusoidal fringe pattern images onto the subject at high speed using a digital light processing projector. Three shifted cosign patterns are projected in sequence to achieve high accuracy.A camera is used to capture these images from another viewing angle. The second step is to compute the wrapped phase from each set of three fringe pattern images. This is accomplished using the arc tangent function and the image intensity values.Next, the phases unwrapped to remove the two pi discontinuities that result from the arc tangent function. The final step is to retrieve the depth from the unwrapped phase of the subject. This is the difference between the unwrapped phase maps of the subject and the calibration plane appropriately scaled and translated by constants found using a reference object.Ultimately, the resulting data frames can be displayed using graphics software. The main advantage of this technique over other existing methods like laser scanning, is that it is capable of both high resolution and high speed. Because known sinusoidal patterns are projected onto the subject, A 3D data point can be retrieved for each pixel of the camera used with a 5 76 by 5 76 camera.We can retrieve over 300, 000 3D data points per frame. Though this method has potential medical applications such as capturing the formation of facial expressions or the beating surface of a heart, it can also be applied to numerous other areas of study. It enables high resolution facial motion capture for use in movies and video games, or an enhanced method of video conferencing.It could also be used to detect defects in a manufacturing environment. Visual demonstration of this method is critical as a calibration. Processing steps are difficult to learn because of the visual age of the system and its measurements.The simplest and the easiest way to detect problems is with a trained visual examination. The first step is to generate the fringe patterns that will be projected. These are prepared in advance using an image programming environment here, matlab.This video will focus on the use of binary patterns. To produce a defocused binary pattern, use a dithering technique to generate sinusoidal patterns using only pure black and pure white pixels. Make three images of the pattern shifted in phase from each other by two pi over three, as called for by the three step phase shifting algorithm.In this demonstration, two additional sets of three are produced for the multi frequency technique, which can capture sharper changes in depth. Next, select a high speed digital light processing projector with a monochromatic setting. Ease the software provided with the projector to upload the images for phase shifting.Now, choose a black and white C, CD or COS camera with the correct capture rate for the system. Keep in mind that the camera will need to capture the entire set of fringe images for each video frame to find the distance at which the projector should be placed from the object. Move the projector relative to a large flat surface when the vertical and horizontal extent of the image is slightly larger than the object to be studied.Measure the distance of the projector to the wall. Use the desired field of view at this distance and the camera sensor size to find the focal length of the lens. The last configuration step is to determine the angular separation between the projector and the camera at a large angle between these components.Triangulation between feature points is obvious, but more features get lost in shadow. At a small angle, triangulation becomes difficult increasing noise in the results. Typically, 10 to 15 degrees is a good compromise.It is best to perform calibration just before data capture. For a binary defocusing system, defocus the projection lens until the patterns at the imaging plane resemble high quality sinusoids. This might require an iterative process of examining test data and adjusting the lens.If the fringes blur together, the projector is too defocused. If dots are visible within the pattern, the projector is too focused. Now, place a flat whiteboard in the fields of view of both the camera and the projector.Project the first of the fringe images on the board. Then capture it with the camera project and record the remaining fringe images. In the same way, save these fringe images for the data processing step, labeling them as the calibration plane.Next place an object of known dimensions in the system's field of view. Here, a rigid foam cube covered with squares of diffuse adhesive foam is used. Project the same series of fringe images onto the cube.Capturing each with the camera. Save the captured images for the processing step, labeling them as the calibration cube. To collect data.Position the subject at the focal plane of the camera project fringe images onto the subject and capture them. High speed is typically required for correct motion capture at high speed. The human eye can only see the fringes.In temporal interference. Use the captured images to assist with adjustments to the camera aperture. To optimize the light level, the fringe images should be as bright as possible, but not saturated.The next step is post-processing of the data. In the three step phase shifting algorithm, the phase is the argument of the cosign function that determines the position of a point within the sinusoidal pattern. An algorithm has been implemented to determine this phase at each point from the fringe images, this computed wrapped phase is in the interval.Negative PI to PI apply this algorithm to the calibration plane and cube and the subject data. Then unwrap the phase maps using another algorithm to add or subtract two pi at phase jumps In the multi frequency technique, the wrapped phase maps for each frequency are combined to yield a single unwrapped phase map, at this point, it is important to revisit the calibration step. Take a horizontal cross section from the center of the phase map of the calibration plane.Remove its bulk profile to obtain a phase error estimate. If the projected pattern was too focused, the error will be large. Adjust the projector lens as needed to obtain error within the range.Negative 0.1 to 0.1 radians. Next, a third algorithm calculates the depth of the calibration cube. This is the difference between the calibration cube and reference plane phase maps.From this, a scale factor is determined. The depth of the subject is found by subtracting the phase map of the reference plane from that of the subject and applying the scale factor. The data can now be saved for visualization in MATLAB or other 3D graphic software.The technique allows real time to high speed three dimensional imaging of a human face at a resolution high enough to reveal fine details. The set of three images on the left is the full face displayed in 2D, texture, overlay, shading, and lighting and wire frame modes. In the center is a wire frame view closeup of the nose area.Note the density of points at right is a closeup view of the region around the eye. These images were produced using sinusoidal fringe patterns. Shown here is a 3D video of the formation of a smile.The video was captured at 60 hertz with a resolution of 640 by 480 sinusoidal fringe patterns were used. It is possible to do live 3D video, capturing, processing, and rendering. In this video, the 3D measurements are displayed at 30 hertz on the computer screen.As a last example of the capabilities of this method, this shows the 3D video imaging of a live rabbit heart. Using binary defocusing, the heart rate was approximately 200 beats per minute. The 3D capture rate was 166 hertz with a resolution of 576 by 576.A high speed was necessary to prevent motion artifacts. Once master the calibration, data capture and data processing can be done in a few hours if performed properly. With processing software designed for speed, many, many processor results can be displayed on computer screen in real time After its development.This technique paved the way for researchers in the field of cardiac surface mechanics to investigate the dynamic surface geometry of a beating rabbit heart using high resolution 3D video data. After watching this video, you should have a basic understanding of how to design and operate high resolution, high speed 3D video system. In particular, you should be familiar with the concepts behind digital fringe projection with the focused binary patterns and the reference plane calibration method.You should also be able to recognize the difference between good and bad unwrapped phase maps.

high-resolution-high-speed-three-dimensional-video-imaging-with

Research

JoVE Journal

Engineering

Non-destructive volume visualization can be achieved only by tomographic techniques, of which the most efficient is the x-ray micro computerized tomography ( CT).

High Resolution 3D Imaging of Ex-Vivo Biological Samples by Micro CT

The complete construction of a custom, real-time confocal scanning imaging system is described.  This system, which can be readily used for video-rate microscopy and microendoscopy, allows for an array of imaging geometries and applications not accessible using standard commercial confocal systems, at a fraction of the cost.

Video-rate Scanning Confocal Microscopy and Microendoscopy

Lensfree optical tomography is a three-dimensional microscopy technique that offers a spatial resolution of  &lt;1 μm × &lt;1 μm × &lt;3 μm in x, y and z dimensions, respectively, over a large imaging-volume of 15-100 mm3, which can be particularly useful for integration with lab-on-a-chip platforms.

Lensfree On-chip Tomographic Microscopy Employing Multi-angle Illumination and Pixel Super-resolution

The three-dimensional locations of weakly-scattering objects can be uniquely identified using digital inline holographic microscopy (DIHM), which involves a minor modification to a standard microscope.  Our software uses a simple imaging heuristic coupled with Rayleigh-Sommerfeld back-propagation to yield the three-dimensional position and geometry of a microscopic phase object.

Digital Inline Holographic Microscopy (DIHM) of Weakly-scattering Subjects

A method for overcoming the optical diffraction limit is presented. The method includes a two-step process: optical phase retrieval using iterative Gerchberg-Saxton algorithm, and imaging system shifting followed by repetition of the first step. A synthetically increased lens aperture is generated along the direction of movement, yielding higher imaging resolution.

Time Multiplexing Super Resolving Technique for Imaging from a Moving Platform

Spatiotemporal information about dynamic proteins inside live cells is crucial for understanding biology. A type of super-resolution microscopy called fast 3D-structured illumination microscopy (f3D-SIM) reveals unique information about the cytokinetic Z ring in bacteria: both its bead-like appearance and the rapid dynamics of FtsZ within the ring.

Super-resolution Imaging of the Cytokinetic Z Ring in Live Bacteria Using Fast 3D-Structured Illumination Microscopy (f3D-SIM)

Four-dimensional (4D) imaging is utilized to study the behavior and interactions among two types of endosomes in living vertebrate nerve terminals.  Movement of these small structures is characterized in three dimensions, permitting confirmation of events such as endosome fusion and exocytosis.

Visualization of Endosome Dynamics in Living Nerve Terminals with Four-dimensional Fluorescence Imaging

We present a compact reflection digital holographic system (CDHM) for inspection and characterization of MEMS devices. A lens-less design using a diverging input wave providing natural geometrical magnification is demonstrated. Both static and dynamic studies are presented.

Compact Lens-less Digital Holographic Microscope for MEMS Inspection and Characterization

We describe a detailed protocol using high-resolution episcopic microscopy to acquire three-dimensional (3D) images of mouse embryos. This improved protocol utilizes a modified tissue preparation method to enhance penetration of the fluorescent dye, thereby permitting morphometric analysis of both small and large-sized specimens.

Rapid Acquisition of 3D Images Using High-resolution Episcopic Microscopy

We present a protocol for creating a real-time movie of a molecular rotational wave packet using a high-resolution Coulomb explosion imaging setup.

Direct Imaging of Laser-driven Ultrafast Molecular Rotation

We present a protocol to record a set of ultra-realistic full-color analog holograms, showing the same brightness, transparency, and homogeneous colors, on ultra-fine-grain silver-halide holographic emulsions for the fabrication of a dynamic holographic 3D display.

Recording Ultra-Realistic Full-Color Analog Holograms for Use in a Moving Hologram Display

Learn digital fringe projection techniques for high-resolution 3D video imaging using a high-speed binary defocusing system. This method captures over 300,000 data points per frame for dynamic surface measurements.

High-resolution, High-speed, Three-dimensional Video Imaging with Digital Fringe Projection Techniques

Explore More Videos