A Machine-Vision Approach to Transmission Electron Microscopy Workflows, Results Analysis and Data Management
June 23rd, 2023
Here, we present a protocol to utilize machine-vision software to stabilize dynamic processes during TEM imaging, while concurrently indexing multiple streams of metadata to each image into a navigable timeline. We demonstrate how this platform enables automated calibration and mapping of the electron dose over the course of an experiment.
This technology was developed to make high performance TEM studies accessible to users of all skill levels. Furthermore, scientific advancements require the collaborative efforts of multiple investigators, so an efficient platform to share and analyze large, complex datasets is essential for successful research. Operando and in situ studies transform a TEM into a realtime nano laboratory, allowing researchers to explore the dynamic nano skill processes that control materials for properties.
One of the greatest challenges for in situ TEM studies is isolating the beam irradiation effects from the intrinsic behavior of the sample. Accurate electron dose measurement and tracking is essential, but without dose management software, one can only measure electron dose rate rather than the total or critical dose. Complex TEM workflows generate large datasets, which must be manually aligned and indexed to their respective images.
In this time consuming process, the key information such as adjustments to imaging conditions or a samples environment may be lost, complicating analysis and reproducibility. We have developed a machine vision workflow to collect and index the images and metadata produced during an experiment into a data-enriched, searchable timeline. Computational and image analysis algorithms can calculate variables between images, apply corrections, and identify hidden trends.
By calculating, synchronizing and indexing new metadata into each image, multimodal analysis is suddenly a new possibility. For example, accurate accumulated dose and dose right information in the zeolite images allows quantitative assessment of sample damage across thousands of images. We will continue to push the capabilities of in-situ TEM by making results accessible and interpretable and improving experimental reproducibility.
This software is a platform to develop workflow-driven modules that target specific historically difficult applications to make in-situ TEM experiments easier and more information-rich. Begin the dose calibration by clicking the connect icon in the MVS software installed on TEM. Select the microscope and activate the connection between the TEM and the MVS software to visualize the images from the camera or detector in the image viewer.
Navigate to the Dose tab and click on Start Dose Area Calibration. Click OK to confirm the calibration hardware and follow the software prompts to enter the requested user configurable values. Then click Start Dose Current Calibration.
Follow the software prompts and enter the requested user configurable values to complete the calibration. Begin by opening the MVS software application on the TEM instrument and selecting Other. Connect the microscope to the MVS software.
Navigate to the Image Metadata tab and select Magnification, Max Dose and Dose Rate to overlay on the image stream seen in the live display. Other metadata may be included if the user desires. Open the Dose tab, select Dose Management and Enable Dose Monitoring.
Then select Show Dose Layer and Show Position Uncertainty to activate automated electron dose tracking and display the dose color overlay. Set the values for the low dose level and high dose level before pressing Save. Then navigate to the Settings tab, select Dose and set the Dose Navigation Map Opacity and Dose Image Overlay Opacity values.
Activate drift correction in the live image viewer window by clicking on Drift Correct. Next, navigate to the Data View tab and plot the metadata values Defocus and Focus Quotient on the Y axis. Activate focus assist, then select Calibrate Focus to run the calibration of the automated focus assist.
Once the calibrate focus routine is complete, close the Data View tab. In the MVS software, open the Image Analysis tab and activate the Live FFT and Quadrants One and Two options. Using the microscope software controls, adjust the beam conditions so that the electron flux is approximately 600 angstrom square per second.
Move to a new region in the sample and center the sample ROI in the live view of the MVS software. Maintain a constant dose rate while continuously imaging the same ROI until the spots corresponding to the atomic structure in the FFT have disappeared. Launch the offline analysis software to view the fully synchronized data sets and open the experiment session file by selecting it from the library.
Display the drift corrected images by activating the DC tab below the image viewport before selecting the desired data overlays by checking their respective overlay data boxes in the Image Metadata tab. Other metadata may be plotted as the user desires. Highlight or scroll through these graphical plots to update the image displayed in the viewport.
Access various tools through the Image Analysis, Toolbox and Data View tabs. Access the FFT for each image through the image analysis to plot Live FFT to update it while scrolling through images. Use the fading of the FFT peaks to determine the time point at which the zeolite nanoparticle structure loses crystallinity.
Note the dose conditions in the software using the tag function. Highlight the tag icon and enter the desired text to denote a specific series of images within the timeline. Images will be tagged with this text until the tag icon is deselected.
To easily filter large data sets into smaller shareable ones without losing their associated metadata, open the filter panel and adjust the sliders to select only a dose rate equal to or above 500 square angstroms per second. Save the new collection using the name 500 dose rate and above. Select the images by highlighting them in the timeline or using the filter options.
Then export the metadata as a CSV file and the image series as a movie file using the same publish option. The zeolite nanoparticle ZSM five image to determine the threshold dose showed that the dose rate read by the MVS software under high dose rate conditions was 519 electrons per square angstrom per second. Nanoparticles in the field of view were imaged continuously until the FFT peaks disappeared, indicating crystalline structure degradation.
The FFT peaks began disappearing after 42 seconds of continuous imaging. The FFT peaks had completely disappeared at one minute and 20 seconds, and a cumulative dose of approximately 60, 000 electrons per square angstrom. The same process was repeated with a different microscope.
The FFT spots fully disappeared after one minute and 50 seconds of continuous imaging, and a cumulative dose of 58, 230 electrons per square angstrom. To begin, select the appropriate workflow option from the MVS software. Follow the workflow prompts to confirm the electrical connection between the holder and the heating E-chip by loading the calibration file and performing a device check.
After connecting the microscope to the MVS software, center the sample ROI in the field of view. Access to the temperature control settings by clicking the Experiment button and Manual control mode. Then set the ramp rate to 25 degrees Celsius per second and the target to 200 degrees Celsius.
Click on Apply to start the experiment. After reaching 200 degrees Celsius, set the ramp rate to 10 degrees Celsius per second. Adjust the target to 600 degrees Celsius and click Apply.
After the set temperature of 600 degrees Celsius is reached, change the ramp rate to two degrees Celsius and the target to 800 degrees Celsius. Click on Apply to start the experiment. Once done, open the analysis software to review the session.
In the timeline, plot the temperature, template morphing factor, dose rate, and cumulative dose. Export images and movies using the Publish option with or without the dose map overlays. The heating experiment performed using a representative nano catalyst sample, gold on iron oxide, showed that at elevated temperatures, the gold nanoparticles within the gold on iron oxide migrated along the surface of the iron oxide support and centered to form larger particles.
An in situ heating experiment recorded a series of TEM snapshots of a porous region within a gold on iron oxide nano catalyst at various time points. The coordinated drift of the sample increased with the increased temperature from a rate of nine to 62 nanometers per minute, and began to decrease towards leveling off with the constant temperature. The MVS software stabilized the particle in the field of view throughout the entire temperature ramp profile, enabling high resolution imaging.
Overview
This study presents a novel machine-vision software protocol for stabilizing dynamic processes during transmission electron microscopy (TEM) imaging, while simultaneously managing metadata indexing into a navigable timeline. The approach significantly enhances the ability to conduct automated calibration and mapping of electron dose across experiments, enabling more efficient analysis of complex datasets.
Key Study Components
Research Area
- Transmission electron microscopy (TEM)
- Machine vision technology
- Data management in microscopy
Background
- The need for accurate electron dose measurement in in situ TEM studies
- Challenges in aligning and indexing complex TEM datasets
- The importance of reproducibility in experimental research
Methods Used
- Machine vision workflow for image and metadata indexing
- Real-time electron dose tracking
- Automated calibration and analysis algorithms
Main Results
- Introduction of a searchable timeline for TEM imaging data
- Enhanced capability for multi-modal analysis of nanoparticle behaviors
- Improved experimental reproducibility through comprehensive data management
Conclusions
- The software platform facilitates easier and more informative in situ TEM experiments
- This advancement holds significance for collaborative research efforts in material sciences and nanotechnology
Frequently Asked Questions
Executive Industry Relevance
High-throughput transmission electron microscopy (TEM) workflows generate vast, complex datasets that challenge reproducibility, data integrity, and collaborative analysis in biopharma R&D. Machine-vision synchronization and automated metadata management enable precise quantification of sample exposure and structural changes, supporting robust target validation and mechanistic de-risking. These capabilities streamline early discovery and preclinical workflows, enhancing predictive confidence and portfolio decision-making.
Strategic Applications in Biopharma R&D
Early Discovery & Target Validation
- Automated dose calibration and tracking enable quantitative assessment of sample integrity under experimental conditions.
- Continuous metadata synchronization supports mechanistic de-risking by isolating beam effects from intrinsic sample behavior.
- Indexed, searchable timelines facilitate rapid identification of key events and reproducibility across experiments.
Screening & Assay Development
- Standardized image and metadata capture prepares datasets for downstream screening and comparative analysis.
- Automated drift correction and focus calibration improve assay reproducibility and quantitative output reliability.
- Data mining and filtering tools enable efficient extraction of relevant subsets for compound evaluation.
Translational & Preclinical Research
- Quantitative dose mapping and structural analysis support translational continuity from discovery to preclinical validation.
- Comprehensive data management enhances cross-study comparability and risk-adjusted advancement decisions.
- Multimodal analysis capabilities align with biomarker discovery and mechanistic studies when supported by experimental design.
Pipeline & Workflow Integration
This machine-vision-enabled TEM workflow integrates from early discovery through preclinical research, supporting hypothesis testing, assay development, and translational studies.
- Discovery Biology: Enables isolation of experimental variables and robust hypothesis interrogation through synchronized imaging and metadata.
- Screening: Provides reproducible, quantitative outputs for reliable assay development and compound triage.
- Analytics: Facilitates trend analysis, threshold identification, and quantitative comparison across experimental conditions.
- Translational Research: Supports continuity of data and mechanistic insights from in vitro to preclinical models.
- Enterprise Reuse: Establishes a scalable, workflow-driven platform for repeated application across diverse R&D programs.
Operational & Enterprise Impact
- Scientific Value: Increases predictive confidence and reduces mechanistic ambiguity in target validation.
- Operational Value: Enhances standardization, reproducibility, and scalability of TEM-based workflows.
- Strategic Value: Improves go/no-go decision quality and capital efficiency by enabling robust data-driven insights.
- Portfolio Impact: Supports risk-adjusted prioritization and advancement of discovery and preclinical assets.
Implementation Considerations
- Requires expertise in TEM operation and machine-vision software configuration.
- Depends on integration with compatible instrumentation and analytical infrastructure.
- Demands cross-team standardization of metadata capture and workflow protocols.
- Adaptation may be needed for different sample types or experimental models.
- Practical limitations include hardware compatibility and user training requirements.
Biopharma FAQs
Chapters in this video
0:00
Introduction
2:23
Dose Calibration of the Transmission Electron Microscope for TEM and Scanning TEM (STEM) Imaging Modes Using Machine‐Vision Approach
3:08
Determination of Dose Threshold Using the Machine‐Vision Synchronization (MVS) and Dose Software
5:14
Metadata and Trend Analysis and Data Export Using the Analysis Software
8:04
In Situ Heating Study of Gold on Iron Oxide Nanoparticles
