Real-time data processing and feedback during in-situ heating

B.K. Miller, C. Czarnik, S.E. Mick


Real-time processing of digitally acquired data dates back to at least the 1980s[1], but decades later, most data processing still takes place long after the acquisition, on a separate computer. Implementing real-time processing is important for in-situ experiments, since the next experimental step at the microscope is often determined by dynamic changes observed in the sample. While a skilled operator can often be incredibly good at picking up subtle changes in complex images, Gatan’s latest in-situ cameras store more frames to disk than can be displayed even with high-end video cards. When using the fastest data collection rates, it is not possible for the experimentalist to monitor every acquired image. Although simple processing, like live FFTs can be used for analyzing data in real-time, only the subset of the acquired frames that are displayed on the computer screen are actually analyzed at the microscope. A new workflow is therefore needed that allows real-time processing of acquired data so that the results of the analysis can be used by the researcher at the microscope to make better decisions about the status and next steps for the experiment being conducted.


Real-Time Feedback

  • In-situ TEM enables observation of dynamic behavior
  • Specific stimuli combinations often are chosen during observation to drive desired dynamics
  • Need to accurately detect dynamics during observation to make decisions about what to do next
  • Computational analysis is needed to detect subtle changes and augment decision making during the experiment


Dataflow for Proof of Concept Experiment

Traditional TEM workflows first acquire all data then transfer data to another PC for processing and analysis after the microscope session is finished. The real-time feedback workflow requires that all raw data be transferred during capture. Some data is then processed and the results passed back to the TEM during acquisition. 


Hardware Used

A scalable parallel file system with hundreds of terabytes of storage from DDN Storage was used to save all data from a Gatan OneView® IS camera as it was acquired during an in-situ heating experiment. While data was still being written to the filesystem, a second computer running Gatan Microscopy Suite® (GMS) software was used to process the incoming data via scripting. In this case, the script converted a selected area electron diffraction pattern into a 1D profile, so that subtle changes could be more easily discerned by the TEM operator.

  • Data collected using OneView IS 4k x 4k camera, GMS 3 software, and integrated DENSsolutions Wildfire heating holder
  • GMS 3 PC linked to DDN GS7K
  • 560 TB raw (400 TiB useable) capacity with 8+2 RAID6 protection
  • Windows server (Intel Xeon 3.2 GHz, 64 GB RAM)


Experimental Details and Results

Experiment: VO2 sample heated to induce phase transition, this is subtle in the 2D pattern below
Algorithm: 2D diffraction pattern is converted to a 1D profile (Radial Maximum) - (Radial Average)
Data Flow: Data captured and stored at 1600 MB/s, 64 MB/s is processed and results in 0.45 MB/s processed data




Future Directions

Computation will be improved by more efficient algorithms and better processing hardware, such as a more powerful server, GPU based processing, or a high-performance cluster.

Other analyses may be performed, including particle/interface tracking, de-noising algorithms, complex reconstructions, and model-based algorithms.



The DDN storage system enables real-time processing of high-frame-rate in-situ images from the OneView IS camera. Diffraction patterns were analyzed in real-time to detect phase transformation in VO2.