de Haan, Kevin, et al., 2020, Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

# Three-line Summary #

1. In recent years, deep learning has been shown to be one of the leading machine learning techniques for a wide variety of inference tasks.
2. In addition to its mainstream applications, such as classification, it has created transformative opportunities for image reconstruction and enhancement in optical microscopy.
3. This article provides an overview of some of the recent work using deep neural networks to advance computational microscopy and sensing systems, also covering their current and future biomedical applications.

 

# Detail Review #

1. Introduction

• Deep learning is a set of machine learning techniques that use multilayered neural networks to automatically analyze signals or data.
  • Deep networks have proven to be very effective for a wide variety of tasks ranging from natural language processing to image classification and playing games, such as Go, among others.
  • Several types of deep networks, such as long short-term memory (LSTM) and convolutional neural networks (CNNs), have been developed over decades of research.
  • Recently, a "perfect storm" of optimized software, hardware (e.g., increased GPU power), and data availability have allowed deep learning (specifically CNNs) to be used as a recipe to tackle complex problems in many fields of research most notably in computer vision.
• In this article, we will focus on deep neural networks' transformative power to solve inverse problems in microscopy using image data.
  • Beyond classification, deep learning presents many interesting opportunities to solve classical inverse problems in imaging, such as deblurring, super-resolution, denoising, and pixel super-resolution (or geometrical super-resolution).
  • In microscopy, variables such as the physical properties of illumination, light-sample interaction, sample preparation, and positioning are fully under the user's control.
  • Therefore, the gold-standard image data can often be experimentally generated rather than simulated.
• In order to solve a microscopy-related inverse problem using deep learning, a neural network must be trained using a set of matching input and ground truth (or gold standard) images.
  • we first discuss the implementation strategies for using deep networks in microscopic image reconstruction and enhancement.
  • we then discuss examples of inverse problems in microscopy that can be solved using deep learning.

* [Ref.] de Haan, Kevin, et al. "Deep-learning-based image reconstruction and enhancement in optical microscopy."  Proceedings of the IEEE  108.1 (2019): 30-50.

 

2. Brief overview of inverse problems in optical microscopy

• The three imaging modalities in optical microscopy
  • (1) Bright-field microscopy: white light illumination is modulated by the light-sample interaction and is collected by an objective lens.
  • (2) Fluorescence microscopy: using an illumination source to excite the fluorophores in the sample being imaged.
  • (3) Digital holographic microscopy: illuminating the sample with coherent light and uses, e.g., a transmission geometry to create an interference pattern of the sample's transmission function.
• For a microscopic imaging system, the discrete imaging forward model, which sets the stage for an inverse imaging problem, can be written as [ g = Hf + n ] (* f = object; g = measurement information; H = mapping operator; n = noise).
  
  • (1) Deconvolution or deblurring: [ g = h*f + n ] (* h = low-pass filter, gaussian kernel or blurring kernel; * = spatial convolution operator)
  • (2) Super-resolution: optical, computational, and statistical techniques have been developed to break the diffraction limit (smallest feature d = ∧/2NA; ∧ = illumination wavelength, NA = numerical aperture) and achieve super-resolution in fluorescence microscopy (some of these fluorescence super-resolution techniques require specialized and expensive equipment, with relatively high-power illumination and a large number of image exposures).

• A primary example of deep learning being used in optical microscopy has been for spatial resolution enhancement using deep learning.
  • Deep learning has been shown to be able to leverage large amounts of well-registered image data to learn statistical transformations through high levels of abstraction in order to improve upon conventional super-resolution and image-enhancement algorithms and achieve superior results.
  • Deep neural networks typically learn to solve inverse imaging problems through supervised learning that utilizes "gold-standard" labels that are known in advance and are matched to corresponding input images.

3. Deep learning as a framework to solve inverse problems in optical microscopy

• To transform images taken by a cost-effective smartphone-based microscope into images equivalent to those taken by a high-quality benchtop microscope.
  • This training dataset will consist of inputs made up of cellphone images and the gold-standard label images of the same samples taken by the benchtop microscope.
  • Once image data have been coregistered(smartphone & benchtop microscope), a GAN that uses a U-net as the generator and a VGGNet style network as the discriminator can be chosen.

* [Ref.] de Haan, Kevin, et al. "Deep-learning-based image reconstruction and enhancement in optical microscopy."  Proceedings of the IEEE  108.1 (2019): 30-50.

 

• The conversion of a low-resolution image into a higher-resolution image, where both the input and ground truth (higher resolution image) are taken by the same microscope.
  • scanning the same samples with a low-NA and high-NA objective lens.
  • This super-resolution image enhancement has been demonstrated for fluorescence, bright field, and coherent (holographic) imaging systems.

* [Ref.] de Haan, Kevin, et al. "Deep-learning-based image reconstruction and enhancement in optical microscopy."  Proceedings of the IEEE  108.1 (2019): 30-50.

 

• To learn a statistical transformation between two different microscopic imaging modalities.
  • Learning these cross-modality transformations where the ground truth image set is made up of images taken by a different microscope allows the network to achieve results that are not possible using standard forward model-based inverse problem solutions.
  • Using this deep learning-based approach, cost-effective or simpler microscopes can take the same quality of measurements as the gold-standard microscopes

* [Ref.] de Haan, Kevin, et al. "Deep-learning-based image reconstruction and enhancement in optical microscopy."  Proceedings of the IEEE  108.1 (2019): 30-50.

 

 

* Reference: de Haan, Kevin, et al. "Deep-learning-based image reconstruction and enhancement in optical microscopy." Proceedings of the IEEE 108.1 (2019): 30-50.