CVIP 2024 · Best IAPR Paper Award

Transforming Single Photon Camera Images to Color High Dynamic Range Images

A computational approach for transforming monochromatic Single Photon Camera (SPC) imagery into colorful HDR images through two-stage and single-stage image translation.

Sumit Sharma · Girish Rongali · Kaushik Mitra

Computational Imaging Lab · Indian Institute of Technology Madras, India

🏆IAPR Best Paper Award
CVIP 2024 · Paper ID 607
Why SPC?

From photon counting to colorful HDR

SPC/SPAD sensors can operate across very bright and very dark regions because their response has a soft saturation behavior. However, the captured images are monochromatic, noisy and limited in resolution. This work studies computational methods to recover visually meaningful color HDR imagery.

High Dynamic Range

A single SPC exposure inherently captures HDR information across challenging illumination conditions.

Monochromatic Input

The input lacks color information and is often noisy and visually unappealing.

Computational Solution

Image-to-image translation and single-image HDR reconstruction are combined and systematically evaluated.

Dataset & Simulator

Building paired SPC–color data

A simulator generates monochromatic SPC images from HDR ground truth. The study uses the Laval indoor HDR dataset, with downsampled resolutions of 1024×512 and 2048×1024. Logarithmic tone mapping is used to create paired color LDR targets.

SPC simulation pipeline

SPC simulation pipeline: HDR ground truth at multiple resolutions is passed through the Single Photon Camera simulator to generate 1-frame and 4-frame averaged monochromatic SPC images.

Two-stage architecture

Colorization followed by HDR reconstruction

The two-stage study evaluates Pix2Pix, Pix2PixHD and CycleGAN for monochromatic-to-color translation, followed by DeepHDR, LHDR, SIHDR and ExpandNet for color-to-HDR reconstruction. This produces 12 possible combinations.

Two-stage SPC to color HDR architecture

Best combination

Pix2PixHD performs best for colorization, while DeepHDR performs best for the subsequent HDR reconstruction. The resulting Pix2PixHD + DeepHDR combination is reported as the strongest two-stage architecture.

Quantitative Results

Pix2PixHD consistently leads colorization

26.53 dB
PSNR · 2048×1024 · 1-frame
0.9056
SSIM · 2048×1024 · 4-frames
0.1995
LPIPS ↓ · 2048×1024 · 4-frames
4.0079
HDR-VDP3 · Pix2PixHD + DeepHDR · 4-frames

The paper reports that 4-frame averaging improves colorization over a single SPC frame. Pix2PixHD + DeepHDR achieves the highest HDR-VDP3 score among the evaluated two-stage combinations.

Quantitative Analysis

Detailed quantitative results

The paper reports three quantitative tables: SPC monochromatic-to-colorization results, SPC monochromatic-to-HDR results, and direct single-stage SPC-to-color-HDR results.

Quantitative Tables 1, 2, and 3 from the paper

Tables 1–3 reproduced from the supplied paper. Table 1 reports LPIPS, PSNR and SSIM for 1-frame and 4-frame SPC inputs; Table 2 reports HDR-VDP3 scores for the two-stage combinations; Table 3 reports single-stage Pix2PixHD HDR results.

Table 1 · Colorization

Pix2PixHD gives the best colorization results. At 2048×1024, 4-frame input reaches 0.1995 LPIPS, 26.50 PSNR and 0.9056 SSIM.

Table 2 · HDR Reconstruction

Pix2PixHD + DeepHDR gives the highest reported HDR-VDP3 scores, reaching 4.0079 for 4-frame input at 2048×1024.

Table 3 · Single-stage

Modified Pix2PixHD reaches an HDR-VDP3 score of 3.9734 at 2048×1024 with 4-frame input.

Qualitative Analysis

Visual quality across the pipeline

Qualitative colorization results

1-frame and 4-frame SPC colorization results at 2048×1024. Pix2PixHD gives the most favorable results.

Qualitative HDR results

Color HDR comparison between the two-stage Pix2PixHD + DeepHDR approach and the single-stage Pix2PixHD approach.

Single-stage vs Two-stage

Comparable direct translation

The study also modifies Pix2PixHD for direct monochromatic SPC → color HDR translation. At 2048×1024, the reported HDR-VDP3 score is 3.9734 for 4-frame input, compared with 4.0079 for Pix2PixHD + DeepHDR in the two-stage pipeline.

Two-stage

SPC monochromatic → color LDR → color HDR. Best reported combination: Pix2PixHD + DeepHDR.

Single-stage

Direct SPC monochromatic → color HDR using a modified Pix2PixHD architecture.

Key observation

The direct approach gives comparable results, while the two-stage approach remains slightly stronger in the reported evaluation.

Conclusion

From monochromatic photons to colorful HDR

The work presents a detailed study of converting SPC images into colorful HDR images using two-stage and single-stage architectures. The analysis demonstrates the potential of SPC's soft-saturation behavior for HDR imaging and identifies strong computational baselines for colorization and HDR reconstruction.