IEEE ICIP 2025

SPC to 3D: Novel View Synthesis from Binary SPC via I2I Translation

Sumit Sharma1, Gopi Raju Matta1, Kaushik Mitra1

1Indian Institute of Technology Madras, India

Accepted for publication at IEEE ICIP 2025 · Paper 2604

SPC to 3D · Demo
Binary SPC measurements → Novel-view synthesis

A modular I2I + 3D reconstruction framework for novel-view synthesis from binary SPC imagery.

Abstract

Single Photon Cameras (SPCs) offer exceptional temporal sensitivity by detecting individual photons, but their binary measurements contain severely limited texture and color information. This makes conventional 3D reconstruction and novel-view synthesis methods challenging to apply directly.

We propose a modular two-stage framework that first translates binary SPC images into plausible color representations using image-to-image translation and subsequently reconstructs the scene using Neural Radiance Fields or 3D Gaussian Splatting. We investigate multiple I2I and 3D reconstruction combinations and demonstrate that Pix2PixHD combined with NeRF or 3DGS produces improved perceptual and geometric quality.

Video Presentation

Full Video Presentation

Watch the complete project presentation hosted on Google Drive.

Motivation

Single-Photon Imaging

SPC/ SPAD imaging can operate with very high sensitivity and temporal resolution.

The Challenge

Binary photon measurements provide limited intensity, texture, and color information.

Our Question

Can meaningful color novel views be recovered from binary SPC measurements?

Method

Our modular pipeline separates appearance recovery from 3D reconstruction.

Proposed two-stage SPC to 3D pipeline.
Overview of the evaluated two-stage approaches for transforming binary SPC measurements into color novel views.
01

Binary SPC → Color

Image-to-image translation converts simulated 3-channel binary SPC measurements into color images.

Pix2PixHDPix2PixCycleGANLookAtMe
02

Color → Novel Views

The translated color representations are used for 3D scene reconstruction and novel-view synthesis.

NeRF3D Gaussian Splatting

SPC Simulation

RGB image→Photon flux→ Poisson detection→SPAD dead-time→ Photon count→Binary SPC
qSPAD0.45
Dead-time τd150 ns
Output1-bit per channel

Experimental Setup

Training

NeRF-LLFF images were augmented using zooming, rotation, shearing, and flipping, producing approximately 15,000 augmented images. SPC measurements were then simulated.

Evaluation

IBRNet-Collected-2 scenes were used for evaluation, with SPC images simulated in the same manner.

50KNeRF iterations
30K3DGS iterations
~15Kaugmented training images

Results

Two-stage approaches consistently provide stronger results than directly reconstructing from binary SPC images.

Best PSNR 22.70 Pix2PixHD + NeRF
Best SSIM 0.6843 Pix2PixHD + NeRF
Best LPIPS 0.4251 Pix2PixHD + 3DGS
ApproachPSNR ↑SSIM ↑LPIPS ↓
Pix2PixHD + NeRF22.700.68430.4949
Pix2PixHD + 3DGS22.210.67720.4251
Pix2Pix + NeRF18.350.55780.6473
CycleGAN + NeRF16.490.53770.6502
Restormer + NeRF20.4160.65620.4886
Restormer + 3DGS19.27330.43250.6063
DiffBIR + NeRF19.3530.61030.5009
DiffBIR + 3DGS17.1950.418250.47815

Single-stage vs. two-stage

Single-stage

Binary SPC → NeRF / 3DGS

VS

Two-stage

Binary SPC → Pix2PixHD → NeRF / 3DGS

Visual Comparison

Qualitative Results

Qualitative comparison of novel-view synthesis results across the evaluated single-stage and two-stage approaches.

Qualitative comparison of SPC input, NeRF, 3DGS, DiffBIR, Restormer, Pix2PixHD, and ground-truth novel views.
Qualitative comparison across the evaluated methods. Pix2PixHD + NeRF/3DGS provides stronger texture detail and visual clarity in the reported comparisons.

Conclusion

We presented a modular two-stage framework for transforming binary SPC images into color novel views. Integrating Pix2PixHD with NeRF or Gaussian Splatting improves texture detail and perceptual quality compared with the evaluated alternatives.

Future work described in the paper includes unified end-to-end optimization and evaluation on real SPC sensor data.

Publication

IEEE International Conference on Image Processing (ICIP) 2025

Paper 2604 · Session TU5.L5 · Generative Image Models

BibTeX

@inproceedings{sharma2025spc,
  title={SPC to 3D: Novel View Synthesis from Binary SPC via I2I Translation},
  author={Sharma, Sumit and Matta, Gopi Raju and Mitra, Kaushik},
  booktitle={IEEE International Conference on Image Processing (ICIP)},
  year={2025}
}