Seeing A 3D World in A Grain of Sand

Our imaging system consists of a camera and a planar catadioptric lens that allows for synchronized full-surround multi-view image acquisition. The conceptual design is illustrated in Fig. 2. Our catadioptric lens is made with eight pairs of flat mirrors that are circularly arranged on the sides of two nested octagonal pyramids. In each pair, the mirrors are facing each other with different tilting angles and shifted locations. Through two times of reflections, the mirror pair guides light from a scene that is underneath to the viewing camera on top. In this way, the light paths are folded such that we are able to image small objects in short distance, as if they are imaged by a zoom lens. Similar idea on light path folding is used in Folded Optics (or Origami Optics) Tremblay et al. 2007; Tremblay et al. 2009 for compact wide-angle lens design.

Then we arrange eight such mirror pairs along a circle, on surfaces of two nested octagonal pyramids, in order to capture full-surround multi-view images. A sample image captured by our imaging system is shown in Fig. 2. Each sub-image is formed by a pair of mirrors. With eight mirror pairs, we obtain eight sub-images that surround the scene in 360^∘ and are time-synchronized by optics. Note that we purposely arrange the mirrors such that light only bounces once on a piece of mirror. Therefore, there is no inter-reflection in our sub-images. Similar design with mirror pairs is studied for stereo image acquisition Kim et al. 2006. But two mirror pairs on opposite sides have non-overlapping views and is not practical for stereo vision, whereas our circular arrangement guarantees overlapping in neighboring views and provides full-surround coverage of the scene.

Next we analyze the light transport inside of our lens and derive the relationship between our viewing volume and mirror parameters. We then use the formulas to optimize the mirror configuration based on the scene dimensions, such that the multi-view coverage of the scene is optimal.

Since our sub-images exhibit weak perspective with long effective focal length, we assume orthographic camera model for simplicity. The effective viewing volume is carved out by eight beams of parallel light as a polyhedron that can be approximated as a symmetric double pyramid. We characterize this volume by its side angle at apex ($\theta$) and the widest length at the base ($l$). WLOG, we illustrate the formation of effective viewing volume in a 2D cross-section (see Fig. 4), since our mirror design is symmetric. We parameterize the mirrors in a pair using their tilting angles ($\alpha_{1,2}$), vertically projected heights ($h_{1,2}$), and distances ($d_{1,2}$) between their upper edges and the central ray (i.e., vertical ray from the imager center). We denote the inner mirror as $M_{1}$ and the outer one as $M_{2}$. Subscript of the parameters indicate their correspondence to the mirrors.

Since light path is reversible, we trace rays from the camera to the scene. Given a vertical ray from camera, it is first bounced on $M_{1}$ and then on $M_{2}$ before reaching the scene. The ray’s incident angle to $M_{1}$ (i.e., the angle between the incident ray and the mirror’s normal) is: $\omega_{1}=\alpha_{1}$. It’s incident angle to $M_{2}$ can then be calculated as: $\omega_{2}=2\alpha_{1}-\alpha_{2}$. Since rays reflected from $M_{2}$ intersect with the symmetric beam from the opposite side to form the effective viewing volume, the pyramid’s side angle at base can be calculated as: $\beta=90^{\circ}-\alpha_{2}+\omega_{2}=90^{\circ}-2(\alpha_{2}-\alpha_{1})$. The side angle at apex is thus:

Let $\Delta\alpha=\alpha_{2}-\alpha_{1}$. We define $2\theta=4\Delta\alpha$ as the field-of-view (FoV) of our catadioptric lens. Detailed derivation of these angles can be found in the supplementary material.

1. (i)

   $45^{\circ}<\alpha_{1}<\alpha_{2}<90^{\circ}$, such that light from scene underneath can be reflected to the imager on top;

2. (ii)

   $h_{2}\geq\dfrac{\sin\alpha_{2}}{\tan\alpha_{1}\cdot\cos(\alpha_{2}-2\alpha_{1})}\cdot h_{1},$ such that $M_{2}$ can cover the entire light beam reflected from $M_{1}$;

3. (iii)

   $d_{2}\geq\dfrac{\tan\alpha_{1}+\cot 2\Delta\alpha}{\tan\alpha_{1}\cdot(\cot 2\Delta\alpha-\cot 2\alpha_{1})}\cdot{h_{1}}+d_{1},$ such that no inter-reflection occurs between $M_{1}$ and $M_{2}$.

Please see the supplementary material for derivation of these conditions. Note that although the size and location of $M_{2}$ ($h_{2}$ and $d_{2}$) would not affect the dimension of effective viewing volume dimension, changing these parameters would result in the viewing volume to be shifted vertically. It is preferable for $d_{2}$ to take smaller values, since the resulting viewing volume would be closer to the camera.

Since the amount of light that can be received by the imager is bounded by its size, we first allow the width of the parallel beam $w$ to take its maximum value, which is half of the sensor size¹¹ 1 We assume orthographic model. Under perspective model, this value also depends on the lens., and we denote it as $w_{\text{max}}$. Since $w=h_{1}/\tan\alpha_{1}$, we can adjust the angle and length of $M_{1}$ to achieve $w_{\text{max}}$. Given $w_{\text{max}}$, the dimension of effective viewing volume is only related to $\Delta\alpha$. The greater the $\Delta\alpha$, the larger the FoV and base length $l$, but the smaller the vertical height $h=w_{\text{max}}/\sin 2\Delta\alpha$. We then set out to find the viewing volume with the largest FoV that can fully enclose a scene with known size. We approximate a scene using its bounding box with size $W\times L\times H$ (where $H$ is the vertical height and we assume $L\geq W$). The FoV takes its largest value when the scene’s bounding box is inscribed in the viewing volume (see Fig. 4). This is because when FoV further increases beyond this value, the vertical height $h$ decreases, which would result in the scene being partially outside of the viewing volume.

Next we derive the FoV value under this circumstance. With similar triangles, we have $(h-H)/h=L/l$. By substituting $h=w_{\text{max}}/\sin 2\Delta\alpha$ and $l=w_{\text{max}}/\cos 2\Delta\alpha$, we have $H\cdot\sin 2\Delta\alpha+L\cdot\cos 2\Delta\alpha=w_{\text{max}}$. By applying the sine angle addition identity, we have:

The largest FoV is $4\Delta\alpha$, which can be calculated given the scene size. Eq. 3 indicates the angle difference between $M_{1}$ and $M_{2}$, which is the optimal mirror configuration to achieve the largest FoV.

Since our multi-view images are combined into a single image, we first segment our captured image into individual multi-view images. We obtain the segmentation mask by taking one image of a solid color background. By applying the mask, we obtain eight images as if captured by surrounding virtual cameras. Since the orientation of virtual cameras are flipped on opposite sides, we re-project the multi-view images onto new viewing planes, such that the transition of view poses is smooth on a circle. We also create foreground masks to segment out the object of interest. This can be done using off-the-shelf segmentation tools, such as Segment Anything Kirillov et al. 2023. Please see examples of our input and processed images in the supplementary material.

We also pre-calibrate our imaging system using a small checkerboard target to obtain the intrinsic and extrinsic parameters of virtual cameras. The extrinsic parameters are re-calculated after view re-projection. The calibration process only needs to run once when our catadioptric lens is fixed on the camera. Pre-calibrating these parameters greatly benefit miniature scene reconstruction, since these tiny scenes often lack textures, and this would result in self-calibration methods (e.g., COLMAP Schönberger and Frahm 2016) to fail. With the camera parameters, we estimate an initial point cloud using structure from motion. We then use the eight multi-view images, along with the camera parameters and the initial point cloud as input for 3DGS-based scene reconstruction.

3DGS, first introduced by Kerbl et al. Kerbl et al. 2023, represents a scene using a set of 3D Gaussian elements. It is a more compact and efficient 3D representation, comparing to NeRF Mildenhall et al. 2020 and classical triangle meshes. It has been demonstrated great success in many rendering applications Tang et al. 2024; Zhou et al. 2024; Kocabas et al. 2024.

A 3D Gaussian is defined by its center position $\boldsymbol{\mu}\in\mathbb{R}^{3}$ and a covariance matrix $\Sigma\in\mathbb{R}^{3\times 3}$:

where $\boldsymbol{x}$ is a point on the Gaussian, and $\Sigma$ can be further decomposed into rotation and scaling matrices ($R$ and $S$ respectively): $\Sigma=RSS^{\top}R^{\top}$.

To render an image, the 3D Gaussians are projected by: $\Sigma^{\prime}=JW\Sigma W^{\top}J^{\top}$, where $W$ is the viewing transformation matrix and $J$ is the Jacobian of the affine approximation of the projective transformation. A pixel’s color $\mathbf{C}$ is calculated by blending the color of ordered Gaussians that overlap the pixel, similar to the NeRF-style rendering:

where $c_{i}$ is the color of the Gaussian, $\alpha_{i}$ is the opacity of the 2D projected Gaussian and $T_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j})$ is the transmittance along the ray.

To optimize the Gaussian parameters, we compare the rendered output against our input reference views using the following loss function:

where $\lambda_{1,2,3}$ are weighting factors for balancing the terms. We use $\lambda_{1}=0.8$, $\lambda_{1}=0.2$ and $\lambda_{3}=0.5$ in our experiments. $\mathcal{L}_{1}$ and $\mathcal{L}_{\text{D-SSIM}}$ are standard losses that evaluate the color similarity between the rendered image and the reference image. Since our input views are sparse with limited overlapping, using the color losses alone results in severe artifacts when synthesizing unseen views. We therefore add the third loss $\mathcal{L}_{\text{depth}}$ (which will be described in the following paragraph), a depth loss constrained by visual hull-generated depth maps, to improve the rendering quality of novel views.

We use the rasterizer provided by Kerbl et al. 2024 to render depth map $\mathbf{D}_{\text{render}}$ from 3D Gaussians. Specifically, the depth is rendered by replacing the color $c_{i}$ in Eq. 5 with the depth $d_{i}$ of the Gaussian’s center: $\mathbf{D}_{\text{render}}=\sum_{i=1}^{n}d_{i}\alpha_{i}T_{i}$. Since $\mathbf{D}_{\text{render}}$ and $\mathbf{D}_{\text{VH}}$ have consistent scale, we calculate the depth loss $\mathcal{L}_{\text{depth}}$ as a weighted L1 norm:

where $\Delta d_{i}=\mathbf{D}_{\text{render}}(\boldsymbol{p}_{i})-\mathbf{D}_{\text{VH}}(\boldsymbol{p}_{i})$, with $\boldsymbol{p}_{i}$ refers to a pixel in image.

We also order a customized mirror lens with sharping imaging quality from a lens maker (see Fig. 7 (d)). We use this lens for capturing real scene data. We use the same 5 megapixel FLIR camera with 12mm lens for image acquisition. Since the lens is compact and portable, we capture miniature scenes from both indoor and outdoor. In total, we capture 10 indoor scenes and 2 outdoor scenes. The size of miniature objects in our scene is in the range of 1cm to 5cm.

We pre-calibrate our imaging system to obtain intrinsic and extrinsic parameters for the virtual cameras. Since mirror reflection doesn’t change intrinsic parameters, we first estimate the intrinsics with all available views and use the same set of parameters for all virtual cameras. We then calibrate the extrinsics after mounting the mirror lens. We also run bundle adjustment to further refine the camera poses. The final re-projection error of our calibration is 0.77.

We validate our approach on our simulated data, and compare our results with recent state-of-the-art 3DGS algorithms: Hierarchical 3DGS Kerbl et al. 2024, FSGS Zhu et al. 2024, DNGaussian Li et al. 2024 and SparseGS Xiong et al. 2023. Most of these methods are optimized for taking sparse view input. It is worth noting that since our scenes are small and lack distinct features, COLMAP fails to run on all of our scenes. For fair comparison, we use our pre-calibrated camera parameters and initial point cloud as input for all compared algorithms. For each scene, we use 8 reference views as input and render 24 novel views. Visual comparison results on five scenes are shown in Fig. 8. Quantitative comparison results using standard metrics (e.g., SSIM, PSNR and LPIPS) are shown in Table 1. The metric values are averaged over all scenes and all rendered novel views.

We can see that our method outperforms the state-of-the-arts in both qualitative and quantitative comparisons. The other sparse view methods do not perform very well. It is likely because that the monocular depth map they use is less accurate than our visual hull depth (see supplementary material for an ablation study on depth loss).

We perform comparisons on real scene images taken with our mirror lens. Fig. 9 show visual comparison results against the state-of-the-arts. We also show our captured image (processed with the multi-view mask) and one re-projected virtual view, which is used as input reference image for 3DGS (we use eight reference views in total). All the scenes are composed with miniature objects with size between 1cm and 4cm. The “tree berry” scene is captured outdoor with our portable lens. Same as the synthetic experiments, all compared methods take our pre-calibrated camera parameters and initial point cloud, since COLMAP fails on our scenes. The “camera man” scene is challenging with thin structures around 1mm (e.g., the tripod legs). Our method is able to recover the thin legs, whereas all other methods cannot.

Since camera parameters are critical, we also compare our method against a recent COLMAP-free method, InstantSplat Fan et al. 2024. The method bypasses COLMAP by jointly optimizing Gaussian attributes and camera parameters. Visual comparison of a synthesized novel view is shown in Fig. 10. We also show their estimated camera poses and our calibrated poses. Their rendering quality is poor likely because of inaccurate camera pose estimation.