We develop a framework for task-specific active next-best-view selection in 3D reconstruction from point clouds, by casting the problem in the language of Bayesian decision theory. Our framework works by (a) placing a prior distribution over the space of implicit surfaces, (b) using recently-developed stochastic surface reconstruction methods to calculate the resulting posterior distribution, then (c) using the posterior distribution to carefully reason about which view to scan next. This enables us to perform camera selection in a manner that is directly optimized for the intended use of the reconstructed data – meaning, we reduce uncertainty only in those regions that make a difference in the task at hand, as opposed to prior approaches that reduce it uniformly across space. We evaluate our method across three distinct downstream tasks: semantic classification, segmentation, and PDE-guided physics simulation. Experimental results demonstrate that our framework achieves superior task performance with fewer views compared to commonly used baselines and prior general uncertainty-reduction techniques.

Method

Task Description

Let $f : [0,1]^3 \to [-1,1]$ be an implicit surface representation of the unknown scene. We assume that information about $f$ is acquired through a scanning operation $\mathrm{scan}(\cdot;f) : SE(3) \to \mathbb{D}$, where $\theta\in\mathrm{SE}(3)$ represents camera angles, and $\mathbb{D}$ is a space representing oriented point clouds. More precisely, $(\boldsymbol{x},\boldsymbol{n}) \in \mathbb{D}$ consists of a dataset of surface locations $\boldsymbol{x}$, and surface normals $\boldsymbol{n}$, of matched but otherwise potentially variable length. We assume the unknown geometry $f$ is only accessible in a black-box manner through these discrete observations, to mimic 3D scanning applications in which no prior knowledge is assumed about the object.

In a sequential scanning setting, at each time step $t$, we have a set of previously selected viewpoints $\theta_\tau$, $\tau=1,..,t$ and the resulting cumulative dataset $\mathcal{D}_t = \bigcup_{\tau=1}^t \mathrm{scan}(\theta_\tau;f)$, where the union symbol denotes concatenation. Our objective is to carefully select a subsequent viewpoint $\theta_{t+1}$.

To introduce a Bayesian setting, we will place a prior distribution $p(f)$ over the implicit surface, obtaining a posterior $p(f \mid \mathcal{D}_t)$. We extend the scanning operation to also allow simulated scans using the posterior distribution, denoted by $\mathrm{scan}(\cdot;f\mid\mathcal{D}_t)$.

Algorithm Overview

With formalizing active point cloud scanning as a Bayesian decision problem, we design our algorithm as the following steps:

1. Given the partially scanned point cloud $\mathcal{D}_t$ at time step $t$, we obtain the posterior geometry reconstruction $p(f \mid \mathcal{D}_t)$ through uncertain geometry reconstruction methods, specifically, we use Stochastic Poisson Surface Reconstruction¹.
2. We define a utility function $u : \mathbb{D} \to \mathbb{R}$, which is task-dependent: describing how useful the information contained in a partial scan is for the task at hand.
3. By conducting the simulated scan using the posterior, we select the next view according to the expected utility improvement acquisition function that which scores the potential value of a candidate camera $\theta$, and is defined as

$$\alpha^{(u)}_{f\mid\mathcal{D}_t}(\theta) = \mathbb{E} \max{\left(0, u(\mathcal{D}_t \cup \mathrm{scan}(\theta; f\mid\mathcal{D}_t))- u(\mathcal{D}_t)\right)} .$$

Using it, we select the next view by solving the optimization problem

$$\theta_{t+1} = \mathop{\arg\max}_{\theta \in \Theta} \alpha^{(u)}_{f\mid\mathcal{D}_t}(\theta) .$$

In practice, the expectation is estimated by a Monte Carlo estimation from multiple samples from the posterior distribution $\hat{f} \sim p(f \mid \mathcal{D}_t)$. Please refer to Algorithm 1 in our paper for the pseudocode.

Task-specific Utility Functions

We provide a general Bayesian next-best-view selection framework that allows for the tailoring of camera selection to the specific requirements of the downstream application, through the selection of utility and acquisition functions. Our paper handlels the following as examples:

1. 3D point cloud classification, to identify the correct class label of the underlying object with the minimum number of viewpoints.
2. 3D Semantic Segmentation and Part Discovery, to discover all semantic parts of the underlying object with minimal observations.
3. Physics-Informed Scanning, driven by the physical properties of the reconstructed scene, e.g., heat diffusion.
4. Traditional task-agnostic 3D reconstruction objective, which typically aims for scene coverage and global scene uncertainty reduction.

Results

Classification

Comparison between our method and baseline methods in the classification task. Our approach achieves a more precise classification result with fewer views:

Segmentation (Part Discovery)

Comparison between our method and furthest point sampling (FPS) in the part discovery task:

Heat Diffusion

Comparison between our method and furthest point sampling (FPS) in the heat diffusion task. With the same number (4) of cameras, our approach achieves a better heat simulation fidelity than the baseline:

Citation

@article{zhu2026bayesian,
  title={A Bayesian Approach for Task-Specific Next-Best-View Selection with Uncertain Geometry},
  author={Zhu, Jingsen and Sell{\'a}n, Silvia and Terenin, Alexander},
  journal={arXiv preprint arXiv:2605.05095},
  year={2026}
}

Reference