论文标题
稳定的视图合成
Stable View Synthesis
论文作者
论文摘要
我们提出稳定的视图合成（SVS）。给定一组源图像，描绘了从自由分布的观点来描绘场景，SVS综合了场景的新视图。该方法在通过结构 - 动作和多视图立体声计算的几何支架上运行。此3D支架上的每个点都与视图射线和相应的特征向量相关联，这些向量编码输入图像中此点的外观。 SV的核心是视图依赖性的地下特征聚合，其中处理每个3D点处的定向特征向量，以生成射线的新功能向量，该射线将此点映射到新的目标视图中。然后，从以这种方式为所有像素合成的特征的张量，通过卷积网络呈现目标视图。该方法由可区分的模块组成，并受到端到端训练。它支持每个点上源图像的空间变化的重要性加权和特征转换。由于在目标视图上表面特征聚集的平稳依赖性引起的空间和时间稳定性；以及依赖视图效应的综合，例如镜面反射。实验结果表明，在三种不同的现实世界数据集上，SVS优于最先进的视图综合方法，在自由观看的视频中实现了前所未有的现实主义级别的现实主义视频，以挑战大型大型场景。代码可从https://github.com/intel-isl/stableviewsynthesis获得
We present Stable View Synthesis (SVS). Given a set of source images depicting a scene from freely distributed viewpoints, SVS synthesizes new views of the scene. The method operates on a geometric scaffold computed via structure-from-motion and multi-view stereo. Each point on this 3D scaffold is associated with view rays and corresponding feature vectors that encode the appearance of this point in the input images. The core of SVS is view-dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view. The target view is then rendered by a convolutional network from a tensor of features synthesized in this way for all pixels. The method is composed of differentiable modules and is trained end-to-end. It supports spatially-varying view-dependent importance weighting and feature transformation of source images at each point; spatial and temporal stability due to the smooth dependence of on-surface feature aggregation on the target view; and synthesis of view-dependent effects such as specular reflection. Experimental results demonstrate that SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse real-world datasets, achieving unprecedented levels of realism in free-viewpoint video of challenging large-scale scenes. Code is available at https://github.com/intel-isl/StableViewSynthesis
