论文标题
RKHS功能空间的分析Kullback-Leibler Divergence估计
Analysis of Discriminator in RKHS Function Space for Kullback-Leibler Divergence Estimation
论文作者
论文摘要
已经提出了两种基于样本的方法来计算两个分布之间的Kullback Leibler（KL）差异，并在大型机器学习模型中应用并应用。尽管发现它们是不稳定的，但问题的理论根本原因尚不清楚。在本文中，我们研究了一种基于生成对抗网络的方法，该方法使用神经网络歧视者来估计KL差异。我们认为，在这种情况下，估计值中的高波动是不控制鉴别函数空间的复杂性的结果。我们通过在繁殖内核希尔伯特空间（RKHS）中首先构建歧视者，为此问题提供理论的基础和补救措施。这使我们能够利用样本复杂性和平均嵌入理论上将KL估计的误差概率结合到RKHS中的歧视器的复杂性。基于这一理论，我们提出了一种可扩展的方法来控制鉴别因子的复杂性，以可靠地估计KL差异。我们支持我们提出的理论和方法，以通过受控实验来控制RKHS歧视者的复杂性。
Several scalable sample-based methods to compute the Kullback Leibler (KL) divergence between two distributions have been proposed and applied in large-scale machine learning models. While they have been found to be unstable, the theoretical root cause of the problem is not clear. In this paper, we study a generative adversarial network based approach that uses a neural network discriminator to estimate KL divergence. We argue that, in such case, high fluctuations in the estimates are a consequence of not controlling the complexity of the discriminator function space. We provide a theoretical underpinning and remedy for this problem by first constructing a discriminator in the Reproducing Kernel Hilbert Space (RKHS). This enables us to leverage sample complexity and mean embedding to theoretically relate the error probability bound of the KL estimates to the complexity of the discriminator in RKHS. Based on this theory, we then present a scalable way to control the complexity of the discriminator for a reliable estimation of KL divergence. We support both our proposed theory and method to control the complexity of the RKHS discriminator through controlled experiments.
