论文标题
连续使用样品的稳定和半稳定的采样方法
Stable and Semi-stable Sampling Approaches for Continuously Used Samples
论文作者
论文摘要
信息检索系统通常是通过标记与用户查询样本相对应的结果的相关性来衡量的。在实用的搜索引擎中，需要连续执行此类测量，例如每日或每周。这在（a）查询样本到产品当前查询流量之间创造了权衡； （b）标签成本：如果我们保留相同的查询样本，结果将是相似的，使我们可以重复使用它们的标签； （c）不断使用同一查询样品引起的过度拟合。在本文中，我们明确地制定了这种权衡，提出了两个新的变体（稳定且半稳定），以简单而加权的随机抽样，并表明它们在连续使用设置上都优于现有方法，包括监视/调试搜索引擎或比较排名者候选者。
Information retrieval systems are usually measured by labeling the relevance of results corresponding to a sample of user queries. In practical search engines, such measurement needs to be performed continuously, such as daily or weekly. This creates a trade-off between (a) representativeness of query sample to current query traffic of the product; (b) labeling cost: if we keep the same query sample, results would be similar allowing us to reuse their labels; and (c) overfitting caused by continuous usage of same query sample. In this paper we explicitly formulate this tradeoff, propose two new variants -- Stable and Semi-stable -- to simple and weighted random sampling and show that they outperform existing approaches for the continuous usage settings, including monitoring/debugging search engine or comparing ranker candidates.
