论文标题
有一个时间和地点超越图像的推理
There is a Time and Place for Reasoning Beyond the Image
论文作者
论文摘要
图像通常比人眼的像素更重要，因为我们可以推断，关联和理由与其他来源的上下文信息相关，以建立更完整的图像。例如，在图1中，我们可以找到一种方法来通过对标志，建筑物，人群等细分的理解来识别与图片相关的新闻文章。这种推理可以提供拍摄图像的时间和放置，这将有助于我们执行后续任务，例如自动故事情节构建，预期效果照片中图像源的校正以及在某些位置或时间上的图像聚类等上游处理。 在这项工作中，我们制定了这个问题，并介绍了Tara：一个带有16K图像的数据集，其相关新闻，时间和位置，自动从New York Times中自动提取，另外还有61k个示例作为WIT的遥远监督。除摘录外，我们提出了一个众包子集，我们认为可以找到图像的时空信息以进行评估目的。我们表明，最先进的联合模型和人类绩效之间存在$ 70 \％$的差距，我们提出的模型略微填补了这种模型，该模型使用细分市场的推理，激励了可以通过世界知识进行开放式推理的高级视觉联合模型。数据和代码可在https://github.com/zeyofu/tara上公开获取。
Images are often more significant than only the pixels to human eyes, as we can infer, associate, and reason with contextual information from other sources to establish a more complete picture. For example, in Figure 1, we can find a way to identify the news articles related to the picture through segment-wise understandings of the signs, the buildings, the crowds, and more. This reasoning could provide the time and place the image was taken, which will help us in subsequent tasks, such as automatic storyline construction, correction of image source in intended effect photographs, and upper-stream processing such as image clustering for certain location or time. In this work, we formulate this problem and introduce TARA: a dataset with 16k images with their associated news, time, and location, automatically extracted from New York Times, and an additional 61k examples as distant supervision from WIT. On top of the extractions, we present a crowdsourced subset in which we believe it is possible to find the images' spatio-temporal information for evaluation purpose. We show that there exists a $70\%$ gap between a state-of-the-art joint model and human performance, which is slightly filled by our proposed model that uses segment-wise reasoning, motivating higher-level vision-language joint models that can conduct open-ended reasoning with world knowledge. The data and code are publicly available at https://github.com/zeyofu/TARA.
