论文标题
探测语言系统性
Probing Linguistic Systematicity
论文作者
论文摘要
最近，人们对深层自然语言理解模型是否表现出系统性的问题引起了很多兴趣。概括这样的单元，像单词这样的单元对它们出现的句子的含义做出了一致的贡献。有积极的证据表明，神经模型通常在系统上概括。我们从语言角度研究了系统性的概念，定义了一组探针和一组指标来衡量系统行为。我们还确定了网络体系结构可以非系统性化的方式，并讨论为什么这种形式的概括可能不满意。作为一个案例研究，我们在自然语言推理（NLI）的设置中进行了一系列实验，表明尽管非系统性化，但某些NLU系统仍取得了高度的整体性能。
Recently, there has been much interest in the question of whether deep natural language understanding models exhibit systematicity; generalizing such that units like words make consistent contributions to the meaning of the sentences in which they appear. There is accumulating evidence that neural models often generalize non-systematically. We examined the notion of systematicity from a linguistic perspective, defining a set of probes and a set of metrics to measure systematic behaviour. We also identified ways in which network architectures can generalize non-systematically, and discuss why such forms of generalization may be unsatisfying. As a case study, we performed a series of experiments in the setting of natural language inference (NLI), demonstrating that some NLU systems achieve high overall performance despite being non-systematic.
