论文标题
Jaket：知识图和语言理解的联合预培训
JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
论文作者
论文摘要
知识图（kgs）包含有关世界知识，实体和关系的丰富信息。因此，它们可以成为现有预训练的语言模型的重要补充。但是，有效地将信息从kg纳入语言建模仍然是一个挑战。对知识图的理解需要相关的上下文。我们提出了一个新颖的联合培训框架Jaket，以模拟知识图和语言。知识模块和语言模块提供了相互协助的基本信息：知识模块在文本中为实体生成嵌入，而语言模块为图表中的实体和关系生成了上下文感知的初始嵌入。我们的设计使预训练的模型可以轻松适应新领域中看不见的知识图。几个知识感知的NLP任务的实验结果表明，我们提出的框架通过有效利用语言理解中的知识来实现卓越的表现。
Knowledge graphs (KGs) contain rich information about world knowledge, entities and relations. Thus, they can be great supplements to existing pre-trained language models. However, it remains a challenge to efficiently integrate information from KG into language modeling. And the understanding of a knowledge graph requires related context. We propose a novel joint pre-training framework, JAKET, to model both the knowledge graph and language. The knowledge module and language module provide essential information to mutually assist each other: the knowledge module produces embeddings for entities in text while the language module generates context-aware initial embeddings for entities and relations in the graph. Our design enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains. Experimental results on several knowledge-aware NLP tasks show that our proposed framework achieves superior performance by effectively leveraging knowledge in language understanding.
