论文标题
通过异构差异组件模型评估评估者间的可靠性：灵活的方法核算上下文变量
Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables
论文作者
论文摘要
评估者间的可靠性（IRR）是高质量评级和评估的先决条件，可能会受到上下文变量（例如评估者的性别，主要或经验）等上下文变量的影响。 IRR中这种异质性源的识别对于实施政策很重要，具有减少测量误差并通过关注最相关的亚组来增加IRR的潜力。在这项研究中，我们提出了一种通过直接对方差成分的差异进行建模，在由于协变量引起的异质性的情况下评估IRR的灵活方法。我们使用贝叶斯因子选择最佳性能模型，建议使用贝叶斯模型平均来作为获得IRR和方差成分估计的替代方法，从而使我们能够考虑模型不确定性。我们使用包容性贝叶斯因素考虑整个模型空间，以提供证据或反对因协变量引起的方差成分差异的证据。在模拟研究中，将所提出的方法与其他贝叶斯和常见主义方法进行了比较，我们在某些情况下证明了它的优势。最后，我们提供了赠款提案同行评审的真实数据示例，证明了该方法的有用性及其在更复杂设计的概括中的灵活性。
Inter-rater reliability (IRR), which is a prerequisite of high-quality ratings and assessments, may be affected by contextual variables such as the rater's or ratee's gender, major, or experience. Identification of such heterogeneity sources in IRR is important for implementation of policies with the potential to decrease measurement error and to increase IRR by focusing on the most relevant subgroups. In this study, we propose a flexible approach for assessing IRR in cases of heterogeneity due to covariates by directly modeling differences in variance components. We use Bayes factors to select the best performing model, and we suggest using Bayesian model-averaging as an alternative approach for obtaining IRR and variance component estimates, allowing us to account for model uncertainty. We use inclusion Bayes factors considering the whole model space to provide evidence for or against differences in variance components due to covariates. The proposed method is compared with other Bayesian and frequentist approaches in a simulation study, and we demonstrate its superiority in some situations. Finally, we provide real data examples from grant proposal peer-review, demonstrating the usefulness of this method and its flexibility in the generalization of more complex designs.
