论文标题
在LSST管道执行中添加工作流管理灵活性
Adding Workflow Management Flexibility to LSST Pipelines Execution
论文作者
论文摘要
数据处理管道需要以尺度执行，从大型运行到大型生产数据发布运行，从而产生数百万个数据产品。作为鲁宾天文台管道执行系统的一部分，BPS是抽象层，为不同的工作流管理系统（WMS）（例如HTCONDOR和PANDA）提供接口。在提交过程中，管道执行系统与数据管家进行交互，以从算法任务中产生面向科学的执行图。 BPS将此执行图转换为工作流程图，然后使用WMS特定的插件提交和管理工作流程。在这里，我们将讨论该界面的架构设计，并简要报告数据预览0.2发行版的最新生产以及管道开发人员如何使用该系统。
Data processing pipelines need to be executed at scales ranging from small runs up through large production data release runs resulting in millions of data products. As part of the Rubin Observatory's pipeline execution system, BPS is the abstraction layer that provides an interface to different Workflow Management Systems (WMS) such as HTCondor and PanDA. During the submission process, the pipeline execution system interacts with the Data Butler to produce a science-oriented execution graph from algorithmic tasks. BPS converts this execution graph to a workflow graph and then uses a WMS-specific plugin to submit and manage the workflow. Here we will discuss the architectural design of this interface and report briefly on the recent production of the Data Preview 0.2 release and how the system is used by pipeline developers.
