EspressoDB: A scientific database for managing high-performance computing workflow

08 Dec 2019

J.Open Source Softw. 5 (2020) no.46, 2007 [arXiv:1912.03580]

Leadership computing facilities around the world support cutting-edge scientific research across a broad spectrum of disciplines including understanding climate change, combating opioid addiction, or simulating the decay of a neutron. While the increase in computational power has allowed scientists to better evaluate the underlying model, the size of these computational projects have grown to a point where a framework is desired to facilitate managing the workflow. A typical scientific computing workflow includes: Defining all input parameters for every step of the computation; Defining dependencies of computational tasks; Storing some of the output data; Post-processing these data files; Performing data analysis on output. EspressoDB is a programmatic object-relational data management framework implemented in Python and based on the Django web framework. EspressoDB was developed to streamline data management workflows, centralize and guarantee data integrity, while providing domain flexibility and ease of use. The framework provided by EspressoDB aims to support the ever increasing complexity of workflows of scientific computing at leadership computing facilities, with the goal of reducing the amount of human time required to manage the jobs, thus giving scientists more time to focus on science.