Continued Integration and Delivery for AWP-ODC-OS in the public Cloud

Yifeng Cui, Alexander N. Breuer, & David Lenz

Published August 15, 2019, SCEC Contribution #9698, 2019 SCEC Annual Meeting Poster #300

Continuous Integration and Delivery (CI/CD) are state-of-the-art software development and engineering techniques that support the process of continuously testing small incremental changes of scientific software. We present the transformation of our CI/CD solutions with AWP-ODC-OS, aiming at other SCEC open source needs in the long run. Unlike our previous XML-based server and workflow configuration, which is challenging to understand for new users and does not generalize to other software products, we now use templatized YAML-configurations instead for increased human-readability. We utilize a light-weight access layer library for distributed computing infrastructure. A data repository, CC0'd to maximize availability, will use Git Large File Storage, which extends Git with pointers for large files. Moreover, this new setup enables commercial cloud solutions for maximum availability and flexibility of the executing hardware. The initial enabling of the AWP-ODC-OS setup will use AWS, while future developments also target other cloud providers. Our computational demands will be matched by using elastic agents on cost-efficient spot instances in AWS. Once the tasks are completed, the agents are destroyed, and the agents’ instances will be released back to the cloud. In summary, Cloud solutions are the best-available approach to tackle this amount of diversity. This continuous development work will provide improved capabilities of SCEC wave propagation community codes, towards automated continuous verification on supercomputing infrastructure.

Key Words
Continued Integration and Delivery, AWP-ODC, Cloud

Citation
Cui, Y., Breuer, A. N., & Lenz, D. (2019, 08). Continued Integration and Delivery for AWP-ODC-OS in the public Cloud. Poster Presentation at 2019 SCEC Annual Meeting.

Related Projects & Working Groups
Computational Science (CS)