Editor's note: Klaus Kraßnitzer presented this work at the Community Workshop on Practical Reproducibility in HPC, which we hosted just around the corner from SC24 in November 2024. We're delighted to share more details about this important contribution to reproducible science.

Reproducibility is the cornerstone of scientific research, yet it remains challenging in computational science. The AutoAppendix project takes a strategic approach to enhance artifact evaluation processes within the SC Conference Series by leveraging Chameleon's flexible infrastructure. This initiative recognizes high-performance computing as a pivotal domain where scientific rigor and reproducibility are paramount, aiming to streamline the often tedious process of reproducing computational research artifacts.

Research Overview

The primary aim of the AutoAppendix project is to conduct a comprehensive evaluation of Artifact Description/Artifact Evaluation (AD/AE) appendices submitted to SC24, particularly those utilizing Chameleon Cloud. By systematically testing these appendices, the project explores current capabilities and limitations in fostering reproducibility, identifying areas where automation can be introduced or enhanced.

Through evaluating multiple appendices from the SC24 conference, we discovered common patterns and challenges in artifact reproduction. Many artifacts suffer from incomplete documentation, failed software installations, or missing experiment data. Even when experiments conclude successfully, it's often unclear what the expected results should be. Our research highlighted the value of containerization solutions (Docker, VirtualBox, Singularity) and breaking experiments into distinct, well-documented steps.

Experimental Implementation on Chameleon

Our experimental methodology involved provisioning various Chameleon nodes to evaluate a subset of SC24 submissions over a period of five weeks. We used hardware ranging from standard compute nodes to specialized GPU configurations based on each artifact's requirements. The evaluation process was labor-intensive, often requiring multiple attempts to successfully execute experiments, particularly those needing specialized hardware like NVIDIA A100 GPUs.

A notable discovery was that only one out of 45 submissions leveraged Chameleon's Jupyter interface and Trovi artifact sharing service, which enables one-click reproducibility. This significantly simplified the evaluation process compared to other submissions that required manual configuration. The stark contrast between this streamlined approach and the tedious setups of other artifacts highlighted the potential for improvement in the reproducibility workflow.

Based on our evaluations, we identified key factors that contributed to successful reproduction:

• Clear documentation of hardware and software requirements
• Automated dependency installation scripts
• Progress indicators for long-running operations
• Containerized environments to ensure consistent execution
• Well-structured experiment steps with expected outputs

Experiment Artifacts

Read the full AutoAppendix report here.

To address the identified challenges and promote better reproducibility practices, we developed three new Trovi templates that authors can use to automate their experiments:

1. Docker Template: Installs the Docker environment and NVIDIA container toolkit when a GPU is available, executing sample operations inside a container.
2. Nix Template: Installs the Nix package manager and launches a reproducible environment for computing tasks like generating Mandelbrot set visualizations.
3. Guix Template: Provisions a node with the Guix package manager installed, demonstrating reproducible artifact execution with a real-world example from a 2006 paper.

These templates all share a common structure with a central configuration file, making it easy for researchers to adapt them to their specific needs.

About the Author

Klaus Kraßnitzer is a Master’s student in the field of Computer Engineering at Vienna University of Technology (TU Wien). His research interests focus on Distributed and High Performance Computing as well as Computer-Aided Verification. Klaus's work with the AutoAppendix project during the UCSC Summer of Reproducibility 2024 builds upon his expertise in systems engineering and scientific computing methodologies.