Experiments are a cornerstone of modern science. Through experiments, we discover and experience the properties of our world. Results from experiments enable researchers to reason about what is. Nevertheless, for this reasoning to be sound, solid data and, thus, experiments are essential. One way to ensure valid data is repeating experiments multiple times: if results are repeatable, we can rely on them. However, relying on the results of other researchers requires trust. And building trust is a challenging endeavor. Reproducible experiments are an approach to ensure trust in experimental results. If we can reproduce the results of fellow researchers in their setup, there is no need to just trust the words of the researchers; we can verify the results by comparison! Despite a globalized world, accessing others’ experimental setups is not always feasible. Thus, researchers started to replicate experiments and results in their setups. Depending on the research domain, repeating, reproducing, or replicating experiments can be challenging. A common challenge is the influence of allegedly immaterial factors, e.g., the exact CPU type for an experiment focusing on networking behavior. The goal of our work is to assist researchers in creating replicable experiments. For this, we extend the researcher's toolkit with the Plain Orchestrating System (pos), both a tool and methodology, designed for this purpose.

Our Research Findings

As introduced, our work aims to assist researchers in creating replicable experiments. Creating replicable experiments is tedious, as experiments tend to have many hidden factors. In previous work, we introduced pos, a tool and a methodology helping to create reproducible experiments. In other words, pos describes a sequence of steps and guidelines to follow and adhering to this workflow leads to reproducible experiments.

To validate this claim, we implemented a tool that follows this methodology and is used at our labs to conduct experiments. This situation, however, raises two related research questions:

• (R1) How can we, as a community, move from reproducible to replicable experiments?
• (R2) How can we best enable external research to conduct pos experiments?

Figure 1: Diagram of the pos methodology.

Given the availability of public research infrastructure, such as Chameleon and CloudLab, we decided to integrate the pos methodology into these existing environments. As depicted in Figure 1, we distinguish a hosting testbed providing crucial resources from a hosted testbed focusing on executing structured experiments. With this approach, researchers reserve infrastructure in one of the supported testbeds, deploy pos as a testbed environment onto their infrastructure, and, ultimately, conduct their experiments inside pos. This approach enabled researchers, regardless of their access to private infrastructure, to conduct pos experiments and, hence, fulfill (R2). Moreover, as the conducted pos experiments can be performed in either Chameleon, CloudLab, or local deployments, published experiments can be tested for replicability by other researchers. As a result, this solution solves (R1), enabling any researcher to package an experiment to be replicable, i.e., understandable, by all.

Running the Experiment on Chameleon

To implement and test our approach, we rely on several features provided by Chameleon. Our work includes an example experiment to highlight that no changes to the experiment are required when moving across the infrastructure. This experiment uses Trovi to share experiments easily and the Jupyter Interface provided by Chameleon for seamless interaction with the experiment environment. Additionally, we used the virtual machines available through KVM@TACC to evaluate our approach. There, we used the Python OpenStack API (python-chi) to automate interactions, such as host creation and configuration.

The extension to pos made as part of this work interacts with Chameleon. For this interaction, we rely on the availability of an API to express our intent. Here, we found the OpenStack API provided by Chameleon particularly helpful. Though OpenStack is a well-known and proven software, to our knowledge, Chameleon is the sole testbed employing it. Yet, the mentioned OpenStack properties make for a pleasant user experience. A description that also fits the Jupyter Interface was used, as mentioned, to structure our sample experiment. Though other testbeds allow installing Jupyter instances, from our experience, only a few do. While not integrated into this work, we are interested in the long-term durability of experiments and reducing the overhead associated with replicating them. Trovi provides a valuable approach that we look forward to integrating into future work.

Experiment Artifacts

Figure 2: Slide from a presentation on pos

Our paper on the pos tool and methodology is available here.

Digital artifacts are available on both GitHub and Chameleon Trovi.

The pos framework will become a component of the European SLICES-RI initiative. SLICES-RI is in the process of creating a European Research Infrastructure consisting of testbeds distributed across multiple sites. The implementation of pos’ portability feature will help creating experiments that can be run on testbeds like Chameleon or on testbeds part of SLICES-RI.

User Interview

Tell us a little bit about yourself.

Henning Stubbe received a B.Sc. in computer science from the University of Rostock, Germany, in 2015 and an M.Sc. degree in computer science from the Technical University of Munich (TUM), Germany, in 2018. He is currently pursuing a Ph.D. degree in computer science at TUM at the Chair of Network Architectures and Services with chairholder Prof. Dr.-Ing. Georg Carle. Since 2019, he has been a Research Associate with TUM at the chair mentioned above. His research interests include software-defined, programmable networking with a particular focus on high-performance benchmarking. Moreover, he investigates the creation of reproducible experiments and the development of the methodology and tools to make them portable between different testbeds.

Sebastian Gallenmüller received his Ph.D. in 2021 from the Technical University of Munich. He currently works as a PostDoc at the Chair of Network Architectures
and Services led by Prof. Georg Carle at the Technical University of Munich. His main research interests are programmable packet processing systems and testbeds for
network experiments with a focus on performance analysis and modeling of packet processing systems.

Georg Carle is professor at the Technical University of Munich, holding the Chair of Network Architectures and Services. He studied at University of Stuttgart, Brunel
University, London, and Ecole Nationale Superieure des Telecommunications, Paris. He did his Ph.D. in Computer Science at University of Karlsruhe, and worked as postdoctoral scientist at Institut Eurecom, Sophia Antipolis, France, at the Fraunhofer Institute for Open Communication Systems, Berlin, and as professor at the University of Tübingen.

How do you stay motivated through a long research project?

As with food, too much of the same stops tasting good. Variety helps. Or a break.

Why did you choose this direction of research?

“One cannot not communicate” stated Paul Watzlawick. As a computer scientist with a networking background, the goal is to assist people in doing so.

Have you ever been in a situation where you couldn't do an experiment that you wanted to do? What prevented you from doing it?

Once in a while experiment attempts are impeded by bottlenecks in the experiment setup. For example, calculations requiring more compute resources than are available. Luckily projects such as Chameleon now enable switching to more powerful machines. However, I find it important to now and then take a step back and reconsider if changing the approach would suffice as well.

Thank you for sharing your knowledge with the Chameleon community!