Rafael, thank you for talking to us. Please, tell us about the SC22 reproducibility initiative – what are its goals and how is it structured? 

Reproducibility is the ability to repeat the research experiments that appear as the validation of a scientific paper. When we can reproduce the experiments, we can confirm the validity of a scientific discovery. Therefore, reproducibility is the foundation of solid and trustworthy scientific and technical research. Furthermore, fostering reproducibility facilitates dissemination, comparison, and adoption of research. Therefore, it can also contribute to increasing the impact and enhancing the visibility of papers.

From my point of view, technology in the past did not allow researchers to make reproducibility easy, requiring titanic amounts of effort and time. However, recent advancements, such as the widespread usage of online software and data repositories (such as github, dataPort or Dryad), or virtualization and packaging technologies such as virtual machines or containers have increased the opportunities for enabling reproducibility and, in consequence, many computer science conferences are already considering it. 

During the last few years, SC has been leading the reproducibility of experiments in computer science, not only from a fantastic technical perspective, but also from its commitment. The SC reproducibility initiative started in 2015, and since then it has been developing continuously. A significant milestone happened in 2019, since then, reproducibility is mandatory for all SC submissions. It is very interesting to see the evolution of the SC reproducibility initiative across the SC websites (there is also a summary each year at the SC reproducibility main website)

On the other hand, the conference has to provide new committees to handle the review of the reproducibility code and data artifacts and it requires an additional effort. In parallel with the Program Committee, in SC there is a Committee of Reproducibility, with different subareas, which include the review of reproducibility artifacts, the reproducibility challenge, and a special issue for reproducibility. 

The reviewing process is similar to that of traditional research papers, but reviewers assess whether they can reproduce the experiments as depicted in the paper. As an output of the process, each paper can receive a badge that quantifies its degree of reproducibility, such as, no code available, code available or results reproduced. Besides, the reproducibility challenge consists of selecting a paper accepted at the previous conference to be used as the source of the Reproducibility Challenge in the Student Cluster Competition (SCC). Then, the critique reproducibility reports obtained are compiled and submitted as a special issue to a journal (typically IEEE TPDS).

What was your role in the reproducibility initiative? 

In 2022, I was involved in the reproducibility initiative of the IEEE TPDS journal, which is one of the first IEEE journals piloting a reproducibility initiative. Along with the editor-in-chief, Prof. Manish Parashar, we believed that it could be interesting to get involved inside the SC reproducibility initiative to learn from their knowhow. Therefore, I participated in the SC22 reproducibility initiative as a reviewer and the experience was absolutely fascinating. 

At SC22, each reviewer handled 1 or 2 software artifacts and the time required to reproduce the experiment varies depending on the type of artifact / experiment and its quality. If the artifact is provided with deployment / installation / execution scripts and it is very well documented, the reviewer may not find many barriers, and it is just letting the machine(s) execute the code. Sometimes, it can be really challenging and an anonymous interaction between the reviewers and the authors is required.

Based on my experience at SC22, I believe that a reproducibility reviewer faces two main challenges. First, to understand the reproducibility artifact description and, then, to obtain the computing resources, to configure them and to execute the experiments on the resources. The problem is that, as researchers, we learn how to structure and write papers as there is a standardized agreement about that in the community. However, reproducibility is something novel and both authors and reviewers need to deal with it. Currently, we are working from the TPDS journal and the SC23 and ICPP23 conferences to standardize the artifact descriptions

On the other hand, another aspect is the computing resources required, their access and their configuration. On this matter, SC is also absolutely fascinating because of the variety of experiment types that can be found, requiring different software and hardware dependencies. For example, one can find from a single computer as a requirement, or a specific supercomputer, or thousands of computing nodes. The configuration and management of software dependencies is another challenge. SC was aware of that and aimed their authors to wrap their artifacts into a container specification. However, not all the experiments can be wrapped in a container specification and still the configuration setup of the computing infrastructure to execute the artifact can be challenging. Again, I believe that a standard reproducibility platform, where authors can prepare all or some of their experiments so that reviewers and eventually the research community can reproduce them could be a solution. I can see that Chameleon Cloud has been working towards that direction.

How do you use Chameleon in this effort and what was your experience? 

One of the artifacts I was reviewing at SC22 required a server with a specific GPU and a version of its OS and compilers. That’s quite difficult to find, unless you have full control of that machine in your lab. Then, I asked the SC reproducibility chair for help, to grant me access to a computing infrastructure and that is when I used Chameleon. I used the GUI interface to look for the hardware and reserve the machine. Then, I requested the installation of the OS and the network and once I had access to it, I executed the artifact directly in the bare metal machine with the command line interface via ssh. Although I did not require a complex experimental set up, I could observe that it can be easily done with current functionality provided in Chameleon.

My experience with Chameleon was absolutely great. I was impressed by the ease of usage, starting from the logging in Chameleon, to the usage of the catalog or the access to the machines. 

What features of the testbed did you need in order to reproduce the evaluated experiments? 

My assignments at SC just required a single computer to be executed. However, I could validate and review the experiments of the server with GPU thanks to Chameleon giving me access to the hardware, and then letting me install the required OS and compilers on it. Furthermore,  looking at other submissions, SC really requires many features that Chameleon can provide, such as Bring Your Own Device (BYOD), when authors have made use of very specific hardware that is very difficult to obtain, otherwise.

Can you point us to artifacts connected with your work that would be of interest to our readership? 

I participated in the first artifact approved by the IEEE TPDS journal, a joint collaboration with Prof. Omer Rana and Prof. Manish Parashar. It can be found here. The artifact is about a reactive resource manager controller for streaming applications. At that time, IEEE was promoting the Codeocean platform, and we used it as the experimental platform. Codeocean is a very interesting reproducibility platform, but, unfortunately, it only supports containerized applications and that can be an important limitation for parallel and distributed systems. The code in Codeocean is a modified version of the one for the paper to let the controller work inside a container. Another more recent code artifact we developed can be found here. This is a simulator of a cloud / edge scenario.

Both artifacts follow the artifact description guidelines that are now at IEEE TPDS, SC23 and ICPP23.

Tell us a little bit about yourself

I am currently associate Professor in the Department of Informatics of the University of Zaragoza, Spain. I have been teaching in the same university for the last 17 years, mostly introduction to programming courses, but, more recently, I am in charge of concurrent programming and distributed systems courses in undergraduate degrees. 

I am also associate editor in chief for reproducibility at IEEE TPDS, and reproducibility co-chair in SC23 and ICPP23. I am really enjoying my current tenure position as a civil servant, because it allows me to assume more risks with research ideas and project proposals, with less bad consequences than in the past. Sometimes, the pressure for publishing of the system can bring more continuist research and “less adventurous” researchers. 

Regarding my hobbies, I love music and nature, and my little girl. In the past, my spare time was for music and nature, now, it is only for her. I hope when she grows up a little more we can enjoy my hobbies together, if not, I can always find some other hobbies.

What is the most powerful piece of advice for students beginning research or finding a new research project?

I particularly like “Labor omnia vincit”, which means something like “work always brings you to success”. We can say just keep on trying and making the effort to do our best. To find research ideas, I think it is key to be very bored. A busy brain is not a good idea for freshness. 

How do you stay motivated through a long research project?

Sometimes it is good to stop and have a break. It works for me to completely change the activity, for instance, to do sports, to read a book, to travel, or to spend good quality time with family and friends.

What has been a tough moment for you either in your life or throughout your career? How did you deal with it? How did it influence your future work direction?

It is fair to acknowledge and assume that learning is always a difficult process, especially during the PhD. It is good to remember that we need to enjoy every moment, tough moments will pass and good moments will come.

What drives your interest in reproducibility and how do you think it can change the way science is done?

I find it fascinating. On one hand, we really owe the society quality research that can be trusted. On the other hand, I believe that it can really boost the productivity of science globally, as it can foster the comparison and the integration of techniques and mechanisms. It is also another way of being organized with our own work.

Thank you for taking the time to talk to us and thank you for sharing your experiences with the Chameleon community!