Build an MLOps Pipeline

In Cloud Computing on Chameleon, we deployed a basic machine learning service to an OpenStack cloud. However, that deployment involved a lot of manual steps ("ClickOps"), and any updates to it would similarly involve lots of manual effort, be difficult to track, etc.

In this tutorial, we will learn how to automate both the initial deployment, and updates during the lifecycle of the application. We will:

  • practice deploying systems following infrastructure-as-code and configuration-as-code principles using automated deployment tools
  • and create an automated pipeline to manage a machine learning model through its lifecycle

Our experiment will use the following automated deployment and lifecycle management tools:

  • Terraform: A declarative Infrastructure as Code (IaC) tool used to provision and manage cloud infrastructure (servers, networks, etc.) by defining the desired end state in configuration files. Here, we use it to provision our infrastructure.
  • Ansible: An imperative Configuration as Code (CaC) tool that automates system configuration, software installation, and application deployment through task-based YAML playbooks describing the steps to achieve a desired setup. Here, we use it to install Kubernetes and the Argo tools on our infrastructure after it is provisioned
  • Argo CD: A declarative GitOps continuous delivery tool for Kubernetes that automatically syncs and deploys applications based on the desired state stored in Git repositories.
  • Argo Workflows: A Kubernetes-native workflow engine where you define workflows, which execute tasks inside containers to run pipelines, jobs, or automation processes.

To run this experiment, you should have already created an account on Chameleon, and become part of a project. You should also have added your SSH key to the KVM@TACC site.

This tutorial uses: three m1.medium VMs at KVM@TACC, and one floating IP.


This material is based upon work supported by the National Science Foundation under Grant No. 2230079.

257 54 50 6 Apr. 25, 2025, 10:19 PM

Authors

Launch on Chameleon

Launching this artifact will open it within Chameleon’s shared Jupyter experiment environment, which is accessible to all Chameleon users with an active allocation.

Download Archive

Download an archive containing the files of this artifact.

Download with git

Clone the git repository for this artifact, and checkout the version's commit

git clone https://github.com/teaching-on-testbeds/mlops-chi.git
# cd into the created directory
git checkout f7d95d3a14dad3aa567b8347b5f4766b5042d054
Feedback

Submit feedback through GitHub issues

Version Stats

61 24 22