This page is the home of the DEMAND MLOps Workgroup. Here we will collect useful resources.

Definition MLOps

The blogpost AI Engineering and MLOps: Building Production-Ready Machine Learning Systems gives our working definition of MLOps (also called AI Engineering) as “an end-to-end machine learning development process to design, build and manage reproducible, testable, and evolvable ML-powered software”. The blogpost Software Engineering for Machine Learning Applications explains the difference with software engineering for rule-based software. In recent work we updated the process from that blogpost to the following picture, indicating that a data engineering stream is also needed.

MLOps Tools

The blogpost A Toolbox for the Applied AI Engineer summarizes a book chapter (available on request) about software engineering for machine learning applications. It contains a bulleted lists of useful tools (methods, techniques, libraries, frameworks, software tools, etc.) for AI engineering. The post Testing Machine Learning Applications contains a more in-depth analysis of tools for testing ML applications.

Data Engineering for MLOps

In the context of the DEMAND project we will focus more on the data-side of MLOps. This is a fairly new topic as can be seen in two recent papers:

1. What About the Data? A Mapping Study on Data Engineering for AI Systems
2. Garbage in, Garbage out. A Mapping Study on Data Quality Engineering for AI Systems

What looks like a promising avenue is to further work out the tooling for so-called DataOps, see the paper by Raj et al. (2020) From Ad-Hoc Data Analytics to DataOps. See also https://lakefs.io/blog/the-state-of-data-engineering-2023/.