Blog

  • Home / AI / How to Compare ML Experiment Tools to Fit Your Data Science Workflow

How to Compare ML Experiment Tools to Fit Your Data Science Workflow

  • October 25, 2021

There are many ML experiment tracking tools that can support different data science workflows. We offer a way to compare alternatives and make an informed choice. We highlight some of the most popular ML experiment tracking tools, including:

  1. What factors to use to compare tools
  2. The types of solutions available
  3. A comparison of the pros and cons of each tool
  4. Provide a useful summary table to share


The rise of experiment tracking in the data science workflow

There is an ever-expanding number of tools to track experiments. Research shows that when we are confronted with a large number of (good) options we tend to get overwhelmed. Choice overload is a real thing. Imagine walking into a supermarket wanting to buy a package of cookies. In the cookie aisle, there are fudge-covered, fat-free, double stuffed, mint-flavored – and those are only the Oreos! The simple task of buying cookies can become paralyzing.

Similarly, the confounding array of options to track experiments can lead to suboptimal decisions. Choice Overload makes it harder to make decisions, and we often end up procrastinating and pushing off decisions and even worse we are less satisfied with the choices we do make.

How to choose the best ML experiment tracking tool

If you are running experiments as part of your data science workflow – you already know how important it is. The iterative nature of experiment tracking requires that it be done in a structured fashion. Even for simpler projects, without proper tools, you’ll quickly get stuck with a mess.

The key question is which of the numerous options out there is best suited for you and your data science workflow. There are a lot of them, much like the cookies described above. We’ll start by covering some of the different factors and considerations when choosing between experiment tracking tools and then we’ll outline the pros and cons of each solution.

What factors to consider?

Based on our analysis we suggest comparing experiment tracking tools based on six key variables.

1. What will I be tracking

Many different factors have to be carefully collected, tracked and saved to get the same code and model to work again. Changing one input can lead to different results.

A non-exhaustive list of factors to track in your experiments might look something like this:

  1. Hyperparameters
  2. Models
  3. Code files
  4. Metrics
  5. Environment


Additionally, for some projects, you might track yet more information such as model weights, prediction distributions, model checkpoints, and hardware resources.

It’s important to check and make sure that the platform you choose covers and tracks all the different elements your data science workflow requires. This is arguably the most acute challenge in experiment tracking, namely saving all the data required and not missing important information.

2. Where is my data being saved

Different tools and platforms save data differently. Some are based on external servers and some are based on local folders with unstructured text files. The more automatic the better; the more manual logging required the more likely something will be forgotten or go wrong.

3. Visualizations

A key element to different tracking tools is their ability to represent the different experiments visually. A good visual representation will enable you to analyze and interpret results quicker. Additionally, the right graph can help communicate results to others, especially to stakeholders with a non-technical background. Visualization is an

4. Ease of Use

Convenience is an important factor that shouldn’t be overlooked. A certain tool might be really powerful and track every single piece of data but it might be a nightmare to use and overkill for any given data science workflow. Another part of this is the UI/UX, some people might place a premium on customizability while others might prioritize a clean and elegant interface.

5. Stability

Some tracking tools are designed to support huge enterprises that require stable and mature solutions that can be depended on. Other tools are newer and might offer more advanced features, but are potentially less stable than some of the established solutions. Different users might balance the amount of risk/reward they are willing to entertain.

6. Scale

The experiment tracking needs of a single user can’t be compared to those of a huge team working collaboratively with multiple data science workflows. A team might need all of their data to be stored in a single repository that is accessible to the whole team. Thus, everyone has one source of information and members of the team can see what others are working on and experimenting with. On the other hand, an individual working alone might not need these features and might prefer to store all the data locally.

What tool fits best in your data science workflow?

There are a lot of tools out there to help track experimentation with different features and approaches. Broadly speaking, you can focus on one of three main options.

1. Open Source

  1. Advantages: Free, customizable, support from community, open standards, avoids lock-In.
  2. Disadvantages:  Hard to “scale”. This can mean challenges sharing work as a team or long-term projects, usability concerns, lack of expert support.


2. Commercial

  1. Advantages: Good UI + UX, ease of use, stability, tailored support.
  2. Disadvantages: Pricey, limited customization, may lead to lock-in and dependency.


3. Platform-Specific

  1. Advantages:  Integrates well with the platform, simple.
  2. Disadvantages: Only works well with the matching platform.

Reviewing ML experiment tracking tools

MLflow

MLflow is an open-source platform that enables you to manage the entire ML lifecycle. Specifically, MLflow Tracking is an API and UI that allows you to log a model’s parameters, metrics, and even the model itself along with various other things during the model training/creation process. MLflow Tracking can be used in any environment and can log results both to local files or to a server. The UI can be used to view and compare results from numerous runs and different users.

Advantages

  1. As an open-source project – MLflow is highly customizable and can be made to fit your data science workflow.
  2. Built to work with any ML library, algorithm, deployment tool, or language.
  3. Easy to add MLflow to existing ML code.
  4. Has a very large and active community behind it and is very widely adopted in the industry.


Cautions:

The UI of MLflow could be improved and visualizations are more limited. This can make sharing information with non-technical stakeholders challenging.

TensorBoard

Tensorboard is Tensorflow’s visualization and tracking toolkit. Tensorboard is open-source and widely integrated with other tools and applications.

Advantages:

  1. A large library of pre-built tracking tools and easily integrates with many other platforms.
  2. Good visualizations help enable good information sharing.
  3. A large community that creates robust community support and problem-solving.

Cautions:

  1. Some may find it complex to use with a long learning curve.
  2. It may not scale well with large amounts of experiments. Slows down when using to view and track large-scale experimentation.
  3. Designed for single-user, local machine usage, not team usage.


ClearML

ClearML is an open-source platform that provides ML researchers with the tools to manage the entire ML lifecycle. ClearML is customizable and integrates with whatever tools a team is already using.

Advantages:

  1. Easy to add auto-logging into many libraries.
  2. Customizable UI that enables users to sort models by different metrics.

Cautions:

  1. Due to a large number of modifications that need to be run for auto-logging (ClearML replaces many built-in functions of other frameworks), the system may be comparatively fragile.
  2. Installing the open-source version on your servers is relatively complicated (compared to MLflow)


Kubeflow

Kubeflow is the machine learning toolkit for Kubernetes. It is an open-source framework based on the way Google runs TensorFlow internally.  Kubeflow is powerful and offers very detailed and accurate tracking. Kubeflow isn’t focused on experiment tracking but does have some tracking features.

Advantages:

  1. Kubeflow is a good fit for Kubernetes users.
  2. Highly scalable and offers great hyperparameter tuning.

Cautions:

  1. Requires adapters to operate and maintain the Kubernetes cluster.
  2. Assumes a high degree of competency with Kubernetes.
  3. Kubeflow is a tool for advanced teams of ML engineers but may be challenging and unintuitive for others.
  4. May have more limited features around experiment tracking compared to other solutions.


Summary

Tracking your experiments in an efficient and organized manner is crucial. Having to try and recreate a model from a couple of months ago that holds an important result for your project is a frustrating situation that can be avoided with some foresight. Having so many options of different experiment management tools and platforms that offer experiment tracking can be more a hindrance than a help. It can be hard to tell the difference between them and can keep us from making a decision and picking one. We hope that this post will help you to discover different experiment tracking tools and pick the one that fits your data science workflow best.


Happy Learning!

-Metacoder