Issue #9 - Interpreting ML Models Is Your Superpower

This Week’s Tutorial

I have a confession to make.

Most machine learning models I've crafted were never intended for production deployments.

If you're unfamiliar with production deployments, here's an easy way to think about it.

Imagine you work for a company that sells automobile insurance. Your employer allows customers to purchase auto insurance via the company's website.

After a customer enters all the required information, the data is fed to a machine learning model that predicts whether the insurance application should be accepted or denied.

That's an example of ML production deployment.

While production deployments are all the rage on social media (e.g., posts about Machine Learning Engineers), here's the reality.

Most organizations have not deployed their own ML models and are not ready.

However, machine learning is still immensely valuable because it is a powerful tool for improving business processes - no production deployment is required.

Successful DIY data scientists use machine learning models to analyze processes and uncover hidden patterns and relationships that can be used for economic gain.

Here are some examples:

• Identifying likely donors for non-profit organizations (i.e., higher revenue).

• Predicting if a claim is fraudulent (i.e., lower costs).

• Analyzing cloud server usage for possible shutdowns (i.e., lower costs).

• Segmenting customers for targeted marketing (i.e., higher revenue and lower costs).

Consider the last example. While this comes from marketing, the pattern applies everywhere (e.g., healthcare, product management, HR, IT, customer service, etc.).

Here's the pattern:

1. Cluster analysis to find the groups (e.g., customer segments).

2. Machine learning predictive models to understand the groups.

3. Use the understanding to improve/change processes (e.g., new marketing campaigns).

BTW - My Cluster Analysis with Python and Introduction to Machine Learning​ online courses will teach you everything you need to implement this pattern yourself.

The critical step for using machine learning effectively for process improvement is interpreting the model in the context of business drivers.

You must know how to translate ML models into a story that business stakeholders can understand.

Make no mistake. This is a DIY data science superpower. A superpower you can learn.

Here's the thing, though.

While there is undoubtedly a technical aspect to interpreting ML models, that's the easy part. The hard part is interpreting models in such a way that it resonates with your business stakeholders.

So this is where we need to start - what are the characteristics of an effective ML model interpretation?

From the perspective of a business stakeholder, an effective ML model interpretation answers the question, "Why?":

• Why is this person a potential donor?

• Why is this claim potentially fraudulent?

• Why should we market to these customers and not others?

To business stakeholders, the model interpretation needs to provide explanations that they find compelling.

When it comes to ML models, here are some characteristics of explanations humans find compelling:

1️⃣ Useful ML models often rely on many features. This makes providing a complete explanation too complex for your business stakeholders.

A compelling explanation instead focuses on comparing two predictions for their differences. For example, "This claim is fraudulent because it was for an amount far higher than is typical for this type of claim."

2️⃣ Another aspect of this complexity is that limiting the number of factors (e.g., features) used in the explanation is best. Humans prefer 1-3 factors in good explanations.

For example, "This claim is fraudulent because it was for an amount far higher than typical for claims for this type of vehicle in this part of the country."

3️⃣ Humans love it when an explanation is both general and accurate. Notice how the example immediately above has both of these characteristics?

4️⃣ Humans are highly attracted to abnormal factors and see them as good explanations.

For example, "This claim is fraudulent because it was filed on the website from an IP address outside the country. Historically, these have overwhelmingly been fraudulent."

The most valuable techniques for interpreting ML models will provide one or more of the above characteristics of good explanations. The more, the better!

As a preview of next week's tutorial, I will mention this.

There's a popular idea in data science that some machine learning algorithms are inherently more interpretable than others.

While this is technically true, social media often portrays the idea in binary terms - an algorithm is interpretable or not.

The above characteristics are super important because they tell us that even an "interpretable algorithm" may not provide a compelling explanation to our business stakeholders.

For example, a logistic regression model where complex interactions are the most important model features. The types of features are very difficult (if not impossible) for business stakeholders to understand.

This means that ML model interpretability is a spectrum, regardless of the algorithm used.

You will learn tools to work across the spectrum.