Issue #14 - Hierarchical Clustering Part 1: Introduction

This Week’s Tutorial

Early in my data science journey, I ignored cluster analysis and missed many opportunities to have impact.

Please don't make the same mistake I did when you get started with machine learning.

There's an unfortunate reality when it comes to how data science is defined in social media and most organizations:

• Data science == machine learning.

• Machine learning == predictive ML models.

• Predictive ML models == production deployments.

Before I get a bunch of 🔥 email replies, let me state something for the record.

When done well, the business value of production ML predictive models can be substantial.

However, these situations are typically the exception rather than the rule. This has been my hands-on experience and is also reported in industry data collected by TDWI, Forrester, and Gartner.

For example, the percentage of ML projects intended for production, but never made it, is very high.

This is unfortunate, because what often gets lost in the discussions about data science is that there are two forms of ML commonly used in business analytics:

1. Supervised Learning: The machine learns from labeled examples.

2. Unsupervised Learning: The machine learns from unlabeled examples.

Supervised Learning is how you craft ML predictive models like decision trees and random forests. These models learn from datasets where each row of data has an outcome of interest (i.e., the label) recorded.

For example, you work for a governmental agency and want to craft an ML model to predict claims fraud. Every row of your historical dataset needs a label indicating whether a claim was fraudulent.

Supervised learning gets all the love in social media, but there's a problem.

Most of the world's data is unlabeled - including the data in your organization.

So what do you do?

You use Unsupervised Learning.

More specifically, you use a form of Unsupervised Learning called cluster analysis. Here's a definition from my favorite machine learning textbook:

"Cluster analysis groups data objects based only on information found in the data that describes the objects and their relationships.

The goal is that the objects within a group be similar (or related) to one another and different from (or unrelated to) the object in other groups.

The greater the similarity (or homogeneity) within a group and the greater the difference between groups, the better or more distinct the clustering."

Because so much real-world data is unlabeled, cluster analysis is a widely used tool in analytics to discover structure in data and produce new insights.

While many forms of cluster analysis have been invented over the years, the three clustering algorithms that are most used in business analytics are:

• K-means clustering

• DBSCAN clustering

• Hierarchical clustering

If you're ready, I can teach you how to use the first two over a weekend with my ​Cluster Analysis with Python​ online course.

The third is the subject of this newsletter tutorial series.

Based on the above definition, hierarchical clustering mines groupings from unlabeled datasets. What distinguishes hierarchical clustering is how the mined groupings are defined.

The easiest way to intuit how hierarchical clustering works is to see a typical real-world example: