How to Standardise Data in Pandas. Z-Score Method

Introduction

In today’s data-driven world, marketing analytics plays a critical role in understanding consumer behavior and driving predictive marketing strategies. One of the key challenges marketers face is effectively segmenting their target audience by managing a lot of different features in the same model. In this blog post we will see how the scaling process works in machine learning.
Among the myriad of data analysis techniques, the Z-Score method has emerged as a powerful tool for marketers seeking to gain deeper insights into their customer data in the EDA process. In this blog post, I delve into the significance of calculating the Z-Score method using Pandas, with a specific focus on its application in unsupervised learning and customer segmentation.

Unsupervised learning techniques, like clustering algorithms, enable marketers to unearth hidden patterns within their customer base, leading to more precise targeting and tailored marketing campaigns. By standardizing the data, marketers can ensure that each feature contributes equally to the clustering process, resulting in more accurate and meaningful customer segmentation. With data science, marketers can transform raw data into actionable insights, laying the foundation for data-driven decision-making..

Are you ready? Let’s get started! 🙂

Why Standardising Data?

Data analysis and machine learning often involve working with multiple independent features to predict a dependent variable. However, the raw data can sometimes be noisy and contain outliers, which can negatively impact the performance of the models. To address this, a common approach is to normalize or standardize the data as part of the feature engineering process.

Data transformations can be of three kinds:

• Data standardization or scaling
• Power transformation
• Data normalisation

Data Standardization or Scaling

Standardization is a technique that helps scale the data to make it more suitable for machine learning algorithms. The goal of standardization is to transform the data so that it has a mean of 0 and a standard deviation of 1. This is different from normalization, where the data is scaled to ensure that different attributes have the same scale, making them directly comparable to a specific range, usually between 0 and 1 (with the method of min-max scaling).

For instance, consider a scenario of data standardization where we want to use income (y-axis) and age (x-axis) of customers as attributes for clustering. Plotting these attributes on the same graph may pose a problem due to the inherent differences in their scales. Income values might range in the thousands or millions, while age values are typically much smaller. The disparity in scale could lead to biased clustering results, as the clustering algorithm may give undue weight to the attribute with larger values (in this case, income).

To address this issue, we apply standardization to both attributes, which involves transforming the data to have a mean of 0 and a standard deviation of 1. This process scales the attributes to a similar, comparable range, allowing the clustering algorithm to treat each attribute equally during the segmentation process. By standardizing the data, the income and age attributes are brought to a common scale, making them suitable for meaningful clustering and more accurate segmentation criteria (Baig, 2022).

In summary, standardization is an essential preprocessing step in marketing when dealing with attributes of varying scales. It ensures that different attributes contribute equally to the clustering process and aids in identifying relevant customer segments, leading to more effective and targeted marketing strategies.

How to Standardise Data?

Among various data rescaling techniques, one simple and widely used method is the Z-Score. However, the selection of the appropriate data standardization approach relies on your data’s specific characteristics and the needs of your analysis or model. Alternative methods include Min-Max Scaling, Robust Scaling, Max Absolute Scaling, LogTransformations, Quantile Transformation, PowerTransformation.

Z-Score Method

This method is usually done when one of the features has a higher variance than the others. The z-score represents the number of standard deviations a data point is from the mean. It is calculated using the following formula:

1. Nominator

• Mean (x): we calculate the mean of the column
• x – mean: we subtract the mean from each data point. This centres the data around 0.

2. Denominator

• We calculate the standard deviation. std (x). The standard deviation is a measure of dispersion or variability of data in statistics. It indicates how much the values in a data set tend to vary from the mean.
• We divide the nominator by the standard deviation for each data points. The greater is the value of the data set (x), the larger will be our standard deviation.

How to read this formula? “Given a normal distribution, 68% of the data will be within 1 standard deviation of the mean, 95% will be within 2 standard deviations, and 99.7% will be within 3 standard deviations. We can then use a threshold of 3 standard deviations to detect anomalies, as only 0.3% of the data points will be above or below this value” (G. Diaz-Bérrio, 2024)

Why this formula works? By dividing the values of different columns (x, y) by their respective standard deviations, we transform the data, ensuring that the values become comparable on a common scale. This standardization process results in all variables having a standard deviation of 1. Consequently, a difference of 1 between different variables indicates a similar level and size.

In marketing, data standardization becomes crucial when creating customer clusters or sales forecast for various purposes and follow a linear distribution. For instance, to develop predictive customer segmentation using unsupervised learning techniques like k-means clustering, it is essential to standardize the data. This ensures that each feature contributes equally to the clustering process, enabling us to group customers more effectively based on their similarities and characteristics.

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler

#Step 2: Load your data into a pandas DataFrame 
df = pd.read_csv('your_data.csv')

#Step 3: Preprocess the data and select the columns to be standardized
#We want to standardize 'column1' and 'column2'
columns_to_standardize = ['column1', 'column2']
data_to_standardize = df[columns_to_standardize].values

#Step 4: Initialize the StandardScaler object
scaler = StandardScaler()

#Step 5: Use the fit method to compute the mean and standard deviation
scaler.fit(data_to_standardize)

#Step 6: Use the transform method to standardize the data
standardized_data = scaler.transform(data_to_standardize)

Now your new variable ‘standardized_data‘ contains the standardized values of ‘column1’ and ‘column2’ to be used for further analysis and modelling.

 

Conclusion

In this hands-on tutorial, we explored the importance of data standardization and learned how to use the Z-Score for marketing analytics. Standardizing data plays a vital role in various data analysis and machine learning tasks if we want to perform predictive customer segmentation in clustering.

Data standardization resolves common issue if we handle in the same data frame different data like income and age. By transforming the data into a common scale we ensuring a level playing field for all features.

The Z-Score, based on the StandardScaler utility from scikit-learn, is a popular and powerful tool in Python. It computes the standardized value for each data point by subtracting the mean and dividing by the standard deviation of the feature. This transformation results in a distribution with a mean of 0 and a standard deviation of 1, which provides a clear and interpretable measure of how many standard deviations a data point deviates from the mean.

In conclusion, by using the Z-Score and the StandardScaler utility from scikit-learn, we can confidently prepare our data for a wide range of applications in marketing. With standardized data, we can improve the accuracy and performance of our models, gain deeper insights from our analysis, and make more informed decisions to improve the performance our startups.

Let’s keep in touch and Reach out to me if you have any questions!

 

---

Bibliography

Diaz-Bérrio, G. (2024). Data analytics for marketing: A practical guide to analyzing marketing data using Python (1st ed.). Packt.

Govindan, G., Baig, M. R., & Shrimali, V. R. (2021). Data Science for Marketing Analytics: A Practical Guide to Forming a Killer Marketing Strategy Through Data Analysis with Python. Packt.

Hadley, W., & Grolemund, G. (2016). R for data science: Import, tidy, transform, visualize, and model data. O’Reilly Media.

McKinney, W. (2018). Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. O’Reilly Media.

Paskhaver, B. (2021). Pandas in Action. Manning Publications.

VanderPlas, J. (2016). Python data science handbook: Essential tools for working with data. O’Reilly Media.

Wickham, H., & Grolemund, G. (2017). R packages: Organize, test, document, and share your code. O’Reilly Media.

Zaki, M. J., & Meira Jr, W. (2014). Data mining and analysis: Fundamental concepts and algorithms. Cambridge University Press.