Section 1.1: Understanding Standardization

Standardization addresses the mean and standard deviation of data points. Raw data can vary significantly, impacting model performance. To mitigate this, we use standardization, which adjusts the mean to zero and sets the standard deviation to one. The formula for standardization is as follows:

z = (feature_value - mean) / standard_deviation

When algorithms process our data, they assume it is centered and that all features have the same variance. If this condition is not met, predictions may be inaccurate.

The sklearn library provides the StandardScaler method in its preprocessing class to standardize datasets.

Example in Python:

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)

X_test = sc.transform(X_test)

Section 1.2: Scaling Techniques for Sparse Data

Scaling transforms feature values into a range between "0" and "1". This can be accomplished using either MinMaxScaler or MaxAbsScaler.

Example in Python:

import numpy as np

from sklearn.preprocessing import MinMaxScaler

X_train = np.array([[1., 0., 2.], [2., 0., -1.], [0., 2., -1.]])

min_max_scaler = MinMaxScaler()

X_train_minmax = min_max_scaler.fit_transform(X_train)

print(X_train_minmax)

When dealing with sparse data, centering may disrupt its structure, so it’s advisable to scale the raw input data.

Scaling with Outliers:

When datasets contain numerous outliers, traditional scaling methods may not perform well. Instead, using a more robust approach, such as the Interquartile Range (IQR) method, can be beneficial. This method focuses on the data range between the 25th and 75th percentiles, effectively minimizing the influence of outliers.

Example in Python:

from sklearn.preprocessing import RobustScaler

X = [[1., 0., 2.], [2., 0., -1.], [0., 2., -1.]]

transformer = RobustScaler().fit(X)

print(transformer.transform(X))

Section 1.3: Normalization Techniques

Normalization adjusts values to their unit norm. The MinMaxScaler is a common example of this process. Normalization is particularly useful in contexts involving quadratic forms, such as kernel-based methods.

There are two main types of normalization:

1. Normalize: Scales input vectors to unit norm using parameters L1, L2, and max (with L2 as the default).
2. Normalizer: Performs similar operations but makes the fit method optional.

Example in Python:

from sklearn.preprocessing import normalize

X = [[1., 0., 2.], [2., 0., -1.], [0., 2., -1.]]

X_normalized = normalize(X, norm='l2')

print(X_normalized)

Section 1.4: Categorical Encoding Techniques

Raw datasets often contain categorical data that requires encoding into numerical values. Common methods include:

• Get Dummies: Utilizes the pandas library to create new feature columns with binary encoding.
• Label Encoder: Converts binary categories into numerical values using sklearn.
• One Hot Encoder: Transforms categorical classes into binary numeric values with additional feature columns.
• Hashing: Efficiently handles high-dimensional data when dealing with features of high cardinality.

Example in Python:

import pandas as pd

df1 = pd.get_dummies(df['State'], drop_first=True)

Section 1.5: Handling Missing Values with Imputation

Imputation involves filling in missing values in a dataset. This process is crucial for maintaining data integrity.

Example of creating a DataFrame with missing values:

import pandas as pd

import numpy as np

df = pd.DataFrame(np.random.randn(4, 3), index=['a', 'c', 'e', 'h'], columns=['First', 'Second', 'Three'])

df = df.reindex(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'])

print(df)

To replace missing values with zero:

print("NaN replaced with '0':")

print(df.fillna(0))

To fill missing values with the mean, you can use the SimpleImputer from sklearn:

from sklearn.impute import SimpleImputer

imp = SimpleImputer(missing_values=np.nan, strategy='mean')

Recommended Articles

1. NLP — Zero to Hero with Python
2. Python Data Structures: Data Types and Objects
3. Python: Zero to Hero with Examples
4. Comprehensive Guide to SVM Classification with Python
5. In-depth Exploration of K-means Clustering with Python
6. Detailed Overview of Linear Regression with Python
7. Thorough Explanation of Logistic Regression with Python
8. Basics of Time Series Analysis with Python
9. NumPy: From Beginner to Expert with Python
10. Understanding the Confusion Matrix in Machine Learning