Understanding Loss Functions for Training Classifiers

Originally published on February 24, 2023

Building a classification model with a collection of annotated training examples requires making the following choices:

  • Classification model
    • This implies whether you should be using a logistic classifier or a multilayer neural network or a convolutional neural network or any other suitable model
  • Loss function
    • A loss function provides a measure of how well the training is proceeding and is needed to adjust the parameters of the classification model being trained. The choice of a loss function depends upon whether the model is being trained for a binary classification task or the number of classes is many.
  • Optimizer
    • The training involves optimizing the chosen loss function by repeatedly going over the training examples to adjust the model parameters. Again, there are several optimizers available in all popular machine learning and deep learning libraries to choose from.

In this blog post, I will focus on three commonly used loss functions for classification to give you a better understanding of these loss functions. These are:

  • Cross Entropy Loss
  • Binary Cross Entropy Loss
  • Negative Log-likelihood Loss

What is Cross Entropy?

Let’s first understand entropy which measures uncertainty of an event. We start by using C to represent a random event/variable which takes on different possible class labels as values in a training set. We use $p(c_i)$ to represent the probability that the class label of a training example is $c_i$, i.e. C equals $c_i$ with probability p. The entropy of the training set labels can be then expressed as below where the summation is carried over all possible labels:

$E(C) = -\sum_i p(c_i)$

It is easy to see that if all training examples are from the same class, then the entropy is zero, and there is no uncertainty about the class label of any training example picked at random. On the other hand, if the training set contains more than one class label, then we have some uncertainty about the class label of a randomly picked training example. As an example, suppose our training data has four labels: cat, dog, horse, and sheep. Let the mix of labels in our training data be cat 40%, dog 10%, horse 25%, and sheep 25%. Then the entropy of the training set using the natural log is given by

Entropy of our Training Set = -(0.4 log0.4 + 0.1log0.1 + 0.25log0.25 +   0.25log0.25 = 1.29

The entropy of a training set will achieve its maximum value when there are equal number of training examples from each category.

Let’s consider another random variable $\hat C$ which denotes the labels predicted by the model for a training example. Now, we have two sets of label distributions, one of true (target) labels in the training set and another of predicted labels. One way to compare these two label distributions is to extend the idea of entropy to cross entropy. It is defined as

$H(C,\hat{C}) = -\sum_i p(c_i)\log p(\hat{c}_i)$

Note that the cross entropy is not a symmetric function. Suppose the classifier that you have trained produces the following distribution of predicted labels: cat 30%, dog 15%, horse 25%, and sheep 30%. The cross entropy of the target and the predicted labels distribution is then given by

Cross Entropy(target labels, predicted labels) = -(0.4log0.3+ 0.1log0.15 + 0.25log0.25 +   0.25log0.3 = 1.32

The difference between the cross entropy value of 1.32 and the entropy of target labels of 1.29 is a measure of how close the predicted label distribution is to the target distribution.

While you are looking at your classifier, your friend pops in to tell you how well his classifier is doing. His classifier is producing the following distribution of predicted labels: cat 30%, dog 20%, horse 20%, and sheep 30%. You look at his numbers and tell him that your classifier is better that his because the cross entropy measure of your classifier, 1.32, is closer to the target entropy of 1.29 than the cross entropy measure of 1.35 of his classifier.

Cross Entropy Loss

The above definition of cross entropy is good for comparing two distributions or classifiers at a global level. However, we are interested in having a measure at the training examples level so that it can be used to adjust the parameters of the classifier being trained. To see how the above concept of cross entropy can be applied to each and every training example, let’s consider a training example inputted to a 3-class classifier to classify images of cats, dogs, and horses. The training example is of a horse. Using one-hot encoding for class labels, the target vector for the input image of the horse will be [0, 0, 1]. Since it is a 3-class problem, the classifier has three outputs as depicted below. where the output of the softmax stage is a vector of probabilities. Note that the classifier output is a vector of numbers, called logits. These are converted to a vector of probabilities by the softmax function as shown.


Thus, we have two sets of probabilities: one given by the target vector t and the second given by the output vector o. We can thus use the cross entropy measure defined earlier to express the cross entropy loss. Plugging in the numbers, the cross entropy loss value is calculated as

-(0*log(0.275) + 0*log(0.300) + 1*log(0.425)) -> 0.856

You can note that this loss would tend towards zero if the output probability for the class label horse goes up. This means that if our classifier is making correct predictions with increasing probabilities, the cross entropy loss will be small.

Since batches of training vectors are inputted at any training instance, the cross entropy loss for the batch is found by summing the loss over all examples.

While using the cross entropy loss in PyTorch, you do not need to worry about the softmax calculations. The cross entropy loss function in PyTorch takes logits as input and thus has a built-in softmax function. You can use the loss function for a single example or for a batch. The following example illustrates the use of cross entropy loss function for a single example.

import torch
import torch.nn.functional as F
out = torch.tensor([3.05, 3.13, 3.48])
target = torch.tensor([0.0, 0.0, 1.0])
loss =F.cross_entropy(out,target)
print(loss)
tensor(0.8566)

Binary Cross Entropy (BCE) Loss

Let’s consider a training model for a two-class problem. Let’s input an example from class 1 to the model, i.e the correct label is y = 1. The model predicts with probability p the input class label to be 1. The probability for the input class label not being 1 is then 1-p. The following formula captures the binary cross entropy loss for this situation:

$BCELoss = -(y*log(p) + (1-y)*log(1-p))$

Assuming p equal to 0.75, the BCELoss is 0.287. It is easy to see that when the predicted probability p approaches 1, the loss approaches 0.

loss = nn.BCELoss()
out= loss(torch.tensor([0.75]),torch.tensor([1.0]))
print(out)
tensor(0.2877)

The BCELoss function is generally used for binary classification problems. However, it can be used for multi-class problems as well. The BCELoss formula for C classes is then expressed as shown below where $y_k$ is the target vector component and $p_k$ is the predicted probability for class k.

$BCELoss = -\frac{1}{C}\sum_k (y_k * log(p_k) + (1-y_k)*log(1-p_k))$

Let’s use the above formula with a three-class problem where the predicted probabilities for an input for three classes are [0.277, 0.299, 0.424]. The training example is from class 3. The target tensor in this case is then [0.0,0.0,1.0]. The BCELoss value for this situation will be then

-(log(1-0.277) + log(1-0.299) + log(0.424))/3 –> 0.5125

We will now use the BCELoss function to validate our calculation.

out = loss(torch.tensor([0.277, 0.299, 0.424]), torch.tensor([0.0,0.0,1.0]))
print(out)
tensor(0.5125)

Note that the first argument in BCELoss() is a tensor of probabilities and the second argument is the target tensor. This means that the model should output probabilities. Often the output layer has the Relu function as the activation function. In such cases, Binary cross entropy with logits loss function should be used which converts the Relu output to probabilities before calculating the loss. This is shown below in the example where the first argument is a tensor of Relu output values. The calulations of the probabilities is also shown using the sigmoid function. You can use these probabilities in the BCELoss function to check whether you get the same loss value or not via these two different calculations.

loss = nn.BCEWithLogitsLoss()
out = loss(torch.tensor([1.8, 0.75]),torch.tensor([1.0,0.0]))
print (out)
print(torch.sigmoid(torch.tensor([1.8,0.75])))# Will output class probabilities
tensor(0.6449)
tensor([0.8581, 0.6792])

If we input the probabilities calculated above using the sigmoid function in the BCELoss function, we should get the same loss value.

loss = nn.BCELoss()
out= loss(torch.tensor([0.858,0.679]),torch.tensor([1.0, 0.0]))
print(out)
tensor(0.6447)

Negative Log Likelihood Loss

The negative log-likelihood loss (NLLLoss in PyTorch) is used for training classification models with C classes. The likelihood means what are the chances that a given set of training examples, $X_1,X_2,⋯,X_n$ was generated by a model that is characterized by a set of parameters represented by ðœ½. The likelihood ð¿ thus can be expressed as 

$𝐿(X_1,X_2,⋯,X_n|\theta)=𝑃(X_1,X_2,⋯,X_n|\theta)$. 

Assuming that all training examples are independent of each other, the right hand side of the likelihood ð¿ expression can be written as

$𝐿(X_1,X_2,⋯,X_n|\theta)= \prod(𝑃(X_1|\theta)𝑃(X_2|\theta)...𝑃(X_n|\theta)$.

Taking the log of the likelihood converts the right hand side multiplications to a summation. Since we are interested in minimizing the loss, the negative of the log likelihood is taken as the loss measure. Thus

$-log𝐿(X_1,X_2,⋯,X_n|\theta) = -\sum_{i=1}^{n}log(𝑃(X_i|\theta)$

The input to the NLLLoss function is log probabilities of each class as a tensor. The size of the input tensor is (minibatch size, C). The target specified in the loss is a class index in the range [0,C−1] where C = number of classes. Let’s take a look at an example of using NLLLoss function.

loss = nn.NLLLoss()
input = torch.tensor([[-0.6, -0.50, -0.30]])# minibatch size is 1 in this example. The log probabilities are all negative as expected.
target = torch.tensor([1])
output = loss(input,target)
print(output)
tensor(0.5000)

It can be noted that the NLLLoss value in this case is nothing but the negative of the log probability of the target class. When the class probabilities are not directly available as usually is the case, the model output needs to go through the LogSoftmax function to get log probabilities.

m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
# input is of size N x C. N=1, C=3 in the example
input = torch.tensor([[-0.8956, 1.1171, 1.3302]])
# each element in target has to have 0 <= value < C
target = torch.tensor([1])
output = loss(m(input), target)
print(output)
tensor(0.8634)

The cross entropy loss and the NLLLoss are mathematically equivalent. The difference between the two arises in how these two loss functions are implemented. As I mentioned earlier the cross entropy loss function in Pytorch expects logits as input, and it includes a softmax function while calculating the cross entropy loss. In the case of NLLLoss, the function expects log probabilities as input. Lacking them, we need to use LogSoftmax function to get the log probabilities as shown above.

There are a few other loss functions available in PyTorch and you can check them at the PyTorch documentation site. I hope you enjoyed reading my explanation of different loss functions. Contact me if you have any question.


CountVectorizer to HashingVectorizer for Numeric Representation of Text

According to some estimates, the unstructured data accounts for about 90% of the data being generated everyday. A large part of unstructured data consists of text in the form of emails, news reports, social media postings, phone transcripts, product reviews etc. Analyzing such data for pattern discovery requires converting text to numeric representation in the form of a vector using words as features. Such a representation is known as vector space model in information retrieval; in machine learning it is known as bag-of-words (BoW) model.

In this post, I will describe different text vectorizers from sklearn library. I will do this using a small corpus of four documents, shown below.

corpus = ['The sky is blue and beautiful',
'The king is old and the queen is beautiful',
'Love this beautiful blue sky',
'The beautiful queen and the old king']

CountVectorizer

The CountVectorizer is the simplest way of converting text to vector. It tokenizes the documents to build a vocabulary of the words present in the corpus and counts how often each word from the vocabulary is present in each and every document in the corpus. Thus, every document is represented by a vector whose size equals the vocabulary size and entries in the vector for a particular document show the count for words in that document. When the document vectors are arranged as rows, the resulting matrix is called document-term matrix; it is a convenient way of representing a small corpus.

For our example corpus, the CountVectorizer produces the following representation.

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())
Doc_Term_Matrix = pd.DataFrame(X.toarray(),columns= vectorizer.get_feature_names())
Doc_Term_Matrix
['and', 'beautiful', 'blue', 'is', 'king', 'love', 'old', 'queen', 'sky', 'the', 'this']

The column headings are the word features arranged in alphabetical order and row indices refer to documents in the corpus. In the present example, the size of the resulting document-term matrix is 4×11, as there are 4 documents in the example corpus and there are 11 distinct words in the corpus. Since common words such as “is”, “the”, “this” etc do not provide any indication about the document content, we can safely remove such words by telling CountVectorizer to perform stop word filtering as shown below.

vectorizer = CountVectorizer(stop_words='english')
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())
Doc_Term_Matrix = pd.DataFrame(X.toarray(),columns= vectorizer.get_feature_names())
Doc_Term_Matrix
['beautiful', 'blue', 'king', 'love', 'old', 'queen', 'sky']

Although the document-term matrix for our small corpus example doesn’t have too many zeros, it is easy to conceive that for any large corpus the resulting matrix will be a sparse matrix. Thus, internally the sparse matrix representation is used to store document vectors.

N-gram Word Features

One issue with the bag of words representation is the loss of context. The BoW representation just focuses on words presence in isolation; it doesn’t use the neighboring words to build a more meaningful representation. The CountVectorizer provides a way to overcome this issue by allowing a vector representation using N-grams of words. In such a model, N successive words are used as features. Thus, in a bi-gram model, N = 2, two successive words will be used as features in the vector representations of documents. The result of such a vectorization for our small corpus example is shown below. Here the parameter ngram_range = (1,2) tells the vectorizer to use two successive words along with each single word as features for the resulting vector representation.

vectorizer = CountVectorizer(ngram_range = (1,2),stop_words='english')
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names())
Doc_Term_Matrix = pd.DataFrame(X.toarray(),columns= vectorizer.get_feature_names())
Doc_Term_Matrix
['beautiful', 'beautiful blue', 'beautiful queen', 'blue', 'blue beautiful', 'blue sky', 'king', 'king old', 'love', 'love beautiful', 'old', 'old king', 'old queen', 'queen', 'queen beautiful', 'queen old', 'sky', 'sky blue']





It is obvious that while N-gram features provide context and consequently better results in pattern discovery, it comes at the cost of increased vector size.

TfidfVectorizer

Simply using the word count as a feature value of a word really doesn’t reflect the importance of that word in a document. For example, if a word is present frequently in all documents in a corpus, then its count value in different documents is not helpful in discriminating between different documents. On other hand, if a word is present only in a few of documents, then its count value in those documents can help discriminating them from the rest of the documents. Thus, the importance of a word, i.e. its feature value, for a document not only depends upon how often it is present in that document but also how is its overall presence in the corpus. This notion of importance of a word in a document is captured by a scheme, known as the term frequency-inverse document frequency (tf-idf ) weighting scheme.

The term frequency is a ratio of the count of a word’s occurrence in a document and the number of words in the document. Thus, it is a normalized measure that takes into consideration the document length. Let us show the count of word i in document j by tf_{ij}. The document frequency of word i represents the number of documents in the corpus with word i in them. Let us represent document frequency for word i by df_i. With N as the number of documents in the corpus, the tf-idf weight w_{ij} for word i in document j is computed by the following formula:

w_{ij} = tf_{ij}\times(1 + \text{log}\frac{1+N}{1+df_{ij}})

The sklearn library offers two ways to generate the tf-idf representations of documents. The TfidfTransformer transforms the count values produced by the CountVectorizer to tf-idf weights.

from sklearn.feature_extraction.text import TfidfTransformer
transformer = TfidfTransformer()
tfidf = transformer.fit_transform(X)
Doc_Term_Matrix = pd.DataFrame(tfidf.toarray(),columns= vectorizer.get_feature_names())
pd.set_option("display.precision", 2)
Doc_Term_Matrix

Another way is to use the TfidfVectorizer which combines both counting and term weighting in a single class as shown below.

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(ngram_range = (1,2),stop_words='english')
tfidf = vectorizer.fit_transform(corpus)
Doc_Term_Matrix = pd.DataFrame(tfidf.toarray(),columns= vectorizer.get_feature_names())
Doc_Term_Matrix

One thing to note is that the tf-idf weights are normalized so that the resulting document vector is of unit length. You can easily check this by squaring and adding the weight values along each row of the document-term matrix; the resulting sum should be one. This sum represents the squared length of the document vector.

HashingVectorizer

There are two main issues with the CountVectorizer and TdidfVectorizer. First, the vocabulary size can grow so much so as not to fit in the available memory for large corpus. In such a case, we need two passes over data. If we were to distribute the vectorization task to several computers, then we will need to synchronize vocabulary building across computing nodes. The other issue arises in the context of an online text classifier built using the count vectorizer, for example spam classifier which needs to decide whether an incoming email is spam or not. When such a classifier encounters words not in its vocabulary, it ignores them. A spammer can take advantage of this by deliberately misspelling words in its message which when ignored by the spam filter will cause the spam message appear normal.  The HashingVectorizer overcomes these limitations.

The HashingVectorizer is based on feature hashing, also known as the hashing trick. Unlike the CountVectorizer where the index assigned to a word in the document vector is determined by the alphabetical order of the word in the vocabulary, the HashingVectorizer maintains no vocabulary and determines the index of a word in an array of fixed size via hashing. Since no vocabulary is maintained, the presence of new or misspelled words doesn’t create any problem. Also the hashing is done on the fly and memory need is diminshed.

You may recall that hashing is a way of converting a key into an address of a table, known as the hash table. As an example, consider the following hash function for a string s of length n:

$\text{hash(s)} = (s[0] + s[1] \cdot p +s[2]\cdot p^2 +\cdots  + s[n-1] \cdot p^{n-1})\text{ mod }m$

The quantities p and m are chosen in practice to minimize collision and set the hash table size. Letting p = 31 and m = 1063, both prime numbers, the above hash function will map the word “blue” to location 493 in an array of size 1064 as per the calculations:

\text{hash(blue)} = (2 + 12 \cdot 31 + 21\cdot 31^2 + 5\cdot 31^3)\text{ mod 1063} = 493\text{, }

where letter b is replaced by 2, l by 12, and so on based on their positions in the alphabet sequence. Similarly, the word “king” will be hashed to location 114 although “king” comes later than “blue” in alphabetical order.

The HashingVectorizer implemented in sklearn uses the  Murmur3 hashing function which returns both positive and negative values. The sign of the hashed value is used as the sign of the value stored in the document-term matrix. By default, the size of the hash table is set to 2^{20}; however, you can specify the size if the corpus is not exceedingly large. The result of applying HashingVectorizer to our example corpus is shown below. You will note that the parameter n_features, which determines the hash table size, is set to 6. This has been done to show collisions since our corpus has 7 distinct words after filtering stop words.

from sklearn.feature_extraction.text import HashingVectorizer
vectorizer = HashingVectorizer(n_features=6,norm = None,stop_words='english')
X = vectorizer.fit_transform(corpus)
Doc_Term_Matrix = pd.DataFrame(X.toarray())
Doc_Term_Matrix

You will note that column headings are integer numbers referring to hash table locations. Also that hash table location indexed 5 shows the presence of collisions. There are three words that are being hashed to this location. These collisions disappear when the hash table size is set to 8 which is more than the vocabulary size of 7. In this case, we get the following document-term matrix.

The HashingVectorizer has a norm parameter that determines whether any normalization of the resulting vectors will be done or not. When norm is set to None as done in the above, the resulting vectors are not normalized and the vector entries, i.e. feature values, are all positive or negative integers. When norm parameter is set to l1, the feature values are normalized so as the sum of all feature values for any document sums to positive/negative 1. In the case of our example corpus, the result of using l1 norm will be as follows.

With norm set to l2, the HashingVectorizer normalizes each document vector to unit length. With this setting, we will get the following document-term matrix for our example corpus.

The HashingVectorizer is not without its drawbacks. First of all, you cannot recover feature words from the hashed values and thus tf-idf weighting cannot be applied. However, the inverse-document frequency part of the tf-idf weighting can be still applied to the resulting hashed vectors, if needed. The second issue is that of collision. To avoid collisions, hash table size should be selected carefully. For very large corpora, the hash table size of 2^{18} or more seems to give good performance. While this size might appear large, some comparative numbers illuminate the advantage of feature hashing. For example, an email classifier with hash table of size 4 million locations has been shown to perform well on a well-known spam filtering dataset having 40 million unique words extracted from 3.2 million emails. That is a ten times reduction in the document vectors size.

To summarize different vectorizers, the TfidfVectorizer appears a good choice and possibly the most popular choice for working with a static corpus or even with a slowing changing corpus provided periodic updating of the vocabulary and the classification model is not problematic. On the other hand, the HashingVectorizer is the best choice when working with a dynamic corpus or in an online setting.

Principal Component Analysis Explained with Examples


(Originally published on August 21, 2018)

Any machine learning model building task begins with a collection of data vectors wherein each vector consists of a fixed number of components. These components represent the measurements, known as attributes or features, deemed useful for the given machine learning task at hand. The number of components, i.e. the size of the vector, is termed as the dimensionality of the feature space. When the number of features is large, we are often interested in reducing their number to limit the number of training examples needed  to strike a proper balance with the number of model parameters.  One way to reduce the number of features is to look for a subset of original features via some suitable search technique. Another way to reduce the number of features or dimensionality is to map or transform the original features in to another feature space of smaller dimensionality. The Principal Component Analysis (PCA) is an example of this feature transformation approach where the new features are constructed by applying a linear transformation on the original set of features. The use of PCA does not require knowledge of the class labels associated with each data vector. Thus, PCA is characterized as a linear, unsupervised  technique for dimensionality reduction.

Basic Idea Behind PCA

The basic idea behind PCA is to exploit the correlations between the original features. To understand this, lets look at the following two plots showing how a pair of variables vary together. In the left plot, there is no relationship in how X-Y values are varying; the values seem to be varying randomly. On the other hand, the variations in the right side plot exhibits a pattern; the Y values are moving up in a linear fashion. In terms of correlation, we say that the values in the left plot show no correlation while the values in the right plot show good correlation. It is not hard to notice that given a X-value from the right plot, we can reasonably guess the Y-value; however this cannot be done for X-values in the left graph. This means that we can represent the data in the right plot with a good approximation lying along a line, that is we can reduce the original two-dimensional data in one dimensions. Thus achieving dimensionality reduction. Of course, such a reduction is not possible for data in the left plot where there is no correlation in X-Y pairs of values.

PCA Steps

Now that we know the basic idea behind the PCA, let's look at the steps needed to perform PCA. These are:

  • We start with N d-dimensionaldata vectors, $\boldsymbol{x}_i, i= 1, \cdots,N$, and find the eigenvalues and eigenvectors of the sample covariance matrix of size d x d using the given data
  • We select the top k eigenvalues,  d, and use the corresponding eigenvectors to define the linear transformation matrix A of size k x d for transforming original features into the new space.
  • Obtain the transformed vectors, $\boldsymbol{y}_i, i= 1, \cdots,N$, using the following relationship. Note the transformation involves first shifting the origin of the original feature space using the mean of the input vectors as shown below, and then applying the transformation. 

$\boldsymbol{y}_i = \bf{A}(\bf{x}_i - \bf{m}_x)$

  • The transformed vectors are the ones we then use for visualization and building our predictive model. We can also recover the original data vectors with certain error by using the following relationship

$\boldsymbol{\hat x}_i = \boldsymbol{A}^t\boldsymbol{y}_i + \boldsymbol{m}_x$

  • The mean square error (mse) between the original and reconstructed vectors is the sum of the eigenvalues whose corresponding eigenvectors are not used in the transformation matrix A.

$ e_{mse} = \sum\limits_{j=k+1}\limits^d \lambda_j$

  • Another way to look at how good the PCA is doing is by calculating the percentage variability, P, captured by the eigenvectors corresponding to top k eigenvalues. This is expressed by the following formula

$ P = \frac{\sum\limits_{j=1}^k \lambda_j}{\sum_{j=1}^d \lambda_j}$

A Simple PCA Example

Let's look at PCA computation in python using 10 vectors in three dimensions. The PCA calculations will be done following the steps given above. Lets first describe the input vectors, calculate the mean vector and the covariance matrix.






Next, we get the eigenvalues and eigenvectors. We are going to reduce the data to two dimensions. So we form the transformation matrix A using the eigenvectors of top two eigenvalues.


With the calculated A matrix, we transform the input vectors to obtain vectors in two dimensions to complete the PCA operation.


Looking at the calculated mean square error, we find that it is not equal to the smallest eigenvalue (0.74992815) as expected. So what is the catch here? It turns out that the formula used in calculating the covariance matrix assums the number of examples, N, to be large. In our case, the number of examples is rather small, only 10. Thus, if we multiply the mse value by N/(N-1), known as the small sample correction, we will get the result identical to the smallest eigenvalue. As N becomes large, the ratio N/(N-1) approaches unity and no such correction is required.

The above PCA computation was deliberately done through a series of steps. In practice, the PCA can be easily done using the scikit-learn implementation as shown below.


Before wrapping up, let me summarize a few takeaways about PCA.

  • You should not expect PCA to provide much reduction in dimensionality if the original features have little correlation with each other.
  • It is often a good practice to perform data normalization prior to applying PCA. The normalization converts each feature to have a zero mean and unit variance. Without normalization, the features showing large variance tend to dominate the result. Such large variances could also be caused by the scales used for measuring different features. This can be easily done by using sklearn.preprocessing.StandardScalar class.
  • Instead of performing PCA using the covariance matrix, we can also use the correlation matrix. The correlation matrix has a built-in normalization of features and thus the data normalization is not needed. Sometimes, the correlation matrix is referred to as the standardized covariance matrix.
  • Eigenvalues and eigenvectors are typically calculated by the singular value decomposion (SVD) method of matrix factorization. Thus, PCA and SVD are often viewed the same. But you should remember that the starting point for PCA is a collection of data vectors that are needed to compute sample covariance/correlation matrices to perform eigenvector decomposition which is often done by SVD.