Python Machine Learning
My notes and highlights on the book.
Author: Sebastian Rashcka
Table of Contents
 Table of Contents
 Ch1. Giving Computers the Ability to Learn from Data
 Ch2. Training Simple Machine Learning Algorithms for Classification
 Ch3. A Tour of Machine Learning Classifiers Using scikitlearn
 Ch4. Building Good Training Datasets – Data Preprocessing
 Ch5. Compressing Data via Dimensionality Reduction
 Ch6. Learning Best Practices for Model Evaluation and Hyperparameter Tuning
 Combining transformers and estimators in a pipeline
 Using kfold crossvalidation to assess model performance
 Kfold crossvalidation
 Choosing K
 Ch7. Combining different models for Ensemble Learning
 Ch8. Applying Machine Learning to Sentiment Analysis
 Ch9. Embedding a Machine Learning Model into a Web Application
 Ch10. Predicting Continuous Target Variables with Regression Analysis
 Linear Regression
 Visualizing the important characteristics of a dataset
 Looking at relationships using a correlation matrix
 Estimating the coefficient of a regression model via scikitlearn
 Fitting a robust regression model using RANSAC
 Evaluating the performance of linear regression models
 Using regularized methods for regression
 Dealing with nonlinear relationships using random forests
 Ch11. Working with Unlabeled Data  Clustering Analysis
 Ch12. Implementing a Multilayer Artificial Neural Network from Scratch
 Ch13. Parallelizing Neural Network Training with TensorFlow
 Ch14. Going Deeper  The Mechanics of TensorFlow
 Ch15. Classifying Images with Deep Convolutional Neural Networks
 Ch16. Modeling Sequential Data Using Recurrent Neural Networks
 Ch17. Generative Adversarial Networks for Synthesizing New Data
 Ch18. Reinforcement Learning for Decision Making in Complex Environments
Ch1. Giving Computers the Ability to Learn from Data
Selflearning algorithms: derive knowledge from data in order to make predictions efficiently
Three types of ML:
 Supervised Learning: Labeled data, direct feedback, predict outcome/future
 Unsupervised Learning: No labels, no feedback, find hidden structure in data
 Reinforcement Learning: Decision process, reward system, learn series of actions
Supervised Learning
 Discrete class labels > classification
 Outcome signal is a continuous value > regression
Predictor variables = “features”
Response variable = “target”
Solving interactive problems with reinforcement learning
Goal: develop a system (agent) that improve its performance based on interactions with the environment
Related to supervised learning, but the feedback is not the ground truth label, but a measure of how well the action was measured by a reward function
The agent tries to maximize the reward through a series of interactions with the environment
Discovering hidden structures with unsupervised learning
Clustering
Allows us to organize data into meaningful subgroups (clusters) without having any prior knowledge of their group memberships. Sometimes called unsupervised classification
Dimensionality reduction for data compression
Commonly used approach in feature preprocessing to remove noise from data, which can degrade the predictive performance of certain algorithms, and compress the data onto a smaller dimensional subspace while retaining most of the relevant information
Useful for visualizing the data > Highdimensional feature set to 2D or 3D
A roadmap for building ML systems
 Preprocessing  getting data into shape: scaling, dimensionality reduction, randomly divide the dataset into training and test sets
 Training and selecting a predictive model: decide a metric to measure performance, compare a handful of different algorithms in order to train and select the best performing model, crossvalidation, hyperparameter optimization
 Evaluating models and predicting unseen data instances: how well the model performs on unseen data (generalization error)
Ch2. Training Simple Machine Learning Algorithms for Classification
Perceptron
The convergence of the perceptron is only guaranteed if the two classes are linearly separable and the learning rate is sufficiently small
Numpy x Python for loop structures: Vectorization means that an elemental arithmetic operation is automatically applied to all elements in an array. By formulating our arithmetic operations as a sequence of instructions on an array, rather than performing a set of operations for each element at a time, we can make better use of our modern CPU architectures with single instruction, multiple data (SIMD) support. Furthermore, NumPy uses highly optimized linear algebra libraries, such as Basic Linear Algebra Subprograms (BLAS) and Linear Algebra Package (LAPACK), that have been written in C or Fortran
Adaptive linear neurons and the convergence of learning
ADAptive LInear NEuron (Adaline): weights are updated based on a linear activation function rather than a unit step function > WidrowHoff rule
Minimizing cost functions with gradient descent
Objective function: often a cost function that we want to minimize
Gradient descent: powerful optimization algorithm to find the weights that minimize the cost function > climbing down a hill until a local or global cost minimum is reached > take steps in the opposite direction of the gradient. Step size defined by the learning rate and slope of the gradient
A logistic regression model is closely related to Adaline, the only difference being its activation and cost function
Improving gradient descent through feature scaling
Gradient descent is one of the many algorithms that benefit from feature scaling
Standardization: gives the data properties of a standard normal distribution > zeromean and unit variance
Largescale machine learning and stochastic gradient descent
 Batch gradient descent: gradient is calculated from the whole training dataset
 Stochastic gradient descent (SGD): iterative/online gradient descent. Each gradient is calculated on a single training example.
Advantages:
 typically reaches convergence much faster because of more frequent weight updates
 can escape shallow local minima more readily if we are working with nonlinear cost functions
 can be used for online learning > model trained on the fly as new data arrives
With SGD it is important to present training data in a random order; also shuffle the training dataset for every epoch to prevent cycles
Minibatch gradient descent: batch gradient descent to smaller subsets of the training data. Compromise between SGD and batch > vectorized operations can improve the computational efficiency
Ch3. A Tour of Machine Learning Classifiers Using scikitlearn
Ch4. Building Good Training Datasets – Data Preprocessing
Missing Data
Unfortunately, most computational tools are unable to handle such missing values or will produce unpredictable results if we simply ignore them. Therefore, it is crucial that we take care of those missing values before we proceed with further analyses.
Nowadays, most scikitlearn functions support DataFrame objects as inputs, but since NumPy array handling is more mature in the scikitlearn API, it is recommended to use NumPy arrays when possible.
# only drop rows where NaN appear in specific columns (here: 'C')
>>> df.dropna(subset=['C'])
Categorical Features
When we are talking about categorical data, we have to further distinguish between ordinal and nominal features. Unfortunately, there is no convenient function that can automatically derive the correct order of the labels of our size feature, so we have to define the mapping manually.
We can simply define a reversemapping dictionary, inv_size_mapping = {v: k for k, v in size_mapping.items()}
Most estimators for classification in scikitlearn convert class labels to integers internally, but it is considered good practice to provide class labels as integer arrays to avoid technical glitches.
Although the color values don’t come in any particular order, a learning algorithm will now assume that green is larger than blue , and red is larger than green . Although this assumption is incorrect, the algorithm could still produce useful results. However, those results would not be optimal.
A common workaround for this problem is to use a technique called onehot encoding. The idea behind this approach is to create a new dummy feature for each unique value in the nominal feature column.
When we are using onehot encoding datasets, we have to keep in mind that this introduces multicollinearity, which can be an issue for certain methods (for instance, methods that require matrix inversion). If features are highly correlated, matrices are computationally difficult to invert, which can lead to numerically unstable estimates. To reduce the correlation among variables, we can simply remove one feature column from the onehot encoded array.
Providing the class label array y as an argument to stratify ensures that both training and test datasets have the same class proportions as the original dataset.
Instead of discarding the allocated test data after model training and evaluation, it is a common practice to retrain a classifier on the entire dataset, as it can improve the predictive performance of the model.
Feature Scaling
Feature scaling is a crucial step in our preprocessing pipeline that can easily be forgotten. Decision trees and random forests are two of the very few machine learning algorithms where we don’t need to worry about feature scaling. Those algorithms are scale invariant. However, the majority of machine learning and optimization algorithms behave much better if features are on the same scale
There are two common approaches to bringing different features onto the same scale: normalization and standardization
Standardization can be more practical for many machine learning algorithms, especially for optimization algorithms such as gradient descent. The reason is that many linear models initialize the weights to 0 or small random values close to 0. Using standardization, we center the feature columns at mean 0 with standard deviation 1 so that the feature columns have the same parameters as a standard normal distribution (zero mean and unit variance), which makes it easier to learn the weights.
Standardization maintains useful information about outliers and makes the algorithm less sensitive to them in contrast to minmax scaling, which scales the data to a limited range of values
We fit the StandardScaler class only once—on the training data—and use those parameters to transform the test dataset or any new data point
The RobustScaler is especially helpful and recommended if we are working with small datasets that contain many outliers. If the machine learning algorithm applied to this dataset is prone to overfitting , the RobustScaler can be a good choice
Overfitting means the model fits the parameters too closely with regard to the particular observations in the training dataset, but does not generalize well to new data; we say that the model has a high variance . The reason for the overfitting is that our model is too complex for the given training data.
Common ways to reduce overfitting by regularization and dimensionality reduction via feature selection, which leads to simpler models by requiring fewer parameters to be fitted to the data.
Regularization

L1 regularization usually yields sparse feature vectors and most feature weights will be zero. Sparsity can be useful in practice if we have a highdimensional dataset with many features that are irrelevant, especially in cases where we have more irrelevant dimensions than training examples.

L2 regularization adds a penalty term to the cost function that effectively results in less extreme weight values compared to a model trained with an unregularized cost function.
There are two main categories of dimensionality reduction techniques: feature selection and feature extraction . Via feature selection, we select a subset of the original features, whereas in feature extraction, we derive information from the feature set to construct a new feature subspace.
Greedy algorithms make locally optimal choices at each stage of a combinatorial search problem and generally yield a suboptimal solution to the problem, in contrast to exhaustive search algorithms , which evaluate all possible combinations and are guaranteed to find the optimal solution
By reducing the number of features, we shrank the size of the dataset, which can be useful in realworld applications that may involve expensive data collection steps. Also, by substantially reducing the number of features, we obtain simpler models, which are easier to interpret.
Random Forest feature importance
Using a random forest, we can measure the feature importance as the averaged impurity decrease computed from all decision trees in the forest, without making any assumptions about whether our data is linearly separable or not.
As far as interpretability is concerned, the random forest technique comes with an important gotcha that is worth mentioning. If two or more features are highly correlated, one feature may be ranked very highly while the information on the other feature(s) may not be fully captured.
SelectFromModel object that selects features based on a userspecified threshold after model fitting, which is useful if we want to use the RandomForestClassifier as a feature selector and intermediate step in a scikitlearn Pipeline object,
Ch5. Compressing Data via Dimensionality Reduction
Feature Extraction
Feature extraction can be understood as an approach to data compression with the goal of maintaining most of the relevant information.
Feature extraction is not only used to improve storage space or the computational efficiency of the learning algorithm, but can also improve the predictive performance by reducing the curse of dimensionality —especially if we are working with nonregularized models.
PCA
PCA aims to find the directions of maximum variance in highdimensional data and projects the data onto a new subspace with equal or fewer dimensions than the original one
Even if the input features are correlated, the resulting principal components will be mutually orthogonal (uncorrelated).
PCA directions are highly sensitive to data scaling, and we need to standardize the features prior to PCA if the features were measured on different scales and we want to assign equal importance to all features
LDA
The general concept behind LDA is very similar to PCA, but whereas PCA attempts to find the orthogonal component axes of maximum variance in a dataset, the goal in LDA is to find the feature subspace that optimizes class separability
Kernel PCA
If we are dealing with nonlinear problems, which we may encounter rather frequently in realworld applications, linear transformation techniques for dimensionality reduction, such as PCA and LDA, may not be the best choice.
Using the kernel trick, we can compute the similarity between two highdimension feature vectors in the original feature space
Ch6. Learning Best Practices for Model Evaluation and Hyperparameter Tuning
We have to reuse the parameters that were obtained during the fitting of the training data to scale and compress any new data, such as the examples in the separate test dataset
Combining transformers and estimators in a pipeline
There is no limit to the number of intermediate steps in a pipeline; however, the last pipeline element has to be an estimator.
If we reuse the same test dataset over and over again during model selection, it will become part of our training data and thus the model will be more likely to overfit. Despite this issue, many people still use the test dataset for model selection, which is not a good machine learning practice
Using kfold crossvalidation to assess model performance
A better way of using the holdout method for model selection is to separate the data into three parts: a training dataset, a validation dataset, and a test dataset. The training dataset is used to fit the different models, and the performance on the validation dataset is then used for the model selection. The advantage of having a test dataset that the model hasn’t seen before during the training and model selection steps is that we can obtain a less biased estimate of its ability to generalize to new data.
A disadvantage of the holdout method is that the performance estimate may be very sensitive to how we partition the training dataset into the training and validation subsets; the estimate will vary for different examples of the data
Kfold crossvalidation
In kfold crossvalidation, we randomly split the training dataset into k folds without replacement, where k – 1 folds are used for the model training, and one fold is used for performance evaluation. This procedure is repeated k times so that we obtain k models and performance estimates.
We use kfold crossvalidation for model tuning, that is, finding the optimal hyperparameter values that yield a satisfying generalization performance, which is estimated from evaluating the model performance on the test folds.
Once we have found satisfactory hyperparameter values, we can retrain the model on the complete training dataset and obtain a final performance estimate using the independent test dataset. The rationale behind fitting a model to the whole training dataset after kfold crossvalidation is that providing more training examples to a learning algorithm usually results in a more accurate and robust model.
Since kfold crossvalidation is a resampling technique without replacement, the advantage of this approach is that each example will be used for training and validation (as part of a test fold) exactly once, which yields a lowervariance estimate of the model performance than the holdout method.
Choosing K
A good standard value for k in kfold crossvalidation is 10, as empirical evidence shows. For instance, experiments by Ron Kohavi on various realworld datasets suggest that 10fold crossvalidation offers the best tradeoff between bias and variance ( A Study of CrossValidation and Bootstrap for Accuracy Estimation and Model Selection , Kohavi, Ron , International Joint Conference on Artificial Intelligence (IJCAI) , 14 (12): 113743, 1995 ).
Small training sets > increase the number of folds. If we increase the value of k , more training data will be used in each iteration, which results in a lower pessimistic bias toward estimating the generalization performance by averaging the individual model estimates.
Large datasets > smaller value for k , for example, k = 5, and still obtain an accurate estimate of the average performance of the model while reducing the computational cost of refitting and evaluating the model on the different folds
Stratified crossvalidation
A slight improvement over the standard kfold crossvalidation approach is stratified kfold crossvalidation, which can yield better bias and variance estimates, especially in cases of unequal class proportions
By plotting the model training and validation accuracies as functions of the training dataset size, we can easily detect whether the model suffers from high variance or high bias, and whether the collection of more data could help to address this problem
Bias x Variance
High bias: This model has both low training and crossvalidation accuracy, which indicates that it underfits the training data. Common ways to address this issue are to increase the number of parameters of the model, for example, by collecting or constructing additional features, or by decreasing the degree of regularization
High variance: which is indicated by the large gap between the training and crossvalidation accuracy. To address this problem of overfitting, we can collect more training data, reduce the complexity of the model, or increase the regularization parameter
For unregularized models, it can also help to decrease the number of features via feature selection
While collecting more training data usually tends to decrease the chance of overfitting, it may not always help, for example, if the training data is extremely noisy or the model is already very close to optimal.
Debugging algorithms with learning and validation curves
Validation curves are a useful tool for improving the performance of a model by addressing issues such as overfitting or underfitting. Validation curves are related to learning curves, but instead of plotting the training and test accuracies as functions of the sample size, we vary the values of the model parameters
Finetuning machine learning models
The grid search approach is quite simple: it’s a bruteforce exhaustive search paradigm where we specify a list of values for different hyperparameters, and the computer evaluates the model performance for each combination to obtain the optimal combination of values from this set
Randomized search usually performs about as well as grid search but is much more cost and timeeffective. In particular, if we only sample 60 parameter combinations via randomized search, we already have a 95 percent probability of obtaining solutions within 5 percent of the optimal performance ( Random search for hyperparameter optimization . Bergstra J , Bengio Y . Journal of Machine Learning Research . pp. 281305, 2012).
Algorithm selection with nested crossvalidation
If we want to select among different machine learning algorithms, though, another recommended approach is nested crossvalidation. In a nice study on the bias in error estimation, Sudhir Varma and Richard Simon concluded that the true error of the estimate is almost unbiased relative to the test dataset when nested crossvalidation is used ( Bias in Error Estimation When Using CrossValidation for Model Selection , BMC Bioinformatics , S. Varma and R. Simon , 7(1): 91, 2006 ).
In nested crossvalidation, we have an outer kfold crossvalidation loop to split the data into training and test folds, and an inner loop is used to select the model using kfold crossvalidation on the training fold. After model selection, the test fold is then used to evaluate the model performance
Looking at different performance evaluation metrics
A confusion matrix is simply a square matrix that reports the counts of the true positive ( TP ), true negative ( TN ), false positive ( FP ), and false negative ( FN ) predictions of a classifier
The true positive rate ( TPR ) and false positive rate ( FPR ) are performance metrics that are especially useful for imbalanced class problems
Receiver operating characteristic ( ROC ) graphs are useful tools to select models for classification based on their performance with respect to the FPR and TPR, which are computed by shifting the decision threshold of the classifier. The diagonal of a ROC graph can be interpreted as random guessing , and classification models that fall below the diagonal are considered as worse than random guessing. A perfect classifier would fall into the topleft corner of the graph with a TPR of 1 and an FPR of 0. Based on the ROC curve, we can then compute the socalled ROC area under the curve ( ROC AUC ) to characterize the performance of a classification model.
Scoring metrics for multiclass classification
Microaveraging is useful if we want to weight each instance or prediction equally, whereas macroaveraging weights all classes equally to evaluate the overall performance of a classifier with regard to the most frequent class labels. If we are using binary performance metrics to evaluate multiclass classification models in scikitlearn, a normalized or weighted variant of the macroaverage is used by default
Dealing with class imbalance
Class imbalance is a quite common problem when working with realworld data—examples from one class or multiple classes are overrepresented in a dataset
The algorithm implicitly learns a model that optimizes the predictions based on the most abundant class in the dataset, in order to minimize the cost or maximize the reward during training.
One way to deal with imbalanced class proportions during model fitting is to assign a larger penalty to wrong predictions on the minority class.
Other popular strategies for dealing with class imbalance include upsampling the minority class, downsampling the majority class, and the generation of synthetic training examples. Unfortunately, there’s no universally best solution or technique that works best across different problem domains. Thus, in practice, it is recommended to try out different strategies on a given problem, evaluate the results, and choose the technique that seems most appropriate
SMOTE
Synthetic Minority Oversampling Technique ( SMOTE ), and you can learn more about this technique in the original research article by Nitesh Chawla and others: SMOTE: Synthetic Minority Oversampling Technique , Journal of Artificial Intelligence Research , 16: 321357, 2002 . It is also highly recommended to check out imbalancedlearn , a Python library that is entirely focused on imbalanced datasets, including an implementation of SMOTE
Ch7. Combining different models for Ensemble Learning
The goal of ensemble methods is to combine different classifiers into a metaclassifier that has better generalization performance than each individual classifier alone.
Voting
Majority voting simply means that we select the class label that has been predicted by the majority of classifiers, that is, received more than 50 percent of the votes.
To predict a class label via simple majority or plurality voting, we can combine the predicted class labels of each individual classifier, , and select the class label, , that received the most votes
Stacking
The stacking algorithm can be understood as a twolevel ensemble, where the first level consists of individual classifiers that feed their predictions to the second level, where another classifier (typically logistic regression) is fit to the levelone classifier predictions to make the final predictions. The stacking algorithm has been described in more detail by David H. Wolpert in Stacked generalization , Neural Networks , 5(2):241–259, 1992 .
Bagging
Instead of using the same training dataset to fit the individual classifiers in the ensemble, we draw bootstrap samples (random samples with replacement) from the initial training dataset, which is why bagging is also known as bootstrap aggregating .
Random forests are a special case of bagging where we also use random feature subsets when fitting the individual decision trees.
Bagging was first proposed by Leo Breiman in a technical report in 1994; he also showed that bagging can improve the accuracy of unstable models and decrease the degree of overfitting.
Bagging algorithm can be an effective approach to reducing the variance of a model. However, bagging is ineffective in reducing model bias, that is, models that are too simple to capture the trend in the data well. This is why we want to perform bagging on an ensemble of classifiers with low bias, for example, unpruned decision trees.
Boosting
In boosting, the ensemble consists of very simple base classifiers, also often referred to as weak learners , which often only have a slight performance advantage over random guessing—a typical example of a weak learner is a decision tree stump.
The key concept behind boosting is to focus on training examples that are hard to classify, that is, to let the weak learners subsequently learn from misclassified training examples to improve the performance of the ensemble.
In contrast to bagging, the initial formulation of the boosting algorithm uses random subsets of training examples drawn from the training dataset without replacement
Boosting can lead to a decrease in bias as well as variance compared to bagging models
Boosting algorithms such as AdaBoost are also known for their high variance, that is, the tendency to overfit the training data
It is worth noting that ensemble learning increases the computational complexity compared to individual classifiers. In practice, we need to think carefully about whether we want to pay the price of increased computational costs for an often relatively modest improvement in predictive performance. An oftencited example of this tradeoff is the famous $1 million Netflix Prize , which was won using ensemble techniques. The details about the algorithm were published in The BigChaos Solution to the Netflix Grand Prize by A. Toescher , M. Jahrer , and R. M. Bell , Netflix Prize documentation , 2009
Gradient Boosting
Another popular variant of boosting is gradient boosting . AdaBoost and gradient boosting share the main overall concept: boosting weak learners (such as decision tree stumps) to strong learners. The two approaches, adaptive and gradient boosting, differ mainly with regard to how the weights are updated and how the (weak) classifiers are combined
XGBoost
XGBoost, which is essentially a computationally efficient implementation of the original gradient boost algorithm ( XGBoost: A scalable tree boosting system . Tianqi Chen and Carlos Guestrin . Proceeding of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM 2016 , pp. 785794)
HistGradientBoosting
Scikitlearn now also includes a substantially faster version of gradient boosting in version 0.21, HistGradientBoostingClassifier , which is even faster than XGBoost
Ensemble methods combine different classification models to cancel out their individual weaknesses, which often results in stable and wellperforming models that are very attractive for industrial applications as well as machine learning competitions.
Ch8. Applying Machine Learning to Sentiment Analysis
Sentiment analysis, sometimes also called opinion mining , is a popular subdiscipline of the broader field of NLP; it is concerned with analyzing the polarity of documents. A popular task in sentiment analysis is the classification of documents based on the expressed opinions or emotions of the authors with regard to a particular topic.
To visualize the progress and estimated time until completion, use the Python Progress Indicator ( PyPrind , https://pypi.python.org/pypi/PyPrind/ )
Bagofwords
The idea behind bagofwords is quite simple and can be summarized as follows: We create a vocabulary of unique tokens—for example, words—from the entire set of documents. We construct a feature vector from each document that contains the counts of how often each word occurs in the particular document.
To construct a bagofwords model based on the word counts in the respective documents, we can use the CountVectorizer class implemented in scikitlearn
Values in the feature vectors are also called the raw term frequencies : tf ( t , d )—the number of times a term, t , occurs in a document, d . It should be noted that, in the bagofwords model, the word or term order in a sentence or document does not matter. The order in which the term frequencies appear in the feature vector is derived from the vocabulary indices, which are usually assigned alphabetically.
Ngrams
The sequence of items in the bagofwords model that we just created is also called the 1gram or unigram model—each item or token in the vocabulary represents a single word. More generally, contiguous sequences of items in NLP—words, letters, or symbols—are also called ngrams
TFIDF
Frequently occurring words typically don’t contain useful or discriminatory information
Term frequencyinverse document frequency ( tfidf ), which can be used to downweight these frequently occurring words in the feature vectors. The tfidf can be defined as the product of the term frequency and the inverse document frequency
Cleaning text
The first important step—before we build our bagofwords model—is to clean the text data by stripping it of all unwanted characters.
One way to tokenize documents is to split them into individual words by splitting the cleaned documents at their whitespace characters
Stemming
In the context of tokenization, another useful technique is word stemming , which is the process of transforming a word into its root form. It allows us to map related words to the same stem
The Porter stemming algorithm is probably the oldest and simplest stemming algorithm. Other popular stemming algorithms include the newer Snowball stemmer (Porter2 or English stemmer) and the Lancaster stemmer (Paice/Husk stemmer)
Lemmatization
While stemming can create nonreal words, such as ‘thu’ (from ‘thus’ ), as shown in the previous example, a technique called lemmatization aims to obtain the canonical (grammatically correct) forms of individual words—the socalled lemmas .
Lemmatization is computationally more difficult and expensive compared to stemming and, in practice, it has been observed that stemming and lemmatization have little impact on the performance of text classification
Stopword removal
Stopwords are simply those words that are extremely common in all sorts of texts and probably bear no (or only a little) useful information that can be used to distinguish between different classes of documents. Examples of stopwords are is , and , has , and like . Removing stopwords can be useful if we are working with raw or normalized term frequencies rather than tfidfs, which are already downweighting frequently occurring words.
TfidfVectorizer , which combines CountVectorizer with the TfidfTransformer
Naive Bayes Classifier
A still very popular classifier for text classification is the naïve Bayes classifier, which gained popularity in applications of email spam filtering. Naïve Bayes classifiers are easy to implement, computationally efficient, and tend to perform particularly well on relatively small datasets
Outofcore learning
Outofcore learning allows us to work with large datasets by fitting the classifier incrementally on smaller batches of a dataset.
Unfortunately, we can’t use CountVectorizer for outofcore learning since it requires holding the complete vocabulary in memory. Also, TfidfVectorizer needs to keep all the feature vectors of the training dataset in memory to calculate the inverse document frequencies. However, another useful vectorizer for text processing implemented in scikitlearn is HashingVectorizer . HashingVectorizer is dataindependent
Outofcore learning is very memory efficient
word2vec
A more modern alternative to the bagofwords model is word2vec , an algorithm that Google released in 2013 ( Efficient Estimation of Word Representations in Vector Space , T. Mikolov , K. Chen , G. Corrado , and J. Dean , arXiv preprint arXiv:1301.3781, 2013 ). The word2vec algorithm is an unsupervised learning algorithm based on neural networks that attempts to automatically learn the relationship between words. The idea behind word2vec is to put words that have similar meanings into similar clusters, and via clever vectorspacing, the model can reproduce certain words using simple vector math, for example, king – man + woman = queen .
Topic Modeling
Topic modeling describes the broad task of assigning topics to unlabeled text documents. For example, a typical application would be the categorization of documents in a large text corpus of newspaper articles. In applications of topic modeling, we then aim to assign category labels to those articles, for example, sports, finance, world news, politics, local news, and so forth
Latent Dirichlet Allocation ( LDA )
LDA is a generative probabilistic model that tries to find groups of words that appear frequently together across different documents
We must define the number of topics beforehand—the number of topics is a hyperparameter of LDA that has to be specified manually.
The scikitlearn library’s implementation of LDA uses the expectationmaximization ( EM ) algorithm to update its parameter estimates iteratively
Ch9. Embedding a Machine Learning Model into a Web Application
Serializing fitted scikitlearn estimators
 One option for model persistence: Python’s inbuilt
pickle
module protocol=4
to choose the latest and most efficient pickle protocoljoblib
: lib, more efficient way to serialize NumPy arrays
Pickle can be a security risk: not secured against malicious code. Pickle was designed to serialize arbitraty objects, the unpickling process will execute code that has been stored in a pickle file
SQLite
SQLite database can be understood as a single, selfcontained database file that allows us to directly access storage files
free DB browser for SQLite app (https://sqlitebrowser.org/dl/) > nice GUI for working with SQLite databases
Flask
Written in Python, provides a convenient interface for embedding existing Python code
Jinja2
Web templates
PythonAnywhere
Lets us run a single web application free of charge
Ch10. Predicting Continuous Target Variables with Regression Analysis
Linear Regression
Regression Line: bestfitting line
Offsets/Residuals: vertical lines from the regression line to the training examples > errors of our prediction
Visualizing the important characteristics of a dataset
 Scatterplot matrix: pairwise correlations between the different features > scatterplotmatrix on MLxtend (https://github.com/rasbt/mlxtend)
Training a linear regression model does not require that the explanatory or target variables are normally distributed > only requirement for certain statistics and hypothesis tests
Looking at relationships using a correlation matrix
 Correlation matrix: square matrix that contains the Pearson productmoment correlation coefficient (Pearson’s r) > linear dependence between pairs of features
 Correlation coefficients are in range 1 to 1. 1 > perfect positive correlation, 0 > no correlation and 1: perfect negative correlation
To fit a linear regression model, we are interested in those features that have a high correlation with our target variable
Estimating the coefficient of a regression model via scikitlearn
The linear regression implementation in scikitlearn works better with unstandardized variables
Fitting a robust regression model using RANSAC

Linear regression models can be heavily impacted by the presence of outliers. In certain situations, a very small subset of our data can have a big effect on the estimated model coefficients

As an alternative to throwing out outliers, we will look at a robust method of regression using the RANdom SAmple Consensus (RANSAC) algorithm, which fits a regression model to a subset of the data, the socalled inliers
Evaluating the performance of linear regression models
 Plot the residuals (the differences or vertical distances between the actual and predicted values) versus the predicted values to diagnose our regression model.
 Residual plots are a commonly used graphical tool for diagnosing regression models. They can help to detect nonlinearity and outliers, and check whether the errors are randomly distributed
Good regression model: errors randomly distributed and the residuals randomly scattered around the centerline

MSE: useful for comparing differente regression models or for tuning their parameters via grid search and crossvalidation

Rˆ2: coefficient of determination. Standardized version of the MSE > better interpretability of the model’s performance. Rˆ2 is the fraction of response variance that is captured by the model
Using regularized methods for regression

Regularization is one approach to tackling the problem of overfitting by adding additional information, and thereby shrinking the parameter values of the model to induce a penalty against complexity.

The most popular approaches to regularized linear regression are the socalled Ridge Regression, least absolute shrinkage and selection operator (LASSO), and elastic Net
Saturation of a model occurs if the number of training examples is equal to the number of features, which is a form of overparameterization. As a consequence, a saturated model can always fit the training data perfectly but is merely a form of interpolation and thus is not expected to generalize well
Dealing with nonlinear relationships using random forests
In the context of decision tree regression, the MSE is often referred to as withinnode variance, which is why the splitting criterion is also better known as variance reduction
If the distribution of the residuals does not seem to be completely random around the zero center point > the model was not able to capture all the exploratory information

The error of the predictions should not be related to any of the information contained in the explanatory variables; rather, it should reflect the randomness of the realworld distributions or patterns. If we find patterns in the prediction errors, for example, by inspecting the residual plot, it means that the residual plots contain predictive information

Improve the model by transforming variables, tuning the hyperparameters of the learning algorithm, choosing simpler or more complex models, removing outliers, or including additional variables
Ch11. Working with Unlabeled Data  Clustering Analysis
Clustering: find natural grouping in data > items in the same cluster are more similar to each other than to those from different clusters
Grouping objects by similarity using kmeans
kmeans
 belongs to the category of prototypebased clustering > each cluster is represented by a prototype (centroid for average or medoid for most representative)
 good at identifying clustrs with a spherical shape
 drawback > have to specify number of clusters,
k
, a priori  Make sure that the features are measured on the same scale > apply zscore standardization or minmax scaling
 clusters do not overlap and are not hierarchical
 assume that there is at least one item in each cluster
kmeans++
 smarter way of placing the inital cluster centroids
Hard versus soft clustering
Hard clustering: each example assigned to exactly one cluster. (e.g., kmeans)
Soft clustering (fuzzy clustering): assign an example to one or more clusters (e.g., fuzzy Cmeans = FCM = soft kmeans = fuzzy kmeans)
Both kmeans and FCM produce very similar clustering outputs
Using the elbow method to find the optimal number of clusters
 withingcluster SSE (distortion) > inertia_ attribute after fitting KMeans using scikitlearn
 elbow method: identify the value of k where the distortion begins to increase most rapidly
Quantifying the quality of clustering via silhouette plots
 silhouette coefficient: range 1 to 1. 0 if the cluster separation and cohesion are equal. 1 (ideal) if how dissimilar an example is from other cluster » how similar it is to the other examples in its own cluster
 silhouette plot with visibly different lengths and widths > evidence of bad or at least suboptimal clustering
Organizing clusters as a hierarchical tree
 allows us to plot dendograms
 do not need to specify the number of clusters upfront
 can be agglomerative (starts with every point as a cluster) or divisive (starts with one cluster and split iteratively)
Grouping clusters in a bottomup fashion
Algorithms for agglomerative hierarchical clustering:
 single linkage
 complete linkage
 average linkage
 Ward’s linkage
Dendograms are often used in combination with a heat map > represent individual values in the data array or matrix containing our training examples with a color code
Locating regions of high density via DBSCAN
DBSCAN > densitybased spatial clustering of applications with noise
 no assumptions about spherical clusters
 no partition of the data into hierarchies that require a manual cutoff point
 assigns cluster labels based on dense regions of points > density: number of points within a specified radius
 doesn’t necessarily assign each point to a cluster > is capable of removing noise points
 two hyperparameters to be optimized to yield good results > MinPts and eta
Disadvantage of the 3 algorithms presented: increasing number of features assuming fixed number of training examples > curse of dimensionality increases
It is common practice to apply dimensionality reduction techniques prior to performing clustering > PCA or KernelPCA
Also common to compress data down to twodimensional subspaces > visualization helps evaluating the results
Graphbased clustering: not covered in the book. e.g, spectral clustering
Ch12. Implementing a Multilayer Artificial Neural Network from Scratch
Singlelayer naming convention: Adaline consists of two layers, one input, and one output. It is called singlelayer network because of its single link between the input and output layers
Introducing the multilayer neural network architecture
Adding additional hidden layers: the error gradients, become increasingly small as more layers are added to a network > vanishing gradient problem
Activating a neural network via forward propagation
 Forward propagate the patterns of training data to generate an output
 Calculate the error to minimize using a cost function between outputs and targets
 Backpropagate the error, find its derivative wrt each weight in the network, update the model
 Multiple epochs of 13, then forward propagation to calculate the output and apply a threshold function to obtain the predicted class labels
MLP: typical example of a feedforward ANN > each layer serves as the input to the next layer without loops
Gradientbased optimization is much more stable under normalized inputs (ranging from 1 to 1). Also, Batch Normalization for improving convergence
Efficient method of save multidimensional NumPy arrays to disk > NumPy’s
savez
/savez_compressed
function > analogous topickle
, but optimized for np arrays. To load:np.load(file.npz)
Training (deep) NN is expensive > stop it early in certain conditions, e.g., starts overfitting, not improving
Common tricks to improve performance:
 Skipconnections
 Learning rate schedulers
 Attaching loss functions to earlier layers
Developing your understanding of backpropagation
Very computationally efficient approach to compute the partial derivatives of a complex cost function in multilayer NNs > Goal: use those derivatives to learn the weight coefficients for parametrizing such a multilayer artificial NN.
Backpropagation is a special case of a reversemode automatic differentiation. Matrixvector multiplication is computationally much cheaper than matrixmatrix multiplication
About the convergence in neural networks
The output function has a rough surface and the optimization algorithm can easily become trapped in local minima
By increasing the learning rate, we can more readily escape such local minima. But, we cal also increase the chance of overshooting the global optimum if the learning rate is too large