Month: December 2020
MMS • Bruno Couriol
Article originally posted on InfoQ. Visit InfoQ
Simon Peyton Jones, lead designer of the Glasgow Haskell compiler, recently announced the establishment of the Haskell Foundation. The Haskell Foundation is a non-profit organization dedicated to broadening the adoption of Haskell, by supporting its ecosystem of tools, libraries, education, and research. The foundation already gathered $200,000 in funding from corporate sponsors.
By Bruno Couriol
MMS • Ben Linders Aino Vonge Corry
Article originally posted on InfoQ. Visit InfoQ
Using the familiar “patterns” approach, the book Retrospectives Antipatterns by Aino Vonge Corry describes unfortunate situations that can sometimes happen in retrospectives. For each situation, described as an antipattern, it also provides solutions for dealing with the situation; this can be a way to solve the problem directly or avoid similar problems in future retrospectives.
By Ben Linders, Aino Vonge Corry
MMS • RSS
Article originally posted on Data Science Central. Visit Data Science Central
If you want some holiday reading, here is something for you
I think this is extraordinarily generous
Applications of Deep Neural Networks 575 page free book by Jeff Heaton
TOC below download links at the end of the post
Happy holidays 🙂
0.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv
1 Python Preliminaries 1
1.1 Part 1.1: Course Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Assignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.2 Your Instructor: Jeff Heaton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.3 Course Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.4 What is Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.5 What is Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.6 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.7 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.8 Beyond Classification and Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.9 What are Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.10 Why Deep Learning? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.11 Python for Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1.12 Software Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.13 Python Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.14 Jupyter Notebooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.15 Python Versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.16 Module 1 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2 Part 1.2: Introduction to Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3 Part 1.3: Python Lists, Dictionaries, Sets and JSON . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.1 Lists and Tuples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.3.2 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.3.3 Maps/Dictionaries/Hash Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.3.4 More Advanced Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.3.5 An Introduction to JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.4 Part 1.4: File Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.4.1 Read a CSV File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.4.2 Read (stream) a Large CSV File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
1.4.3 Read a Text File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
1.4.4 Read an Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.5 Part 1.5: Functions, Lambdas, and Map/Reduce . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.5.1 Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.5.2 Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.5.3 Lambda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.5.4 Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
v
vi CONTENTS
2 Python for Machine Learning 33
2.1 Part 2.1: Introduction to Pandas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.1.1 Missing Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.1.2 Dealing with Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.1.3 Dropping Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.1.4 Concatenating Rows and Columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.1.5 Training and Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.1.6 Converting a Dataframe to a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.1.7 Saving a Dataframe to CSV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.1.8 Saving a Dataframe to Pickle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.1.9 Module 2 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.2 Part 2.2: Categorical and Continuous Values . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.2.1 Encoding Continuous Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.2 Encoding Categorical Values as Dummies . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.2.3 Target Encoding for Categoricals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.2.4 Encoding Categorical Values as Ordinal . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.3 Part 2.3: Grouping, Sorting, and Shuffling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.3.1 Shuffling a Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.3.2 Sorting a Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.3.3 Grouping a Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.4 Part 2.4: Apply and Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.4.1 Using Map with Dataframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.4.2 Using Apply with Dataframes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.4.3 Feature Engineering with Apply and Map . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.5 Part 2.5: Feature Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.5.1 Calculated Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
2.5.2 Google API Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.5.3 Other Examples: Dealing with Addresses . . . . . . . . . . . . . . . . . . . . . . . . . 62
3 Introduction to TensorFlow 67
3.1 Part 3.1: Deep Learning and Neural Network Introduction . . . . . . . . . . . . . . . . . . . . 67
3.1.1 Classification or Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.1.2 Neurons and Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.1.3 Types of Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1.4 Input and Output Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.1.5 Hidden Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.1.6 Bias Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.1.7 Context Neurons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.1.8 Other Neuron Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.1.9 Why are Bias Neurons Needed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.1.10 Modern Activation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.1.11 Linear Activation Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.1.12 Rectified Linear Units (ReLU) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.1.13 Softmax Activation Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.1.14 Classic Activation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.1.15 Step Activation Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.1.16 Sigmoid Activation Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.1.17 Hyperbolic Tangent Activation Function . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.1.18 Why ReLU? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
CONTENTS vii
3.1.19 Module 3 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.2 Part 3.2: Introduction to Tensorflow and Keras . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.2.1 Why TensorFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.2.2 Deep Learning Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.2.3 Using TensorFlow Directly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.2.4 TensorFlow Linear Algebra Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.2.5 TensorFlow Mandelbrot Set Example . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.2.6 Introduction to Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.2.7 Simple TensorFlow Regression: MPG . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.2.8 Introduction to Neural Network Hyperparameters . . . . . . . . . . . . . . . . . . . . 89
3.2.9 Controlling the Amount of Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.2.10 Regression Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.2.11 Simple TensorFlow Classification: Iris . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.3 Part 3.3: Saving and Loading a Keras Neural Network . . . . . . . . . . . . . . . . . . . . . . 97
3.4 Part 3.4: Early Stopping in Keras to Prevent Overfitting . . . . . . . . . . . . . . . . . . . . . 99
3.4.1 Early Stopping with Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
3.4.2 Early Stopping with Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
3.5 Part 3.5: Extracting Weights and Manual Network Calculation . . . . . . . . . . . . . . . . . 104
3.5.1 Weight Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.5.2 Manual Neural Network Calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4 Training for Tabular Data 111
4.1 Part 4.1: Encoding a Feature Vector for Keras Deep Learning . . . . . . . . . . . . . . . . . . 111
4.1.1 Generate X and Y for a Classification Neural Network . . . . . . . . . . . . . . . . . . 115
4.1.2 Generate X and Y for a Regression Neural Network . . . . . . . . . . . . . . . . . . . 116
4.1.3 Module 4 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.2 Part 4.2: Multiclass Classification with ROC and AUC . . . . . . . . . . . . . . . . . . . . . . 116
4.2.1 Binary Classification and ROC Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.2.2 ROC Chart Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.2.3 Multiclass Classification Error Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.2.4 Calculate Classification Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.2.5 Calculate Classification Log Loss . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.3 Part 4.3: Keras Regression for Deep Neural Networks with RMSE . . . . . . . . . . . . . . . 128
4.3.1 Mean Square Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
4.3.2 Root Mean Square Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.3.3 Lift Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
4.4 Part 4.4: Training Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.4.1 Classic Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.4.2 Momentum Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
4.4.3 Batch and Online Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.4.4 Stochastic Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
4.4.5 Other Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.4.6 ADAM Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.4.7 Methods Compared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.4.8 Specifying the Update Rule in Tensorflow . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.5 Part 4.5: Error Calculation from Scratch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.5.1 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.5.2 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
viii CONTENTS
5 Regularization and Dropout 143
5.1 Part 5.1: Introduction to Regularization: Ridge and Lasso . . . . . . . . . . . . . . . . . . . . 143
5.1.1 L1 and L2 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.1.2 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
5.1.3 L1 (Lasso) Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
5.1.4 L2 (Ridge) Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.1.5 ElasticNet Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
5.2 Part 5.2: Using K-Fold Cross-validation with Keras . . . . . . . . . . . . . . . . . . . . . . . . 152
5.2.1 Regression vs Classification K-Fold Cross-Validation . . . . . . . . . . . . . . . . . . . 152
5.2.2 Out-of-Sample Regression Predictions with K-Fold Cross-Validation . . . . . . . . . . 152
5.2.3 Classification with Stratified K-Fold Cross-Validation . . . . . . . . . . . . . . . . . . 155
5.2.4 Training with both a Cross-Validation and a Holdout Set . . . . . . . . . . . . . . . . 158
5.3 Part 5.3: L1 and L2 Regularization to Decrease Overfitting . . . . . . . . . . . . . . . . . . . 160
5.4 Part 5.4: Drop Out for Keras to Decrease Overfitting . . . . . . . . . . . . . . . . . . . . . . . 163
5.5 Part 5.5: Benchmarking Regularization Techniques . . . . . . . . . . . . . . . . . . . . . . . . 167
5.5.1 Additional Reading on Hyperparameter Tuning . . . . . . . . . . . . . . . . . . . . . . 168
5.5.2 Bootstrapping for Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
5.5.3 Bootstrapping for Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.5.4 Benchmarking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
6 Convolutional Neural Networks (CNN) for Computer Vision 181
6.1 Part 6.1: Image Processing in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
6.1.1 Creating Images (from pixels) in Python . . . . . . . . . . . . . . . . . . . . . . . . . . 182
6.1.2 Transform Images in Python (at the pixel level) . . . . . . . . . . . . . . . . . . . . . 183
6.1.3 Standardize Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
6.1.4 Adding Noise to an Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
6.1.5 Module 6 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.2 Part 6.2: Keras Neural Networks for Digits and Fashion MNIST . . . . . . . . . . . . . . . . 189
6.2.1 Computer Vision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.2.2 Computer Vision Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
6.2.3 MNIST Digits Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
6.2.4 MNIST Fashion Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
6.2.5 CIFAR Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
6.2.6 Other Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
6.2.7 Convolutional Neural Networks (CNNs) . . . . . . . . . . . . . . . . . . . . . . . . . . 191
6.2.8 Convolution Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
6.2.9 Max Pooling Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6.2.10 TensorFlow with CNNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6.2.11 Access to Data Sets – DIGITS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
6.2.12 Display the Digits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
6.2.13 Training/Fitting CNN – DIGITS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
6.2.14 Evaluate Accuracy – DIGITS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
6.2.15 MNIST Fashion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
6.2.16 Display the Apparel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
6.2.17 Training/Fitting CNN – Fashion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
6.3 Part 6.3: Implementing a ResNet in Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
6.3.1 Keras Sequence vs Functional Model API . . . . . . . . . . . . . . . . . . . . . . . . . 209
6.3.2 CIFAR Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
6.3.3 ResNet V1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
CONTENTS ix
6.3.4 ResNet V2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
6.4 Part 6.4: Using Your Own Images with Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
6.5 Part 6.5: Recognizing Multiple Images with Darknet . . . . . . . . . . . . . . . . . . . . . . . 225
6.5.1 How Does DarkNet/YOLO Work? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
6.5.2 Using YOLO in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
6.5.3 Installing YoloV3-TF2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
6.5.4 Transfering Weights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
6.5.5 Running DarkFlow (YOLO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
6.5.6 Module 6 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
7 Generative Adversarial Networks 237
7.1 Part 7.1: Introduction to GANS for Image and Data Generation . . . . . . . . . . . . . . . . 237
7.2 Part 7.2: Implementing DCGANs in Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
7.3 Part 7.3: Face Generation with StyleGAN and Python . . . . . . . . . . . . . . . . . . . . . . 250
7.3.1 Keras Sequence vs Functional Model API . . . . . . . . . . . . . . . . . . . . . . . . . 251
7.3.2 Generating High Rez GAN Faces with Google CoLab . . . . . . . . . . . . . . . . . . 252
7.3.3 Run StyleGan2 From Command Line . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
7.3.4 Run StyleGAN2 From Python Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
7.3.5 Examining the Latent Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
7.4 Part 7.4: GANS for Semi-Supervised Training in Keras . . . . . . . . . . . . . . . . . . . . . . 259
7.4.1 Semi-Supervised Classification Training . . . . . . . . . . . . . . . . . . . . . . . . . . 259
7.4.2 Semi-Supervised Regression Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
7.4.3 Application of Semi-Supervised Regression . . . . . . . . . . . . . . . . . . . . . . . . 260
7.5 Part 7.5: An Overview of GAN Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
7.5.1 Select Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
8 Kaggle Data Sets 263
8.1 Part 8.1: Introduction to Kaggle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
8.1.1 Kaggle Ranks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
8.1.2 Typical Kaggle Competition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
8.1.3 How Kaggle Competition Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
8.1.4 Preparing a Kaggle Submission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
8.1.5 Select Kaggle Competitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
8.1.6 Module 8 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
8.2 Part 8.2: Building Ensembles with Scikit-Learn and Keras . . . . . . . . . . . . . . . . . . . . 265
8.2.1 Evaluating Feature Importance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
8.2.2 Classification and Input Perturbation Ranking . . . . . . . . . . . . . . . . . . . . . . 266
8.2.3 Regression and Input Perturbation Ranking . . . . . . . . . . . . . . . . . . . . . . . . 269
8.2.4 Biological Response with Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . 271
8.2.5 What Features/Columns are Important . . . . . . . . . . . . . . . . . . . . . . . . . . 273
8.2.6 Neural Network Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
8.3 Part 8.3: Architecting Network: Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . 277
8.3.1 Number of Hidden Layers and Neuron Counts . . . . . . . . . . . . . . . . . . . . . . . 278
8.3.2 Activation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
8.3.3 Advanced Activation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
8.3.4 Regularization: L1, L2, Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
8.3.5 Batch Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
8.3.6 Training Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
8.3.7 Experimenting with Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
x CONTENTS
8.4 Part 8.4: Bayesian Hyperparameter Optimization for Keras . . . . . . . . . . . . . . . . . . . 282
8.5 Part 8.5: Current Semester’s Kaggle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
8.5.1 Iris as a Kaggle Competition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
8.5.2 MPG as a Kaggle Competition (Regression) . . . . . . . . . . . . . . . . . . . . . . . . 292
8.5.3 Module 8 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
9 Transfer Learning 297
9.1 Part 9.1: Introduction to Keras Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . 297
9.1.1 Transfer Learning Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
9.1.2 Module 9 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
9.2 Part 9.2: Popular Pretrained Neural Networks for Keras . . . . . . . . . . . . . . . . . . . . . 304
9.2.1 DenseNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
9.2.2 InceptionResNetV2 and InceptionV3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
9.2.3 MobileNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
9.2.4 MobileNetV2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
9.2.5 NASNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
9.2.6 ResNet, ResNetV2, ResNeXt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
9.2.7 VGG16 and VGG19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
9.2.8 Xception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
9.3 Part 9.3: Transfer Learning for Computer Vision and Keras . . . . . . . . . . . . . . . . . . . 306
9.3.1 Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
9.4 Part 9.4: Transfer Learning for Languages and Keras . . . . . . . . . . . . . . . . . . . . . . . 316
9.5 Part 9.5: Transfer Learning for Keras Feature Engineering . . . . . . . . . . . . . . . . . . . . 323
10 Time Series in Keras 327
10.1 Part 10.1: Time Series Data Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
10.1.1 Module 10 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
10.2 Part 10.2: Programming LSTM with Keras and TensorFlow . . . . . . . . . . . . . . . . . . . 331
10.2.1 Understanding LSTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
10.2.2 Simple TensorFlow LSTM Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
10.2.3 Sun Spots Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
10.2.4 Further Reading for LSTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
10.3 Part 10.3: Text Generation with LSTM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
10.3.1 Additional Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
10.3.2 Character-Level Text Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
10.4 Part 10.4: Image Captioning with Keras and TensorFlow . . . . . . . . . . . . . . . . . . . . 350
10.4.1 Needed Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
10.4.2 Running Locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
10.4.3 Clean/Build Dataset From Flickr8k . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
10.4.4 Choosing a Computer Vision Neural Network to Transfer . . . . . . . . . . . . . . . . 355
10.4.5 Creating the Training Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
10.4.6 Using a Data Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
10.4.7 Loading Glove Embeddings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
10.4.8 Building the Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
10.4.9 Train the Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
10.4.10 Generating Captions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
10.4.11Evaluate Performance on Test Data from Flicker8k . . . . . . . . . . . . . . . . . . . . 364
10.4.12Evaluate Performance on My Photos . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
10.4.13Module 10 Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
CONTENTS xi
10.5 Part 10.5: Temporal CNN in Keras and TensorFlow . . . . . . . . . . . . . . . . . . . . . . . 368
10.5.1 Sun Spots Example – CNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370
11 Natural Language Processing and Speech Recognition 375
11.1 Part 11.1: Getting Started with Spacy in Python . . . . . . . . . . . . . . . . . . . . . . . . . 375
11.1.1 Installing Spacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
11.1.2 Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376
11.1.3 Sentence Diagramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
11.1.4 Stop Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
11.2 Part 11.2: Word2Vec and Text Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
11.2.1 Suggested Software for Word2Vec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380
11.3 Part 11.3: What are Embedding Layers in Keras . . . . . . . . . . . . . . . . . . . . . . . . . 383
11.3.1 Simple Embedding Layer Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
11.3.2 Transferring An Embedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
11.3.3 Training an Embedding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
11.4 Part 11.4: Natural Language Processing with Spacy and Keras . . . . . . . . . . . . . . . . . 391
11.4.1 Word-Level Text Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
11.5 Part 11.5: Learning English from Scratch with Keras and TensorFlow . . . . . . . . . . . . . 399
11.5.1 Imports and Utility Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
11.5.2 Getting the Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
11.5.3 Building the Vocabulary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
11.5.4 Building the Training and Test Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
11.5.5 Compile the Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
11.5.6 Train the Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
11.5.7 Evaluate Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
11.5.8 Adhoc Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
12 Reinforcement Learning 413
12.1 Part 12.1: Introduction to the OpenAI Gym . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
12.1.1 OpenAI Gym Leaderboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
12.1.2 Looking at Gym Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
12.1.3 Render OpenAI Gym Environments from CoLab . . . . . . . . . . . . . . . . . . . . . 416
12.2 Part 12.2: Introduction to Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
12.2.1 Introducing the Mountain Car . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
12.2.2 Programmed Car . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
12.2.3 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424
12.2.4 Running and Observing the Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
12.2.5 Inspecting the Q-Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429
12.3 Part 12.3: Keras Q-Learning in the OpenAI Gym . . . . . . . . . . . . . . . . . . . . . . . . . 430
12.3.1 DQN and the Cart-Pole Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430
12.3.2 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
12.3.3 Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433
12.3.4 Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
12.3.5 Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
12.3.6 Metrics and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
12.3.7 Replay Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
12.3.8 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
12.3.9 Training the agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
12.3.10 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442
xii CONTENTS
12.3.11 Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442
12.3.12 Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443
12.4 Part 12.4: Atari Games with Keras Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 444
12.4.1 Actual Atari 2600 Specs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444
12.4.2 OpenAI Lab Atari Pong . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
12.4.3 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446
12.4.4 Atari Environment’s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
12.4.5 Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
12.4.6 Metrics and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
12.4.7 Replay Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
12.4.8 Random Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
12.4.9 Training the agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
12.4.10 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
12.4.11 Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454
12.5 Part 12.5: Application of Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . 455
12.5.1 Create an Environment of your Own . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456
12.5.2 Testing the Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464
12.5.3 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
12.5.4 Instanciate the Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
12.5.5 Metrics and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
12.5.6 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
12.5.7 Training the agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
12.5.8 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
12.5.9 Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
12.5.10 Videos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473
13 Advanced/Other Topics 475
13.1 Part 13.1: Flask and Deep Learning Web Services . . . . . . . . . . . . . . . . . . . . . . . . 475
13.1.1 Flask Hello World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
13.1.2 MPG Flask . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 476
13.1.3 Flask MPG Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
13.1.4 Images and Web Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
13.2 Part 13.2: Deploying a Model to AWS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
13.2.1 Train Model (optionally, outside of AWS) . . . . . . . . . . . . . . . . . . . . . . . . . 483
13.2.2 Next Step: Convert Model (must use AWS SageMaker Notebook) . . . . . . . . . . . 484
13.2.3 Step 1. Set up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484
13.2.4 Step 2. Load the Keras model using the JSON and weights file . . . . . . . . . . . . . 485
13.2.5 Step 3. Export the Keras model to the TensorFlow ProtoBuf format (must use AWS
SageMaker Notebook) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
13.2.6 Step 4. Convert TensorFlow model to a SageMaker readable format (must use AWS
SageMaker Notebook) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
13.2.7 Tar the entire directory and upload to S3 . . . . . . . . . . . . . . . . . . . . . . . . . 487
13.2.8 Step 5. Deploy the trained model (must use AWS SageMaker Notebook) . . . . . . . . 488
13.2.9 Part 3: Test Model Deployment (optionally, outside of AWS) . . . . . . . . . . . . . . 489
13.2.10 Call the end point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489
13.2.11Additional Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 490
13.3 Part 13.3: Using a Keras Deep Neural Network with a Web Application . . . . . . . . . . . . 490
13.4 Part 13.4: When to Retrain Your Neural Network . . . . . . . . . . . . . . . . . . . . . . . . 491
13.4.1 Preprocessing the Sberbank Russian Housing Market Data . . . . . . . . . . . . . . . 493
CONTENTS xiii
13.4.2 KS-Statistic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494
13.4.3 Detecting Drift between Training and Testing Datasets by Training . . . . . . . . . . 496
13.5 Part 13.5: Using a Keras Deep Neural Network with a Web Application . . . . . . . . . . . . 497
13.5.1 Converting Keras to CoreML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 498
13.5.2 Creating an IOS CoreML Application . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
13.5.3 More Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500
14 Other Neural Network Techniques 503
14.1 Part 14.1: What is AutoML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
14.1.1 AutoML from your Local Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
14.1.2 AutoML from Google Cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
14.1.3 A Simple AutoML System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
14.1.4 Running My Sample AutoML Program . . . . . . . . . . . . . . . . . . . . . . . . . . 514
14.2 Part 14.2: Using Denoising AutoEncoders in Keras . . . . . . . . . . . . . . . . . . . . . . . . 516
14.2.1 Function Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516
14.2.2 Multi-Output Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
14.2.3 Simple Autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 520
14.2.4 Autoencode (single image) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
14.2.5 Standardize Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522
14.2.6 Image Autoencoder (multi-image) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525
14.2.7 Adding Noise to an Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
14.2.8 Denoising Autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
14.3 Part 14.3: Anomaly Detection in Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
14.3.1 Read in KDD99 Data Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
14.3.2 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540
14.3.3 Training the Autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543
14.3.4 Detecting an Anomaly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
14.4 Part 14.4: Training an Intrusion Detection System with KDD99 . . . . . . . . . . . . . . . . . 545
14.4.1 Read in Raw KDD-99 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 545
14.4.2 Analyzing a Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
14.4.3 Encode the feature vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549
14.4.4 Train the Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 551
14.5 Part 14.5: New Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
14.5.1 Neural Structured Learning (NSL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
14.5.2 Bert, AlBert, and Other NLP Technologies . . . . . . . . . . . . . . . . . . . . . . . . 554
14.5.3 Explainability Frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557
link to download HERE
created by heaton research
youtube channnel heaton research
Image source pixabay
MMS • RSS
Article originally posted on Data Science Central. Visit Data Science Central
This article was written by Johannes Rieke.
Abstract
A very lightweight tutorial to object detection in images. We will bootstrap simple images and apply increasingly complex neural networks to them. In the end, the algorithm will be able to detect multiple objects of varying shapes and colors (image below). You should have a basic understanding of neural networks to follow along.
Image analysis is one of the most prominent fields in deep learning. Images are easy to generate and handle, and they are exactly the right type of data for machine learning: easy to understand for human beings, but difficult for computers. Not surprisingly, image analysis played a key role in the history of deep neural networks.
In this blog post, we’ll look at object detection — finding out which objects are in an image. For example, imagine a self-driving car that needs to detect other cars on the road. There are lots of complicated algorithms for object detection. They often require huge datasets, very deep convolutional networks and long training times. To make this tutorial easy to follow along, we’ll apply two simplifications: 1) We don’t use real photographs, but images with abstract geometric shapes. This allows us to bootstrap the image data and use simpler neural networks. 2) We predict a fixed number of objects in each image. This makes the entire algorithm a lot, lot easier (it’s actually surprisingly simple besides a few tricks). At the end of the post, I will outline how one can expand on this approach to detect many more objects in an image.
I tried to make this tutorial as simple as possible: I will go step by step, starting with detection of a single object. For each step, there’s a Jupyter notebook with the complete code in this github repo. You don’t need to download any additional dataset. The code is in Python plus keras, so the networks should be easy to understand even for beginners. Also, the networks I use are (mostly) very simple feedforward networks, so you can train them within minutes.
Table of contents
- Detecting a single object
- Detecting multiple objects
- Putting it all together: Shapes, Colors, and Convolutional Neural Networks
- Real-world objects
To read the whole article, with each point detailed and illustrated, click here.
DSC Ressources
- Free Book and Resources for DSC Members
- New Perspectives on Statistical Distributions and Deep Learning
- Deep Analytical Thinking and Data Science Wizardry
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- Comprehensive Repository of Data Science and ML Resources
- Advanced Machine Learning with Basic Excel
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- Selected Business Analytics, Data Science and ML articles
- Hire a Data Scientist | Search DSC | Find a Job
- Post a Blog | Forum Questions
MMS • RSS
Article originally posted on Data Science Central. Visit Data Science Central
This article was written by Berkeley AI Research (BAIR) Laboratory.
Abstract
We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss func- tion to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demon- strate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this pa- per, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.
Table of contents
1. Introduction
2. Related work
3. Method
3.1. Objective
3.2. Network architectures
3.2.1 Generator with skips
3.2.2 Markovian discriminator (PatchGAN)
3.3. Optimization and inference
4. Experiments
4.1. Evaluation metrics
4.2. Analysis of the objective function
4.3. Analysis of the generator architecture
4.4. From PixelGANs to PatchGANs to ImageGANs
4.5. Perceptual validation
4.6. Semantic segmentation
4.7. Community-driven Research
5. Conclusion
To read the whole article, with each point detailed and illustrated, click here.
DSC Ressources
- Free Book and Resources for DSC Members
- New Perspectives on Statistical Distributions and Deep Learning
- Deep Analytical Thinking and Data Science Wizardry
- Statistical Concepts Explained in Simple English
- Machine Learning Concepts Explained in One Picture
- Comprehensive Repository of Data Science and ML Resources
- Advanced Machine Learning with Basic Excel
- Difference between ML, Data Science, AI, Deep Learning, and Statistics
- Selected Business Analytics, Data Science and ML articles
- Hire a Data Scientist | Search DSC | Find a Job
- Post a Blog | Forum Questions
Gross Underestimates and Overestimates from the Same Data: Covid-19 Death Rates Example
MMS • RSS
Article originally posted on Data Science Central. Visit Data Science Central
- Ten-fold differences have been reported in Covid-19 death rates.
- Estimate issues are because of how the data is calculated.
- Why the same data can yield gross overestimates and underestimates.
What is the actual death rate for Covid-19? After nearly a year of the pandemic, no one can agree on an answer. Depending on which expert you ask, it’s somewhere between 0.53% and 6% for the general population and possibly as high as 35% for older patients with certain pre-existing conditions. These widely varying figures illustrate how difficult it is to take data and make predictions–even with the best data and machine learning tools at your disposal.
Part of the problem is clarity: news sources and blogs in particular cite statistics without clarifying exactly what statistic they are talking about. For example, one MedPageToday article [1] mentions a “fatality rate”; The article doesn’t make it immediately clear if that fatality rate applies to hospitalized cases, cases who have tested positive, or the population as a whole (each of these rates would be vastly different).
Despite Covid-19- fueling one of the largest explosions of scientific literature in history [2], we’re not even close to accurately figuring out what percentage of the population the virus actually kills. All of the reported figures amount to nothing more than data-driven guesswork.
The Anatomy of a Death Rate Calculation
A recent episode of The Guardian’s Science Weekly Update [3] addressed the question of why Covid-19 fatality rates vary so much. In the program, Paul Hunter, professor of medicine at the University of East Anglia, explains that the figures such as the World Health Organization’s reported death rate of 3.4 percent was calculated by the number of Covid-19 related deaths (as recorded on death certificates) divided by the number of confirmed cases (based on positive Covid-19 tests). That figure, called the “case fatality rate (CFR)” is is a statistic that Dr. Robert Pearl M.D. calls “inaccurate and misleading” [4]. Why? Depending on how you look at it, it’s either a gross underestimate or overestimate.
The estimation issues arise because of how the figures are calculated. The CFR is calculated by recording people at the beginning of their illness and at the end of their illness–people who are still ill when the data is being recorded may still go on to die after the figures have been tallied. The 3.4% then is an underestimate–the people who are currently sick and go on to die will push that percentage up to around 5 to 6%. Although around 5% might seem like a reasonable estimate (probably one that matches figures you’ve often seen in the news), note how the above figure is obtained in the first place– the number of deaths divided by the number of cases. There are an unknown number of people, possibly up to 10 times higher than the official counts [5] of people with the virus who don’t get tested. If we could count all of these cases, most of which are probably mild or asymptomatic, then the death rate would be significantly lower, meaning that 3.4% is actually an overestimate. The actual number of deaths as it relates to the actual number of cases in the population is called the “true infection fatality rate (IFR)” and may be as low as 0.53% [7].
The solution to more accurate reporting seems clear: find more cases. But this isn’t as easy as it sounds. One recent study showed that in France, a paltry 10% of Covid-19 cases were actually detected [8].
Throwing in a Few More Complications
Complicating matters even further is that, geographically speaking, detection rates also vary widely . When you try to compare death rate between countries, there may be more than a 20-fold difference in identified cases [5].
Other issues that have lead to overestimates include not accounting for an aging population [6] or the presence of pre-existing medical conditions; The fatality rate for younger, healthier individuals is significantly lower than for older individuals with pre-existing conditions. Researchers at Johns Hopkins used machine learning to discover that age is the strongest predictor of who dies from Covid-19, ranging from a 1% fatality rate for the under-50s to a whopping 34 percent for those over age 85. However, those figures were also based on patients who are symptomatic and is therefore also an overestimate of death risk.
Calculating Covid-19 death rates in the population is a challenge, and case counts are unreliable. In general, we can say that the organizations that are better at identifying mild cases will have the most accurate figures. However, identifying which organization is more “accurate” at this task is a challenge in itself.
Data Doesn’t Always Tell the Right Picture
The fact is, an analysis is only going to be as good as the data at hand. Collecting and analyzing data opens up a myriad of possible statistical biases, [no term] all of which can completely ruin your analysis. And then–assuming you have reliable [no term] data–it then becomes a matter of clearly communicating your results to the general public: a matter which as the above example shows, is no easy task.
References
[1] Here’s Why COVID-19 Mortality Has Dropped
[2] Scientists are drowning in COVID-19 papers. Can new tools keep them afloat?
[3] Covid-19: why are there different fatality rates? – Science Weekly Podcast
[4] Three Misleading, Dangerous Coronavirus Statistics
[8] COVID research updates: How 90% of French COVID cases evaded detection
Image: CDC (Public Domain)
MMS • RSS
Article originally posted on Data Science Central. Visit Data Science Central
Is it accurate to mention that you are simply checking out top software development trends in 2021? All things considered, you’re within the perfect spot! Here you’ll peruse different software development trends of 2021. The greater a part of the IT anticipates come up short? It’s hardly any stunning for a big number of folks who realize the complexities and dynamic nature of the software development industry. So on make due in this capricious industry of consistent developments and headways, it’s mandatory to coordinate its pace.
2020 has been a year pretty much all the inclining advances like computer-generated experience, expanded reality, AI, AI, blockchain, and various others. 2021 will bring back us some new advancements. These advances are acquainted with satisfying the stress of clients by proficient software engineers. it’s safe to mention that you simply are desirous to use the drifting advances in your software development venture? we should always determine about the software development trends for 2021.
Know The Software Development Trends 2021
Like it or not, 2021 is practically round the bend and, with it, plenty of latest advancements, trends and approaches are getting to the software development world. Given the speed at which a number of them are rising at this moment, all things considered, 2021 are going to be the year some of those trends detonate and become standards. From the increasing strength of AI and therefore the universality of the web of Things to the moving idea of outsourcing services, there are many desires around what 2021 will bring.
5G Available for Wider Regions
One of the foremost foreseen innovative advances within the most up-to-date decade could at long last be available to the general population in 2021. In spite of the very fact that present in chosen urban communities and zones starting at now, everything focuses to at least one year from now because the year where 5G bring its 20 Gbps fast remote networking, lower inertness, and more prominent bandwidth accessibility. 5G makes bound to alter the gadgets we use regularly, with cell phones at the highest.
However, 5G is something aside from improved paces for our versatile perusing. The upgraded qualities will evacuate certain impediments present in 4G. during this manner, designers will have the choice to form increasingly powerful apps, with new functionalities, particularly those encompassing video and enlarged reality. The technology will likewise mean an enormous improvement in security and therefore the exactness of data for development based apps.
Internet of Things (IoT)
IoT has become extremely popular among tons of individuals nowadays. The all-out introduced base of IoT associated gadgets goes to require an immense jump forward in 2021, towards a fivefold increment from 2015 which will carry the mixture sum to 75.44 billion gadgets worldwide in 2025. In other words, this may get intrigued by these gadgets to the foremost extreme which, thusly, will support demand for IoT software.
As that happens, an ever-increasing a number of engineers and software outsourcing organizations will have some expertise altogether things IoT, with stress on abilities, for instance, cloud processing, robotization, and information investigation taking the spotlight.
The predominance of Artificial Intelligence (AI)
Though AI has been around for several years, it continues to grow and enhance per annum, becoming foundational to several technologies round the world. Developers predict that AI will still dominate as more industries adopt it in their infrastructures.
Artificial intelligence-based software isn’t exactly another thing however the in no time speed at which it’s being embraced across ventures will make them cement its situation in 2021. As indicated by ongoing forecasts, the event of the AI software showcase on an overall level will keep its mind-boggling pace one year from now. Accordingly, the worldwide AI software advertise is relied upon to become 154% to a surprising size of $14.7 billion.
Such development is really the equivalent anticipated for the finish of 2019 and an identical one expected for 2021. Those figures come to point out that AI isn’t only a pattern any more it’s become a prevailing force within the innovative universe of today. From the pervasive chatbots that are presently overhauling an outsized number of internet sites from around the globe to prescriptive investigative arrangements fit recommending answers for business issues hooked into immense measures of data, AI (and AI, one among its most famous branches) are going to be highlighted across an ever-increasing number of things throughout most ventures.
Python Predicted to Rising Alongside AI and Ml Developments
From ML research to video games development to web development, Python consistently proves to be a well-liked and well-loved language. Since ML and AI development is on the increase, Python is predicted to continue during this steady growth and recognition, particularly for impressive innovations including ML-powered chatbots.
Although the concept of the “fastest-growing” language is often hard to pin down, data suggests that Python may take the cake. Not only is Python used for a spread of popular fields and jobs, but it also features a low entry barrier and a supportive community fostered by the most recent generation of developers.
Outsourcing for the Experience Economy
The probability that we, as a general public, are revering experiences over material things is getting progressively obvious as time passes. For what reason would 2021 be any unique? All things considered, there’s a big motivation behind why it’s worth to research the experience economy over the span of 1 year from now. that’s because reports gauge that 2021 are going to be the year where client experience will surpass cost and item because of the key brand differentiator.
That is an enormous move since the experience will become a definitive objective for all businesses, particularly for the software business. The experience economy will without a doubt open new pathways for advancement and market openings, so it’s no big surprise that the whole software development world will change to deal with that. the best difference within the mall? you’ll discover it in the redistribute development field.
Progressive Web Apps (PWAs)
Everything focuses on 2021 a bit like the year where we at long last begin to bid farewell to versatile locales for progressive web apps (PWAs). These are a kind of website that seems like an area versatile application in spite of the very fact that it runs within the program. It resembles a mix of the simplest of the 2 universes – and organizations are starting to acknowledge it.
In this way, instead of creating versatile sorts of their websites (or investing energy and cash building portable apps that have a high possibility of failing to be used), PWAs show up as a perfect account the 2 organizations and guests. On one hand, organizations can build up PWA tons of snappier and obtain similar advantages they might get from an application. but, guests can interface with an application like condition that heaps quicker and it’s a minimum of somewhat secure.
Blockchain Technology
It doesn’t make a difference within the event that you simply pursue Bitcoin or not, you’re more likely than not heard the unexpected ascent and fall within the estimation of Bitcoin in December of 2018. All things considered, we won’t discuss Bitcoin or Ethereum here however the technology behind cryptographic sorts of money – Blockchain.
At first, Blockchain was restricted to the fund business yet it’s currently moving to other areas too, for instance, medicinal services, coordination’s, open organization software development, and more. The blockchain technology is one among the software development trends which will enable your business to reach new statures.
Cross-Platform Development
The period of local apps goes down and cross-platform development is taking up due to its undeniable advantages. Engineers and entrepreneurs are putting resources into cross-platform development utilizing the freshest systems and programming dialects for it.
Software engineers are slanting towards programming dialects that incorporate highlights of a minimum of two. Perhaps the simplest case of this is: Kotlin and Scala assuming control over past JAVA in android application development. While JAVA is an item situated programing language, Kotlin and Scala offer articles arranged programming and utilitarian programming simultaneously.
Use of Scanners and Sensors
Of late, versatile apps that use sensors and scanners to satisfy their motivation are getting an integral factor, particularly with the increasing business of IoT gadgets. the entire market is so far advancing and sensors increment the chances of latest applications. for instance, there are portable apps that permit you to control your TV utilizing your telephone.
Cybersecurity – Secure Internet
The idea of cybersecurity for instance security over the web isn’t new. because the whole populace is bitten by bit moving on the web, the necessity to stay it secure seems to be increasingly critical. watching things objectively, you depend upon the web for various things, for instance, bank exchanges, cloud stockpiling, correspondence, and whatnot!
Cybersecurity assures that your own and expert professional information is secured from digital assaults. Following the continual software pattern, by 2021, the necessity to use cybersecurity for the wellbeing of users and software will increment altogether. This must be executed by procuring engineers who can plan custom applications and software for your business
Increasing Popularity of Cloud Services
Cloud has been within the marketplace for quite while now. Indeed, even advancements during this rundown, for instance, IoT and Blockchain work as a cloud administration. the difficulty of data stockpiling is genuine and cloud offers a solid and secure option in contrast thereto. The market is moving from utilizing physical information servers to cloud and your business should become a bit of it at the foremost punctual.
Latest technology in software development – Explore some current and latest software development trends 2021 like AI, Blockchain, IoT, etc.