House Price Prediction using a Random Forest Classifier

House Price Prediction using a Random Forest Classifier

Concerns of Autonomous ARTIFICIAL INTELLIGENCE in Medical Imaging

HUMAN IDENTIFICATION BASED ON IRIS DETECTION TECHNOLOGY

DRIVERLESS CARS -THE ETHICS OF AUTONOMOUS VEHICLES

ARTIFICIAL INTELLIGENCE's IMPACT ON FIN-TECH

Security and Privacy issues with IoT (Internet of Things) - Top barriers to IoT success

Real World Examples of IoT (Internet of Things) - How will IoT change our lives?

Internet of Things (IoT) - Next Stage of Information Revolution

Solving a regression problem using a Sequential Neural Network Model in Keras

All about Keras Framework in Deep Learning

Building a CNN model in Keras using MNIST dataset

Building a simple sequential neural network with dense layers in Keras

Transfer Learning and Fine Tuning a model in Deep Learning

Difference between Sigmoid and Softmax function in deep learning

Regularization Techniques used in Neural Networks in Deep Learning

Hyperparameter Tuning in Neural Networks in Deep Learning

Hyperparameter Tuning in Neural Networks in Deep Learning

In order to minimize the loss and determine optimal values of weight and bias, we need to tune our neural network hyper-parameters. Hyperparameters are the parameters that the neural network can’t learn itself via gradient descent or some other variant.

Hyper-parameters are opposite of learnable parameters. Learnable parameters are automatically learned and then optimized by the neural network. For example, weights and bias are learnable by the neural networks. These are also called trainable parameters as these are optimized during the training process using gradient descent.

This is our responsibility to provide optimal values for these hyper-parameters from our experience, domain knowledge and cross-validation. We need to manually tweak these hyperparameters to get better accuracy from the neural network.

Following is the list of hyperparameters used in neural networks:

1. Number of hidden layers: Keep adding the hidden layers until the loss function does not minimize to a certain extent. General rule is that we should a use a large number of hidden layers with proper regularization technique.

2. Number of units or neurons in a layer: Larger number of units in a layer may cause overfitting. Smaller number of units may cause underfitting. So, try to maintain a balance and use dropout technique.

3. Dropout: Dropout is regularization technique to avoid overfitting thus increasing the generalizing capabilities of the neural network. In this technique, we deliberately drop some units in a hidden layer to introduce generalization capabilities into it. Dropout value should range in between 20%-50% of number of neurons in a layer.

4. Activation Function: Activation functions introduce non-linearity in a neural network. Sigmoid, Step, Tanh, ReLU, Softmax are the activation functions. Mainly we use ReLU activation function for hidden layers and softmax for output layer.

5. Learning Rate: Learning rate determines how quickly weights and bias are updated in a neural network. If the learning rate is very small, learning process will significantly slow down and the model will converge too slowly. It may also also end up in local minima and never reach global minima. Larger learning rate speeds up the learning but may not converge.

Learning rate is normally set somewhere between 0.01 to 0.0001. Usually a decaying learning rate is preferred.

6. Momentum: Momentum helps in accelerating SGD in a relevant direction. Momentum helps to know the direction of the next step with the knowledge of the previous steps. It helps to prevent oscillations by adding up the speed. A typical choice of momentum should be between 0.5 to 0.9.

7. Number of epochs: Number of epochs is the number of times the whole training data is shown to the network while training. Default number of epochs is 1.

8. Batch size: Batch size is the number of samples passed to the network at one time after which parameter update happens. This is also called mini-batch. It should be in power of 2. Default batch size is 128.

9. Weight Initialization: Biases are typically initialized to 0 (or close to 0), but weights must be initialized carefully. Their initialization can have a big impact on the local minimum found by the training algorithm.

If weight is too large: During back-propagation, it will lead to exploding gradient problem. It means, the gradients of the cost with the respect to the parameters are too big. This leads the cost to oscillate around its minimum value.

If weight is too small: During back-propagation, it will lead to vanishing gradient problem. The gradients of the cost with respect to the parameters are too small, leading to convergence of the cost before it has reached the minimum value.

So, initializing weights with inappropriate values will lead to divergence or a slow-down in the training of the neural network.

To prevent this vanishing and exploding problem, we usually assign random numbers for weights in such a way that weights are normally distributed (mean = 0, standard deviation = 1).

For more details on weight initialization, please visit my this post.

10. Loss Function: The loss function compares the network's output for a training example against the intended output. A common general-purpose loss function is the Squared Errors loss function. When the output of the neural network is being treated as a probability distribution (e.g. a softmax output layer is being used), we generally use the cross-entropy as a loss function.

Hyperparameter Tuning: Following are some ways to tune hyperparameters in a neural network:

1. Coordinate Descent: It keeps all hyperparameters fixed except for one, and adjust that hyperparameter to minimize the validation error.

2. Grid Search: Grid search tries each and every hyperparameter setting over a specified range of values. This involves a cross-product of all intervals, so the computational expense is exponential in the number of parameters. Good part is that it can be easily parallelized.

3. Random Search: This is opposite of grid search. Instead of taking cross-product of all the intervals, it samples the hyperparameter space randomly. It performs better than grid search because grid search can take an exponentially long time to reach a good hyperparameter subspace. This can also be parallelized.

4. Cross-validation: We can also try cross-validation by trying different portions of dataset during training and testing.

Global and Local Minima in Gradient Descent in Deep Learning

What is Dropout? How does it prevent overfitting in a neural network?

Basic Frameworks to implement Deep Learning Algorithms

Gradient Descent Algorithm in Deep Learning: Batch, Stochastic and Mini Batch

Autoencoders in Deep Learning: Components, Types and Applications

A comparison between Machine Learning and Deep Learning (Machine Learning vs Deep Learning)

Activation (Squashing) Functions in Deep Learning: Step, Sigmoid, Tanh and ReLu

Basic introduction of various layers in CNN (Convolutional Neural Network)

TensorFlow: Tensors, Computational Graphs, Nodes, Estimators and TensorBoard

Tuples in Python: Indexing, Slicing, Packing, Unpacking, Concatenation, Repetition, Comparison, Membership, Iteration

Basic introduction of RNN (Recurrent Neural Network) in Deep Learning

Difference between Decision Tree and Random Forest in Machine Learning

Advantages and Disadvantages of Linear Regression in Machine Learning

Data Visualization using Pair Grid and Pair Plot (Seaborn Library)

Data Visualization using Regression Plot (Seaborn Library)

Data Visualization using FacetGrid (Seaborn Library)

Data Visualization using Heatmap (Seaborn Library)

Data Visualization using Bar Plot (Seaborn Library)

Artificial Intelligence in Sports

Beautifulsoup: Scraping Google with Python for Searching Job

Beautifulsoup: Scraping Google with Python for Searching Job !

Neural Networks (Introduction & Architecture)

Correlation vs Covariance

Outlier Detection

What are type I and type II errors?

Linear Regression

Artificial Neural Networks : Interview Questions

DIFFERENCE BETWEEN LINEAR REGRESSION AND LOGISTIC REGRESSION

Digit classification using convolutional neural network in keras.

practical coding Text Classification using neural networks part 1

create a chatbot using python part 1

Word Stemming

You may have heard me talk about word stemming in the previous tutorial. Stemming a word is attempting to find the root of the word. For example, the word "thats" stem might be "that" and the word "happening" would have the stem of "happen". We will use this process of stemming words to reduce the vocabulary of our model and attempt to find the more general meaning behind sentences.

words = [stemmer.stem(w.lower()) for w in words if w != "?"]
words = sorted(list(set(words)))

labels = sorted(labels)

This code will simply create a unique list of stemmed words to use in the next step of our data preprocessing.

Bag of Words

Now that we have loaded in our data and created a stemmed vocabulary it's time to talk about a bag of words. As we know neural networks and machine learning algorithms require numerical input. So out list of strings wont cut it. We need some way to represent our sentences with numbers and this is where a bag of words comes in. What we are going to do is represent each sentence with a list the length of the amount of words in our models vocabulary. Each position in the list will represent a word from our vocabulary. If the position in the list is a 1 then that will mean that the word exists in our sentence, if it is a 0 then the word is nor present. We call this a bag of words because the order in which the words appear in the sentence is lost, we only know the presence of words in our models vocabulary.

As well as formatting our input we need to format our output to make sense to the neural network. Similarly to a bag of words we will create output lists which are the length of the amount of labels/tags we have in our dataset. Each position in the list will represent one distinct label/tag, a 1 in any of those positions will show which label/tag is represented.

training = []
output = []

out_empty = [0 for _ in range(len(labels))]

for x, doc in enumerate(docs_x):
    bag = []

    wrds = [stemmer.stem(w.lower()) for w in doc]

    for w in words:
        if w in wrds:
            bag.append(1)
        else:
            bag.append(0)

    output_row = out_empty[:]
    output_row[labels.index(docs_y[x])] = 1

    training.append(bag)
    output.append(output_row)

Finally we will convert our training data and output to numpy arrays.

training = numpy.array(training)
output = numpy.array(output)

Full Code

import nltk
from nltk.stem.lancaster import LancasterStemmer
stemmer = LancasterStemmer()

import numpy
import tflearn
import tensorflow
import random
import json


with open("intents.json") as file:
    data = json.load(file)



words = []
labels = []
docs_x = []
docs_y = []

for intent in data["intents"]:
    for pattern in intent["patterns"]:
        wrds = nltk.word_tokenize(pattern)
        words.extend(wrds)
        docs_x.append(wrds)
        docs_y.append(intent["tag"])

    if intent["tag"] not in labels:
        labels.append(intent["tag"])

words = [stemmer.stem(w.lower()) for w in words if w != "?"]
words = sorted(list(set(words)))

labels = sorted(labels)

training = []
output = []

out_empty = [0 for _ in range(len(labels))]

for x, doc in enumerate(docs_x):
    bag = []

    wrds = [stemmer.stem(w.lower()) for w in doc]

    for w in words:
        if w in wrds:
            bag.append(1)
        else:
            bag.append(0)

    output_row = out_empty[:]
    output_row[labels.index(docs_y[x])] = 1

    training.append(bag)
    output.append(output_row)


training = numpy.array(training)
output = numpy.array(output)