Text Messages Classification using LSTM, Bi-LSTM, and GRU

Text classification also known as text tagging or text categorization is the process of categorizing text into organized groups. By using Natural Language Processing (NLP), text classifiers can automatically analyze text and then assign a set of pre-defined tags or categories based on its content.

This article aims to conduct a binary classification model to detect which text messages are spam or not spam (ham). Moreover, we also added a section to predict or detect ham or spam using text message which we’ve never seen before.

We use public dataset from: UCL datasets. It contains 5.574 SMS phone messages. The data were collected for the purpose of the mobile phone spam research. The data have been labeled as either spam or not spam (ham).

We will use the Dense classifier, Long Short Term Memory (LSTM), Bi-directional Long Short Term Memory (Bi-LSTM) and Gated Recurrent Unit (GRU) as our method and compare all of those methods in terms of the model performance.

Here are the steps to do the experiment:

• Import library
• Load the dataset
• Visualize ham or spam message using wordcloud
• Handling imbalance data
• Text preprocessing
• Define the model architecture and train the four model
• Comparing the result of the four different models
• Use the final trained model to classify the new messages

Step 1. Import Library

Let’s import the libraries that we need:

# Load, explore and plot data
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
%matplotlib inline
# Train test split
from sklearn.model_selection import train_test_split
# Text pre-processing
import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.callbacks import EarlyStopping
# Modeling
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, GRU, Dense, Embedding, Dropout, GlobalAveragePooling1D, Flatten, SpatialDropout1D, Bidirectional

Step 2. Load the Dataset

The dataset that we used in this article is from UCI Machine Learning Repository: SMS Spam Collection Dataset. However, to make things easier in this article, we try to access the dataset from this github : https://raw.githubusercontent.com/kenneth-lee-ch/SMS-Spam-Classification/master/spam.csv .

df = pd.read_csv('https://raw.githubusercontent.com/kenneth-lee-ch/SMS-Spam-Classification/master/spam.csv', encoding='ISO-8859-1')
# rename the columns
df = df[['v1','v2']]
df.rename(columns={'v1':'label', 'v2':'message'}, inplace=True)
df.head()

The summary statistics to better understand the data:

df.describe()

df.groupby('label').describe().T

Step 3. Visualize ham or spam message using wordcloud

The next step is to visualize the most frequently occurring words in a given text using WordCloud.

The WordCloud of the ‘ham’ message:

ham_msg_cloud = WordCloud(width =520, height =260, stopwords = STOPWORDS, max_font_size = 50, background_color = "black", colormap = 'Pastel1').generate(ham_msg_text)
plt.figure(figsize=(16,10))
plt.imshow(ham_msg_cloud, interpolation = 'bilinear')
plt.axis('off') # turn off axis
plt.show()

The words that appear most frequently in ham messages based on WordCloud above are: now, will, ok, today, Sorry etc.

The WordCloud of the ‘spam’ message:

spam_msg_cloud = WordCloud(width =520,
                          height =260,
                          stopwords=STOPWORDS,
                          max_font_size=50, 
                          background_color ="black",
                         colormap='Pastel1').generate(spam_msg_text)
plt.figure(figsize=(16,10))
plt.imshow(spam_msg_cloud, interpolation='bilinear')
plt.axis('off') # turn off axis
plt.show()

The words that appear most frequently in spam messages from WordCloud above are: FREE, call, URGENT, mobile, etc.

Step 4. Handling imbalance data

The next step: Let’s look at how ham and spam messages are distributed.

plt.figure(figsize=(8,6))
sns.countplot(df.label)
plt.title('The distribution of ham and spam messages')

Based on the visualization of the data distribution above, it can be concluded that the data is imbalanced. There are most frequent ham messages than the spam messages.

Approach to deal with the imbalanced dataset problem:

• Choose proper evaluation metric
• Resampling (oversampling and undersampling)
• Synthetic Minority Oversampling Technique (SMOTE)
• BalancedBaggingClassifier
• Threshold moving

In this article, we use the undersampling method for handling the unbalanced data. Undersampling is to under-sample the majority class randomly and uniformly. This can potentially lead to the loss of information. But if the examples of the majority class are near to others, this method might yield good results.

# downsample the ham msg
ham_msg_df = ham_msg.sample(n = len(spam_msg), random_state = 44)
spam_msg_df = spam_msg

The distribution of ham and sample messages (after downsampling) below, shows a similar distribution across the message types after accounting for the imbalanced data:

Step 5. Text preprocessing

5.1. Get length column for each text and convert the text label to numeric value:

After we get a final dataframe, next we add the text_length column (the length of each of the text message) and the msg_type column (the converted numeric label of the data).

# Get length column for each text
msg_df['text_length'] = msg_df['message'].apply(len)

msg_df['msg_type'] = msg_df['label'].map({'ham':0, 'spam':1})
msg_label = msg_df['msg_type'].values
msg_df.head()

5.2. Train test split

After that we do a train test split to divide the data into 80% train data and 20% test data.

x_train, x_test, y_train, y_test = train_test_split(msg_df['message'], msg_label, test_size=0.2, random_state=434)

5.3. Tokenization

We need to convert the text message data into numerical representation, so the model will understand it.

# Defining pre-processing parameters
max_len = 50 
trunc_type = 'post'
padding_type = 'post'
oov_tok = '<OOV>' # out of vocabulary token
vocab_size = 500

The Tokenizer API from TensorFlow Keras can split sentences into words and encode them into integers.

The Tokenizer will perform all the necessary pre-processing steps:

• tokenize into word character (word level)
• num_words for maximum number of unique tokens
• filter out the punctuation terms
• convert all words to lower case
• convert all words to integer index

tokenizer = Tokenizer(num_words = vocab_size, 
                      char_level = False,
                      oov_token = oov_tok)
tokenizer.fit_on_texts(x_train)

• num_words : how many unique word that we want to load in training and testing data
• oov_token : out of vocabulary token will be added to word index in the corpus which is used to build the model. This is used to replace out of vocabulary words (words that are not in our corpus) during text_to_sequence calls.

# Get the word_index
word_index = tokenizer.word_index
total_words = len(word_index)
total_words

5.4. Sequence and padding

The next step : Let’s represent each sentence by sequences of numbers using texts_to_sequencesfrom Tokenizer object. After that, we padded the sequence so that we can have same length of each sequence.

For data train:

training_sequences = tokenizer.texts_to_sequences(x_train)
training_padded = pad_sequences(training_sequences,
                                maxlen = max_len,
                                padding = padding_type,
                                truncating = trunc_type)

For data test:

testing_sequences = tokenizer.texts_to_sequences(x_test)
testing_padded = pad_sequences(testing_sequences,
                               maxlen = max_len,
                               padding = padding_type,
                               truncating = trunc_type)

• padding : ‘pre’ or ‘post (default pre). By using pre, we’ll pad before each sequence and post will pad after each sequence.
• maxlen : maximum length of all sequences. If not provided, by default it will use the maximum length of the longest sentence.
• truncating : ‘pre’ or ‘post’ (default ‘pre’). If a sequence length is larger than the provided maxlen value then, these values will be truncated to maxlen. ‘pre’ option will truncate at the beginning where as ‘post’ will truncate at the end of the sequences.

The shape of training and testing padded (tensor):

print('Shape of training tensor: ', training_padded.shape)
print('Shape of testing tensor: ', testing_padded.shape)

Step 6. Define the model architecture and train the model

6.1 Dense Model

Define the dense classifier model architecture:

# Define parameter
vocab_size = 500 
embedding_dim = 16
drop_value = 0.2
n_dense = 24
# Define Dense Model Architecture
model = Sequential()
model.add(Embedding(vocab_size,
                    embedding_dim,
                    input_length = max_len))
model.add(GlobalAveragePooling1D())
model.add(Dense(24, activation='relu'))
model.add(Dropout(drop_value))
model.add(Dense(1, activation='sigmoid'))

Sequential calls for Keras sequential model in which layer added in a sequence. The embedding layer maps each word to a N-dimensional vector of real numbers. The embedding_dim is the size of the word_vector , in this case we use 16. Because the embedding layer is the first hidden layer in our model, we need to set our input layer as defined by the input_length = max_len

Next, we use the GlobalAveragePooling1D as the pooling layer, which helps to reduce the number of parameters in the model and to avoid overfitting.

Next, we use a dense layer with activation function relu followed by a dropout layer to avoid overfitting and a final output layer with sigmoid activation function. The sigmoid activation function outputs probabilities will between 0 and 1.

The summary of the dense model:

model.summary()

Compile the model:

model.compile(loss = 'binary_crossentropy', optimizer = 'adam' , metrics = ['accuracy'])

We use thebinary_crossentropy as a loss function because the output of the model is binary and for the optimizer, we use adam which makes use of momentum to avoid local minima.

Train the model:

Next, let’s train the model using model.fit . It uses padded training data

num_epochs = 30
early_stop = EarlyStopping(monitor='val_loss', patience=3)
history = model.fit(training_padded,
                    y_train,
                    epochs=num_epochs, 
                    validation_data=(testing_padded, y_test),
                    callbacks =[early_stop],
                    verbose=2)

• epoch : number of times the learning algorithm will work through the entire training data.
• callbacks : to pass the early stopping parameter. EarlyStopping(monitor=’val_loss’, patience=2) was used to define that we want to monitor the validation loss and if the validation loss is not improved after 2 epochs, then the model training will stop. This technique helps to avoid overfitting problem.
• verbose : 2 , it will show us loss and accuracy on each epoch.

model.evaluate(testing_padded, y_test)
0s 10ms/step - loss: 0.0873 - accuracy: 0.9599

The model resulted, the training loss: 0.07, training accuracy: 97.49%, validation loss: 0.0873 and validation accuracy: 95.99%.

Plots the graph of accuracy:

From the graph of accuracy above, the accuracy is increasing over epochs. The model is performing better in train data than in the valid data.

Plots the graph of loss:

From the graph of loss below, we can conclude if the loss is decreasing as the number of the epochs increases.

Print the accuracy of train and valid data:

train_dense_results = model.evaluate(training_padded, np.asarray(y_train), verbose=2, batch_size=256)
valid_dense_results = model.evaluate(testing_padded, np.asarray(y_test), verbose=2, batch_size=256)
print(f'Train accuracy: {train_dense_results[1]*100:0.2f}')
print(f'Valid accuracy: {valid_dense_results[1]*100:0.2f}')

6.2 Long Short Term Memory (LSTM)

Long Short Term Memory (LSTM) was designed to overcome the problems of simple Recurrent Neural Network (RNN) by allowing the network to store data in a sort of memory that it can access at a later times.

Source: https://colah.github.io/posts/2015-08-Understanding-LSTMs/

The key of the LSTM model is the cell state. The cell state is updated twice with few computations that resulting stabilize gradients. It has also a hidden state that acts like a short term memory.

In LSTM there are Forget Gate, Input Gate and Output Gate.

• The first step is to decide what information we’re going to throw away from the cell state. This decision is made by a sigmoid layer called the “Forget Gate” layer.
• The second step is to decide what new information that we’re going to store in the cell state. This has two parts. First, a sigmoid layer called the “Input Gate” layer decides which values we’ll update. Next, a tanh layer which creates a vector of new candidate values that could be added to the state.
• Finally, we need to decide what we are going to output. This output will be based on our cell state, but will be a filtered version. First, we run a sigmoid layer which decides what parts of the cell state we’re going to output. Then, we put the cell state through tanh (to push the values to be between -1 and 1) and multiply it by the output of the sigmoid gate, so that we only output the parts we decided

Define the LSTM model architecture:

# Define parameter
n_lstm = 128
drop_lstm = 0.2
# Define LSTM Model 
model1 = Sequential()
model1.add(Embedding(vocab_size, embedding_dim, input_length=max_len))
model1.add(SpatialDropout1D(drop_lstm))
model1.add(LSTM(n_lstm, return_sequences=False))
model1.add(Dropout(drop_lstm))
model1.add(Dense(1, activation='sigmoid'))

The summary of the model:

model1.summary()

Compile the model:

model1.compile(loss = 'binary_crossentropy',
               optimizer = 'adam',
               metrics = ['accuracy'])

Train the model:

num_epochs = 30
early_stop = EarlyStopping(monitor='val_loss', patience=2)
history = model1.fit(training_padded,
                     y_train,
                     epochs=num_epochs, 
                     validation_data=(testing_padded, y_test),
                     callbacks =[early_stop],
                     verbose=2)

Plots the graph of accuracy:

Plots the graph of loss:

Print the accuracy of train and valid data:

6.3 Bidirectional Long Short Term Memory (Bi-LSTM)

A Bidirectional LSTM, or biLSTM, is a sequence processing model that consists of two LSTMs: one taking the input in a forward direction, and the other in a backwards direction. BiLSTMs effectively increase the amount of information available to the network, improving the context available to the algorithm (e.g. knowing what words immediately follow and precede a word in a sentence). Unlike standard LSTM, the input flows of Bi-LSTM in both directions, and it’s capable of utilizing information from both sides. It’s also a powerful tool for modeling the sequential dependencies between words and phrases in both directions of the sequence.

BiLSTM adds one more LSTM layer, which reverses the direction of information flow. Briefly, it means that the input sequence flows backward in the additional LSTM layer. Then we combine the outputs from both LSTM layers in several ways, such as average, sum, multiplication, or concatenation.

To illustrate, the unrolled BiLSTM is presented in the figure below:

Source: https://www.baeldung.com/cs/bidirectional-vs-unidirectional-lstm

Define the Bi-LSTM model architecture:

model2 = Sequential()
model2.add(Embedding(vocab_size,
                     embedding_dim,
                     input_length = max_len))
model2.add(Bidirectional(LSTM(n_lstm,
                              return_sequences = False)))
model2.add(Dropout(drop_lstm))
model2.add(Dense(1, activation='sigmoid'))

The summary of the model:

model2.summary()

Compile the model:

model2.compile(loss = 'binary_crossentropy',
               optimizer = 'adam',
               metrics=['accuracy'])

Train the model:

num_epochs = 30
early_stop = EarlyStopping(monitor = 'val_loss',
                           patience = 2)
history = model2.fit(training_padded,
                     y_train,
                     epochs = num_epochs,
                     validation_data = (testing_padded, y_test),
                     callbacks = [early_stop],
                     verbose = 2)

Plots the graph of accuracy:

Plots the graph of loss:

Print the accuracy of train and valid data:

6.4 Gated Recurrent Unit (GRU)

A Gated Recurrent Unit, or GRU, is a type of recurrent neural network. It is similar to an LSTM, but only has two gates — a reset gate and an update gate and notably lacks an output gate. Fewer parameters means GRUs are generally easier/faster to train than their LSTM counterparts.

Source: https://paperswithcode.com/method/gru

Define the GRU model architecture:

model3 = Sequential()
model3.add(Embedding(vocab_size,
                     embedding_dim,
                     input_length = max_len))
model3.add(SpatialDropout1D(0.2))
model3.add(GRU(128, return_sequences = False))
model3.add(Dropout(0.2))
model3.add(Dense(1, activation = 'sigmoid'))

The summary of the model:

model3.summary()

Compile the model:

model3.compile(loss = 'binary_crossentropy',
                       optimizer = 'adam',
                       metrics=['accuracy'])

Train the model:

num_epochs = 30
early_stop = EarlyStopping(monitor='val_loss', patience=2)
history = model3.fit(training_padded,
                     y_train,
                     epochs=num_epochs, 
                     validation_data=(testing_padded, y_test),
                     callbacks =[early_stop],
                     verbose=2)

Plots the graph of accuracy:

Plots the graph of loss:

Print the accuracy of train and valid data:

Comparing the four different models

The next step is to compare the four models that we use,

# Comparing the four different models
print(f"Dense model loss and accuracy: {model.evaluate(testing_padded, y_test)} " )
print(f"LSTM model loss and accuracy: {model1.evaluate(testing_padded, y_test)} " )
print(f"Bi-LSTM model loss and accuracy: {model2.evaluate(testing_padded, y_test)} " )
print(f"GRU model loss and accuracy: {model3.evaluate(testing_padded, y_test)}")

The validation loss for these four models are 0.087, 0.095, 0.084 and 0.69, respectively. And the validation accuracy are 95.98%, 96.98%, 97.32% and 48.16%.

Based on the loss, accuracy and the plots, we can conclude if the Bi-LSTM model is the best model for this classification case, with the validation accuracy = 97.32 % and the loss = 0.084.

Predict the Ham or Spam for the new messages

Evaluate how the Dense model predicts/classifies whether its spam or ham given the text from our original data. First message below are spam, whereas the second one is a ham message.

predict_msg = ["Have friends and colleagues who could benefit from
                these weekly updates? Send them to this link to
                subscribe",
               "Call me"]
def predict_spam(predict_msg):
  new_seq = tokenizer.texts_to_sequences(predict_msg)
  padded = pad_sequences(new_seq,
                         maxlen = max_len,
                         padding = padding_type,
                         truncating = trunc_type)
  return(model.predict(padded))
predict_spam(predict_msg)

As shown below, the model correctly predicts the first sentence as SPAM and the second sentence as not spam or HAM. There is 95% chance that the first sentence is SPAM.

Summary:

In this article, we have performed the binary classification on UCL datasets using several deep learning models including: Dense, LSTM, Bi-LSTM and GRU. Based on the experiments that have been carried out, it is concluded that the Bi-LSTM model is the model that has the best performance (in this case), with the accuracy value of 97.32% and the loss value = 0.084.

References:

NLP: Spam Detection in SMS (text) data using Deep Learning

SMS_Spam_Detection_DNN_LSTM_BiLSTM/NLP_SMS_Spam_Detection_FinalVersion_Downsampling_Dense_LSTM_BiLST…

What is Imbalanced Data | Techniques to Handle Imbalanced Data

Multi Class Text Classification with Keras and LSTM

Papers with Code - BiLSTM Explained

Papers with Code - GRU Explained

https://www.baeldung.com/cs/bidirectional-vs-unidirectional-lstm

Mlearning.ai Submission Suggestions