A Comprehensive Guide to Automating Article Categorization with Machine Learning



Introduction

How Do ML models perform Document Classification?

Document Classification and Its Role

How Document Classification Works



Data Collection: The first step is to find the source of a labeled dataset, which consists of a document or digital content and its associated category or label. Ensure the dataset is representative and balanced across categories.
Data Preprocessing: This process involves multiple steps, cleaning the data in each step. The text is cleaned by removing noise such as special characters, HTML tags, punctuation, and stop words. The text is tokenized into words or subwords. Additionally, techniques like stemming or lemmatization can be further applied to normalize words.
Feature Extraction: Now, the preprocessed text data is converted into numerical features that machine learning algorithms can understand. Some commonly used text representation techniques include TF-IDF (Term Frequency-Inverse Document Frequency) or word embeddings like Word2Vec, GloVe, or BERT for capturing semantic meanings.
Split Dataset: Divide the dataset into training, validation, and testing sets to train and evaluate the model's performance. Sometimes, the dataset is divided into training and testing; no validation is required as the task is simple.
Model Selection: We must select a suitable machine-learning model for document classification. Some commonly used models include Logistic Regression, Naive Bayes, Support Vector Machines (SVM), and Neural Networks.
Model Training & Testing: The selected model is trained using the training dataset and evaluated on a separate testing dataset. Evaluation metrics such as accuracy, precision, recall, and F1 score are used to assess the model's effectiveness.

Supervised vs Unsupervised Learning for Text Classification

Source

The Newsroom Challenge - Organizing Information

Manual vs. Automated Classification

Costs: Manual categorization of news articles involves human effort, time, and resources. Hiring skilled personnel or assigning existing staff to categorize articles incurs costs.
Benefits: Human judgment and expertise can lead to nuanced categorization, especially for complex or ambiguous topics. It allows for fine-tuning categories based on editorial guidelines and audience preferences.
Limitations: Manual classification is labor-intensive, prone to errors, and may not scale well with the growing volume of news content. It can also be subjective, leading to inconsistencies in categorization.

Costs: Implementing automated categorization using machine learning models requires an initial investment in technology infrastructure, model development, and training data. However, the long-term operational costs can be significantly lower than those associated with manual efforts.
Benefits: Automated classification leverages ML algorithms to process large volumes of text data rapidly and consistently. It reduces human errors, improves scalability, and enables real-time categorization of incoming news articles.
Limitations: Automated systems may face challenges in accurately categorizing nuanced or context-dependent topics. They require regular monitoring and updates to maintain classification accuracy as news trends and topics evolve.

How Automated Categorization Enhances News Discovery, Reading, and Archiving

News Discovery: Automated categorization enables users to discover relevant news articles more efficiently by grouping content into meaningful categories or topics. This improves user experience by facilitating targeted content recommendations and personalized news feeds based on user preferences.
Reading Experience: Automated categorization can enhance the reading experience by presenting articles in organized sections or topic clusters, allowing users to seamlessly navigate and explore related content.
Archiving and Retrieval: Automated categorization aids in systematically archiving news content, making it easier to retrieve historical articles based on specific categories, dates, or keywords. This archival system supports research, analysis, and content repurposing efforts within news organizations.

Practical Guide For Building Your Personal Document Classifier Using Weights and Biases

Dataset Selection

Step 1: Importing the required libraries

import os
import pandas as pd
import numpy as np
import random
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
import re
import string
import nltk
from nltk.corpus import stopwords
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
import wandb

Step 2: Initialize Weights & Biases

config = {
    "seed": 42,
    "lr": 0.001,
    "epochs": 5
}

wandb.init(project='article_classifier', config=config)



Step 3: Seeding the Environment

random.seed(config["seed"])
os.environ["PYTHONHASHSEED"] = str(config["seed"])
np.random.seed(config["seed"])
torch.manual_seed(config["seed"])
torch.cuda.manual_seed(config["seed"])
torch.backends.cudnn.deterministic = True

Step 4: Downloading the Stopwords

nltk.download('stopwords')

Step 5: Data Loading & Preprocessing

Converting text to lowercase.
Removing punctuations.
Removing stopwords.
Removing whitespaces.

# Load dataset from CSV
data = pd.read_csv('Articles.csv', encoding='ISO-8859-1')

# Preprocessing
def clean_text(text):
    text = text.lower()  # Convert text to lowercase
    text = re.sub(r'\d+', '', text)  # Remove digits
    text = text.translate(str.maketrans('', '', string.punctuation))  # Remove punctuation
    text = ' '.join([word for word in text.split() if word not in stopwords.words('english')])  # Remove stop words
    text = text.strip()  # Remove leading and trailing whitespaces
    return text

data['Cleaned_Article'] = data['Article'].apply(clean_text)

X = data['Cleaned_Article']
y = data['NewsType']





Step 6: Label Encoding and Vectorization

# Label Encoding
label_encoder = LabelEncoder()
y = label_encoder.fit_transform(y)

class_labels = label_encoder.classes_

# Vectorization
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(X)

# Convert sparse matrix to dense tensor
X = torch.tensor(X.toarray(), dtype=torch.float32)
y = torch.tensor(y, dtype=torch.long)

Step 7: Splitting the Dataset

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42)
print(f"Training samples: {len(X_train)} - Testing samples: {len(X_test)}")

Step 8: Defining the Model Architecture

class NewsClassifier(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(NewsClassifier, self).__init__()
        self.fc = nn.Sequential(
            nn.Linear(input_dim, input_dim//8),
            nn.ReLU(),
            nn.Linear(input_dim//8, output_dim)
        )

    def forward(self, x):
        x = self.fc(x)
        return x

input_dim = X_train.shape[1]
output_dim = len(y_train.unique())
model = NewsClassifier(input_dim, output_dim)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

Step 9: Logging Initial Results

# Logging articles to the table
pred_table = wandb.Table(columns=["Name", "Sample 1", "Sample 2", "Sample 3", "Sample 4", "Sample 5", "Sample 6", "Sample 7", "Sample 8", "Sample 9", "Sample 10"])

# Split
_, test_x, _, test_y = train_test_split(data['Cleaned_Article'].values, data['NewsType'].values, test_size=0.5, random_state=42)
pred_table.add_data("Articles", test_x[0], test_x[1], test_x[2], test_x[3], test_x[4], test_x[5], test_x[6], test_x[7], test_x[8], test_x[9])

# Initial results
initial_samples = X_test[:10]  # Get 10 initial samples

initial_predictions = torch.argmax(torch.softmax(model(initial_samples), dim=1), dim=1)
initial_predictions_labels = [class_labels[i] for i in initial_predictions.tolist()]
print("Initial Predictions: \t", initial_predictions_labels)

pred_table.add_data("Initial Predictions", initial_predictions_labels[0], 
initial_predictions_labels[1], initial_predictions_labels[2], initial_predictions_labels[3], 
initial_predictions_labels[4], initial_predictions_labels[5], initial_predictions_labels[6],
initial_predictions_labels[7], initial_predictions_labels[8], initial_predictions_labels[9])

Step 10: Training and Logging Predictions

# Training loop
for epoch in range(config["epochs"]):
    model.train()
    optimizer.zero_grad()
    outputs = model(X_train)
    loss = criterion(outputs, y_train)
    print(f"Epoch: {epoch+1:2d} - Loss: {loss:1.4f}")
    wandb.log({"Epoch":epoch+1, "Loss":loss})
    loss.backward()
    optimizer.step()

    # Logging final results
    final_predictions = torch.argmax(torch.softmax(model(initial_samples), dim=1), dim=1)
    final_predictions_labels = [class_labels[i] for i in final_predictions.tolist()]
    pred_table.add_data(f"Epoch {epoch+1}", final_predictions_labels[0], final_predictions_labels[1], final_predictions_labels[2], final_predictions_labels[3], final_predictions_labels[4], final_predictions_labels[5], final_predictions_labels[6], final_predictions_labels[7], final_predictions_labels[8], final_predictions_labels[9])

print("Final Predictions: \t", final_predictions_labels)





Step 11: Testing & Evaluation

model.eval()
with torch.no_grad():
    test_outputs = model(X_test)
    predicted = torch.argmax(torch.softmax(test_outputs, dim=1), dim=1)

    accuracy = accuracy_score(y_test, predicted)
    confusion_mat = confusion_matrix(y_test, predicted)
    classification_rep = classification_report(y_test, predicted)

# Logging evaluation metrics to W&B
wandb.log({"Accuracy": accuracy})
wandb.log({"Confusion Matrix": confusion_mat.tolist()})
wandb.log({"Classification Report": classification_rep})

print("Evaluation ->")
print("Accuracy: ", accuracy)
print("Confusion Matrix:\n", confusion_mat)
print("Classification Report:\n", classification_rep)





Step 12: Saving the Model

torch.save(model.state_dict(), 'article_categorization_model.pth')

wandb.finish()

Output





Publishing the Results - Model Evaluation and Implementation

Accuracy
Confusion Matrix
Classification Report



True Positives (TP): 648
False Positives (FP): 1
False Negatives (FN): 6
True Negatives (TN): 691

Challenges faced throughout the Guide

Dataset: Finding a good enough dataset is a minor challenge. Sometimes, you may not find the right kind and need to annotate the dataset.
Class Imbalance: Unequal distribution of classes in the dataset can result in biased models that perform well on majority classes but poorly on minority ones. Techniques like class weighting may be needed to address this issue.
Text Preprocessing: It is an essential step in data cleaning, but with the present dataset, text preprocessing does not have any major effect on performance. So, comparing the performance with and without text preprocessing is better. Text preprocessing is also a time-consuming process, as it takes time to clean the text.

Conclusion