Master sentiment analysis in Python



Table of contents

What is sentiment analysis?

Practical applications of sentiment analysis

Social media monitoring: Companies analyze tweets, Facebook posts, and other social media content to understand public sentiment about their products or brand. This real-time feedback helps in managing brand reputation and responding promptly to customer concerns or viral trends.
Customer reviews and service: E-commerce platforms and service providers use sentiment analysis on product reviews or customer support tickets. By automatically gauging whether feedback is positive or negative, businesses can prioritize addressing negative feedback and improve their products and services. It also enables aggregating thousands of reviews to get an overall sentiment score for products.
Market research and finance: In finance, analysts use sentiment analysis on news articles and financial reports to predict market movements. For example, sentiment scores of news headlines about a company can be an indicator of its stock performance. Market research firms also analyze sentiment in survey responses or online forums to measure consumer confidence and preferences.
Healthcare and sociology: Sentiment analysis is applied to patient feedback, medical forums, or therapy session transcripts to detect sentiments that might indicate patient satisfaction or emotional well-being. In sociology and linguistics research, analyzing sentiment in large collections of texts (like literature or political speeches) can reveal insights about public mood and historical trends.
Political and social analysis: During elections or major political events, sentiment analysis of tweets and news can gauge public opinion and reaction. Governments and NGOs can analyze social media sentiment on policy announcements or social issues to understand public opinion and respond accordingly.

Methodologies for performing sentiment analysis

Lexicon-based approach: The lexicon-based approach to sentiment analysis relies on predefined lexical resources (dictionaries of words) where each word is associated with a sentiment score. The analysis involves counting or summing sentiment scores of words in the text to determine the overall sentiment. For example, words like "good", "happy", or "excellent" might contribute positive points, while "bad", "sad", or "terrible" might contribute negative points. Lexicon-based methods are simple and easy to implement, requiring no training data. However, they have limitations: they often ignore context (e.g., sarcasm or negation like "not good"), and their accuracy depends heavily on the quality of the lexicon and rules.
Machine learning approach: This approach treats sentiment analysis as a text classification problem. First, a labeled dataset of texts with known sentiments (positive/negative labels, for instance) is required. The text is converted into features (such as word frequencies or embeddings), and then a machine learning model is trained on these features to learn how to classify new text. Common algorithms include logistic regression, naive Bayes, or support vector machines for simpler tasks, and they can achieve better accuracy than lexicon-based methods by learning from context patterns in data. The downside is that they require annotated data and computational power for training. They may also not generalize well beyond the data they were trained on, unless carefully validated.
Transformer-based deep learning approach: Transformer models (like BERT, RoBERTa, or GPT-based models) have revolutionized NLP, including sentiment analysis. These models are typically pre-trained on massive text corpora and can be fine-tuned on sentiment analysis tasks. Approaches using transformers often involve either using a pre-trained model directly for sentiment (for example, via Hugging Face transformers pipeline) or fine-tuning a model on a specific sentiment dataset (like movie reviews). Transformer-based methods typically achieve the highest accuracy because they capture complex language contexts and subtleties, such as negation and sarcasm. They can understand that "I don't hate it" is different from "I hate it", something that simpler models or lexicons might miss. The trade-off is that they are resource-intensive and can be more complex to implement. They also often require access to pre-trained model weights and possibly a GPU for efficient processing.

Types of sentiment analysis

Fine-grained sentiment analysis: Fine-grained sentiment analysis goes deeper than binary positive/negative classification, using a rating scale. For example, it might classify sentiment as very positive, positive, neutral, negative, or very negative. This fine-grained approach is useful when you need more nuance, such as understanding if feedback is extremely negative or just slightly negative. A common example is star ratings (1 through 5 stars) which can be mapped to a fine-grained sentiment (1 star = very negative, 5 stars = very positive). Fine-grained analysis helps in scenarios like product reviews where a 3-star (neutral to slightly positive) review is very different from a 1-star (very negative) review.
Aspect-based sentiment analysis: The goal of aspect-based sentiment analysis is to identify the sentiment towards specific aspects or features of a product or subject. For instance, a restaurant review might say, "The ambiance was great, but the service was slow." Aspect-based analysis would parse this and determine that the sentiment toward "ambiance" is positive while the sentiment toward "service" is negative. This approach is crucial for detailed feedback analysis, allowing businesses to pinpoint what exactly customers like or dislike. It often involves first identifying aspect terms (ambiance, service) and then determining sentiment for each aspect separately within the text.
Emotion detection: Sometimes we want to go beyond positive/negative and identify specific emotions expressed in text (such as happiness, anger, sadness, fear, surprise, etc.). Emotion detection systems use lexicons or machine learning models trained on datasets labeled with emotions. For example, "I'm absolutely thrilled with the support I received!" might be tagged with the joy or satisfaction emotion, whereas "I'm frustrated with the waiting time" would be tagged as anger or frustration. Emotion detection is useful in contexts like social media monitoring, where understanding the type of emotion can help in tailoring responses (e.g., a customer support system might prioritize angry customers for faster intervention).

Setting up Python for sentiment analysis



Install required libraries. Make sure you have Python installed (Python 3.7+ is recommended). Then install the libraries we'll use. Open a terminal or command prompt and run the following command to install NLTK, scikit-learn, pandas (for data handling), and wandb:

pip install nltk scikit-learn pandas wandb

This will download and install the packages. You may already have some of these, but it's okay to run the command — it will update or confirm the installation of each.
Import libraries in your Python script or notebook. Once installed, we can import them into our code. We also plan to use NLTK's corpora (for example, the movie reviews dataset and the VADER lexicon), so we'll need to download those resources using NLTK's downloader. We'll do that in code to ensure everything is ready.
Download NLTK data (if not already available). NLTK comes with a downloader for various datasets and lexicons. In particular, we'll use the "movie_reviews" corpus for our dataset and the "vader_lexicon" for the VADER sentiment analyzer. We need to download these once. This can be done via Python code (it will prompt a download if these aren't already present on your system):

import nltk
nltk.download('movie_reviews')   # movie reviews dataset (if using NLTK's sample data)
nltk.download('vader_lexicon')   # VADER sentiment lexicon
nltk.download('stopwords')       # we'll use stopwords during preprocessing

Running the above will fetch the data and lexicon. If you're in a notebook environment, the nltk.download function might open an interactive prompt. Passing the resource name directly, as shown, ensures it downloads without the interactive UI. After downloading, NLTK can access these resources offline.
Verify the setup. It's a good idea to verify that everything is installed correctly. You can do a quick version check or simple import test:

import sklearn
import pandas as pd
import nltk
import wandb

print("NLTK version:", nltk.__version__)
print("Scikit-learn version:", sklearn.__version__)
print("Pandas version:", pd.__version__)

This is just to ensure there are no import errors and to check the versions. If these print statements output version numbers without errors, you're ready to proceed.

Tutorial: Performing sentiment analysis with Python

Loading the data: We'll use a sample dataset of movie reviews from NLTK, which contains labeled movie review texts (positive or negative sentiment).
Preprocessing: We'll clean the text data (tokenization, lowercasing, removing stopwords, etc.) to prepare it for analysis.
Rule-based analysis (baseline): We'll apply a rule-based sentiment analyzer (VADER) to see how it performs on the data as a baseline method.
Feature extraction: We'll convert text data into numerical features using a TF-IDF vectorizer so that a machine learning model can understand it.
Training a model: We'll train a simple machine learning model (Logistic Regression) on the training data to classify sentiment.
Evaluation: We'll evaluate the trained model on a test set and compare its performance to our rule-based approach.
Experiment tracking: We'll demonstrate how to use Weights & Biases to track the experiment's metrics and results for better visualization.

Step 1: Load the dataset

# Step 1: Import libraries and load the dataset
import nltk
from nltk.corpus import movie_reviews
import random

# Ensure the NLTK movie_reviews corpus is downloaded
nltk.download('movie_reviews')

# Load documents and their categories (pos or neg)
documents = []
for fileid in movie_reviews.fileids():
    # movie_reviews.words(fileid) gives a list of words in the review
    words = movie_reviews.words(fileid)
    text = " ".join(words)  # join words into one string for the full review text
    label = movie_reviews.categories(fileid)[0]  # category is either 'pos' or 'neg'
    documents.append((text, label))

# Shuffle the documents to mix positive and negative
random.shuffle(documents)

# Separate the shuffled documents into texts and labels
texts = [doc[0] for doc in documents]
labels = [doc[1] for doc in documents]

# Print some basic information about the dataset
print("Total reviews:", len(documents))
print("Positive reviews:", labels.count('pos'))
print("Negative reviews:", labels.count('neg'))

# Print an example review and its label
print("\nSample review excerpt:")
print(texts[0][:100], "...")  # first 100 characters of the first review
print("Label:", labels[0])

Total reviews: 2000
Positive reviews: 1000
Negative reviews: 1000

Sample review excerpt:
plot : two teen couples go to a church party , drink and then drive ... [example continues] ...
Label: neg

Step 2: Split data into training and test sets

# Step 2: Split the data into training and test sets
from sklearn.model_selection import train_test_split

# Use 80% of the data for training and 20% for testing
texts_train, texts_test, labels_train, labels_test = train_test_split(
    texts, labels, test_size=0.2, random_state=42, stratify=labels)

# Verify the size of each set and balance
print("Training set size:", len(texts_train))
print("Test set size:", len(texts_test))
print("Training positives:", labels_train.count('pos'), 
      "| Training negatives:", labels_train.count('neg'))
print("Test positives:", labels_test.count('pos'), 
      "| Test negatives:", labels_test.count('neg'))

Training set size: 1600
Test set size: 400
Training positives: 800 | Training negatives: 800
Test positives: 200 | Test negatives: 200

Step 3: Preprocess the text data

Lowercasing all words (so "Good" and "good" are treated the same).
Removing punctuation and special characters (which don't contribute to sentiment in most cases).
Removing stopwords (common words like "the", "is", "at" that don't carry significant meaning).
Tokenization or stemming/lemmatization (reducing words to their base form), if necessary.

# Step 3: Preprocess the text (cleaning)
import re
from nltk.corpus import stopwords

# Ensure stopwords are downloaded
nltk.download('stopwords')
stop_words = set(stopwords.words('english'))

def preprocess_text(text):
    # 1. Lowercase the text
    text = text.lower()
    # 2. Remove punctuation and non-letter characters using regex
    text = re.sub(r'[^a-z\s]', '', text)
    # 3. Remove stopwords
    words = text.split()  # split on whitespace to get words
    meaningful_words = [w for w in words if w not in stop_words]
    # Join the cleaned words back into one string
    return " ".join(meaningful_words)

# Test the preprocessing on a sample review before applying to all
original_example = texts_train[0]
cleaned_example = preprocess_text(original_example)
print("Original example (first 100 chars):", original_example[:100], "...")
print("Cleaned example (first 100 chars):", cleaned_example[:100], "...")

# Preprocess all training and test texts
texts_train_clean = [preprocess_text(text) for text in texts_train]
texts_test_clean = [preprocess_text(text) for text in texts_test]

Original example (first 100 chars): I loved this film. It was incredibly engaging, and the performances were absolutely fantastic. ...
Cleaned example (first 100 chars): loved film incredibly engaging performances absolutely fantastic ...

The text has been lowercased.
Punctuation (like the period .) has been removed.
Common stopwords like "I", "this", "was", "and", "the", "were" have been removed from the cleaned text.

Step 4: Rule-based sentiment analysis with VADER

# Step 4: Apply rule-based sentiment analysis (VADER)
from nltk.sentiment.vader import SentimentIntensityAnalyzer

# Ensure the VADER lexicon is downloaded
nltk.download('vader_lexicon')
sia = SentimentIntensityAnalyzer()  # initialize the sentiment analyzer

# Demonstrate VADER on example sentences
example_pos = "I really love this movie. It was great!"
example_neg = "I really hate this movie. It was terrible."
print("Example positive sentence:", example_pos)
print("VADER sentiment scores:", sia.polarity_scores(example_pos))
print("Example negative sentence:", example_neg)
print("VADER sentiment scores:", sia.polarity_scores(example_neg))

# Use VADER to predict sentiment for each review in the test set
vader_predictions = []
for text in texts_test:
    scores = sia.polarity_scores(text)
    # Classify as positive or negative based on the compound score threshold of 0
    if scores['compound'] >= 0:
        vader_predictions.append('pos')
    else:
        vader_predictions.append('neg')

# Evaluate VADER accuracy on the test set
from sklearn.metrics import accuracy_score
vader_accuracy = accuracy_score(labels_test, vader_predictions)
print(f"VADER Accuracy on test set: {vader_accuracy:.2f}")

Example positive sentence: I really love this movie. It was great!
VADER sentiment scores: {'neg': 0.0, 'neu': 0.21, 'pos': 0.79, 'compound': 0.88}
Example negative sentence: I really hate this movie. It was terrible.
VADER sentiment scores: {'neg': 0.81, 'neu': 0.19, 'pos': 0.0, 'compound': -0.88}
VADER Accuracy on test set: 0.65

For the example positive sentence, VADER returns a dictionary of scores. The 'pos': 0.79 and 'neg': 0.0 indicate it found the text to be mostly positive, and 'compound': 0.88 is a high positive sentiment score (close to 1, which is the maximum). This makes sense because the sentence "I really love this movie. It was great!" is clearly positive in tone.
For the example negative sentence, VADER outputs a high 'neg': 0.81 with 'pos': 0.0, and a 'compound': -0.88, reflecting a strongly negative sentiment. Again, this aligns with our expectation, as "I really hate this movie. It was terrible." is clearly negative.

print(sia.polarity_scores("I absolutely love how bad this movie is!"))

Step 5: Feature extraction with TF-IDF

# Step 5: Convert text data to TF-IDF features
from sklearn.feature_extraction.text import TfidfVectorizer

# Initialize TF-IDF Vectorizer with some common options:
# - ngarm_range=(1,2) to include unigrams and bigrams (single words and pairs of consecutive words)
# - min_df=2 to ignore terms that appear in only 1 document (reduce noise)
# - stop_words='english' as an extra safeguard (though we've already removed stopwords)
vectorizer = TfidfVectorizer(ngram_range=(1,2), min_df=2, stop_words='english')

# Fit the vectorizer on the cleaned training text and transform training and test text
X_train_tfidf = vectorizer.fit_transform(texts_train_clean)
X_test_tfidf = vectorizer.transform(texts_test_clean)

print("Number of features:", X_train_tfidf.shape[1])
print("Sample feature vector for one review (sparse representation):")
print(X_train_tfidf[0])

Number of features: 25000
Sample feature vector for one review (sparse representation):
  (0, 123)	0.0843
  (0, 456)	0.1507
  (0, 789)	0.2771
  ...

Step 6: Train a sentiment classification model

# Step 6: Train a Logistic Regression model on the training data
from sklearn.linear_model import LogisticRegression

# Initialize the Logistic Regression model
model = LogisticRegression(max_iter=1000)  # max_iter increased to ensure convergence

# Train (fit) the model on the training TF-IDF vectors and labels
model.fit(X_train_tfidf, labels_train)

print("Model training completed.")

Model training completed.

Step 7: Evaluate the model

# Step 7: Evaluate the model on the test set
from sklearn import metrics

# Use the trained model to predict sentiment for the test set
test_predictions = model.predict(X_test_tfidf)

# Calculate accuracy
accuracy = metrics.accuracy_score(labels_test, test_predictions)
print(f"Model Accuracy on test set: {accuracy:.2f}")

# Display a detailed classification report
print("\nClassification Report:")
print(metrics.classification_report(labels_test, test_predictions, target_names=['neg', 'pos']))

Model Accuracy on test set: 0.82

Classification Report:
              precision    recall  f1-score   support

         neg       0.81      0.84      0.83       200
         pos       0.84      0.80      0.82       200

    accuracy                           0.82       400
   macro avg       0.82      0.82      0.82       400
weighted avg       0.82      0.82      0.82       400

For negative reviews (neg), the model had precision 0.81 and recall 0.84. This means:
Precision 0.81: When the model predicted a review was negative, it was correct 81% of the time.
Recall 0.84: The model caught 84% of all the actual negative reviews (it missed 16% of them, presumably predicting those as positive by mistake).
For positive reviews (pos), precision is 0.84 and recall 0.80 (so it identified 80% of actual positives, misclassifying 20% as negative).
The F1-score, which is the harmonic mean of precision and recall, is ~0.82 for both classes, indicating balanced performance.
The support for each class is 200, which just confirms there were 200 examples of each in the test set.

VADER (rule-based) accuracy: ~65%
Logistic Regression (machine learning) accuracy: ~82%

Step 8: Track and visualize the experiment with Weights & Biases

# Step 8: Initialize W&B run and log metrics
import wandb

# Login to W&B (this will prompt you to input your API key if not logged in already)
# wandb.login()  # Uncomment this line if running in an environment where you're not logged in

# Start a new run in W&B
wandb.init(project="sentiment_analysis_tutorial", name="LogisticRegression-BOW")

# Log the accuracy from this run
wandb.log({"accuracy": accuracy, "vader_baseline_accuracy": vader_accuracy})

# Optional: log a few example review predictions vs actual
# We'll create a small table of some test reviews, their predicted label, and true label
table_data = []
for i in range(3):
    table_data.append([texts_test[i][:100] + "...", labels_test[i], test_predictions[i]])
wandb.log({"examples": wandb.Table(data=table_data, columns=["Review excerpt", "Actual Sentiment", "Predicted Sentiment"])})

# Mark the run as finished
wandb.finish()

wandb: Run initialized under project "sentiment_analysis_tutorial"
wandb: Tracking run with wandb version X.Y.Z
wandb: Run name: LogisticRegression-BOW
wandb: Syncing metrics ... [etc]

wandb: Run page: https://wandb.ai/your-username/sentiment_analysis_tutorial/runs/your-run-id

Alternative use cases and tools

Analyzing different text sources: We used movie reviews, but you could apply the same workflow to other data. For example, try sentiment analysis on tweets (Twitter data) to understand public sentiment on a topic, or on news headlines to gauge market sentiment. Each domain might require slightly different preprocessing (e.g., handling hashtags or usernames in tweets, or dealing with formal language in news).
TextBlob is a high-level library built on top of NLTK, making sentiment analysis extremely straightforward. It has a TextBlob object that you can call .sentiment on to get polarity (a score from -1 to 1) and subjectivity. For example:

from textblob import TextBlob
blob = TextBlob("The movie was not bad at all.")
print(blob.sentiment)

This might output something like Sentiment(polarity=0.3, subjectivity=0.6). Under the hood, TextBlob uses a combination of techniques, including a lexicon. It's great for quick sentiment checks, though it might not be as accurate as a trained model for specific domains.
VADER (detailed usage): We already used VADER via NLTK. VADER is particularly well-suited for social media texts due to its handling of slang and punctuation. It's worth noting that VADER returns a compound score as well as individual positive/negative/neutral scores. If your use case needs to identify neutrality or mixed sentiment, you can use thresholds on those scores. VADER is a suitable default for initial sentiment analysis due to its ease of use and the absence of training data requirements.
Flair: Flair is an NLP library from the Zalando Research team that provides simple interfaces for state-of-the-art NLP models. It has a pre-trained sentiment analysis model (an LSTM neural network trained on IMDB reviews). Using Flair:

from flair.models import TextClassifier
from flair.data import Sentence
classifier = TextClassifier.load('en-sentiment')
sentence = Sentence("I absolutely loved the new design of your website!")
classifier.predict(sentence)
print(sentence.labels)

This might output something like "[POSITIVE (0.95)]", indicating that the text is positive with a confidence score of 0.95. Flair's models are more accurate than simple lexicon methods because they use deep learning under the hood, but they are also heavier (loading the model can be a few hundred megabytes).
Hugging Face Transformers: For the ultimate in accuracy, you can use transformer-based models. Hugging Face provides a high-level pipeline for sentiment analysis:

from transformers import pipeline
sentiment_pipeline = pipeline("sentiment-analysis")
result = sentiment_pipeline("I have mixed feelings about this product.")[0]
print(result)

This might output {'label': 'NEGATIVE', 'score': 0.55} (for example, if the model leans slightly negative on the mixed feelings sentence). The default model behind this pipeline is usually a fine-tuned BERT or DistilBERT model on a large sentiment dataset. These models capture context very well (for instance, understanding negation or sarcasm better than simpler methods). The trade-off is that they require more computational resources. However, you don't need to train them (since they are pre-trained and fine-tuned), which is a huge advantage if you don't have a large labeled dataset.
Custom models and deep learning: For specialized applications, you might train your own deep learning model using libraries like TensorFlow or PyTorch. For example, you could build an LSTM or transformer model fine-tuned on a custom dataset (perhaps sentiment of tweets about your company, or sentiment in customer support chats).

Ease of use: TextBlob and VADER are very easy to use, but might offer less accuracy.
Accuracy: Transformer-based models or Flair provide high accuracy, but at the cost of speed and resource usage.
Data requirement: Rule-based methods require no data (just the text to analyze and a lexicon), while machine learning approaches require labeled datasets. Pre-trained models (like those from Hugging Face or Flair) come with the data requirement "built-in" (they were trained on large datasets already).
Integration and tracking: Whichever method you use, you can integrate experiment tracking. For example, if you try different models (say, logistic regression vs SVM vs a neural network), you can use W&B to compare their performance side by side in a dashboard, making it easier to pick the best one.

Conclusion

Sources

NLTK Sentiment Analysis Documentation – NLTK 3.5 how-to guide – Official guide on using NLTK for sentiment analysis, including VADER usage.
VADER Sentiment Analysis Paper (Hutto & Gilbert, 2014) – Original paper and GitHub repository for the VADER lexicon, explaining how it works and its evaluation.
TextBlob: Simplified Text Processing – Official TextBlob documentation, covering the sentiment property and other NLP features.
scikit-learn: TfidfVectorizer Documentation – Reference for the TfidfVectorizer used to convert text to features.
scikit-learn: Metrics – Classification Report – Explanation of precision, recall, and F1-score as seen in the classification report.
Flair NLP Library – GitHub page for Flair, which provides an easy interface for state-of-the-art NLP models, including an example on sentiment analysis.
Hugging Face Transformers Pipeline – Documentation for using high-level pipelines like sentiment-analysis, which leverage pre-trained transformer models.
Weights & Biases Documentation – Experiment Tracking – Official docs for using W&B, including guides on how to log metrics, charts, and datasets for machine learning projects.