NLTK

NLTK is the classic library for teaching and researching NLP. While slower than spaCy, it offers comprehensive linguistic data.

When to Use

• Classical NLP & Linguistics: Tokenization, stemming, lemmatization, part-of-speech (POS) tagging, and syntactic parsing.
• Rule-Based Sentiment Analysis: Quick heuristic sentiment scoring with VADER for social media texts.
• Text Preprocessing for Classical ML: Cleaning corpus text for TF-IDF, Naive Bayes, and SVM models.
• Educational & Linguistic Research: Analyzing word frequencies, collocations, and lexical dispersion across corpora.

Quick Start

import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

nltk.download('punkt_tab')
nltk.download('stopwords')

text = "Natural Language Toolkit provides classical algorithms for text analysis."
tokens = word_tokenize(text)
stop_words = set(stopwords.words('english'))
filtered = [w for w in tokens if w.lower() not in stop_words and w.isalnum()]

print("Filtered tokens:", filtered)

Core Concepts

Tokenization, Stopwords & WordNet Lemmatization

Standard text normalization pipeline:

import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
import string

# Ensure required corpora are downloaded
nltk.download('punkt_tab', quiet=True)
nltk.download('stopwords', quiet=True)
nltk.download('wordnet', quiet=True)

def preprocess_text(raw_text: str) -> list[str]:
    # 1. Lowercase and tokenize
    tokens = word_tokenize(raw_text.lower())

    # 2. Filter punctuation and English stopwords
    stop_words = set(stopwords.words('english'))
    filtered = [w for w in tokens if w.isalnum() and w not in stop_words]

    # 3. Lemmatize words to base dictionary form
    lemmatizer = WordNetLemmatizer()
    lemmatized = [lemmatizer.lemmatize(w) for w in filtered]

    return lemmatized

sample = "The distributed microservices were running exceptionally fast and scaling reliably!"
print("Cleaned Tokens:", preprocess_text(sample))

VADER Sentiment Analysis for Fast Heuristic Scoring

Rule-based sentiment intensity scoring:

from nltk.sentiment.vader import SentimentIntensityAnalyzer

nltk.download('vader_lexicon', quiet=True)
sia = SentimentIntensityAnalyzer()

reviews = [
    "Antigravity IDE is amazingly responsive and saves hours of tedious refactoring!",
    "The legacy codebase crashes constantly with unhelpful error messages.",
    "The software was delivered on Tuesday afternoon."
]

for review in reviews:
    scores = sia.polarity_scores(review)
    compound = scores['compound']
    sentiment = 'positive' if compound >= 0.05 else ('negative' if compound <= -0.05 else 'neutral')
    print(f"Sentiment: {sentiment:8s} (Score: {compound:+.2f}) | {review[:50]}...")

Part-of-Speech (POS) Tagging & Named Entity Chunking

Extracting grammatical structures from text:

nltk.download('averaged_perceptron_tagger_eng', quiet=True)
nltk.download('maxent_ne_chunker_tab', quiet=True)
nltk.download('words', quiet=True)

sentence = "Sundar Pichai announced new Gemini models at Google headquarters."
tokens = word_tokenize(sentence)
pos_tags = nltk.pos_tag(tokens)

# Extract named entities
chunks = nltk.ne_chunk(pos_tags)
for chunk in chunks:
    if hasattr(chunk, 'label'):
        print(f"Entity: {' '.join(c[0] for c in chunk)} [{chunk.label()}]")

Common Patterns

Lemmatization with POS (Part of Speech) Tagging

Problem: Simple stemmers truncate words unnaturally (running -> run, but better -> better).

Solution: Combine WordNetLemmatizer with POS tags:

from nltk.stem import WordNetLemmatizer
from nltk.corpus import wordnet
import nltk

nltk.download('averaged_perceptron_tagger_eng')
nltk.download('wordnet')

lemmatizer = WordNetLemmatizer()
# Lemmatize with explicit verb POS tag
print(lemmatizer.lemmatize("running", wordnet.VERB)) # Output: run
print(lemmatizer.lemmatize("better", wordnet.ADJ))   # Output: good

Best Practices

Do:

• Prefer lemmatization over stemming (PorterStemmer) for human-readable root words.
• Download specific corpora explicitly in setup scripts (nltk.download('punkt_tab')) to prevent runtime failures in Docker.
• Leverage VADER for social media text where punctuation and capitalization convey sentiment.
• Migrate to spaCy or Hugging Face transformers when semantic understanding or modern deep learning is required.

Don't:

• Call nltk.download() repeatedly on every request inside web handlers; download once at build/container time.
• Use NLTK for high-throughput production tokenization; use Hugging Face tokenizers or spaCy.
• Use WordNet lemmatizer without specifying POS tags if fine-grained verb/noun distinction is needed.

Troubleshooting

References

• NLTK Documentation