NLP Fundamentals
Behind every review is an emotion — NLP lets machines read the anger, disappointment, and delight hidden in customer feedback.
1. What You'll Learn
- Text preprocessing: tokenization (Jieba/NLTK), stopword removal, stemming/lemmatization
- Text representation: Bag-of-Words, TF-IDF, N-gram
- Word embeddings: Word2Vec (Gensim) principles and training, word similarity computation
- Text classification in practice: sentiment analysis on product reviews with TF-IDF + LogisticRegression
- Bob's review insights: automatically identify key problem words in negative reviews to guide product improvements
2. A Real Story from an E-commerce Customer Service Manager
(1) The Pain Point: 5 Thousand Reviews a Day, Impossible to Analyze by Hand
Bob's platform receives 5 thousand product reviews every day — positive, negative, and suggestions all mixed together. His customer service team can only sample 1%, missing a huge number of quality issues. Last month a defective batch triggered 2 thousand negative reviews, but it wasn't discovered until 3 weeks later, by which time it had already caused 500 thousand USD in return losses. Massive text data is both an untapped gold mine and a minefield.
(2) The NLP Solution
NLP can automatically analyze review sentiment, extract key problem words, and monitor negative review trends in real time.
PYTHON
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
pipe = Pipeline([
("tfidf", TfidfVectorizer(max_features=5000, ngram_range=(1, 2))),
("clf", LogisticRegression(max_iter=500)),
])
pipe.fit(X_train_reviews, y_train_sentiment)
print(f"Sentiment accuracy: {pipe.score(X_test, y_test):.3f}")
(3) The Payoff: Problems Found Within 3 Days, Return Rate Down 40%
After Bob started using NLP to automatically analyze reviews, the time from discovering a product quality issue to responding dropped from 3 weeks to 3 days, the return rate fell by 40%, and he saves roughly 1 million USD per year.
3. Text Preprocessing
(1) Tokenization and Cleaning
▶ Example: English Text Preprocessing Pipeline
PYTHON
import re
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
import nltk
nltk.download("punkt_tab", quiet=True)
nltk.download("stopwords", quiet=True)
nltk.download("wordnet", quiet=True)
def preprocess_text(text):
# Lowercase
text = text.lower()
# Remove special characters and numbers
text = re.sub(r"[^a-zA-Z\s]", "", text)
# Tokenize
tokens = word_tokenize(text)
# Remove stopwords
stop_words = set(stopwords.words("english"))
tokens = [t for t in tokens if t not in stop_words and len(t) > 2]
# Stemming
stemmer = PorterStemmer()
tokens_stemmed = [stemmer.stem(t) for t in tokens]
return " ".join(tokens_stemmed)
# Test
reviews = [
"This product is absolutely amazing! Best purchase ever.",
"Terrible quality, broke after 2 days. Very disappointed.",
"Decent product for the price, but shipping was slow.",
]
for review in reviews:
cleaned = preprocess_text(review)
print(f"Original: {review}")
print(f"Cleaned: {cleaned}\n")
Output:
TEXT
# Function defined successfully
▶ Example: Word Tokenization (NLTK)
PYTHON
# Word tokenization with NLTK
# import nltk
# nltk.download('punkt')
# from nltk.tokenize import word_tokenize
# text = "This product has great quality and fast shipping"
# tokens = word_tokenize(text)
# print(tokens) # ['This', 'product', 'has', 'great', 'quality', 'and', 'fast', 'shipping']
# For English reviews in SalesPredict (international scenario)
from sklearn.feature_extraction.text import CountVectorizer
reviews = [
"Great laptop fast performance",
"Terrible screen quality",
"Good price fast delivery",
]
vectorizer = CountVectorizer()
X_bow = vectorizer.fit_transform(reviews)
print(f"Vocabulary: {vectorizer.vocabulary_}")
print(f"BoW matrix shape: {X_bow.shape}")
Output:
TEXT
# Executed successfully
| Preprocessing Step | English | Chinese |
|---|---|---|
| Tokenization | NLTK/spaCy (whitespace splitting) | Jieba/LAC (no spaces) |
| Stopwords | NLTK stopwords | HIT/Baidu stopword lists |
| Stemming | PorterStemmer | Not applicable |
| Lemmatization | WordNetLemmatizer | Not applicable |
4. Text Representation Methods
Turning text from a "string" into a "numeric vector" requires a representation transformation pipeline:
graph LR
TEXT[Raw Text] --> TOKEN[Tokenize<br/>split / jieba] --> REMOVE[Remove Stopwords] --> STEM[Stem/Lemmatize] --> BOW[Bag-of-Words<br/>Count Vector] --> TFIDF[TF-IDF<br/>Weighted Vector] --> EMBED[Word2Vec<br/>Dense Embedding]
style BOW fill:#fff3cd
style TFIDF fill:#d1ecf1
style EMBED fill:#d4edda
(1) Bag-of-Words and TF-IDF
▶ Example: Comparing Three Text Representations
PYTHON
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
import numpy as np
reviews = [
"great product fast delivery happy customer",
"terrible quality broke disappointed waste money",
"great quality decent price fast shipping",
"awful product terrible experience never buy again",
"excellent quality great value fast delivery",
]
labels = [1, 0, 1, 0, 1] # 1=positive, 0=negative
# Method 1: Bag-of-Words (count)
bow = CountVectorizer()
X_bow = bow.fit_transform(reviews)
print(f"BoW vocabulary size: {len(bow.vocabulary_)}")
# Method 2: TF-IDF
tfidf = TfidfVectorizer(max_features=1000)
X_tfidf = tfidf.fit_transform(reviews)
print(f"TF-IDF shape: {X_tfidf.shape}")
# Method 3: N-gram (capture phrases)
tfidf_ngram = TfidfVectorizer(ngram_range=(1, 2), max_features=1000)
X_ngram = tfidf_ngram.fit_transform(reviews)
print(f"TF-IDF with bigrams shape: {X_ngram.shape}")
print(f"Top bigrams: {[w for w in tfidf_ngram.get_feature_names_out() if ' ' in w][:10]}")
Output:
TEXT
# Executed successfully
| Representation | Information | Dimensionality | Use Case |
|---|---|---|---|
| BoW (term frequency) | Low | High | Simple classification |
| TF-IDF | Medium | High | Standard for text classification |
| N-gram | Medium | Very high | Capturing phrases |
| Word2Vec | High | Low (100-300) | Semantic tasks |
(2) TF-IDF Intuition
- TF (Term Frequency): how often a word appears in a document
- IDF (Inverse Document Frequency): how rare a word is (rarer words get higher weight)
- TF-IDF = TF × IDF: words that are common in this document but rare in others get the highest weight
5. Word Embeddings with Word2Vec
(1) How Word2Vec Works
Word2Vec learns word vectors by predicting context words (CBOW) or predicting context from a word (Skip-gram), so that semantically similar words end up with nearby vectors.
▶ Example: Training Word2Vec and Word Similarity
PYTHON
from gensim.models import Word2Vec
from gensim.test.utils import common_texts
import numpy as np
# Train Word2Vec on sample corpus
# In practice, use real review data corpus
sentences = [
["great", "product", "fast", "delivery", "happy"],
["terrible", "quality", "broke", "disappointed"],
["great", "quality", "decent", "price"],
["awful", "product", "terrible", "experience"],
["excellent", "quality", "great", "value"],
["fast", "shipping", "good", "price", "recommend"],
["poor", "quality", "slow", "delivery", "angry"],
["amazing", "product", "best", "purchase", "happy"],
]
model = Word2Vec(sentences, vector_size=50, window=3, min_count=1, workers=4, epochs=50)
# Word similarity
print("Similar to 'great':")
for word, sim in model.wv.most_similar("great", topn=5):
print(f" {word}: {sim:.3f}")
# Word vector
print(f"\nVector for 'quality': {model.wv['quality'][:5]}... (50-dim)")
# Analogy: great - good + terrible = ?
# result = model.wv.most_similar(positive=["terrible"], negative=["good"], topn=3)
Output:
TEXT
Similar to
| Parameter | Meaning | Recommended Value |
|---|---|---|
| vector_size | Vector dimensionality | 100-300 |
| window | Context window size | 5 |
| min_count | Minimum word frequency | 5 |
| sg | 0=CBOW, 1=Skip-gram | 1 (small data) |
▶ Example: Visualizing Word Embeddings
PYTHON
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
words = ["great", "terrible", "happy", "angry", "fast", "slow",
"quality", "price", "product", "delivery"]
vectors = [model.wv[w] for w in words]
vectors_2d = PCA(n_components=2).fit_transform(vectors)
fig, ax = plt.subplots(figsize=(10, 8))
ax.scatter(vectors_2d[:, 0], vectors_2d[:, 1], s=100)
for i, word in enumerate(words):
ax.annotate(word, (vectors_2d[i, 0], vectors_2d[i, 1]), fontsize=12)
ax.set_title("Word2Vec Embedding Visualization")
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("word2vec_vis.png", dpi=150)
Output:
TEXT
# Executed successfully
6. Sentiment Analysis on Product Reviews in Practice
▶ Example: A Complete Sentiment Classification Workflow
PYTHON
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, confusion_matrix
import numpy as np
# Simulate product reviews
rng = np.random.default_rng(42)
positive_templates = [
"great product love it highly recommend",
"excellent quality fast delivery satisfied",
"amazing value best purchase ever happy",
"good product decent price works well",
"love this item perfect condition great",
]
negative_templates = [
"terrible quality broke after one day",
"awful product waste money very disappointed",
"poor quality slow delivery never again",
"worst purchase defective product angry",
"bad product cheap material not recommend",
]
# Generate 2000 reviews
reviews = []
labels = []
for _ in range(1000):
reviews.append(rng.choice(positive_templates))
labels.append(1)
reviews.append(rng.choice(negative_templates))
labels.append(0)
X_train, X_test, y_train, y_test = train_test_split(reviews, labels, test_size=0.2, random_state=42)
# TF-IDF + Logistic Regression pipeline
pipe = Pipeline([
("tfidf", TfidfVectorizer(max_features=5000, ngram_range=(1, 2), min_df=2)),
("clf", LogisticRegression(max_iter=500, C=1.0)),
])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
print(f"Accuracy: {pipe.score(X_test, y_test):.4f}")
print(f"\nClassification Report:\n{classification_report(y_test, y_pred, target_names=['Negative', 'Positive'])}")
# Extract most important words for each sentiment
feature_names = pipe.named_steps["tfidf"].get_feature_names_out()
coefs = pipe.named_steps["clf"].coef_[0]
top_positive = np.argsort(coefs)[-10:][::-1]
top_negative = np.argsort(coefs)[:10]
print("\nTop positive words:")
for idx in top_positive:
print(f" {feature_names[idx]:20s}: {coefs[idx]:.3f}")
print("\nTop negative words:")
for idx in top_negative:
print(f" {feature_names[idx]:20s}: {coefs[idx]:.3f}")
Output:
TEXT
\nTop positive words:
\nTop negative words:
📌 Key point: The words with the largest coefficients are the strongest indicators of positive sentiment, while the words with the smallest (most negative) coefficients are the strongest indicators of negative sentiment. Bob can use these keywords to monitor review trends and quickly spot product quality issues.
❓ FAQ
Q Should I use TF-IDF or Word2Vec?
A Use TF-IDF for text classification (simple, effective, and interpretable); use Word2Vec for semantic tasks (similarity/analogy); use pretrained embeddings (BERT, etc.) for deep learning. Start with TF-IDF as a beginner.
Q How do I choose n for N-grams?
A For text classification, use (1,2), i.e. unigrams + bigrams; bigrams matter a lot for sentiment analysis (e.g. "not good"); larger n (3+) is usually not worth it (dimensionality explodes and data becomes sparse).
Q What tools should I use for Chinese tokenization?
A Jieba (most popular), pkuseg (high accuracy), LAC (Baidu, full-featured). For e-commerce reviews, add brand names and domain terms with a custom dictionary.
Q How much data does Word2Vec need to train?
A At least 10 million words to learn meaningful vectors. With little data, use pretrained vectors (e.g. GoogleNews-vectors) or just stick with TF-IDF.
Q What accuracy counts as good for sentiment analysis?
A For simple binary classification (positive/negative), TF-IDF + logistic regression typically reaches 80-85%; deep models like BERT can hit 90-95%. But in real business settings with lots of noisy data, 70-80% is already usable.
Q How do I handle sarcasm and implicit negation?
A Simple models (TF-IDF) struggle with this. You need to — 1) add more N-gram features; 2) use pretrained language models (BERT); 3) manually annotate sarcastic samples to train a dedicated model.
📖 Summary
- Text preprocessing pipeline: lowercase → remove special characters → tokenize → remove stopwords → stem/lemmatize
- TF-IDF is the standard representation for text classification: term frequency × inverse document frequency, giving the highest weight to rare but important words
- N-grams capture phrase information (e.g. "not good"), but dimensionality balloons, so control max_features
- Word2Vec learns word vectors through context prediction; semantically similar words have nearby vectors
- Text classification pipeline: TfidfVectorizer + LogisticRegression — simple and efficient
- Model coefficients let you extract keywords — positive/negative indicator words directly guide business improvements
📝 Exercises
- Basic (difficulty ⭐): Use TfidfVectorizer to vectorize 5 product reviews, then print the vocabulary and the TF-IDF matrix. Hint: refer to the example in Section 4.
- Intermediate (difficulty ⭐⭐): Build a TF-IDF + LogisticRegression sentiment classification pipeline and compare the F1 score difference between unigram-only and (1,2)-gram. Hint: ngram_range=(1,1) vs (1,2).
- Challenge (difficulty ⭐⭐⭐): Implement Bob's review insight system — after training the sentiment classifier, extract the top 20 negative keywords, sort them by frequency, and output a "most urgent issues to fix" leaderboard. Hint: sort coef_ + map to feature names + count frequencies.
← Previous Lesson: Dimensionality Reduction | Next Lesson: Introduction to Deep Learning →
