The question
Sentiment classification used to require hand-built word lists or models trained on thousands of labelled examples. Today a large language model can do it from a one-line instruction. How much better are the new models really, and is the largest model always the best choice?
Approach
- Lexicon baselines: VADER and TextBlob score each review from a fixed dictionary of word polarities.
- Neural networks: a simple network, a CNN and an LSTM, all on pre-trained GloVe word embeddings, evaluated on 10,000 held-out reviews.
- Fine-tuned transformer:
bert-base-uncased, fine-tuned for two epochs on 9,600 reviews and tested on 1,200. - Zero-shot LLMs: six models (GPT-5.3 Instant, GPT-4.1 mini, Claude Haiku 4.5, Claude Sonnet 4.6, Gemini 2.5 Flash, Gemini 2.5 Flash-Lite). Each labelled the same fixed random sample of 1,000 reviews with a prompt that allowed only "positive" or "negative" as the answer. The code includes error handling and retries, and API keys are read from secrets, never from the code.
Every model was evaluated with accuracy, precision, recall, F1 and a confusion matrix, followed by a qualitative review of every misclassified text.
Results
| Zero-shot LLM | Accuracy | Precision | Recall | F1 | Errors |
|---|---|---|---|---|---|
| GPT-5.3 Instant | 0.969 | 0.963 | 0.973 | 0.968 | 31 |
| Claude Haiku 4.5 | 0.962 | 0.945 | 0.977 | 0.961 | 38 |
| Claude Sonnet 4.6 | 0.961 | 0.947 | 0.973 | 0.960 | 39 |
| GPT-4.1 mini | 0.958 | 0.954 | 0.958 | 0.956 | 42 |
| Gemini 2.5 Flash-Lite | 0.945 | 0.953 | 0.931 | 0.942 | 55 |
| Gemini 2.5 Flash (634 valid) | 0.934 | 0.958 | 0.910 | 0.933 | 42 |
What the numbers say
- LLMs win without any training data. Every zero-shot LLM beat the fine-tuned BERT (90.5%) by about 3 to 6 percentage points.
- Bigger is not automatically better. Claude Haiku 4.5 matched Claude Sonnet 4.6 (96.2% vs 96.1%) at a fraction of the cost.
- Lexicons lean positive. TextBlob's recall is 0.95, but its precision is only 0.63: it labels many negative reviews as positive.
- Among the classic networks, the LSTM generalises best. Its train–test gap is 1.1 points, against about 10 points for the CNN and the simple network.
- Every error is inspected. The notebooks list each misclassified review for qualitative analysis; ironic and mixed reviews are typical difficult cases.
Reporting failures. API errors are part of the result. Gemini 2.5 Flash returned a valid label for only 634 of 1,000 reviews, and it is reported with that caveat rather than left out.