WebSentiment: Free Sentence-Level Tone Audit for SEOs

Reading time: 11 min

A single “positive” review score tells you almost nothing. A product page, an editorial piece, or a forum thread mixes neutral passages, praise, caveats, and sometimes one line that sinks the whole thing. What a SEO or a copywriter actually needs is to know which sentences carry which tone, and where they sit on the page.

Let me show you the data. In early November 2026 I read Dan Petrovic’s write-up on his seven-level sentiment classifier, scored from -3 to +3, that reads each sentence with surrounding text for context. That clicked with something I’d been chewing on since my TeknSEO 2026 workshop (“Google classifies everything, including you”): class text with small local models instead of firing a LLM API call at every sentence.

His model was trained on synthetic English text. I kept the scale and swapped most of the synthetic data for real web text in six languages — English, French, Spanish, German, Italian, and Portuguese. The result is WebSentiment: an open dataset and two models, free for commercial use. And you can help improve it without writing a single line of code.

The project needs human labels on real web sentences. If you read any of the six languages, come score a few at https://sdeaure-websentiment-gold.hf.space. Every contributor who wants it gets credited in the dataset card — details at the end.

A restaurant review, scored sentence by sentence

Here’s a TripAdvisor review from the test set with three scores: the labeler (the LLM that annotated the dataset), the base model, and the context model that reads each sentence alongside the full review.

Real web text — typos and exclamation marks included. The TripAdvisor boilerplate (“Do you own or manage this property?”) stays at 0, and the last sentence carries the verdict. Sentence two is a gorgeous trap: four compliments about the decor, then the food disappoints at the very end of the line. Read alone, it sounds like praise — the base model slaps +3 on it. With the whole review in view, the context model lands at -2, much closer to the human labeler.

Six ways a practitioner can actually use this

1. Audit the tone of a page or a block. Break the page into sentences, score each one, and look at the heat map by block: intro, body, callouts, footer. On a test-set page from Le Télégramme, a piece about an English workshop at a school reads clearly positive (“Thanks to this method, the students picked up many everyday words”: +2), but the “Top stories” block that follows stacks a rail strike, a stock market slip, and “At least 62 dead in an attack on a mosque” (-2 for the model, -3 for the labeler). The reader scrolling sees both. The model scores the tone of the words, so a tragedy reported soberly comes out heavily negative. That’s your judgment call: does the adjacency bother you?

2. Track reviews and forum threads. Score every sentence of the reviews, follow the rolling average over time, and read the -2 and -3 lines first. Across a few hundred reviews, that sorts the reading list fast. Irony (“great, late again”) remains the weak spot of every system I’ve measured — including the LLMs.

3. Compare a brand’s mentions across countries. The same model scores all six languages on the same scale, so you can compare tone in France, Germany, or Spain without switching tools. Careful: it scores the sentence’s tone and ignores who it’s aimed at. A line comparing your brand to a competitor can come out positive while working against you. Comparisons are one of the constructions that trip it up.

4. Look at the tone of your headlines. A headline is roughly ten words — about the size of the training segments (median 12 words). You can score your headlines, your Discover cards, or your title tags and see where your editorial line lands. The score says nothing about clicks: to know whether a tone works for you, cross it with your own Google Search Console numbers.

5. Compare your pages to competitors on a query. Pull the top-10 pages for a query, score the paragraphs, and see whether the SERP is informative and neutral or loaded with emotion (enthusiastic comparisons, hot takes). Is your page in the tone of what’s being shown? To compare a batch of pages against each other, use the corpus-centering option — it corrects the average offset specific to each batch.

6. Flag the very negative passages in user content. Comments, hosted reviews, Q&A — push the -3 lines to the top of a moderation queue. The extreme levels (-3 and +3) are the model’s least reliable, so treat it as a triage tool. A human decides. This isn’t a take — it’s a pattern I’ve seen in every light automation project I’ve run.

Try it in the browser before installing anything

Paste text, and each sentence is scored from -3 to +3 and highlighted: green for positive, red for negative, darker as the score strengthens. You can read each sentence alone or in the context of the whole text: https://huggingface.co/spaces/SDeaure/websentiment-demo

The base model — for isolated text like reviews, headlines, or posts — is a small 119 KB layer on EmbeddingGemma-300m, Google’s open embedding model. It runs on a laptop and even in the browser with transformers.js. In Python:

EmbeddingGemma is gated: accept its terms on Hugging Face and log in (hf auth login) before the first call.

The context model is for sentences within a page: it reads the page once, then scores each sentence alongside it. That’s the one that rescues the second sentence in the review above. Treat it as experimental — its gain has only been measured against the labeler’s scores, which is exactly where your contribution comes in.

The dataset is there to train your own classifier on your own scale (three classes, or a 1-to-5 score). Every sentence ships with its source page, its format (product page, article, forum), and its URL.

On the licensing side: the dataset is under ODC-By 1.0, the models under CC BY 4.0 — commercial use included. Redistributing? An attribution notice is mandatory. Using them in a tool or site without redistributing? A link to the project page is requested.

Everything is on https://sylvain-deaure.fr/websentiment/ : Hugging Face links and documentation.

Score some sentences and improve a free model

All the training labels in WebSentiment come from a single LLM (Claude Sonnet). The model learned its intuitions — and its blind spots. Take that second sentence from the restaurant review again: the base model says +3, the context model says -2, the labeling LLM said -1. Which one is right?

Nobody talks about this part. To answer it, you need human labels on this kind of sentence, and there aren’t enough. It needs readers.

That’s what the gold reference set is for: 1,200 web sentences in the six languages, each shown with the passage it came from. You read, pick a score from -3 to +3, and move on. There’s no right answer to find — I want your first reading. One score already helps, and you stop whenever you want.

With your scores I can:

• check whether the context model really beats the base model, against humans rather than only against the LLM;
• see which constructions it gets wrong (irony, mixed sentences, soberly reported incidents) — and therefore what to fix in the next version;
• publish an open reference so anyone can benchmark their own sentiment model in French, German, or Portuguese.

The deal is simple: the model is free, commercial use included, and every sentence you score improves it for everyone using it — you included.

You’ll be credited. The tool has an optional “How to credit you?” field that accepts a name, a handle, or a link to your profile, and you’ll be thanked by name in the published reference dataset card. Leave it blank, and you appear under a number.

Sign-in uses a free Hugging Face account. The more of you per language, the more cases the reference covers.

It’s here: https://sdeaure-websentiment-gold.hf.space

The numbers, and the honest limits

Across nine human-annotated test sets in the six languages (film reviews, tweets, press headlines, financial news), the model reaches 0.673 macro-F1 in three classes (negative, neutral, positive).

On random web text, it matches the labeler’s exact level in 67% of cases and comes within one level in 95%.

Training labels come from an LLM, not humans. The model inherits its biases and can hardly outperform it. I’ve seen this play out before with every “AI-labeled” dataset I’ve audited: the ceiling is the teacher, until humans step in.

Irony and mixed sentences (praise and criticism in the same line) remain the hard part. Six languages are measured. The others may work through the multilingual encoder — nothing is measured there.

For long documents, split into sentences or paragraphs, then average.

Slow down. Think. If you run this on your pages, show me what you get.