A detector that ranks environmental claims by how far the language runs ahead of the commitment. It tracks expert judgement closely — Pearson 0.906 — and still finds at best 40% of what a reviewer would flag. Published here because that second number is the one that decides how it can be used.
Illustrative system study
Quick read
Research · document intelligence
Research
The problem
A claim that outruns the commitment. Evaluated on 29 claims — and it misses most of them.
Who it is for
Sustainability and ESG analysts, and the risk teams who have to justify why a particular claim was escalated.
What you receive
Per-claim scores, a validation harness, and the numbers that failed
Evidence status
Research evidencePublished evaluation on 29 claims. Read the method and limitations below.
Main limitation
Twenty-nine evaluated claims. Every figure here has wide uncertainty and none of it generalises to a new corpus without re-evaluation.
A claim that outruns the commitment. Evaluated on 29 claims — and it misses most of them.
Research
Investigative work with a published artefact. Not deployed, and not in use by anyone.
The problem
A sustainability analyst reading corporate disclosures has to decide which environmental claims are worth challenging. The claims that matter are rarely false outright; they are vague where they should be specific, and the reading is slow, subjective and hard to defend when two analysts disagree.
Who it is for
Sustainability and ESG analysts, and the risk teams who have to justify why a particular claim was escalated.
Context
This began as my master’s thesis at Bocconi and became a working codebase with a validation harness. It is research: it was never deployed to a client and has no production users.
My role
I designed and built the whole system — claim extraction, the scoring components, the questionnaire used to collect expert ratings, and the validation scripts that produce the numbers below.
Data provenance
Environmental claims extracted from public corporate communications. The reference ratings are human judgements collected through a structured questionnaire, stored in the repository alongside the code.
Constraints
A reference set of twenty-nine rated claims. That was the honest ceiling on how much expert rating time was available, and it constrains every conclusion drawn from it.
Approach
Extract discrete environmental claims from a document with spaCy rather than scoring the document as a whole, so a score can always be traced back to a sentence.
Score each claim on several components — lexical density of promotional language, semantic specificity via ClimateBERT, and similarity to known narrative patterns via Sentence Transformers — and combine them under weights held in one config file.
Approach
Validate against the human ratings with correlation and error metrics, not accuracy alone, because the underlying quantity is a degree of risk rather than a class.
Keep three detector versions side by side and compare them on the same set, instead of reporting only the one that looks best.
Decisions that mattered
Reported correlation and error against continuous expert ratings before reporting any binary accuracy.
Greenwashing risk is a matter of degree. Binarising it first would have hidden that the model tracks reviewer judgement well while still being a poor filter.
Decisions that mattered
Kept the simple lexical version (v1) as the reference model even though a transformer version existed.
The semantic version (v2) correlated worse — 0.557 against v1’s 0.906 — and was retired. Using the more sophisticated architecture would have made a better story and a worse tool.
Decisions that mattered
Tuned v3 toward recall while holding precision at 1.00.
A reviewer aid that raises a false alarm burns trust much faster than one that stays quiet. Missing items is recoverable by reading; a wrong accusation is not.
Figure
Recorded example
Figure A comparison of three detector versions against expert ratings on 29 claims. Version 1 has the highest correlation at Pearson 0.906 and mean absolute error 0.206; version 3 reaches 0.793 with error 0.256; version 2 is the weakest at 0.557 and was retired. On binary flagging, version 1 recalls 15% of flaggable claims and version 3 recalls 40%, both at 100% precision.
Detector score against expert rating across the 29 evaluated claims, with the three detector versions compared. Reproduced from the repository’s validation output, 29 January 2026.
Baseline
Two internal comparisons: a keyword-density detector, and the earlier v1 and v2 detectors scored on the same claims. There is no external published benchmark for this task to compare against.
How it was evaluated
Every detector version is scored on the same 29 human-rated claims. Continuous agreement is measured with Pearson, Spearman and Kendall correlation plus MAE and RMSE; flag/no-flag behaviour is measured with precision, recall and F1. Rater reliability is reported so the human ceiling is visible. The published figures were re-run and reproduced on 29 January 2026.
Results
The v1 detector tracks expert ratings closely: Pearson r = 0.906, Spearman 0.811, MAE 0.206.
As a flagging tool it is weak. v1 catches 15% of flaggable claims; the tuned v3 rule-based detector reaches 40%, at F1 0.571.
Neither version produced a false positive on this set — precision was 1.00 — but on twenty-nine items that is a handful of correct calls, not a property of the method.
The human raters agreed at Krippendorff’s α = 0.69, which is the ceiling any model is being measured against.
What was delivered
A Python package that turns a document into per-claim scores with the contributing components exposed.
A validation harness that regenerates every published metric from the stored ratings.
Written reports covering the algorithm, the detector versions and the performance comparison.
What it changes in practice
Used as intended, this shortens the queue rather than clearing it: an analyst reads the highest-scoring claims first and has a component breakdown to point at when explaining the ranking. It does not reduce the number of claims that must ultimately be read by a person.
Limitations
Twenty-nine evaluated claims. Every figure here has wide uncertainty and none of it generalises to a new corpus without re-evaluation.
The score is a ranking signal. It is not a calibrated probability, and a high score is not a finding that a company has greenwashed.
Recall of 0.40 at best. Most flaggable claims are still missed.
Evaluated on English-language corporate communications only.
The scoring weights were tuned on the same small set they are reported against, so the figures are optimistic.
Human review
Every score is reviewed by a person before it reaches a conclusion. The system ranks and explains; it never issues a verdict about a company.
Evidence record · 01
Every figure quoted above, with its source and what it does not tell you.
Environmental claims extracted from corporate communications and rated for greenwashing risk by human reviewers, forming the reference set the detector is scored against. Human expert ratings collected via a structured questionnaire, then used as ground truth.
As of
2026-01-29
Limitations
Twenty-nine items is a small evaluation set. Every figure derived from it carries wide uncertainty and none of it should be read as a population estimate.
Agreement between the human raters who produced the reference ratings. Krippendorff's alpha across raters; pairwise agreement ≈ 0.852.
As of
2026-01-29
Limitations
α = 0.69 is substantial but not strong agreement: the humans themselves disagree on roughly a third of the signal, which caps how well any model can be expected to match them.
Correlation between the v1 detector score and the human reference rating, over the 29-claim set. Pearson correlation on continuous scores. Spearman 0.811, Kendall 0.627, MAE 0.206, RMSE 0.237.
Compared with
v3 rule-based detector, r = 0.793; v2 semantic detector, r = 0.557.
As of
2026-01-29
Limitations
Correlation on 29 items. It says the score moves with reviewer judgement, not that the score is calibrated as a probability of greenwashing.
Binary flag / no-flag performance of the v3 rule-based detector against the human reference labels. Thresholded score compared with binarised human labels on the same 29-claim set.
Compared with
v1 baseline: recall 0.15 at precision 1.00 (F1 0.261). v3 reaches F1 0.571.
As of
2026-01-29
Limitations
Recall of 0.40 means roughly three in five flaggable claims are missed. Precision of 1.00 on a set this small is a handful of correct positives, not a guarantee. The tool is a reviewer aid; it cannot be a filter that runs unattended.
Nothing on this site is accounting, audit, tax, legal or investment advice, and no figure here is a promise of a result. Where a number appears, its source and its limits appear with it.